Meta pronounces Voicebox, a generative mannequin for a number of voice synthesis duties

Final week, Meta Platforms’ synthetic intelligence analysis arm launched Voicebox, a machine studying mannequin that may generate speech from textual content. What units Voicebox aside from different text-to-speech fashions is its capability to carry out many duties that it has not been educated for, together with modifying, noise elimination, and magnificence switch.

The mannequin was educated utilizing a particular technique developed by Meta researchers. Whereas Meta has not launched Voicebox as a consequence of moral considerations about misuse, the preliminary outcomes are promising and might energy many functions sooner or later.

‘Circulate Matching’

Voicebox is a generative mannequin that may synthesize speech throughout six languages, together with English, French, Spanish, German, Polish, and Portuguese. Like massive language fashions, it has been educated on a really normal job that can be utilized for a lot of functions. However whereas LLMs attempt to study the statistical regularities of phrases and textual content sequences, Voicebox has been educated to study the patterns that map voice audio samples to their transcripts. 

Such a mannequin can then be utilized to many downstream duties with little or no fine-tuning. “The objective is to construct a single mannequin that may carry out many text-guided speech era duties by in-context studying,” Meta’s researchers write in their paper (PDF) describing the technical particulars of Voicebox.

The mannequin was educated Meta’s “Circulate Matching” method, which is extra environment friendly and generalizable than diffusion-based studying strategies utilized in different generative fashions. The method allows Voicebox to “study from diverse speech knowledge with out these variations having to be rigorously labeled.” With out the necessity for guide labeling, the researchers had been in a position to practice Voicebox on 50,000 hours of speech and transcripts from audiobooks.

The mannequin makes use of “text-guided speech infilling” as its coaching objective, which suggests it should predict a phase of speech given its surrounding audio and the whole textual content transcript. Mainly, it implies that throughout coaching, the mannequin is supplied with an audio pattern and its corresponding textual content. Components of the audio are then masked and the mannequin tries to generate the masked half utilizing the encircling audio and the transcript as context. By doing this time and again, the mannequin learns to generate natural-sounding speech from textual content in a generalizable means.

Replicating voices throughout languages, modifying out errors in speech, and extra

Not like generative fashions which might be educated for a selected software, Voicebox can carry out many duties that it has not been educated for. For instance, the mannequin can use a two-second voice pattern to generate speech for brand spanking new textual content. Meta says this functionality can be utilized to carry speech to people who find themselves unable to talk or customise the voices of non-playable recreation characters and digital assistants.

Voicebox additionally performs fashion switch in numerous methods. For instance, you possibly can present the mannequin with two audio and textual content samples. It should use the primary audio pattern as fashion reference and modify the second to match the voice and tone of the reference. Curiously, the mannequin can do the identical factor throughout totally different languages, which might be used to “assist individuals talk in a pure, genuine means — even when they don’t converse the identical languages.”

The mannequin may also do quite a lot of modifying duties. For instance, if a canine barks within the background when you’re recording your voice, you possibly can present the audio and transcript to Voicebox and masks out the phase with the background noise. The mannequin will use the transcript to generate the lacking portion of the audio with out the background noise. 

The identical method can be utilized to edit speech. For instance, if in case you have misspoken a phrase, you possibly can masks that portion of the audio pattern and cross it to Voicebox together with a transcript of the edited textual content. The mannequin will generate the lacking half with the brand new textual content in a means that matches the encircling voice and tone.

One of many fascinating functions of Voicebox is voice sampling. The mannequin can generate varied speech samples from a single textual content sequence. This functionality can be utilized to generate artificial knowledge to coach different speech processing fashions. “Our outcomes present that speech recognition fashions educated on Voicebox-generated artificial speech carry out virtually in addition to fashions educated on actual speech, with 1 % error fee degradation versus 45 to 70 % degradation with artificial speech from earlier text-to-speech fashions,” Meta writes.

Voicebox has limits too. Because it has been educated on audiobook knowledge, it doesn’t switch effectively to conversational speech that’s informal and accommodates non-verbal sounds. It additionally doesn’t present full management over totally different attributes of the generated speech, reminiscent of voice fashion, tone, emotion, and acoustic situation. The Meta analysis staff is exploring methods to beat these limitations sooner or later.

Mannequin not launched

There’s rising concern concerning the threats of AI-generated content material. For instance, cybercriminals not too long ago tried to rip-off a girl by calling her and utilizing AI-generated voice to impersonate her grandson. Superior speech synthesis methods reminiscent of Voicebox might be used for related functions or different nefarious deeds, reminiscent of creating pretend proof or manipulating actual audio.

“As with different highly effective new AI improvements, we acknowledge that this expertise brings the potential for misuse and unintended hurt,” Meta wrote on its AI weblog. As a result of these considerations, Meta didn’t launch the mannequin however supplied technical particulars on the structure and coaching course of within the technical paper. The paper additionally accommodates particulars a couple of classifier mannequin that may detect speech and audio generated by Voicebox to mitigate the dangers of utilizing the mannequin.