Meta unveils ‘Seamless’ speech-to-speech translator

Meta, proprietor of Fb, Instagram, and WhatsApp, on Tuesday unveiled its newest effort in machine translation, this one geared towards speech translation. 

This system, SeamlessM4T, surpasses present fashions which can be skilled particularly for speech-to-speech translation between languages, in addition to fashions that convert between speech and textual content in a number of language pairs. Therefore, SeamlessM4T is an instance not simply of generality however of what’s known as multi-modality — the power for one program to function on a number of knowledge varieties, on this case, each speech and textual content knowledge. 

Additionally: Meta to launch open-source industrial AI mannequin to compete with OpenAI and Google

Beforehand, Meta has targeted on giant language fashions that may translate textual content between 200 totally different languages. That target textual content is an issue, say lead creator Loïc Barrault and colleagues at each Meta and UC California at Berkeley. 

“Whereas single, unimodal fashions comparable to No Language Left Behind (NLLB) push text-to-text translation (T2TT) protection to greater than 200 languages, unified S2ST [speech-to-speech-to-text] fashions are removed from attaining comparable scope or efficiency,” write Barrault and workforce.

The formal paper, “SeamlessM4T — Massively Multilingual & Multimodal Machine Translation,” is posted on Meta’s devoted website for the general challenge, Seamless Communication. There may be additionally a companion GitHub website. 

Speech has been left behind partly as a result of much less speech knowledge is available within the public area to coach neural networks, write the authors. However there is a deeper level: Speech knowledge is essentially richer as a sign for neural networks.

“The very problem round why speech is more durable to deal with from a machine translation standpoint — that it encodes extra info and expressive elements — can also be why it’s superior at conveying intent and forging stronger social bonds between interlocutors,” they write. 

The objective of SeamlessM4T is to create one program that’s skilled on each speech knowledge and textual content knowledge on the similar time. The “M4T” stands for “Massively Multilingual & Multimodal Machine Translation.” Multi-modality is an express a part of this system. 

Additionally: Meta’s newest AI mannequin will make content material accessible in a whole bunch of languages

Such a program is typically known as an “end-to-end” program as a result of it does not break up the elements which can be about textual content and the elements which can be about speech into separate features, as within the case of “cascaded fashions,” the place this system first is skilled on one factor, comparable to speech to textual content, after which one other factor, comparable to speech to speech.

As this system’s authors put it, “most S2ST [speech-to-speech translation] methods at the moment rely closely on cascaded methods composed of a number of subsystems that carry out translation progressively — e.g., from computerized speech recognition (ASR) to T2TT [text-to-text translation], and subsequently text-to-speech (TTS) synthesis in a 3-stage system.”

As a substitute, the authors constructed a program that mixes a number of present elements skilled collectively. They included “SeamlessM4T-NLLB a massively multilingual T2TT mannequin,” plus a program known as w2v-BERT 2.0, “a speech illustration studying mannequin that leverages unlabeled speech audio knowledge,” plus T2U, “a text-to-unit sequence-to-sequence mannequin,” and multilingual HiFi-GAN, a “unit vocoder for synthesizing speech from items.” 

Additionally: Meta’s ‘data2vec’ is a step towards One Neural Network to Rule Them All

All 4 elements are plugged collectively like a Lego set right into a single program, additionally launched this yr by Meta, known as UnitY, which will be described as “a two-pass modeling framework that first generates textual content and subsequently predicts discrete acoustic items.” 

The entire group is seen within the diagram beneath.

This system manages to do higher than a number of other forms of applications on checks of speech recognition, speech translation, and speech-to-text, the authors report. That features beating each taint applications which can be additionally end-to-end, in addition to applications designed for speech explicitly:

We discover that SeamlessM4T-Giant, the bigger mannequin of the 2 we launch, outper- types the earlier state-of-the-art (SOTA) end-to-end S2TT mannequin (AudioPaLM-2-8B- AST [Rubenstein et al., 2023]) by 4.2 BLEU factors on Fleurs [Conneau et al., 2022] when translating into English (i.e., an enchancment of 20%). In comparison with cascaded mod- els, SeamlessM4T-Giant improves translation accuracy by over 2 BLEU factors. When translating from English, SeamlessM4T-Giant improves on the earlier SOTA (XLS- R-2B-S2T [Babu et al., 2022]) by 2.8 BLEU factors on CoVoST 2 [Wang et al., 2021c], and its efficiency is on par with cascaded methods on Fleurs. On the S2ST activity, SeamlessM4T-Giant outperforms robust 3-stage cascaded fashions (ASR, T2TT and TTS) by 2.6 ASR-BLEU factors on Fleurs. On CVSS, SeamlessM4T-Giant outperforms a 2-stage cascaded mannequin (Whisper-Giant-v2 + YourTTS [Casanova et al., 2022]) by a big margin of 8.5 ASR-BLEU factors (a 50% enchancment). Preliminary human evalua- tions of S2TT outputs evinced equally spectacular outcomes. For translations from English, XSTS scores for twenty-four evaluated languages are constantly above 4 (out of 5); for into English instructions, we see vital enchancment over Whisper-Giant-v2’s baseline for 7 out of 24 languages.

Additionally: Google’s ‘translation glasses’ had been truly at I/O 2023, and proper in entrance of our eyes

The companion GitHub website gives not simply this system code but in addition SONAR, a brand new expertise for “embedding” multi-modal knowledge, and BLASAR 2.0, a brand new model of a metric by which to robotically consider multi-modal duties.