Transcription and translation when people mix languages

Why mixed speech is hard for translator apps

Phone translator apps ask for two languages before anyone speaks, then treat every sentence as one of them. Some meeting notetakers transcribe one language per meeting, chosen before recording. A sentence that starts in German and ends in English fits neither setup.

All Voices does not fix the languages in advance. It works out the language of each line from the speech, keeps the words in the language they were spoken, and translates the line for you.

How it works

• Every line keeps its language. The transcript shows each line in the language and script it was spoken in, with the speaker’s name and the translation underneath.
• One language to read, or two to interpret. Choose the language you read, and everything else is translated into it. When two people speak two languages, All Voices can interpret both ways instead. In that mode, a line in a third language is translated into one of the two.
• Expected languages help. In Settings you can list the languages you expect. They help All Voices recognise them; other languages are still identified. When you know exactly which languages will be spoken, you can restrict recognition to them during the session.
• Terms for names and jargon. Add product names, acronyms and names of people as terms, and they are recognised in every new session, in any language.

What we have checked

We test a meeting with three speakers who speak English, German and Mandarin Chinese in turn, including a German sentence that switches to English halfway. In our latest test, every run completed, and each sentence was kept in the language in which it was spoken. That test used generated voices, not a recording of a real meeting; a real room with crosstalk and accents is harder.

So the mixes we can name are Chinese and English, German and English, and meetings that move between all three. Other combinations of the languages in the app may work as well, but we have not measured them. The free 30 minutes are enough to try your own mix before you buy.

Where mixing happens

• International teams where one language is the meeting language and another is the working language.
• Families in which parents and children answer each other in different languages.
• Workplaces where product names and technical terms stay in English inside sentences in another language.

After the meeting

The saved meeting keeps the recording in step with the transcript, so you can replay any mixed sentence and hear exactly what was said. The summary, decisions and action items are written in the language you choose, and can be written again in a second language. See meeting notes in two languages.

Limits

• Live transcription and translation need an internet connection. If it drops, All Voices keeps the recording while it reconnects.
• Some languages in the app are marked experimental. Mixed speech in those languages is less reliable.
• Strong dialects are harder for every speech system. When the result matters, check names and numbers against the recording.