On Monday, Spotify rolled out a restricted pilot program that makes use of AI to routinely translate podcasts into varied languages, utilizing voice synthesis know-how from OpenAI to protect the unique speaker’s voice. The characteristic goals to supply a extra genuine listening expertise in comparison with conventional dubbing. It may additionally introduce language errors which can be troublesome for non-native audio system to detect, since machine translation is far from a perfect know-how.
In its press launch asserting this system, Spotify says it’s a platform that permits creators to share their work world wide. Then it asks a query: “With current developments, we’ve been questioning: Are there extra methods we are able to bridge the language hole in order that these voices might be heard worldwide?”
Spotify’s reply is Voice Translation, which might reportedly translate English voices into Spanish, French, and German whereas retaining the distinctive vocal traits of the speaker. The characteristic is presently getting used with solely choose podcasters, equivalent to Dax Shepard, Monica Padman, Lex Fridman, Invoice Simmons, and Steven Bartlett.
“We consider {that a} considerate strategy to AI may help construct deeper connections between listeners and creators, a key part of Spotify’s mission to unlock the potential of human creativity,” stated Ziad Sultan, Spotify’s VP of personalization, within the announcement.
Spotify says the translated episodes will likely be accessible worldwide to each Premium and Free customers. Customers can entry the translations by way of Spotify’s “Now Enjoying View” for supported episodes or a devoted Voice Translations Hub that may proceed so as to add extra translated content material.
On X, Lex Friedman posted a sample of his voice cloned and translated into Spanish, writing, “That is me talking Spanish, due to superb work by Spotify AI engineers. The interpretation & voice-cloning are totally accomplished by AI. Language can create obstacles of understanding & thus gas division. I am unable to anticipate AI to interrupt down this barrier & reveal our widespread humanity.”
Misplaced in translation
However not all podcasters are excited in regards to the potential for automated AI translations. Reacting to the information on BlueSky, Retronauts co-creator and co-host Jeremy Parish posted, “One more reason to roll my eyes when folks ask why we don’t make the podcast accessible on Spotify.”
Previously, we have seen voice-cloning know-how from each Microsoft and Meta analyze samples of supply audio, then increase that audio with a big coaching information set of voices to synthesize a brand new, related voice. That know-how can doubtlessly fail when an individual’s vocal type is not represented effectively within the information set of coaching samples, particularly with certain accents.
Right here, Spotify is including a further layer of complexity, hoping to seamlessly translate which means between languages with out making errors, one thing Meta has additionally tried with SeamlessM4T. Over the previous decade, AI-driven translation has made massive strides, however it hasn’t knocked human translators fully out of the sport. Trade consultants point out that these methods nonetheless get tripped up on nuance and do not perceive cultural context, affecting the standard of the translated materials.
Tech-savvy customers possible anticipate translation errors when the supply is correctly framed as a machine translation, however when the errors come within the podcaster’s personal voice, it might add a brand new dimension of hassle, particularly if the translated audio is taken out of context and later presumed to be authentic. Moreover, if the unique speaker would not know the translated language, they cannot examine to see if the interpretation precisely displays their authentic intentions. That is placing loads of belief—and private status—within the palms of unproven automation know-how.
For now, it seems that Spotify’s program is engaged on a restricted, opt-in foundation amongst choose podcasters solely, so problems with consent over cloning podcast visitor voices do not seem like at play except this rolls out extra broadly. Going ahead, Spotify says it hopes to assemble suggestions from creators and listeners to refine the voice translation characteristic. Nevertheless, with over 100 million common podcast listeners on the platform, that is 100 million methods this experiment may go poorly if the interpretation know-how makes embarrassing errors.
