Lipdub empowers everyday users to automatically translate and dub their voices into 28 languages

Captions, the leading AI-powered creative studio, announced the launch of Lipdub, the new app that automatically translates and dubs video content into 28 different languages and counting. With Lipdub, anyone can dub themselves, friends, or family — and experience familiar voices and faces in a suite of new languages.

“AI Dubbing’s success inspired us to push this technology to the next level and introduce synced lip movement to the mix. Since then we’ve been focused on making the technology widely available, leading us to create and launch Lipdub as its own, separate app.”

Lipdub’s video translations are complete with the ability to not only translate the subject’s exact voice but also sync their lip movement to match the translated audio, thereby delivering a viewing experience that retains the essence and context of the original content. Users can simply record or upload a video, and the app will change the subject’s exact voice and lip movements to any language they select — including French, Hindi, Spanish, Italian, Portuguese, and more. Lipdub not only translates languages, but also incorporates dialects and even vocabulary, with options like Gen Z and Texas slang. You can watch a demo video showcasing Lipdub here.

Marketing Technology News: Skai Strengthens its Creative Capabilities With First-of-its-Kind Creative Center

“Captions entered the translation space in the fall of 2022 with the release of translated captioning. We then launched our voice translation feature, AI Dubbing, in early 2023, and were quickly met with enthusiastic adoption from users around the world,” said Gaurav Misra, co-founder and CEO of Captions. “AI Dubbing’s success inspired us to push this technology to the next level and introduce synced lip movement to the mix. Since then we’ve been focused on making the technology widely available, leading us to create and launch Lipdub as its own, separate app.”

At the core of Lipdub’s innovative technology is generative AI that facilitates the app’s high-quality voice translations and realistic synchronization of voice to lip movement. The algorithm can pick up on subtleties of various dialects and tones of voice, making it a vital component in Lipdub’s ability to authentically translate content. All videos produced using Lipdub are marked with a “Lipdub™ AI-generated” label to ensure transparency.

In 2021, Captions launched its original, eponymous app, which has provided more than 5 million creators of all skill levels — from real estate agents to podcasters to online coaches — the tools needed to develop and publish studio-grade content with just a few quick taps. A one-stop, AI-powered creative studio, Captions simplifies and automates key aspects of video production, from captioning, editing, and dubbing, to music integration.

To date, Captions has raised $40 million from investors including Kleiner Perkins, Sequoia Capital, Andreessen Horowitz, and SV Angel.

Marketing Technology News: MarTech Interview with Tobyn Sowden, CEO at Redbrick