AI Dubbing

Sieve’s AI Dubbing is a completely automated dubbing API built for developers. It can take in an input video or audio file and dub it across many languages, completely automatically.

Note: The API currently accepts maximum 15 minute long files. If you need to dub longer files, please contact us (sales@sievedata.com) for access.

Key Features

• Speaker Style Preservation: Preserve the tone and style of the original speaker.
• Multiple Speaker Support: Multi-speaker support with distinct voices for each speaker.
• Broad Range of Languages: Support for 29 popular languages.
• Background Noise Addition: Add original background noise back to the dubbed audio for a more natural sounding dub.

Unique Features

• Language styles: You can specify language styles such as "informal french", "shakespearean english", "brazilian portuguese", etc (only available with gpt4 translation).
• Faster than realtime: Faster than realtime processing of dubs.
• Voice engines: Pick from a variety of voice engines depending on your cost, quality, and speed requirements.
• Safe words: Specify safe words that you don't want to translate such as names or places.
• Translation Dictionary: Customize translations by specifying mappings for specific words or phrases to control the output dub.
• Multi-language inputs: Specify multiple target languages as a comma-separated list to get multiple language dubs at once.
• Metadata Output: Option to output transcription, translation, and diarization metadata along with the dubbed video or audio.

Supported Languages

Sieve's AI Dubbing supports various languages depending on the voice_engine you use. If you use elevenlabs for TTS, you can only dub into the 29 following languages. If you use openai for TTS, you can dub into 100+ languages -- though voice cloning is not supported for openai.

• 🇺🇸 English
• 🇮🇳 Hindi
• 🇵🇹 Portuguese
• 🇨🇳 Mandarin (Chinese)
• 🇪🇸 Spanish
• 🇫🇷 French
• 🇩🇪 German
• 🇯🇵 Japanese
• 🇦🇪 Arabic
• 🇷🇺 Russian
• 🇰🇷 Korean
• 🇮🇩 Indonesian
• 🇮🇹 Italian
• 🇳🇱 Dutch
• 🇹🇷 Turkish
• 🇵🇱 Polish
• 🇸🇪 Swedish
• 🇵🇭 Tagalog (Filipino)
• 🇲🇾 Malay
• 🇷🇴 Romanian
• 🇺🇦 Ukrainian
• 🇬🇷 Greek
• 🇨🇿 Czech
• 🇩🇰 Danish
• 🇫🇮 Finnish
• 🇧🇬 Bulgarian
• 🇭🇷 Croatian
• 🇸🇰 Slovak
• 🇮🇳 Tamil

Translation Dictionary Usage

If you have certain words you wish to be translated to specific words, rather than letting the backend decide for itself you can simply pass a JSON with the requried word mappings.

For Example:

{
    "China": "Africa",
    "New York": "NYU"
}

Note You can either specify the desired translated word in the target language or in the source language and it will be automatically translated for you.

You can also use the translation dictionary to make edits in your speech, By replacing certain words or phrases with desired words or phrases.

For Example:

{
    "Oh my God!": "Oh Lord!",
    "It's bad news" : "It's great news"
}

Experimental Features

• Lipsyncing: The app has an enable_lipsyncing option that will sync the lips of the source video to the dubbed audio. This is useful for creating more natural-looking dubs, however it is still experimental and expected to fail at times.