Translate any video or audio content with natural sounding translations and voices.

OPENAI_API_KEY

ELEVEN_LABS_API_KEY

source_file

A comma-separated list of languages to which the audio will be translated. Default is "spanish". See supported languages in README.

target_language

The translation engine to use. Default is "gpt4". Supported backends are "seamless" and "gpt4". See the README for more information.

translation_engine

The TTS engine to use. Default is "elevenlabs". Supported backends are "elevenlabs" and "openai". See the README for more information.

voice_engine

Whether to return the transcript of the video along that includes the translated text. Default is False.

return_transcript

A comma-separated list of words that should not be translated. Default is "".

safewords

a JSON object that allows more control over how words are translated. See README for more information on using this parameter.

translation_dictionary

The start time of the media to dub from. Default is 0.

start_time

The end time of the media to dub to. If -1, the end time is the length of the media.

end_time

(ALPHA FEATURE) Whether to enable lip-syncing on the original video to the dubbed audio. Defaults to False. Only applicable for video inputs.

enable_lipsyncing

dubbing

Dubbing

Smart, automatic cropping of a video to a given aspect ratio based on subject detection and speaker tracking.

file

whether or not to determine crops based on who's actively speaking in the video. Defaults to False.

active_speaker_detection

the aspect ratio of the video to create. Must be "1:1", "4:3", "3:4", "4:5", "5:4", or "9:16". You may also specify multiple aspect ratios separated by commas to get metadata for each one. Defaults to "9:16".

aspect_ratio

whether or not to return the fully edited video file or just the metadata so you can render on your own. Defaults to False. If multiple aspect ratios are specified, while return_video is True, only the first aspect ratio will be used.

return_video

the seconds into the video to start processing from. Defaults to 0.

the seconds into the video to stop processing at. Defaults to -1, which means the end of the video.

whether or not to use a faster, but less accurate, model for the object detection that powers the cropping. Defaults to False.

speed_boost

whether or not to let the function auto-pick the main subject to focus on. Note: this setting is still experimental but it could yield better results on a broader range of videos. Defaults to False.

smart_edit

whether or not to return a visualization of the crops and bounding boxes overlaid onto the original video (for debugging purposes). Defaults to False.

visualize

whether or not to include the information about the subjects (people, faces, etc) of the crops in the metadata. Defaults to False.

include_subjects

whether or not to include the versions of the layouts if speaker detection was off in the output metadata. This is useful if you want the option to switch between active speaker and non-active speaker layouts without having to reprocess the video. Defaults to False and is only used if active_speaker_detection is True and return_video is False.

include_non_active_layouts

a comma-separated list of natural language class prompts to detect in the video. Defaults to "person". You can specify any class in natural language.

prompt

a comma-separated list of natural language class prompts to exclude from the detection in the video. Defaults to "".

negative_prompt

whether or not to only allow single crop layout versus multi-crop layouts. Defaults to False.

single_crop_only

the speed at which the crops moves from one frame to another. Defaults to 0.1. Must be between 0 and 1.

crop_movement_speed

the number of seconds between each sample taken for an updated crop position. Should be lower for faster moving subjects, and higher for still subjects. Defaults to 3.

crop_sampling_interval

autocrop

Auto Crop

A comprehensive solution for video lipsyncing with a suite of different model and enhancements options.

Audio file to sync with the driver image/video file.

audio

Whether to enhance the output using the default enhancers like CodeFormer. Enabled  by default.

enhance

Backend to use for lipsync. Default is "musetalk".

backend

Whether to downsample the video to 720p. Default is true. Recommended for MuseTalk backend for the best performance.

downsample

Method of cutting the audio and video when their durations mismatch. Default is "audio" which cuts the video to the audio's duration. "video" cuts the audio to the video's duration. "shortest" cuts the longer file to the shorter file's duration. More info in README.

cut_by

Whether to enhance the audio using the audio enhancement function. Default is true. Note: This is only for the lipsync process, not the final output.

enhance_audio

lipsync

Lipsync

Given a video or audio, generate a title, chapters, summary, tags, and highlights.

Whether to generate chapters or not. Defaults to True.

generate_chapters

Whether to generate highlights or not. Defaults to False.

generate_highlights

Maximum number of sentences in summary. Defaults to 5.

max_summary_length

Maximum number of words in title. Defaults to 10.

max_title_length

Number of tags to generate. Defaults to 5.

num_tags

Whether to denoise audio before analysis. Results in better transcription but slower processing. Defaults to True.

denoise_audio

Whether to use voice activity detection to split audio into segments. Results in less repetition on borders of transcription segments. Defaults to False.

use_vad

Whether to speed up processing by using a slightly faster transcription backend will less accurate word-timestamps. Defaults to False.

Topic(s) of highlights to generate, can be multiple comma-separated phrases. Can be anything from "Most interesting" to "Technology". Defaults to "Most interesting".

highlight_search_phrases

Whether to return each output as a JSON file (useful for efficient fetching of large payloads). Defaults to False.

return_as_json_file

video_transcript_analyzer

Analyze Transcripts

State-of-the-art audio-visual active speaker detection based on new, efficent face and speaker detection models.

Whether to use the faster but less accurate object detection model when processing the video.

The maximum number of faces to return per frame. If there are more than this number of faces, only the largest x faces will be returned.

max_num_faces

Whether to only return the frame data at the start of each scene cut or to return the frame data for every frame in the video.

return_scene_cuts_only

Whether to return the scene data along with the frame data. If True, the scene data will be returned in the "related_scene" field of the output.

return_scene_data

The seconds into the video to start processing from. Defaults to 0.

The seconds into the video to stop processing at. Defaults to -1, which means the end of the video.

The framerate to run object detection at for speaker detection. Defaults to 2.

processing_fps

A threshold that determines the minimum size of a face to run speaker detection on. Defaults to 0.5. Lower values allow for smaller faces to be used.

face_size_threshold

Active Speaker Detection

Fast, high quality speech transcription with many available backends, word-level timestamps, speaker diarization, and translation capabilities.

Whether to return word-level timestamps. Defaults to True.

word_level_timestamps

Whether to perform speaker diarization. Defaults to False.

speaker_diarization

Whether to use a smaller, less accurate model for faster speed. Defaults to False.

A choice between different model backends. Choices between "stable-ts", "whisper-timestamped", "whisperx" and "whisper-zero". See README for more information.

Language of the audio. Defaults to auto-detect if not specified. See README for supported language codes.

source_language

Language code of the language to translate to (doesn't translate if left blank). See README for supported language codes.

Minimum number of speakers to detect for diarization. Defaults to auto-detect when set to -1.

min_speakers

Maximum number of speakers to detect for diarization. Defaults to auto-detect when set to -1.

max_speakers

Minimum length of silence in seconds to use for splitting audio for parallel processing. Defaults to 0.4.

min_silence_length

Minimum length of audio segment in seconds to use for splitting audio for parallel processing. If set to -1, we pick a value based on your settings.

min_segment_length

A parameter to manually specify the start and end times of each chunk when splitting audio for parallel processing. If set to "", we use silence detection to split the audio. If set to a string formatted with a start and end second on each line, we use the specified chunks. Example: '0,10' and '10,20' on separate lines.

chunks

Whether to apply denoising to the audio to get rid of background noise before transcription. Defaults to False.

Whether to use Silero VAD for splitting audio into segments. Defaults to False. More accurate than ffmpeg silence detection.

Whether to use Pyannote segmentation for splitting audio into segments. Defaults to False.

use_pyannote_segmentation

vad_threshold

The threshold for Pyannote segmentation when merging segments. This value determines the number of seconds between words to merge segments. Defaults to 0.8.

pyannote_segmentation_threshold

A prompt to correct misspellings and style. Defaults to "".

initial_prompt

speech_transcriber

Speech Transcription

Generate audio-visual descriptions of videos with timestamp references.

video

The level of detail for the final description. Pick from 'concise', 'medium', or 'detailed'.

conciseness

The level of visual detail for the final description. Pick from 'low', 'medium', 'high', or 'ultra'.

visual_detail

Whether to use the transcript when generating the final description.

spoken_context

Whether to use object detection when generating the final description. BETA FEATURE.

object_context

If true, we prompt the underlying models to return even more details in their responses. This can be useful if the initial responses are too vague or lacking in detail.

detail_boost

If true, the video will be chunked by scene instead of by 60s intervals. This can be useful to extract more detailed descriptions for videos with multiple scenes or cuts.

chunk_by_scene

The minimum duration of a scene in seconds. By default, the function will set a minimum duration of 1/20 of the video duration. If the scene is shorter than this duration, it will be merged with the next scene. Only used if chunk_by_scene is enabled.

minimum_scene_duration

If true, the function will return the references used to generate the description as citations to each sentence, including the timestamps, visual captions and transcripts. chunk_by_scene is auto-enabled if this option is selected.

enable_references

If true, the function will return all the granular data used to generate the description, including the keyframes, visual captions, object detections, and summaries.

return_metadata

By default, describe makes a combination of calls (some which include OpenAI) that generate the most vivid descriptions. This variable instead allows you to simply sample the middle frame of the video for a pure visual description that is less detailed, but doesn't require any external API calls.

image_only

Any additional instructions on the questions to answer or the details to emphasize in the final description.

additional_instructions

llm_backend

describe

Describe

Filters for removing background noise, enhancing speech, and more in audio files.

Task to perform. See README for more details.

filter_type

(only applicable to `noise` and `enhance` settings). The number of steps (1-100) to take in enhancing the audio. Higher values yield better results but can make the process slow.

enhancement_steps

audio_enhancement

Audio Enhance

Generate and render video or audio highlights for long-form content based on search phrases.

If True, the function will render the clips and return the clips as files. If False, the function will just return the metadata of the highlights.

render_clips

The search phrases to use to generate highlights.

highlights

Highlight Generation

A set of text-to-speech models and tooling that helps generate natural-sounding speech, clone voices, control emotions, access word timestamps, and more.

CARTESIA_AI_API_KEY

voice

text

The reference audio to be used for voice cloning. Only required when a voice-cloning backend is selected.

reference_audio

The emotion you want the voice to have. Currently available for cartesia voice engine only.

emotion

The speaking rate of the voice. Controls wether the voice should speak fast, slow or normally.

pace

The stability determines how stable the voice is and the randomness between each generation. Currently available for elevenlabs. range ∈ [0-1].

stability

The style determines how much the voice is influenced by the reference audio. Currently available for elevenlabs. range ∈ [0-1].

style

Whether to return the timestamps of the spoken audio.

word_timestamps

Text to Speech

Detect scene changes in a video with PySceneDetect.

Threshold the average change in pixel intensity must exceed to trigger a cut. Only used if adaptive_threshold is False.

threshold

Whether to use adaptive thresholding, which compares the difference in content between adjacent frames without a fixed threshold, and instead a rolling average of adjacent frame changes. This can help mitigate false detections in situations such as fast camera motions.