Smart, automatic cropping of a video to a given aspect ratio based on subject detection and speaker tracking.

file

whether or not to determine crops based on who's actively speaking in the video. Defaults to False.

active_speaker_detection

the aspect ratio of the video to create. Must be "1:1", "4:3", "3:4", "4:5", "5:4", or "9:16". You may also specify multiple aspect ratios separated by commas to get metadata for each one. Defaults to "9:16".

aspect_ratio

whether or not to return the fully edited video file or just the metadata so you can render on your own. Defaults to False. If multiple aspect ratios are specified, while return_video is True, only the first aspect ratio will be used.

return_video

the seconds into the video to start processing from. Defaults to 0.

start_time

the seconds into the video to stop processing at. Defaults to -1, which means the end of the video.

end_time

whether or not to use a faster, but less accurate, model for the object detection that powers the cropping. Defaults to False.

speed_boost

whether or not to let the function auto-pick the main subject to focus on. Note: this setting is still experimental but it could yield better results on a broader range of videos. Defaults to False.

smart_edit

whether or not to return a visualization of the crops and bounding boxes overlaid onto the original video (for debugging purposes). Defaults to False.

visualize

whether or not to include the information about the subjects (people, faces, etc) of the crops in the metadata. Defaults to False.

include_subjects

whether or not to include the versions of the layouts if speaker detection was off in the output metadata. This is useful if you want the option to switch between active speaker and non-active speaker layouts without having to reprocess the video. Defaults to False and is only used if active_speaker_detection is True and return_video is False.

include_non_active_layouts

a comma-separated list of natural language class prompts to detect in the video. Defaults to "person". You can specify any class in natural language.

prompt

a comma-separated list of natural language class prompts to exclude from the detection in the video. Defaults to "".

negative_prompt

whether or not to only allow single crop layout versus multi-crop layouts. Defaults to False.

single_crop_only

the speed at which the crops moves from one frame to another. Defaults to 0.1. Must be between 0 and 1.

crop_movement_speed

the number of seconds between each sample taken for an updated crop position. Should be lower for faster moving subjects, and higher for still subjects. Defaults to 3.

crop_sampling_interval

autocrop

Auto Crop

Given a video or audio, generate a title, chapters, summary, tags, and highlights.

OPENAI_API_KEY

Whether to generate chapters or not. Defaults to True.

generate_chapters

Whether to generate highlights or not. Defaults to False.

generate_highlights

Maximum number of sentences in summary. Defaults to 5.

max_summary_length

Maximum number of words in title. Defaults to 10.

max_title_length

Number of tags to generate. Defaults to 5.

num_tags

Whether to denoise audio before analysis. Results in better transcription but slower processing. Defaults to True.

denoise_audio

Whether to use voice activity detection to split audio into segments. Results in less repetition on borders of transcription segments. Defaults to False.

use_vad

Whether to speed up processing by using a slightly faster transcription backend will less accurate word-timestamps. Defaults to False.

Topic(s) of highlights to generate, can be multiple comma-separated phrases. Can be anything from "Most interesting" to "Technology". Defaults to "Most interesting".

highlight_search_phrases

video_transcript_analyzer

Analyze Transcripts

source_video

target_audio

Whether to reduce the resolution of the video to half of the original on each axis. Significantly speeds up inference. Defaults to False.

low_resolution

Whether to reduce the fps of the video to half of the original. Significantly speeds up inference. Defaults to False.

low_fps

video_retalking

Lip Sync

Fast, high quality speech transcription with word-level timestamps and translation capabilities

Whether to return word-level timestamps. Defaults to True.

word_level_timestamps

Whether to perform speaker diarization. Defaults to False.

speaker_diarization

Whether to use a smaller, less accurate model for faster speed. Defaults to False.

A choice between different model backends. Choices between "stable-ts", "whisper-timestamped", and "whisperx". See README for more information.

backend

Language of the audio. Defaults to auto-detect if not specified. See README for supported language codes.

source_language

Language code of the language to translate to (doesn't translate if left blank). See README for supported language codes.

target_language

Minimum number of speakers to detect for diarization. Defaults to auto-detect when set to -1.

min_speakers

Maximum number of speakers to detect for diarization. Defaults to auto-detect when set to -1.

max_speakers

Minimum length of silence in seconds to use for splitting audio for parallel processing. Defaults to 0.4.

min_silence_length

Minimum length of audio segment in seconds to use for splitting audio for parallel processing. If set to -1, we pick a value based on your settings.

min_segment_length

A parameter to manually specify the start and end times of each chunk when splitting audio for parallel processing. If set to "", we use silence detection to split the audio. If set to a string formatted with a start and end second on each line, we use the specified chunks. Example: '0,10' and '10,20' on separate lines.

chunks

Whether to apply denoising to the audio to get rid of background noise before transcription. Defaults to False.

Whether to use Silero VAD for splitting audio into segments. Defaults to False. More accurate than ffmpeg silence detection.

Whether to use Pyannote segmentation for splitting audio into segments. Defaults to False.

use_pyannote_segmentation

vad_threshold

A prompt to correct misspellings and style. Defaults to "".

initial_prompt

speech_transcriber

Transcribe Speech

Translate any video or audio to several languages

ELEVEN_LABS_API_KEY

source_file

The language to which the audio will be translated. Default is "spanish". See supported languages in README.

Whether to enable lip-syncing on the original video to the dubbed audio. Defaults to False. Only applicable for video inputs.

enable_lipsyncing

dubbing

Dub Content

Generate a description of a video using both audio and visual context.

TOGETHERAI_API_KEY

video

The level of detail for the final description. Pick from 'concise', 'medium', or 'detailed'

conciseness

The level of visual detail for the final description. Pick from 'low', 'medium', or 'high'

visual_detail

Whether to use the transcript when generating the final description

spoken_context

Whether to use object detection when generating the final description. BETA FEATURE.

object_context

If true, we prompt the underlying models to return even more details in their responses. This can be useful if the initial responses are too vague or lacking in detail.

detail_boost

If true, the video will be chunked by scene instead of by 60s intervals. This can be useful for videos with multiple scenes or cuts.

chunk_by_scene

If true, the function will return all the granular data used to generate the description, including the keyframes, visual captions, object detections, and summaries.

return_metadata

By default, describe makes a combination of calls (some which include OpenAI) that generate the most vivid descriptions. This variable instead allows you to simply sample the middle frame of the video for a pure visual description that is less detailed, but doesn't require any external API calls.

image_only

Any additional instructions on the questions to answer or the details to emphasize in the final description

additional_instructions

The backend to use for the LLM model. Pick from 'openai' or 'mixtral'. Requires 3rd party API keys. See the README for more information.

llm_backend

describe

Generate a Video Description

Remove background noise from audio and upsample it.

audio

Task to perform, one of ["upsample", "noise", "all"]

filter_type

If True, use a faster, more experimental model for audio enhancement. Defaults to False.

Number of enhancement steps applied to the audio between 10 and 150. Higher values may improve quality but will take longer to process. Defaults to 50. Only applicable if enhance_speed_boost is False.

enhancement_steps

audio_enhancement

Enhance Audio

Reliable text translation through complex tokenization and sentence splitting.

text

The source language of the text in ISO 639-1 format. If 'auto', the language will be detected automatically.

source

The target language of the text in ISO 639-1 format. Default is 'es' (Spanish).

target

translate

Text Translation

State-of-the-art active speaker detection based on new, efficent face and speaker detection models.

Whether to use the faster but less accurate object detection model when processing the video.

The maximum number of faces to return per frame. If there are more than this number of faces, only the largest x faces will be returned.

max_num_faces

Whether to only return the frame data at the start of each scene cut or to return the frame data for every frame in the video.

return_scene_cuts_only

Whether to return the scene data along with the frame data. If True, the scene data will be returned in the "related_scene" field of the output.

return_scene_data

The seconds into the video to start processing from. Defaults to 0.

The seconds into the video to stop processing at. Defaults to -1, which means the end of the video.

The framerate to run object detection at for speaker detection. Defaults to 2.

processing_fps

A threshold that determines the minimum size of a face to run speaker detection on. Defaults to 0.5. Lower values allow for smaller faces to be used.

face_size_threshold

Detect Active Speakers

Generate and render video or audio highlights for long-form content based on search phrases.

If True, the function will render the clips and return the clips as files. If False, the function will just return the metadata of the highlights.

render_clips

The search phrases to use to generate highlights.

highlights

Generate Content Highlights

Transcribe and translate live audio streams in real-time with high accuracy.

A URL to a live audio stream (RTMP, HLS, etc.). Needs to be supported by FFMPEG.

Language code of the language of the transcript (defaults to English if blank). See README for supported language codes.

Language code of the provided audio. Defaults to blank for auto-detection, but faster inference if the language is known.

stream_language

The interval at which to process transcripts. Must be > than 3 seconds.

chunk_size

a list of segments, each with a start time, end time, and text

live_speech_transcriber

Transcribe a Live Stream

Generate sound effects for stock videos using AI.

The video to generate a sound effect for.

The duration of the sound effect in seconds. Must be less than 20 seconds.

duration

video-sound-effect

Generate AI Sound Effects

Detect scene changes in a video with PySceneDetect.

Threshold the average change in pixel intensity must exceed to trigger a cut. Only used if adaptive_threshold is False.

threshold

Whether to use adaptive thresholding, which compares the difference in content between adjacent frames without a fixed threshold, and instead a rolling average of adjacent frame changes. This can help mitigate false detections in situations such as fast camera motions.

adaptive_threshold

pyscenedetect

Detect Scenes in Video

Generate a video of a person speaking a given text using lipsyncing.

PLAYHT_API_KEY

PLAYHT_API_USER_ID

TTS model to use. Supported models: "xtts", "elevenlabs", or "playht". "elevenlabs" or "playht" is recommended for better quality but requires an API key.

tts_model

Value between 0 and 1. Increasing variability can make speech more expressive with output varying between re-generations. It can also lead to instabilities.

speech_stability

Value between 0 and 1. Low values are recommended if background artifacts are present in generated speech.

speech_similarity_boost

The ID of the ElevenLabs or Play.ht voice to use. If none are set, the voice will be cloned from the source audio and used. Only applicable if tts_model is "elevenlabs" or "playht".

voice_id

Whether to delete the voice after use. Only applicable if tts_model is "elevenlabs" or "playht".