Given a video or audio, generate a title, chapters, summary and tags

OPENAI_API_KEY

file

Maximum number of sentences in summary. Defaults to 5.

max_summary_length

Maximum number of words in title. Defaults to 10.

max_title_length

Number of tags to generate. Defaults to 5.

num_tags

Whether to generate chapters or not. Defaults to True.

generate_chapters

video_transcript_analyzer

source_video

target_audio

Whether to reduce the resolution of the video to half of the original on each axis. Significantly speeds up inference. Defaults to False.

low_resolution

Whether to reduce the fps of the video to half of the original. Significantly speeds up inference. Defaults to False.

low_fps

video_retalking

Remove background noise from audio and upsample to 48kHz.

audio

Task to perform, one of ["upsample", "noise", "all"]

filter_type

If True, use a faster but less accurate model for audio enhancement. Defaults to False.

enhance_speed_boost

Number of enhancement steps applied to the audio between 10 and 150. Higher values may improve quality but will take longer to process. Defaults to 50. Only applicable if enhance_speed_boost is False.

enhancement_steps

audio_enhancement

dis_background_remover

Generate a video of a person speaking a given text using lipsyncing.

ELEVEN_LABS_API_KEY

PLAYHT_API_KEY

PLAYHT_API_USER_ID

text

TTS model to use. Supported models: "xtts", "elevenlabs", or "playht". "elevenlabs" or "playht" is recommended for better quality but requires an API key.

tts_model

Value between 0 and 1. Increasing variability can make speech more expressive with output varying between re-generations. It can also lead to instabilities.

speech_stability

Value between 0 and 1. Low values are recommended if background artifacts are present in generated speech.

speech_similarity_boost

The ID of the ElevenLabs or Play.ht voice to use. If none are set, the voice will be cloned from the source audio and used. Only applicable if tts_model is "elevenlabs" or "playht".

voice_id

Whether to delete the voice after use. Only applicable if tts_model is "elevenlabs" or "playht".

cleanup_voice_id

Whether to refine the source audio using sieve/audio_enhancement.

refine_source_audio

Whether to refine the generated target audio using sieve/audio_enhancement.

refine_target_audio

Whether to reduce the resolution of the output video to half of the original on each axis; significantly speeds up inference.

Whether to reduce the fps of the output video to half of the original; significantly speeds up inference.

text_to_video_lipsync

Transcribe and translate live audio streams in real-time with high accuracy.

A URL to a live audio stream (RTMP, HLS, etc.). Needs to be supported by FFMPEG.

Language code of the language of the transcript (defaults to English if blank). See README for supported language codes.

target_language

Language code of the provided audio. Defaults to blank for auto-detection, but faster inference if the language is known.

stream_language

The interval at which to process transcripts. Must be > than 3 seconds.

chunk_size

a list of segments, each with a start time, end time, and text

live_speech_transcriber

Detect scene changes in a video with PySceneDetect.

video

Threshold the average change in pixel intensity must exceed to trigger a cut

threshold

pyscenedetect

Fast, high quality speech transcription with word-level timestamps and translation capabilities

Whether to return word-level timestamps. Defaults to True.

word_level_timestamps

Whether to perform speaker diarization. Defaults to False.

speaker_diarization

Whether to use a smaller, less accurate model for faster speed. Defaults to False.

speed_boost

Whether to enable a more accurate post-processing step at the cost of speed. Defaults to False.

decode_boost

Language of the audio. Defaults to auto-detect if not specified. Otherwise, specify the language code {en, fr, de, es, it, ja, zh, nl, uk, pt}. This may improve transcription speed.

source_language

Language code of the language to translate to (doesn't translate if left blank). See README for supported language codes.

Minimum number of speakers to detect for diarization. Defaults to auto-detect when set to -1.

min_speakers

Maximum number of speakers to detect for diarization. Defaults to auto-detect when set to -1.

max_speakers

Minimum length of silence in seconds to use for splitting audio for parallel processing. Defaults to 0.8.

min_silence_length

Minimum length of audio segment in seconds to use for splitting audio for parallel processing. If set to -1, we pick a value based on your settings.

min_segment_length

speech_transcriber

Generate captions for an image with ViT and GPT-2.

image

vit-gpt2-image-captioner

target_phrase

The text to replace the target phrase with

replacement_text

The URL of the voice model to use for voice conversion

voice_model_url

speech_editor

Urhythmic: Rhythm Modeling for Voice Conversion

length of audio segments during training in seconds

segment_length

training_timeout

audio_files

urhythmic_trainer

Download highest-resolution version of YouTube video as an MP4.

whether or not to ensure the audio is included in the video. if set to false, audio may still be included on occassion.

include_audio

youtube_to_mp4

Unofficial implementation of HiFi-GAN+ from the paper "Bandwidth Extension is All You Need" by Su, et al.

hifi_gan_plus

Detect the language of a text with LangID, a lightweight language identification tool

language code and confidence as a JSON object with keys `language_code` in ISO 639-1 format and `confidence` as a float

langid

AudioSR: Versatile Audio Super-resolution at Scale

ddim_steps

guidance_scale

audiosr

reference audio to use when cloning voice

reference_audio

if set, function will delete the specified voice ID instead of cloning a voice and returning the ID

delete_voice_id

playht_voice_cloning

The unique ID for a PlayHT or Cloned Voice.

voice

The quality of the generated audio. Choose between "draft", "low", "medium", "high", and "premium". The default is "high".

quality

Control how fast the generated audio should be. A number greater than 0 and less than or equal to 5.0.

speed

An integer number greater than or equal to 0. If equal to null or not provided, a random seed will be used. Useful to control the reproducibility of the generated audio. Assuming all other properties didn't change, a fixed seed should always generate the exact same audio file.

seed

A floating point number between 0, inclusive, and 2, inclusive. If equal to null or not provided, the model's default temperature will be used. The temperature parameter controls variance. Lower temperatures result in more predictable results, higher temperatures allow each run to vary more, so the voice may sound less like the baseline voice.

temperature

The voice engine used to synthesize the voice. Choose between "PlayHT2.0-turbo", "PlayHT2.0", and "PlayHT1.0". The default is "PlayHT2.0-turbo".

voice_engine

An emotion to be applied to the speech. Only supported when voice_engine is set to PlayHT2.0 or PlayHT2.0-turbo, and voice uses that engine.

emotion

A number between 1 and 6. Use lower numbers to reduce how unique your chosen voice will be compared to other voices. Higher numbers will maximize its individuality. Only supported when voice_engine is set to PlayHT2.0 or PlayHT2.0-turbo, and voice uses that engine.

voice_guidance

A number between 1 and 30. Use lower numbers to to reduce how strong your chosen emotion will be. Higher numbers will create a very emotional performance. Only supported when voice_engine is set to PlayHT2.0 or PlayHT2.0-turbo, and voice uses that engine.

style_guidance

A number between 1 and 2. This number influences how closely the generated speech adheres to the input text. Use lower values to create more fluid speech, but with a higher chance of deviating from the input text. Higher numbers will make the generated speech more accurate to the input text, ensuring that the words spoken align closely with the provided text. Only supported when voice_engine is set to PlayHT2.0, and voice uses that engine.

text_guidance

playht_speech_synthesis

A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) using on Deep Filtering.

deepfilternet_v2

elevenlabs_voice_cloning

the ID of the ElevenLabs voice to use. If none are set, the ID of Rachel will be used.

stability

similarity_boost

Value between 0 and 1. High values are recommended if the style of the speech should be exaggerated compared to the uploaded audio. Higher values can lead to more instability in the generated speech. Setting this to 0.0 will greatly increase generation speed and is the default setting.

style

Boost the similarity of the synthesized speech and the voice at the cost of some generation speed.

use_speaker_boost

pick between eleven_multilingual_v2, eleven_multilingual_v1, and eleven_monolingual_v1

model_id

elevenlabs_speech_synthesis

Practical Algorithms for General Image/Video Restoration.

the factor to scale the image or video by

scale

real-esrgan

Upscale an input video by a given factor.

upscale the video by a factor of scale. Default is 2.

video_upscaling

Ultralytics YOLOv8, the latest version of the acclaimed real-time object detection model.

Image or video file. If video, a generator is returned with the results for each frame.

Confidence threshold for the predictions.

confidence_threshold

The class that should be detected. There are 80 classes to choose from for detections (see README). Entering -1 for the classes parameter detects all of them. To detect any one, you can enter the corresponding class number as input for classes.

classes

yolov8l

Detect faces in image and video with MediaPipe.

List of faces with their bounding boxes, classes, and scores

mediapipe_face_detector

Generate audio that sounds similar to a reference speaker in 13 languages

language code of the text to generate. try "en" for English, "es" for Spanish, or others below. If left blank, the language will be detected automatically.

language_code

xtts

Translate text into 96 different languages

Source language. Try "eng" for English, "spa" for Spanish, or others below.

Target language. Try "eng" for English, "spa" for Spanish, or others below.

seamless_text2text

Visual question answering with GPT-4 level capabilities.

prompt to use for text generation. the prompt is typically a question or a caption.

prompt

when decoding text, samples from the top p percentage of most likely tokens; lower to ignore less likely tokens

top_p

adjusts randomness of outputs, greater than 1 is random and 0 is deterministic

maximum number of tokens to generate. A word is generally 2-3 tokens

max_tokens

llava-vl-13b

start time of the audio in seconds. Defaults to 0.

start_time

end time of the audio in seconds. Defaults to -1 (end of audio).

end_time

Minimum number of speakers to detect. If set to -1, the number of speakers is automatically detected.

Maximum number of speakers to detect. If set to -1, the number of speakers is automatically detected.

pyannote-diarization

An version of Whisper that offers accurate word-level timestamps and little hallucination.

whether to use the smaller, faster model (large-v3 vs base)

A prompt to correct misspellings and style.

initial_prompt

Language code of the audio (defaults to English), faster inference if the language is known.