
<div class='w-full border border-gray-200 rounded-md p-4 mb-4'>
    <span class='text-md text-gray-600 text-semibold'>
        <b>Note:</b>
        This article was also posted on Zight's blog. You can read it
        <a href="https://zight.com/blog/zight-uses-sieve-for-video-ai" class="text-sieve-purple-500">
            here.
        </a>
    </span>
</div>

At [Zight](https://zight.com/), the product and engineering teams rely on Sieve for the development and deployment of all the capabilities that come with [Zight’s AI offerings](https://zight.com/ai/). We discuss the importance of AI in video communication and why Zight chose Sieve to power these new features.

<div class="w-full border border-gray-200 rounded-md px-8 pt-2 pb-8 mb-4" role="alert">
  <p class="text-3xl font-bold">Partnership Outcomes</p>
  <div>
    <p>🕒 <span class="font-semibold">160 engineering hours saved per month.</span> Zight streamlined GPU infrastructure maintenance, model setups, and output quality enhancements.</p>
    <p>⚡ <span class="font-semibold">10x faster time to market.</span> Zight launched their AI summaries feature in under 2 weeks, a task that would have otherwise taken 5 months.</p>
    <p>💰 <span class="font-semibold">$600k saved over 5 months.</span> By leveraging Sieve, Zight avoided hiring 1-2 in-house ML engineers and circumvented the inefficient use of on-demand compute resources from AWS.</p>
  </div>
</div>

Below, we interview Zight’s Head of Product, [Phin](https://www.linkedin.com/in/phin/), who leads the efforts on all of [Zight’s AI offerings](https://zight.com/ai/).

## The importance of AI in workplace communication

### **Can you describe what your company does in a few sentences?**

The [Zight](https://zight.com/) platform lets you capture and share content like [videos](https://zight.com/screen-recorder/), [screenshots](https://zight.com/screenshot-app/), gifs, and files. You also get notified when your content is viewed or interacted with.

Different departments in a company use it. For instance, the support and customer success teams send personalized troubleshooting videos or screenshots with step-by-step instructions. The sales team uses it for short outreach videos. The engineering and product teams use it for demos and bug reports. They can send device log files, a video showing how-to steps, or a screenshot of their console while debugging with a teammate — all in one tool.

### When did you realize you needed to integrate AI capabilities?

Customers can create lots of content, but it can become disorganized quickly. We provide options to rename and sort content into folders, but this requires user action, which many people don't do.

Sharing content, like a 5-minute video, without any context can make it hard for viewers to determine its relevance. Often, you might receive a link via Slack, Teams, or Email with little information, making it the viewer's responsibility to watch and then respond.

We realized we could potentially automate quite a bit of this to build a nicer end-user experience within Zight.

## Why we chose Sieve for smart titles and summaries

We used Sieve to automatically add titles, descriptions, and transcripts to each video recorded in Zight. This helps make videos more engaging and easier to understand. It also makes videos searchable, which means it is easier to find the information you need over time.

### Time to Market

Our initial integration with Sieve took just under a day from signing up for an account to having it live in our staging environment. We could just call one endpoint and get properly formatted content for everything we were doing along with all the nice-to-haves like error handling, webhooks, monitoring, and more. This allowed us to get the closed beta in front of customers extremely quickly.

### Modularity and Consolidation

More broadly, we were also thinking about our long-term AI strategy. Previously, we were using a different provider for AI transcriptions but we knew we wanted to build a lot more on top of this — with titles and descriptions being just the start. We realized that many of the other features we wanted to build like translations, automated chapters, removing filler words, tagging screenshots, dubbing video, etc. all could use some common set of models under the hood.

Sieve’s [explore](https://www.sievedata.com/explore) page was my first view into realizing how easy Sieve makes building each one of these features. The pre-built apps were a great way for us to start, but unlike other providers, Sieve made it very easy to build custom apps through their core building blocks like the [transcription](https://www.sievedata.com/functions/sieve/speech_transcriber), [visual question-answering](https://www.sievedata.com/functions/sieve/cogvlm-chat), and the ability to [deploy custom functions](https://docs.sievedata.com/guide/custom-apps/deploying-functions) while only needing to think about the logic on top.

Every app comes with a nice playground you can use without writing any code. This is especially helpful for me as a product leader that may not always want to dig into code.

![Transcript Analysis App](https://storage.googleapis.com/sieve-public-data/zight-case-study/zight-transcript-analysis.webp)

Sieve also allows our developers to deploy custom apps depending on our needs extremely quickly. This gives us a set of abstraction that we can peek at when we want to, without adding unnecessary complexity when we don’t need it.

![Custom Function Example](https://storage.googleapis.com/sieve-public-data/zight-case-study/zight-custom-function.webp)

![Video Dubbing Example](https://storage.googleapis.com/sieve-public-data/zight-case-study/zight-video-dubbing.webp)

### Cost

AI is known for notoriously high costs but Sieve’s platform gave us control over the tradeoffs we wanted to make on **quality, cost, and speed** in a few ways — unlike most other black box solutions out there.

- They support incredibly bursty workloads that let us spin up and down from 0→100→0 concurrent requests without loss in performance.
- The out-of-the-box apps let us select options to control the tradeoff between quality and speed. For example, [on the transcription app](https://www.sievedata.com/functions/sieve/speech_transcriber) we used, it was easy to test the `speed_bost` and `decode_boost` options to fit our use case.
- Sieve also allows for private custom deployments of these public functions as we see fit for our use case and scale.

## Sieve can do so much more

At Zight, we know the immense value of AI in our product roadmap moving forward and we love tools that can help our team solve the complex problems around it. Sieve proved to be such a useful platform, that we thought we’d share the experience on our blog. On top of what we mentioned in this post, Sieve’s platform can do so much more.

Check out other apps on their [explore](https://www.sievedata.com/explore) page or [their blog](https://www.sievedata.com/blog) to read about other AI use cases and how they’ve built an easy-to-use experience, while prioritizing performance.


Zight and Sieve: Using AI to build better video communication

Finding highlights in long-form video content automatically

How developers are changing video creation once again with AI

Introducing Describe: Incredibly descriptive audiovisual summaries for videos


Our team recently saw the new [“sound effect” feature from Pika Labs](https://twitter.com/pika_labs/status/1766554610188095642) and found it extremely fun to play with. So we decided to make our own version! Basically the goal is to take an input 5-10 second clip, and generated the corresponding sounds that match.

<iframe width="100%" height="315" src="https://www.youtube.com/embed/gj6oHmRRX6Q" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

Generally we think something like this could be useful for folks trying to add sound effects to stock video content. Instead of using stock audio and sound effect libraries like [Storyblocks](https://www.storyblocks.com/audio), [Adobe](https://stock.adobe.com/audio), or [PremiumBeat](https://www.premiumbeat.com/royalty-free/stock-audio) — why not generate it with AI?

Adding appropriate sound effects to videos means being able to understand what’s going on in the video, what it could sound like, and then generating appropriate sounds given all of that context.

## How it Works

We’ve seen multi-modal AI models like OpenAI’s GPT-4 take in simple text prompts to generate images. This is done by getting the LLM to first expand the text prompt, which it then feeds into DALLE 3. Similarly, our idea is to do the following:

- extract some frames in the stock video
- get a model to describe what that video is or might sound like
- get a model to generate audio using that description

And turns out there’s some great open-source models that allow us to do just this. Vision language models (VLMs) are starting to get really good. These models can take in an image along with a prompt and then respond to it based on the image. We have a couple of these available on Sieve such as [CogVLM](https://www.sievedata.com/functions/sieve/cogvlm-chat), [InternLM](https://www.sievedata.com/functions/sieve/internlmx-composer-2q), and [Moondream](https://www.sievedata.com/functions/sieve/moondream) — each of which come with a certain cost / quality tradeoff. For this app, we decided to pick CogVLM which we found to be really descriptive. Here’s how we use Sieve to prompt it.

```python
import sieve

image = sieve.Image(path="./some_image.jpg")
prompt = "describe what you might hear in this image in detail."

cogvlm_chat = sieve.function.get("sieve/cogvlm-chat")
output = cogvlm_chat.run(image, prompt)

print(output)
```

We then take this prompt and feed it into [AudioLDM](https://www.sievedata.com/functions/sieve/audioldm), a state-of-the-art audio generation model.

```python
audioldm = sieve.function.get("sieve/audioldm")
audio = audioldm.run(output)
print(audio)
```

In the case of video, we decide to simply sample the middle frame to pass into our VLM though there could be more robust approaches to this using video-native models in the future. One all of this is complete, we’re able to create videos like these. **All of the sound you hear was generated with AI!**

<script src="https://cdn.jsdelivr.net/npm/@mux/mux-player"></script>

<mux-player
  playback-id="00oJbhQcTEyvdPpiFKwsfJa01vHZV02701ow2zxtkDeyzvg"
  metadata-video-title="Bee Video"
  primary-color="#FFFFFF"
  accent-color="#7240FF"
  secondary-color="#000000"
/>

<mux-player
  playback-id="01vHHGPZkH5uDtNbg01I3Zd8tYVHv00kAqJ01XWJB8aDlQs"
  metadata-video-title="Bacon Video"
  primary-color="#FFFFFF"
  accent-color="#7240FF"
  secondary-color="#000000"
/>

You can try the app for yourself [here](https://www.sievedata.com/functions/sieve/video-sound-effect) and we’ve open-source all the code [here](https://github.com/sieve-community/video-sound-effect/tree/main) so you can modify it however you like.


Adding Sound Effects to Stock Videos with AI


In recent months, the demand for smaller and smaller AI models has continued to grow. Thanks to some clever tricks like quantization, better curated training data, synthetic data, and other smart tweaks, these models are shrinking in parameter size without losing their core capabilities. We’re excited about this because it means our customers can build more interactive features accessible to more of their users without bearing the expense of running large models.

Take [moondream2](https://github.com/vikhyat/moondream) for example. It’s a tiny vision-language model that can run on devices as small as the Raspberry Pi. I asked it the following.

> Question: What logo is on the player’s shirt and what is he doing?

> Answer: Nike is on the player’s shirt, and they are walking into the stadium.

![soccer player image](https://storage.googleapis.com/sieve-public-data/gpu-sharing-blog/soccer-moondream.webp)

Pretty good! As models get smaller, they’ll get easier to run on the edge, but it also means you’ll need less resources to run them in the cloud.

At Sieve, we want to make it as easy as possible to discover, design, and run AI. A big part of that means running AI efficiently and so today, we’re excited to announce official support for GPU sharing on Sieve, with some of our public functions already making use of it.

## What is it?

GPU time-sharing allows multiple containers to share a single physical GPU. This lets each container more efficiently use GPU resources to save running costs.

Time sharing works by allocating short time slices to different containers that share a single physical GPU in a round-robin fashion. More technically, the GPU context switches between all of the containers that are sharing the GPU. At any point in time, only one container can occupy the GPU. However, over a fixed time interval, the context switching ensures that each container gets the right proportion of the time-slice.

## What is it useful for?

If you’re already seeing maximal utilization on a GPU, this feature isn’t really useful. However, most workloads aren’t like this. Many computer vision models, for example, perform CPU-intensive operations such as OpenCV preprocessing, video frame reading, and more, which are intertwined with the actual model's operation on a GPU. You could call these types of workloads “GPU bursty” since they are only using the GPU when they’re not doing the other pre or post processing steps.

Turning on GPU timesharing means that while your code may be doing some pre-processing, another process can be utilizing the GPU for a model operation at full speed. This makes timesharing especially useful for these sorts of “GPU bursty” workloads.

## How is this different than Multi-Instance GPU (MIG)?

The latest NVIDIA GPUs have an operation mode called Multi-Instance GPU (MIG). [NVIDIA’s docs](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html) compare MIG with time-sharing as follows:

> MIG allows you to partition a GPU into several smaller, predefined instances, each of which looks like a mini-GPU that provides memory and fault isolation at the hardware layer. You can share access to a GPU by running workloads on one of these predefined instances instead of the full native GPU.

> Time-slicing trades the memory and fault-isolation that is provided by MIG for the ability to share a GPU by a larger number of users. Time-slicing also provides a way to provide shared access to a GPU for older generation GPUs that do not support MIG. However, you can combine MIG and time-slicing to provide shared access to MIG instances.

On Sieve, we also offer the A100 20GB compute type, which is an A100 using the MIG operation mode. You can choose to further slice this with timesharing if you’d like maximal utilization along with hardware guarantees.

## Sample Benchmark

Let’s run a benchmark with the [video-retalking](https://github.com/OpenTalker/video-retalking) model, an AI model that performs relatively high quality lipsync. The interesting thing about this repository is that it’s actually a combination of models that detect face landmarks, perform varying image transformation, generate lips, upscale images, and more, to then generate the output video.

We optimized this model and used to run it on an A100 40GB GPU, which would take ~10 seconds to generate a single second of video. This was already much faster than the original repository, which would takes 60s to generate a single second of video ([read more about our work here](https://www.sievedata.com/blog/fast-high-quality-ai-lipsyncing)!).

We recently moved this model to run on an A100 with GPU time-sharing set to a split of 2. Previously, the model would use the whole GPU for 10 seconds to generate a single second of video. With time-sharing enabled, the model now uses half the A100 to generate a single second of video in 12.5 seconds. While the individual generation is slower, two generations can now run concurrently.

What used to be 10 seconds of A100 compute time for 1 second of generated video is now 12.5 seconds of compute time for 2 seconds of generated video. That’s almost a **40% efficiency gain with no other code changes**!

<div className="border border-gray-200 rounded-md py-4 px-2">
  <img src="https://storage.googleapis.com/sieve-public-data/gpu-sharing-blog/perf-graph.webp" alt="performance graph" />
</div>

## How do I use it?

If you’re using public functions on Sieve, we’re slowly transitioning various functions to leverage GPU sharing. [YOLO](https://www.sievedata.com/functions/sieve/yolov8), for example, splits a T4 two ways, [XTTS](https://www.sievedata.com/functions/sieve/xtts) splits an A100 4 ways, [TalkNet](https://sievedata.com/functions/sieve/talknet-asd) splits an L4 3 ways, and [Video Retalking](https://www.sievedata.com/functions/sieve/video_retalking) splits an A100 two ways now. This generally means greater throughput and lower cost for these functions.

If you’re deploying custom functions on Sieve, enable time-sharing is as simple as adding a `split` option to your decorator.

```python
@sieve.function(
	name="gpu_function_sharing",
	gpu=sieve.gpu.T4(split=3)
)
def runs_on_gpu(video: sieve.File) -> sieve.File:
	...
```

If no split is specified, the GPU will be used regularly without time sharing. Read more about it [in our docs](https://docs.sievedata.com/guide/custom-apps/gpu-acceleration#gpu-sharing).

## Conclusion

A core belief we have at Sieve is the idea that combining many models together makes building [useful AI apps easier and faster](https://www.sievedata.com/blog/fast-active-speaker-detection). With GPU sharing, it means each of these models will naturally become even faster and more cost-effective.

We’ve already seen this feature magically speed up certain customer workloads and we’re excited to see what you build with the gain in efficiency :)


Introducing GPU sharing on Sieve

Fast, efficient active speaker detection on videos

Announcing the most cost-effective audio transcription API

Improving on open-source for fast, high-quality AI lipsyncing


In the world of multimedia and podcasting, user experience often hinges on audio quality. Superior quality has the capacity to captivate audiences, encouraging prolonged engagement with your content, whether they be podcasts, tutorials, or promos. More specifically, audio quality is not just about enhancing user experience but also plays a crucial role in ensuring the accuracy of downstream AI tasks such as automated transcription services, like transcription with [OpenAI's Whisper](https://github.com/openai/whisper). In fact, an entire industry of startups like [Descript](https://www.descript.com), [Krisp AI](https://krisp.ai), and more are working in the space.

With the help of a few open source models, namely [AudioSR](https://github.com/haoheliu/versatile_audio_super_resolution) and [DeepFilterNet](https://github.com/rikorose/deepfilternet), we’ve launched an audio enhancement app that helps remove background noise and enhance speech, making sure your audio quality is always up to par.

Here’s a quick example of the sound quality difference using open-source models:

<script src="https://cdn.jsdelivr.net/npm/@mux/mux-player"></script>

<mux-player
  stream-type="on-demand"
  playback-id="mu3U00wldTaxIg7aCcdo00fS2ObM01oZqtUpUeQkO6lVW4"
  metadata-video-title="Placeholder (optional)"
  metadata-viewer-user-id="Placeholder (optional)"
  primary-color="#FFFFFF"
  secondary-color="#000000"
/>

In this post, we’ll go through a quick demonstration of how you might integrate this solution into your AI project and some background on the models we used.

## Trying it out

You can try out the pre-built audio enhancement app in a few clicks here with your own audio samples [here](https://www.sievedata.com/functions/sieve/audio_enhancement). Here’s a few more samples from podcasts and YouTube videos:

**Sample 1**

<div class="sm:flex sm:space-x-6">
  <div class="sm:w-1/2">
    <span class="block mt-2 mb-2 sm:mt-0">Original</span>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample1/original.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
  <div class="sm:w-1/2">
    <span class="block mt-2 mb-2 sm:mt-0">Enhanced</span>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample1/sieve.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
</div>

**Sample 2**

<div class="sm:flex sm:space-x-6">
  <div class="sm:w-1/2">
    <span class="block mt-2 mb-2 sm:mt-0">Original</span>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample2/original.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
  <div class="sm:w-1/2">
    <span class="block mt-2 mb-2 sm:mt-0">Enhanced</span>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample2/sieve.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
</div>

### Run via API or Python

You can also integrate the app into your current workflow through an API call or Python call with the following steps:

1. [Sign up](https://www.sievedata.com/dashboard) for a Sieve account and [find your API key here](https://www.sievedata.com/dashboard/settings).
2. Run audio enhancement via API (or see below for Python)
   ```bash
   curl -X POST https://mango.sievedata.com/v2/push \
   -H "X-API-Key: <your-api-key>" \
   -d '{
     "function": "sieve/audio_enhancement",
     "inputs": {
       "audio": {
         "url": "<your-audio-url>"
       }
     }
   }'
   ```
3. Run via Python client

   - Install the Python client

   ```
   pip install sievedata
   ```

   - Login with your [API key](https://www.sievedata.com/dashboard/settings)

   ```bash
   sieve login
   ```

   - Run this Python script with your own audio!

   ```python
   import sieve

   audio_enhancer = sieve.function.get('sieve/audio_enhancement')

   # Specify "upsample", "noise" or "all" for the filtering type
   enhanced_audio = audio_enhancer.run(sieve.Audio(path="./speech.wav"), "upsample")

   # View results on Sieve dashboard or locally from this path
   print(enhanced_audio.path)
   ```

It’s as simple as that! Results will now be viewable on your Sieve dashboard or directly from your Python code.

## **How it works**

The magic lies in the tandem of two open-source AI models - AudioSR and DeepFilterNet.

### AudioSR

AudioSR is a generative model that uses a diffusion-based approach to estimate the high-frequency components of a low-resolution audio signal. It does this by training a latent diffusion model to learn the conditional generation of high-resolution spectrograms from low-resolution spectrograms. The model can handle a flexible input sampling rate between 4kHz and 32kHz, covering most real-world scenarios. AudioSR has achieved promising results on speech, music, and sound effects with different input sampling rate settings and has been verified to be a plug-and-play module for enhancing the audio quality of various audio generation models.

### DeepFilterNet

DeepFilterNet is a deep learning-based speech enhancement framework that utilizes harmonic structure of speech to efficiently enhance speech quality by removing unwanted noise from audio files. It operates in two stages, with the first stage enhancing the speech envelope in the ERB domain, and the second stage using deep filtering to enhance the periodic component. Several optimizations have been made to the training procedure, data augmentation, and network structure resulting in state-of-the-art speech enhancement performance while reducing processing, making it applicable to run on embedded devices in real-time.

## **Other Solutions**

Let's take a look at how open source stands against the other prominent ones in the market, namely the [Dolby Enhance API](https://dolby.io/products/enhance/):

<div class="grid grid-cols-1 md:grid-cols-3 gap-4 w-max">
  <div class="mx-3 flex flex-col gap-3">
    <p class="text-center">Original</p>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample1/original.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample2/original.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
  <div class="mx-3 flex flex-col gap-3">
    <p class="text-center">Dolby</p>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample1/dolby.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample2/dolby.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
  <div class="mx-3 flex flex-col gap-3">
    <p class="text-center">AudioSR + DeepFilterNet</p>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample1/sieve.mp3" type="audio/wav">
      Your browser does not support the audio element.
    </audio>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample2/sieve.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
</div>

Very promising results from a purely open-source solution!

## What next?

Audio enhancement is a feature that can be used upstream of most other AI audio functionality. For example, a feature like speech editing ([also featured on Sieve](https://www.sievedata.com/functions/sieve/speech_editor)) can use audio enhancement capabilities to enhance the quality of the generated voices.

Sieve's cloud platform makes combining functionality in this way easy. To try it, [create an account](https://www.sievedata.com/dashboard)!


State of the art audio enhancement in 5 minutes


<div class='w-full border border-gray-200 rounded-md p-4 mb-4'>
    <span class='text-md text-gray-600 text-semibold'>
        <b>Note:</b>
        This blog is outdated. If you're interested in a production-ready app for this capability, please check out the
        <a href="https://www.sievedata.com/functions/sieve/video_transcript_analyzer" class="text-sieve-purple-500 hover:text-sieve-purple-700">
            video transcript analyzer
        </a>
        on Sieve.
    </span>
</div>

Recently, we've seen content platforms like YouTube implement [video chapters](https://support.google.com/youtube/answer/9884579?hl=en) to make their content more digestable for viewers.

With the rise of generative AI, we thought it'd be useful to build a workflow to auto-generate these video chapters using public models including OpenAI's [Whisper](https://openai.com/research/whisper), [GPT-3](https://openai.com/blog/gpt-3-apps), and [nltk's Text Tiling](https://www.nltk.org/_modules/nltk/tokenize/texttiling.html) implementation. In this blog post, we'll dive into the specifics of our workflow and how it works.

If you'd like to try it for yourself or integrate into your own app, try it on Sieve by clicking the button below. Keep reading if you'd like to learn about how it works.

<div class='w-full text-center'>
  <a href='https://sievedata.com/dashboard'
    className='group inline-flex items-center h-9 rounded-md text-sm font-medium whitespace-nowrap px-3 focus:outline-none focus:ring-none transition-colors duration-200 ease-in-out reverse flex-row-reverse text-gray-100 hover:text-sieve-purple-700 bg-sieve-purple-500 hover:bg-transparent border border-sieve-purple-500 mr-2'
    style="text-decoration: none"
  >
    Try it on Sieve
  </a>
  <a href='https://github.com/sieve-community/examples/tree/main/auto_chapter_title'
    className='group inline-flex items-center h-9 rounded-md text-sm font-medium whitespace-nowrap px-3 focus:outline-none focus:ring-none transition-colors duration-200 ease-in-out reverse flex-row-reverse text-gray-100 hover:text-sieve-purple-700 bg-sieve-purple-500 hover:bg-transparent border border-sieve-purple-500 ml-2'
    style="text-decoration: none"
  >
    See implementation
  </a>
</div>

## Audio Transcription

Speech-to-text models have been around for more than a few years, but none rival OpenAI's Whisper model in accuracy & speed. With OpenAI's Whisper model running on Sieve, we're able to generate an accurate transcript of a 15 minute video in less than a minute.

Here's a quick breakdown how the Whisper model works:

1. Preprocessing: The input audio is first converted to a spectrogram, which represents the frequencies present in the audio over time.
2. Encoding: The spectrogram is then passed through a convolutional neural network (CNN) that processes the audio and generates a set of encodings that help with future steps.
3. Generation: The encoding is used as input to a decoder, which generates the output text word by word, conditioned on the previously generated words. The decoder uses a technique called beam search to generate multiple candidate outputs and selects the most likely output based on a scoring function that takes into account both the model's confidence in the generated text and the coherence and relevance of the generated text.

## Text Segmentation

To be effective in generating chapters from video, we need some method of segmenting different topics. In our workflow, we use the [Natural Language Python Toolkit (nltk)](https://www.nltk.org/) and its [implementation of Text Tiling](https://www.nltk.org/_modules/nltk/tokenize/texttiling.html).

Text Tiling is a technique for text segmentation that identifies major topics or themes in a document and divides the text into coherent segments. Here's a brief overview of how Text Tiling works:

1. Preprocessing: The input text is first tokenized into individual words and stop words (common words such as "the," "a," and "and") are removed. Words are also reduced to their base form in a process called lemmatization to reduce the number of unique words.
2. Similarity calculation: Next, adjacent blocks of text are compared using a sliding window approach. A sliding window of fixed size is moved across the text, and the cosine similarity between the term frequency vectors of adjacent windows is calculated. The term frequency vector represents the frequency of each word in the text segment.
3. Segmentation: The text is then segmented into blocks using a threshold score. Blocks of text with similarity scores above the threshold are grouped together, while blocks with scores below the threshold are considered to represent a different topic or theme and are assigned to a new segment.

## GPT-3 Prompting

After we have our transcript segmented into topics, we can prompt GPT-3 to generate chapter headings.

For those unfamiliar, GPT-3 is a language model that uses a deep neural network with a transformer architecture to generate natural language text. It is trained on a massive corpus of text using unsupervised learning techniques, which allows it to generate coherent and contextually relevant text in response to a given prompt or task.

However, GPT-3 does have a limitation in the number of tokens it can take in as its prompt, which means we can't simply pass in a long transcript and generate chapter headings.

To work around this, we prompt GPT-3 twice:

1. First, we generate chapter headings for each section by itself
2. Then, we prompt GPT-3 to consolidate all chapter headings and remove any unnecessary ones

## Our final workflow

<img src="/blog-assets/autochapter/workflow.png">

We ran our workflow on this sample video of popular YouTuber, MKBHD, talking about AI and in less than a minute, we have chapter headings!

<iframe class="w-full" style="aspect-ratio:16/9;" src="https://www.youtube.com/embed/0gNauGdOkro"></iframe>

```json
[
  {
    "name": "Introduction",
    "start": 0,
    "end": 102.32
  },
  {
    "name": "OpenAI's ChatGPT AI Chatbot",
    "start": 102.32,
    "end": 193.72
  },
  {
    "name": "The Use of AI Tools as a Creative Tool",
    "start": 193.72,
    "end": 314.59999999999997
  },
  {
    "name": "AI Art and Impacting the Art World",
    "start": 314.59999999999997,
    "end": 631.48
  },
  {
    "name": "AI Art and Copyright Infringement",
    "start": 631.48,
    "end": 932.4000000000001
  }
]
```

## What next?

To start generating your own chapter headings for custom videos or integrate this into your app, you can run our template workflow from our [dashboard](https://www.sievedata.com/dashboard). We offer \$20 of credit, so you can get started right away.

In addition to pre-built building blocks and models, Sieve deploys your custom models at scale, so you don't have to worry about infra and can focus on building your video AI features fast. If you have any models you would love to see on Sieve's platform or have any questions about custom use cases, please don't hesitate to [reach out](https://calendly.com/sievedata/30min?month=2023-02).

<div class='w-full text-center'>
  <a href='https://sievedata.com/dashboard'
    className='group inline-flex items-center h-9 rounded-md text-sm font-medium whitespace-nowrap px-3 focus:outline-none focus:ring-none transition-colors duration-200 ease-in-out reverse flex-row-reverse text-gray-100 hover:text-sieve-purple-700 bg-sieve-purple-500 hover:bg-transparent border border-sieve-purple-500 mr-2'
    style="text-decoration: none"
  >
    Try it on Sieve
  </a>
  <a href='https://github.com/sieve-community/examples/tree/main/auto_chapter_title'
    className='group inline-flex items-center h-9 rounded-md text-sm font-medium whitespace-nowrap px-3 focus:outline-none focus:ring-none transition-colors duration-200 ease-in-out reverse flex-row-reverse text-gray-100 hover:text-sieve-purple-700 bg-sieve-purple-500 hover:bg-transparent border border-sieve-purple-500 ml-2'
    style="text-decoration: none"
  >
    See implementation
  </a>
</div>


Automatically generating video chapter titles with AI


<div class='w-full border border-gray-200 rounded-md p-4 mb-4'>
    <span class='text-md text-gray-600 text-semibold'>
        <b>Note:</b>
        This blog is outdated. If you're interested in a production-ready app for this capability, please check out the
        <a href="https://www.sievedata.com/functions/sieve/video_retalking" class="text-sieve-purple-500 hover:text-sieve-purple-700">
            video retalking app
        </a>
        on Sieve.
    </span>
</div>

In the past few months, we’ve seen the rise of all sorts of generative AI tools, including GPT-3, Stable Diffusion, DALL-E, Meta AI's [Make-A-Video](https://makeavideo.studio/) and much more. Specifically, AI video generation companies like [Synthesia](synthesia.io) and [Elai.io](http://Elai.io) have shown us the power in generating AI avatars, for use cases ranging from customer support tools to innocent Twitter trolling.

This past weekend, we set out to build a workflow that could generate avatar AI videos using as many **open-source models** as possible, given an avatar image and a few voice samples. We launched our workflow as a Twitter AI bot that can mimic others ([Stephen A. Smith](https://twitter.com/StephenAAvatar), [Mark Zuckerberg](https://twitter.com/ZuckAvatar), and [Sam Altman](https://twitter.com/SamaAvatar)) and answer all of Twitter’s burning questions in video form. You can check out the latest responses [here](https://projects.sievedata.com/avatars), or go ahead and @ them on Twitter yourself!

<video src="/blog-assets/avatar_bot/side2side.mp4" controls></video>

In this blog post, we’ll be going through how to make an AI generated avatar video using a variety of open-source models. If you’d like to just try out our template workflow and start generating videos in minutes, skip down below.

## Lip-syncing audio to video

The first critical step in generating AI avatar videos is making the avatar’s mouth sync up to the provided audio. Luckily, there’s a popular open source model, [Wav2Lip](https://github.com/Rudrabha/Wav2Lip), that helps with this.

Wav2Lip is a state-of-the-art lip-syncing technology that uses deep neural networks to generate realistic lip movements from an input audio signal. It consists of two main components: an audio encoder and a lip-sync decoder. The audio encoder extracts features from the provided audio signal that help the lip-sync decoder, a convolutional neural net, predict lip movements.

We also found that Wav2Lip gives us much better results when we input a video of just the cropped face, so we preprocess our videos by running an [open-source face detector (MediaPipe)](https://google.github.io/mediapipe/solutions/face_detection.html) and tracking those faces with a [SORT implementation](https://github.com/sieve-community/examples/blob/main/yolo_object_tracking/tracker.py).

You can find our [full implementation of Wav2Lip here](https://github.com/sieve-community/examples/tree/main/video_lipsyncing), with its associated Sieve workflow.

To generate a video of a person saying something else, you can run Wav2Lip on the reference video along with a generated audio track. However, as you’ll notice below — the results don’t look that amazing. The mouth is a bit blurry and feels a bit unnatural. We'd ideally want Wav2Lip to look a bit higher resolution.

<div class="grid grid-cols-2">
  <video class="col-span-1 pr-2" src="/blog-assets/avatar_bot/mok_driving.mp4" controls></video>
  <video class="col-span-1 pl-2" src="/blog-assets/avatar_bot/mok_wav2lip.mp4" controls></video>
</div>

There are a couple approaches we could take to make Wav2Lip generate higher resolution videos by retraining the base model but due to the lengthy process that is, we take a slightly different approach. We take our Wav2Lip output video and feed it to a separate model that was trained to “animate” a source image to look like the “driving video”. As we’ll see in the next section, this makes the lips look a lot clearer!

## Animating avatar image

To animate an image and mimic our reference video (generated by Wav2Lip), we can use the recently released open-source model, the [Thin-Plate Spline Motion model](https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model).

<div class="w-full text-center text-sm">
  <img src='/blog-assets/avatar_bot/vox.gif' alt='Thin-plate animation demo' style='margin-bottom: 8px'></img>
  <a href='https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model'>Source</a>
</div>

The Thin-Plate Spline motion model performs motion transfer from a “driving video” (in our case, the Wav2Lip video) and a static image. This model works by producing optical flow matrices, or the transformation from one frame to the next, and applying those transformations on the source image. Additionally, to smooth out the resulting images, the model uses multi-resolution occlusion masks to fill in missing parts of the avatar image animation.

Check out our full implementation of [the model here](https://github.com/sieve-community/examples/blob/main/talking_head_avatars/main.py). After just these 2 steps using open-source models, we’re able to generate avatar videos that look somewhat realistic.

<div class="grid grid-cols-2 content-evenly">
  <video class="col-span-1 pr-2" src="/blog-assets/avatar_bot/avatar_sample_1.mp4" controls></video>
  <video class="col-span-1" src="/blog-assets/avatar_bot/avatar_sample_2.mp4" controls></video>
</div>

## Aside: Twitter bot / voice cloning

To round our project out for our Twitter bot, we wanted to experiment with voice cloning with a few samples. The most promising open-source model was [Tortoise-TTS](https://github.com/neonbjb/tortoise-tts). However, the results still ended up being subpar and sounded more robotic than we had hoped.

Instead, we opted for a more complete and custom solution: [Eleven Labs](https://beta.elevenlabs.io/speech-synthesis). With just a few samples, we’re able to clone voices and generate scripts with an API call and the results were pretty incredible:

<audio class='w-full mb-2' src="/blog-assets/avatar_bot/stephen_cloned.mp3" controls></audio>
<audio class='w-full' src="/blog-assets/avatar_bot/sama_cloned.mp3" controls></audio>

## Generate Avatars!

Now that we have some background on the models we used, let’s generate some avatar videos!

1. To follow along, you’ll need to sign up for a [Sieve account](https://www.sievedata.com/) and get your [Sieve API key](https://sievedata.com/dashboard/settings). You'll also need to install the Sieve CLI:

```bash
export SIEVE_API_KEY={YOUR_SIEVE_KEY}
pip install https://mango.sievedata.com/v1/client_package/sievedata-0.0.1.1.2-py3-none-any.whl
```

2. Write your workflow! Open up a new file in a new folder: `avatar_gen/wf.py` and copy the following code:

```python
import sieve
@sieve.workflow(name="my_avatar_gen")
def avatar_generation(driving_video: sieve.Video, driving_audio: sieve.Audio, avatar: sieve.Image) -> sieve.Video:
    images = sieve.reference("sieve-developer/video-splitter")(driving_video)
    faces = sieve.reference("sieve-developer/mediapipe-face-detector")(images)
    tracked_faces = sieve.reference("sieve-developer/sort")(faces)
    synced = sieve.reference("sieve-developer/wav2lip")(driving_video, driving_audio, tracked_faces)
    return sieve.reference("sieve-developer/thin_plate_spline_motion_model")(avatar, synced)
```

This workflow uses Sieve building blocks (`sieve.reference`), so we can skip all the boilerplate code and the nuances with each model, empowering us to build features like avatar generation fast. If you want to dive deeper into the implementation of each of these models, check out our [examples repo](https://github.com/sieve-community/examples).

3. Deploy to Sieve. From the `avatar_gen` directory, run `sieve deploy`.
   ![deploy output](/blog-assets/avatar_bot/deploy-output.png)
4. Generate your first video!

- From the UI, navigate to your [workflow](https://www.sievedata.com/dashboard), input a driving video, driving audio and avatar image and push your first job! Here’s a few sample inputs to get you started:

  - Driving video: [https://storage.googleapis.com/sieve-public-videos/mok-best-driving.mp4](https://storage.googleapis.com/sieve-public-videos/mok-best-driving.mp4)
  - Driving audio: [https://storage.googleapis.com/sieve-public-videos-grapefruit/mark_future.wav](https://storage.googleapis.com/sieve-public-videos-grapefruit/mark_future.wav)
  - Avatar image: [https://storage.googleapis.com/sieve-public-videos-grapefruit/zuck_avatar2.jpeg](https://storage.googleapis.com/sieve-public-videos-grapefruit/zuck_avatar2.jpeg)

  <img src='/blog-assets/avatar_bot/demo_dashboard.png' alt="Dashboard Inputs" class="border">

- Profit!
  <video src="/blog-assets/avatar_bot/mark_future.mp4" controls></video>

## What next?

To integrate your own AI avatar videos into your app in a few minutes, you can follow the guide above or clone our template from the UI and access any of your job outputs [via our API](https://docs.sievedata.com/reference/api/jobs/get). We offer \$20 of credit, so you can get started right away.

In addition to pre-built building blocks and models, Sieve deploys your custom models at scale, so you don’t have to worry about infra and can focus on building your video AI features fast. If you have any models you would love to see on Sieve’s platform or have any questions about custom use cases, please don’t hesitate to [reach out](https://calendly.com/sievedata/30min?month=2023-02).


Building realistic video AI avatars in an hour from scratch


When Google was founded, most of the internet was just text. Today, **~80% of the internets IP traffic is video**. TikTok is expected to reach **1.8B monthly active users** by the end of this year, everyone uses Zoom, smart cameras are being placed everywhere from our homes, to farms, factories, warehouses, and construction sites, and we’ve been ushered into a new age of creative tooling with products like Figma.

As video has become more integral to products, the ability of software developers to quickly deliver great AI experiences has lagged behind. Most companies we talk to today face these massive bottlenecks in running ML workflows at video-scale, improving ML models over time to fit their use case, and integrating those outputs into end-user applications.

That’s why we built [Sieve](https://sievedata.com/): a cloud native platform that makes it easy to process, understand, and search video.

Some people use Sieve as a structured database for video. Define a workflow, run a query, integrate into an app, and submit feedback to improve models over time.

![Sieve Use Cases](/blog-assets/graphics/launch/sieve-use-case.png)

Others take querying to a whole different level. Because Sieve can expose information as a video is being processed, users can poll Sieve’s query API to build highly interactive features without setting up any separate backend.

![Rotoscoping Demo](/blog-assets/graphics/launch/rotoscoping.gif)

Sieve was built while working closely with many software startups who themselves were trying to build smart features and experiences with video. Till now, the product has only been open to them to use but today we’re opening up our beta for anyone to try. Just sign up at the top right of this page.

We’re also annoucing that we've raised a ~\$4M seed round led by [Matrix Partners](https://www.matrixpartners.com/) with participation from [Y Combinator](https://www.ycombinator.com/), [Swift Ventures](https://www.swift.vc/), [AI Grant](https://aigrant.org/), and a wide host of angels (Lucy Guo, Eric Jang, Alfredo Andere, Melisa Tokmak, Ryan Chan, Everett Berry, Phillip Kuznetsov, JP Ren). This financing will allow us to stay laser focused on building the best products for everyone.

Please [join our community](https://join.slack.com/t/sievecommunity/shared_invite/zt-16yj3jgzb-MUuamHJU7WJPGyFqiOs0MA), [try the product](https://www.sievedata.com/dashboard), and [read the docs](https://docs.sievedata.com/guide/introduction). We know it's not perfect yet, but we're excited to learn from your feedback and build a better product together.

Also if you’re interested in the work we do, check out our [jobs page](https://www.ycombinator.com/companies/sieve/jobs) or [reach out directly](mailto:jobs@sievedata.com).
