Improving on open-source for fast, high-quality AI lipsyncing


In the world of multimedia and podcasting, user experience often hinges on audio quality. Superior quality has the capacity to captivate audiences, encouraging prolonged engagement with your content, whether they be podcasts, tutorials, or promos. More specifically, audio quality is not just about enhancing user experience but also plays a crucial role in ensuring the accuracy of downstream AI tasks such as automated transcription services, like transcription with [OpenAI's Whisper](https://github.com/openai/whisper). In fact, an entire industry of startups like [Descript](https://www.descript.com), [Krisp AI](https://krisp.ai), and more are working in the space.

With the help of a few open source models, namely [AudioSR](https://github.com/haoheliu/versatile_audio_super_resolution) and [DeepFilterNet](https://github.com/rikorose/deepfilternet), we’ve launched an audio enhancement app that helps remove background noise and enhance speech, making sure your audio quality is always up to par.

Here’s a quick example of the sound quality difference using open-source models:

<script src="https://cdn.jsdelivr.net/npm/@mux/mux-player"></script>

<mux-player
  stream-type="on-demand"
  playback-id="mu3U00wldTaxIg7aCcdo00fS2ObM01oZqtUpUeQkO6lVW4"
  metadata-video-title="Placeholder (optional)"
  metadata-viewer-user-id="Placeholder (optional)"
  primary-color="#FFFFFF"
  secondary-color="#000000"
/>

In this post, we’ll go through a quick demonstration of how you might integrate this solution into your AI project and some background on the models we used.

## Trying it out

You can try out the pre-built audio enhancement app in a few clicks here with your own audio samples [here](https://www.sievedata.com/functions/sieve/audio_enhancement). Here’s a few more samples from podcasts and YouTube videos:

**Sample 1**

<div class="sm:flex sm:space-x-6">
  <div class="sm:w-1/2">
    <span class="block mt-2 mb-2 sm:mt-0">Original</span>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample1/original.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
  <div class="sm:w-1/2">
    <span class="block mt-2 mb-2 sm:mt-0">Enhanced</span>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample1/sieve.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
</div>

**Sample 2**

<div class="sm:flex sm:space-x-6">
  <div class="sm:w-1/2">
    <span class="block mt-2 mb-2 sm:mt-0">Original</span>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample2/original.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
  <div class="sm:w-1/2">
    <span class="block mt-2 mb-2 sm:mt-0">Enhanced</span>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample2/sieve.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
</div>

### Run via API or Python

You can also integrate the app into your current workflow through an API call or Python call with the following steps:

1. [Sign up](https://www.sievedata.com/dashboard) for a Sieve account and [find your API key here](https://www.sievedata.com/dashboard/settings).
2. Run audio enhancement via API (or see below for Python)
   ```bash
   curl -X POST https://mango.sievedata.com/v2/push \
   -H "X-API-Key: <your-api-key>" \
   -d '{
     "function": "sieve/audio_enhancement",
     "inputs": {
       "audio": {
         "url": "<your-audio-url>"
       }
     }
   }'
   ```
3. Run via Python client

   - Install the Python client

   ```
   pip install sievedata
   ```

   - Login with your [API key](https://www.sievedata.com/dashboard/settings)

   ```bash
   sieve login
   ```

   - Run this Python script with your own audio!

   ```python
   import sieve

   audio_enhancer = sieve.function.get('sieve/audio_enhancement')

   # Specify "upsample", "noise" or "all" for the filtering type
   enhanced_audio = audio_enhancer.run(sieve.Audio(path="./speech.wav"), "upsample")

   # View results on Sieve dashboard or locally from this path
   print(enhanced_audio.path)
   ```

It’s as simple as that! Results will now be viewable on your Sieve dashboard or directly from your Python code.

## **How it works**

The magic lies in the tandem of two open-source AI models - AudioSR and DeepFilterNet.

### AudioSR

AudioSR is a generative model that uses a diffusion-based approach to estimate the high-frequency components of a low-resolution audio signal. It does this by training a latent diffusion model to learn the conditional generation of high-resolution spectrograms from low-resolution spectrograms. The model can handle a flexible input sampling rate between 4kHz and 32kHz, covering most real-world scenarios. AudioSR has achieved promising results on speech, music, and sound effects with different input sampling rate settings and has been verified to be a plug-and-play module for enhancing the audio quality of various audio generation models.

### DeepFilterNet

DeepFilterNet is a deep learning-based speech enhancement framework that utilizes harmonic structure of speech to efficiently enhance speech quality by removing unwanted noise from audio files. It operates in two stages, with the first stage enhancing the speech envelope in the ERB domain, and the second stage using deep filtering to enhance the periodic component. Several optimizations have been made to the training procedure, data augmentation, and network structure resulting in state-of-the-art speech enhancement performance while reducing processing, making it applicable to run on embedded devices in real-time.

## **Other Solutions**

Let's take a look at how open source stands against the other prominent ones in the market, namely the [Dolby Enhance API](https://dolby.io/products/enhance/):

<div class="grid grid-cols-1 md:grid-cols-3 gap-4 w-max">
  <div class="mx-3 flex flex-col gap-3">
    <p class="text-center">Original</p>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample1/original.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample2/original.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
  <div class="mx-3 flex flex-col gap-3">
    <p class="text-center">Dolby</p>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample1/dolby.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample2/dolby.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
  <div class="mx-3 flex flex-col gap-3">
    <p class="text-center">AudioSR + DeepFilterNet</p>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample1/sieve.mp3" type="audio/wav">
      Your browser does not support the audio element.
    </audio>
    <audio controls>
      <source src="https://storage.googleapis.com/sieve-public-data/enhancement-blog/sample2/sieve.mp3" type="audio/mpeg">
      Your browser does not support the audio element.
    </audio>
  </div>
</div>

Very promising results from a purely open-source solution!

## What next?

Audio enhancement is a feature that can be used upstream of most other AI audio functionality. For example, a feature like speech editing ([also featured on Sieve](https://www.sievedata.com/functions/sieve/speech_editor)) can use audio enhancement capabilities to enhance the quality of the generated voices.

Sieve's cloud platform makes combining functionality in this way easy. To try it, [create an account](https://www.sievedata.com/dashboard)!


State of the art audio enhancement in 5 minutes


<div class='w-full border border-gray-200 rounded-md p-4 mb-4'>
    <span class='text-md text-gray-600 text-semibold'>
        <b>Note:</b>
        This blog is outdated. If you're interested in a production-ready app for this capability, please check out the
        <a href="https://www.sievedata.com/functions/sieve/video_transcript_analyzer" class="text-sieve-purple-500 hover:text-sieve-purple-700">
            video transcript analyzer
        </a>
        on Sieve.
    </span>
</div>

Recently, we've seen content platforms like YouTube implement [video chapters](https://support.google.com/youtube/answer/9884579?hl=en) to make their content more digestable for viewers.

With the rise of generative AI, we thought it'd be useful to build a workflow to auto-generate these video chapters using public models including OpenAI's [Whisper](https://openai.com/research/whisper), [GPT-3](https://openai.com/blog/gpt-3-apps), and [nltk's Text Tiling](https://www.nltk.org/_modules/nltk/tokenize/texttiling.html) implementation. In this blog post, we'll dive into the specifics of our workflow and how it works.

If you'd like to try it for yourself or integrate into your own app, try it on Sieve by clicking the button below. Keep reading if you'd like to learn about how it works.

<div class='w-full text-center'>
  <a href='https://sievedata.com/dashboard'
    className='group inline-flex items-center h-9 rounded-md text-sm font-medium whitespace-nowrap px-3 focus:outline-none focus:ring-none transition-colors duration-200 ease-in-out reverse flex-row-reverse text-gray-100 hover:text-sieve-purple-700 bg-sieve-purple-500 hover:bg-transparent border border-sieve-purple-500 mr-2'
    style="text-decoration: none"
  >
    Try it on Sieve
  </a>
  <a href='https://github.com/sieve-community/examples/tree/main/auto_chapter_title'
    className='group inline-flex items-center h-9 rounded-md text-sm font-medium whitespace-nowrap px-3 focus:outline-none focus:ring-none transition-colors duration-200 ease-in-out reverse flex-row-reverse text-gray-100 hover:text-sieve-purple-700 bg-sieve-purple-500 hover:bg-transparent border border-sieve-purple-500 ml-2'
    style="text-decoration: none"
  >
    See implementation
  </a>
</div>

## Audio Transcription

Speech-to-text models have been around for more than a few years, but none rival OpenAI's Whisper model in accuracy & speed. With OpenAI's Whisper model running on Sieve, we're able to generate an accurate transcript of a 15 minute video in less than a minute.

Here's a quick breakdown how the Whisper model works:

1. Preprocessing: The input audio is first converted to a spectrogram, which represents the frequencies present in the audio over time.
2. Encoding: The spectrogram is then passed through a convolutional neural network (CNN) that processes the audio and generates a set of encodings that help with future steps.
3. Generation: The encoding is used as input to a decoder, which generates the output text word by word, conditioned on the previously generated words. The decoder uses a technique called beam search to generate multiple candidate outputs and selects the most likely output based on a scoring function that takes into account both the model's confidence in the generated text and the coherence and relevance of the generated text.

## Text Segmentation

To be effective in generating chapters from video, we need some method of segmenting different topics. In our workflow, we use the [Natural Language Python Toolkit (nltk)](https://www.nltk.org/) and its [implementation of Text Tiling](https://www.nltk.org/_modules/nltk/tokenize/texttiling.html).

Text Tiling is a technique for text segmentation that identifies major topics or themes in a document and divides the text into coherent segments. Here's a brief overview of how Text Tiling works:

1. Preprocessing: The input text is first tokenized into individual words and stop words (common words such as "the," "a," and "and") are removed. Words are also reduced to their base form in a process called lemmatization to reduce the number of unique words.
2. Similarity calculation: Next, adjacent blocks of text are compared using a sliding window approach. A sliding window of fixed size is moved across the text, and the cosine similarity between the term frequency vectors of adjacent windows is calculated. The term frequency vector represents the frequency of each word in the text segment.
3. Segmentation: The text is then segmented into blocks using a threshold score. Blocks of text with similarity scores above the threshold are grouped together, while blocks with scores below the threshold are considered to represent a different topic or theme and are assigned to a new segment.

## GPT-3 Prompting

After we have our transcript segmented into topics, we can prompt GPT-3 to generate chapter headings.

For those unfamiliar, GPT-3 is a language model that uses a deep neural network with a transformer architecture to generate natural language text. It is trained on a massive corpus of text using unsupervised learning techniques, which allows it to generate coherent and contextually relevant text in response to a given prompt or task.

However, GPT-3 does have a limitation in the number of tokens it can take in as its prompt, which means we can't simply pass in a long transcript and generate chapter headings.

To work around this, we prompt GPT-3 twice:

1. First, we generate chapter headings for each section by itself
2. Then, we prompt GPT-3 to consolidate all chapter headings and remove any unnecessary ones

## Our final workflow

<img src="/blog-assets/autochapter/workflow.png">

We ran our workflow on this sample video of popular YouTuber, MKBHD, talking about AI and in less than a minute, we have chapter headings!

<iframe class="w-full" style="aspect-ratio:16/9;" src="https://www.youtube.com/embed/0gNauGdOkro"></iframe>

```json
[
  {
    "name": "Introduction",
    "start": 0,
    "end": 102.32
  },
  {
    "name": "OpenAI's ChatGPT AI Chatbot",
    "start": 102.32,
    "end": 193.72
  },
  {
    "name": "The Use of AI Tools as a Creative Tool",
    "start": 193.72,
    "end": 314.59999999999997
  },
  {
    "name": "AI Art and Impacting the Art World",
    "start": 314.59999999999997,
    "end": 631.48
  },
  {
    "name": "AI Art and Copyright Infringement",
    "start": 631.48,
    "end": 932.4000000000001
  }
]
```

## What next?

To start generating your own chapter headings for custom videos or integrate this into your app, you can run our template workflow from our [dashboard](https://www.sievedata.com/dashboard). We offer $20 of credit, so you can get started right away.

In addition to pre-built building blocks and models, Sieve deploys your custom models at scale, so you don't have to worry about infra and can focus on building your video AI features fast. If you have any models you would love to see on Sieve's platform or have any questions about custom use cases, please don't hesitate to [reach out](https://calendly.com/sievedata/30min?month=2023-02).

<div class='w-full text-center'>
  <a href='https://sievedata.com/dashboard'
    className='group inline-flex items-center h-9 rounded-md text-sm font-medium whitespace-nowrap px-3 focus:outline-none focus:ring-none transition-colors duration-200 ease-in-out reverse flex-row-reverse text-gray-100 hover:text-sieve-purple-700 bg-sieve-purple-500 hover:bg-transparent border border-sieve-purple-500 mr-2'
    style="text-decoration: none"
  >
    Try it on Sieve
  </a>
  <a href='https://github.com/sieve-community/examples/tree/main/auto_chapter_title'
    className='group inline-flex items-center h-9 rounded-md text-sm font-medium whitespace-nowrap px-3 focus:outline-none focus:ring-none transition-colors duration-200 ease-in-out reverse flex-row-reverse text-gray-100 hover:text-sieve-purple-700 bg-sieve-purple-500 hover:bg-transparent border border-sieve-purple-500 ml-2'
    style="text-decoration: none"
  >
    See implementation
  </a>
</div>


Automatically generating video chapter titles with AI


<div class='w-full border border-gray-200 rounded-md p-4 mb-4'>
    <span class='text-md text-gray-600 text-semibold'>
        <b>Note:</b>
        This blog is outdated. If you're interested in a production-ready app for this capability, please check out the
        <a href="https://www.sievedata.com/functions/sieve/video_retalking" class="text-sieve-purple-500 hover:text-sieve-purple-700">
            video retalking app
        </a>
        on Sieve.
    </span>
</div>

In the past few months, we’ve seen the rise of all sorts of generative AI tools, including GPT-3, Stable Diffusion, DALL-E, Meta AI's [Make-A-Video](https://makeavideo.studio/) and much more. Specifically, AI video generation companies like [Synthesia](synthesia.io) and [Elai.io](http://Elai.io) have shown us the power in generating AI avatars, for use cases ranging from customer support tools to innocent Twitter trolling.

This past weekend, we set out to build a workflow that could generate avatar AI videos using as many **open-source models** as possible, given an avatar image and a few voice samples. We launched our workflow as a Twitter AI bot that can mimic others ([Stephen A. Smith](https://twitter.com/StephenAAvatar), [Mark Zuckerberg](https://twitter.com/ZuckAvatar), and [Sam Altman](https://twitter.com/SamaAvatar)) and answer all of Twitter’s burning questions in video form. You can check out the latest responses [here](https://projects.sievedata.com/avatars), or go ahead and @ them on Twitter yourself!

<video src="/blog-assets/avatar_bot/side2side.mp4" controls></video>

In this blog post, we’ll be going through how to make an AI generated avatar video using a variety of open-source models. If you’d like to just try out our template workflow and start generating videos in minutes, skip down below.

## Lip-syncing audio to video

The first critical step in generating AI avatar videos is making the avatar’s mouth sync up to the provided audio. Luckily, there’s a popular open source model, [Wav2Lip](https://github.com/Rudrabha/Wav2Lip), that helps with this.

Wav2Lip is a state-of-the-art lip-syncing technology that uses deep neural networks to generate realistic lip movements from an input audio signal. It consists of two main components: an audio encoder and a lip-sync decoder. The audio encoder extracts features from the provided audio signal that help the lip-sync decoder, a convolutional neural net, predict lip movements.

We also found that Wav2Lip gives us much better results when we input a video of just the cropped face, so we preprocess our videos by running an [open-source face detector (MediaPipe)](https://google.github.io/mediapipe/solutions/face_detection.html) and tracking those faces with a [SORT implementation](https://github.com/sieve-community/examples/blob/main/yolo_object_tracking/tracker.py).

You can find our [full implementation of Wav2Lip here](https://github.com/sieve-community/examples/tree/main/video_lipsyncing), with its associated Sieve workflow.

To generate a video of a person saying something else, you can run Wav2Lip on the reference video along with a generated audio track. However, as you’ll notice below — the results don’t look that amazing. The mouth is a bit blurry and feels a bit unnatural. We'd ideally want Wav2Lip to look a bit higher resolution.

<div class="grid grid-cols-2">
  <video class="col-span-1 pr-2" src="/blog-assets/avatar_bot/mok_driving.mp4" controls></video>
  <video class="col-span-1 pl-2" src="/blog-assets/avatar_bot/mok_wav2lip.mp4" controls></video>
</div>

There are a couple approaches we could take to make Wav2Lip generate higher resolution videos by retraining the base model but due to the lengthy process that is, we take a slightly different approach. We take our Wav2Lip output video and feed it to a separate model that was trained to “animate” a source image to look like the “driving video”. As we’ll see in the next section, this makes the lips look a lot clearer!

## Animating avatar image

To animate an image and mimic our reference video (generated by Wav2Lip), we can use the recently released open-source model, the [Thin-Plate Spline Motion model](https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model).

<div class="w-full text-center text-sm">
  <img src='/blog-assets/avatar_bot/vox.gif' alt='Thin-plate animation demo' style='margin-bottom: 8px'></img>
  <a href='https://github.com/yoyo-nb/Thin-Plate-Spline-Motion-Model'>Source</a>
</div>

The Thin-Plate Spline motion model performs motion transfer from a “driving video” (in our case, the Wav2Lip video) and a static image. This model works by producing optical flow matrices, or the transformation from one frame to the next, and applying those transformations on the source image. Additionally, to smooth out the resulting images, the model uses multi-resolution occlusion masks to fill in missing parts of the avatar image animation.

Check out our full implementation of [the model here](https://github.com/sieve-community/examples/blob/main/talking_head_avatars/main.py). After just these 2 steps using open-source models, we’re able to generate avatar videos that look somewhat realistic.

<div class="grid grid-cols-2 content-evenly">
  <video class="col-span-1 pr-2" src="/blog-assets/avatar_bot/avatar_sample_1.mp4" controls></video>
  <video class="col-span-1" src="/blog-assets/avatar_bot/avatar_sample_2.mp4" controls></video>
</div>

## Aside: Twitter bot / voice cloning

To round our project out for our Twitter bot, we wanted to experiment with voice cloning with a few samples. The most promising open-source model was [Tortoise-TTS](https://github.com/neonbjb/tortoise-tts). However, the results still ended up being subpar and sounded more robotic than we had hoped.

Instead, we opted for a more complete and custom solution: [Eleven Labs](https://beta.elevenlabs.io/speech-synthesis). With just a few samples, we’re able to clone voices and generate scripts with an API call and the results were pretty incredible:

<audio class='w-full mb-2' src="/blog-assets/avatar_bot/stephen_cloned.mp3" controls></audio>
<audio class='w-full' src="/blog-assets/avatar_bot/sama_cloned.mp3" controls></audio>

## Generate Avatars!

Now that we have some background on the models we used, let’s generate some avatar videos!

1. To follow along, you’ll need to sign up for a [Sieve account](https://www.sievedata.com/) and get your [Sieve API key](https://sievedata.com/dashboard/settings). You'll also need to install the Sieve CLI:

```bash
export SIEVE_API_KEY={YOUR_SIEVE_KEY}
pip install https://mango.sievedata.com/v1/client_package/sievedata-0.0.1.1.2-py3-none-any.whl
```

2. Write your workflow! Open up a new file in a new folder: `avatar_gen/wf.py` and copy the following code:

```python
import sieve
@sieve.workflow(name="my_avatar_gen")
def avatar_generation(driving_video: sieve.Video, driving_audio: sieve.Audio, avatar: sieve.Image) -> sieve.Video:
    images = sieve.reference("sieve-developer/video-splitter")(driving_video)
    faces = sieve.reference("sieve-developer/mediapipe-face-detector")(images)
    tracked_faces = sieve.reference("sieve-developer/sort")(faces)
    synced = sieve.reference("sieve-developer/wav2lip")(driving_video, driving_audio, tracked_faces)
    return sieve.reference("sieve-developer/thin_plate_spline_motion_model")(avatar, synced)
```

This workflow uses Sieve building blocks (`sieve.reference`), so we can skip all the boilerplate code and the nuances with each model, empowering us to build features like avatar generation fast. If you want to dive deeper into the implementation of each of these models, check out our [examples repo](https://github.com/sieve-community/examples).

3. Deploy to Sieve. From the `avatar_gen` directory, run `sieve deploy`.
   ![deploy output](/blog-assets/avatar_bot/deploy-output.png)
4. Generate your first video!

- From the UI, navigate to your [workflow](https://www.sievedata.com/dashboard), input a driving video, driving audio and avatar image and push your first job! Here’s a few sample inputs to get you started:

  - Driving video: [https://storage.googleapis.com/sieve-public-videos/mok-best-driving.mp4](https://storage.googleapis.com/sieve-public-videos/mok-best-driving.mp4)
  - Driving audio: [https://storage.googleapis.com/sieve-public-videos-grapefruit/mark_future.wav](https://storage.googleapis.com/sieve-public-videos-grapefruit/mark_future.wav)
  - Avatar image: [https://storage.googleapis.com/sieve-public-videos-grapefruit/zuck_avatar2.jpeg](https://storage.googleapis.com/sieve-public-videos-grapefruit/zuck_avatar2.jpeg)

  <img src='/blog-assets/avatar_bot/demo_dashboard.png' alt="Dashboard Inputs" class="border">

- Profit!
  <video src="/blog-assets/avatar_bot/mark_future.mp4" controls></video>

## What next?

To integrate your own AI avatar videos into your app in a few minutes, you can follow the guide above or clone our template from the UI and access any of your job outputs [via our API](https://docs.sievedata.com/reference/api/jobs/get). We offer \$20 of credit, so you can get started right away.

In addition to pre-built building blocks and models, Sieve deploys your custom models at scale, so you don’t have to worry about infra and can focus on building your video AI features fast. If you have any models you would love to see on Sieve’s platform or have any questions about custom use cases, please don’t hesitate to [reach out](https://calendly.com/sievedata/30min?month=2023-02).


Building realistic video AI avatars in an hour from scratch


When Google was founded, most of the internet was just text. Today, **~80% of the internets IP traffic is video**. TikTok is expected to reach **1.8B monthly active users** by the end of this year, everyone uses Zoom, smart cameras are being placed everywhere from our homes, to farms, factories, warehouses, and construction sites, and we’ve been ushered into a new age of creative tooling with products like Figma.

As video has become more integral to products, the ability of software developers to quickly deliver great AI experiences has lagged behind. Most companies we talk to today face these massive bottlenecks in running ML workflows at video-scale, improving ML models over time to fit their use case, and integrating those outputs into end-user applications.

That’s why we built [Sieve](https://sievedata.com/): a cloud native platform that makes it easy to process, understand, and search video.

Some people use Sieve as a structured database for video. Define a workflow, run a query, integrate into an app, and submit feedback to improve models over time.

![Sieve Use Cases](/blog-assets/graphics/launch/sieve-use-case.png)

Others take querying to a whole different level. Because Sieve can expose information as a video is being processed, users can poll Sieve’s query API to build highly interactive features without setting up any separate backend.

![Rotoscoping Demo](/blog-assets/graphics/launch/rotoscoping.gif)

Sieve was built while working closely with many software startups who themselves were trying to build smart features and experiences with video. Till now, the product has only been open to them to use but today we’re opening up our beta for anyone to try. Just sign up at the top right of this page.

We’re also annoucing that we've raised a ~\$4M seed round led by [Matrix Partners](https://www.matrixpartners.com/) with participation from [Y Combinator](https://www.ycombinator.com/), [Swift Ventures](https://www.swift.vc/), [AI Grant](https://aigrant.org/), and a wide host of angels (Lucy Guo, Eric Jang, Alfredo Andere, Melisa Tokmak, Ryan Chan, Everett Berry, Phillip Kuznetsov, JP Ren). This financing will allow us to stay laser focused on building the best products for everyone.

Please [join our community](https://join.slack.com/t/sievecommunity/shared_invite/zt-16yj3jgzb-MUuamHJU7WJPGyFqiOs0MA), [try the product](https://www.sievedata.com/dashboard), and [read the docs](https://docs.sievedata.com/guide/introduction). We know it's not perfect yet, but we're excited to learn from your feedback and build a better product together.

Also if you’re interested in the work we do, check out our [jobs page](https://www.ycombinator.com/companies/sieve/jobs) or [reach out directly](mailto:jobs@sievedata.com).
