Video to JSON (Early Preview)

This function can automatically return structured data about a video. It can be customized for a variety of tasks like sports analysis, video summarization, screen recordings, etc.

Input

• video: The video file to be analyzed.
• task: The preset task to be performed.
• custom_prompt: A custom prompt or task you want to specify over the preset task.
• custom_json_schema: A custom JSON schema to validate the output of the task. See README for more information.
• custom_granularity: You may specify "full video", "by scene", "every minute", or "every 5 seconds". If you want to manually specify the scene boundaries, provide a string of comma-separated numbers on newlines. Each line is a scene. The first number is the start time of the scene, the second is the end time, and the third is an optional description. See README for more information.

Example Custom JSON Schema

Below is an example of a custom JSON schema. If specified, the app will return a JSON object that matches the schema. This allows you to customize the output of the function to your needs, for your specific use case. For example, you might use the following schema to extract information about sports games.

{
  "is_a_play": {
    "type": "boolean",
    "description": "Indicates if the scene is part of a play."
  },
  "description": {
    "type": "string",
    "description": "A textual description of the scene."
  },
  "teams_in_scene": {
    "type": "array",
    "items": {
      "type": "string"
    },
    "description": "List of teams present in the scene."
  },
  "players_in_scene": {
    "type": "array",
    "items": {
      "type": "object",
      "properties": {
        "name": {
          "type": "string",
          "description": "Name of the player. Null if you don't know the name."
        },
        "number": {
          "type": ["string", "null"],
          "description": "Number of the player. Null if the number is not visible."
        }
      }
    },
    "description": "List of players present in the scene, including their names and numbers. If the number is not visible, it will be null."
  },
  "score": {
    "type": ["object", "null"],
    "properties": {
      "teamname": {
        "type": "string",
        "description": "The full name of the team."
      }
    },
    "additionalProperties": {
      "type": "integer",
      "description": "The score of the team."
    },
    "description": "The score details of the teams in the scene."
  }
}

Example Custom Scenes

Below is an example of how to specify custom granularity. The first scene is from 0 to 5 seconds, and the second scene is from 5 to 50 seconds.

0, 5
5, 50

Alternatively, you can provide a description for each scene. The first scene is from 0 to 10 seconds and the second scene is from 15 to 40 seconds.

0, 10, the first scene
15, 40, the far away shot where you should focus on the people in the background