Activities, Relations and Attributes: Adding Context and Interactions

While masks and categories define what is in the video, additional annotation types allow you to describe what is happening and how objects relate.

Depending on your specific use case, you can mix and match these types to create the depth of data you need. There is no requirement to use every type for every project – you can choose the combination that best fits your goals.

Activities

Activities describe higher-level work steps or global actions in a scene (e.g., pouring water, assembling part, safety check).

  • Automation: Activities can be created fully automatically by our pipelines. A generated activity consists of a Title, a Description, and references to the specific objects involved in the action.
  • With Vocabulary: For the best results, we recommend defining a Vocabulary (or Standard Operating Procedures). We have seen a significant jump in output quality when the model is provided with even a small amount of domain context, such as a list of expected steps.
  • Manual Creation: You can also create activities manually or edit automated ones.

Note: To access these tools, ensure you are in Edit Mode. See Edit Mode Basics for instructions.

Add an Activity Manually:

  1. Select the Frame Range: Click and drag on the Timeline to mark the duration of the activity.
  2. Click Add: Click the + icon on the Activity portion of the Timeline.
  3. Define Details: In the Inspector Panel, choose a predefined activity from your vocabulary or enter a custom Title and Description.
  4. Save: Click Save.

Modify an Activity:

  1. Select the Activity: Click the Activity on the Timeline.
  2. Enter Edit Mode: In the Inspector Panel, click the Edit icon.
  3. Update Details: Modify the title, description, or frame range.
  4. Save: Click Save.

Relations and Attributes

Relations and attributes add granular detail to your objects. Because these are attached to specific instances, masks must be created first before you can define attributes or relations.

  • Relations: Describe how objects interact with one another (e.g., hand → holds → tool).
  • Attributes: Describe properties or states of an object (e.g., open, filled, damaged).
  • Automation: By default, nearly every pipeline automatically generates attributes and relations on keyframes.

Note: To access these tools, ensure you are in Edit Mode. See Edit Mode Basics for instructions.

Add Relations Manually:

  1. Select the Primary Object: Click the mask (e.g., hand).
  2. Define Frame Range: Select the frames where the interaction occurs.
  3. Add: In the Inspector Panel → Relations section, click +.
  4. Define Details: Choose the relation type (e.g., holds) and select the target object (e.g., tool).
  5. Save: Click the Checkmark icon to save.

Modify a Relation:

  1. Select the Object: Click the mask or the timeline object.
  2. Edit: Hover over the existing relation in the Inspector Panel and click the Edit icon.
  3. Update: Edit the value and click the Checkmark icon to save.

Add Attributes Manually:

  1. Select the Object: Click the mask or the timeline object.
  2. Define Frame Range: (Optional) Select specific frames if the attribute only applies temporarily.
  3. Add: In the Inspector Panel → Attributes section, click +.
  4. Define Value: Choose from your vocabulary or type a custom value, then click the Checkmark icon to save.

Modify an Attribute:

  1. Select the Object: Click the mask or the timeline object.
  2. Edit: Hover over the existing attribute in the Inspector Panel and click the Edit icon.
  3. Update: Edit the value and click the Checkmark icon to save.