feat: OPTIC-2125: Add sync between TimeSeries, Video, Audio #7376

cloudmark · 2025-04-15T14:50:31Z

TimeSeries Synchronization

timeseries.mp4

Overview

TimeSeries can synchronize with Audio and Video components, allowing coordinated playback and seeking across different media types.

Time Units

All sync messages use relative seconds from the start of each component
TimeSeries converts these relative seconds based on its configuration:
- Time-based: Converts to absolute timestamps
- Index-based: Uses relative seconds as indices

Configuration

Time-based TimeSeries:

<TimeSeries name="ts" timeColumn="timestamp" timeFormat="%Y-%m-%d %H:%M:%S">

Index-based TimeSeries:
```
<TimeSeries name="ts">
```

Synchronization Behavior

Time-based TimeSeries:
- Converts relative seconds to absolute timestamps
- Maintains precise temporal alignment with media, any custom sampling rate
- Example: 30s in video = 30s from start in TimeSeries
- If the TimeSeries starts at the 5th second, it will serve as the starting point for other media. Therefore, the 0 second of the video will begin at the 5th second of the TimeSeries, and this offset will always be applied.
Index-based TimeSeries:
- Uses direct indices as relative seconds
- Each second in media = one index in TimeSeries, sampling rate is 1 Hz always
- Example: 30s in video = index 30 in TimeSeries
- Suitable mostly for debugging and tests

Length Mismatches

Sync works up to the length of the shorter component
Components stop at their respective ends
No data loss occurs, but sync stops at the shorter component's end

Best Practices

Use time-based TimeSeries when:
- Data has actual timestamps
- Precise temporal alignment is needed
- Working with multiple media types
Use index-based TimeSeries when:
- Data is sequential
- No actual timestamps are available
- Simple 1 sample <=> 1 second mapping with media time is sufficient

Configurations

Index based TimeSeries + Video + TimeSeriesLabels

Index-based TimeSeries (no timestamps at X axis)
Note: One value equals one second because the time axis is not specified in the TimeSeries tag. The video is synced with this idea — one sample equals one second if timestamps are not provided.

<View>
  <Video name="video" value="$video" sync="group_a"/>
  
  <TimeSeries name="timeseries"
              value="$ts" valueType="json"
              sync="group_a" sep=","
              overviewWidth="10%"
              >
    <Channel column="value" strokeColor="#FF0000"/>
    <Channel column="value" strokeColor="#00FF00"/>
  </TimeSeries>
  
  <TimeSeriesLabels name="labels" toName="timeseries">
    <Label value="action"/>
    <Label value="pause"/>
  </TimeSeriesLabels>

</View>

<!-- {
  "video": "/static/samples/opossum_snow.mp4",
  "ts": {
      "value": [
        10.7036820361892644,
        -0.18120536109567212,
        -0.39251488391214157,
        1.3384817293995075,
        0.8779675446349394,
        -0.1511946071051955,
        -0.7955547028255082,
        1.0736798948078534,
        1.1266164855584428,
        -0.440291574562604,
        -0.8436786901744359,
        -0.24956239687939094,
        1.268049926141147,
        0.6300808834120004,
        1.7946935071842107,
        -0.37700464705843,
        0.706518542026297,
        -0.45787451607104046,
        -2.3643354623876607,
        0.13984274721398307,
        0.3174445171792305,
        -1.8162371732091722,
        -0.30289394872251374,
        -0.730112449190387,
        -1.6852497246079239,
        -1.0473893262227658,
        0.10416951356137397,
        -2.0266185534759633,
        -0.05196549263706541,
        0.4436085233243668,
        -0.0956064205420074,
        -1.1790065141112944,
        -0.015063840978932763,
        0.28691755509866407,
        1.4122332721986657,
        0.40127732957527523,
        1.546243544663401,
        0.11119508061291504,
        -0.499517691828469,
        -0.02922576888373752,
        -0.8454178734108769,
        0.19122400060485445,
        0.6914340334390281,
        -0.18047241277757645,
        -0.6394589243120249,
        1.0019886671810008
      ]
  }
} -->

Time-based multiple TimeSeries + Audio + Video + TimeSeriesLabels

Time-based TimeSeries
TimeSeries, Audio and Video are synced together
Choices and timeline labels are used as control tags for labeling

<View>
  <Video name="video" value="$video" sync="group_a"/>
  <Audio name="audio" value="$video" sync="group_a"/>
  
  <TimeSeriesLabels name="timelinelabels" toName="accel_timeseries">
    <Label value="A"/>
    <Label value="B"/>
  </TimeSeriesLabels>	

  <TimeSeries 
              name="accel_timeseries"
              value="$accel_data"
              sync="group_a"
              timeColumn="time"
              timeFormat="%H:%M:%S.%f"
              timeDisplayFormat="%H:%M:%S.%f"
              overviewWidth="10%"
  >
    <Channel column="accel_x" strokeColor="#FF0000" height="100"/>
    <Channel column="accel_y" strokeColor="#00FF00" height="100"/>
  </TimeSeries>
  
  <TimeSeries 
              name="gyro_timeseries" 
              value="$gyro_data" 
              sync="group_a"
              timeColumn="time"
              timeFormat="%H:%M:%S.%f"
              timeDisplayFormat="%H:%M:%S.%f"
              overviewWidth="10%"
  >
    <Channel column="gyro_x" strokeColor="#0000FF" height="100"/>
    <Channel column="gyro_y" strokeColor="#FF00FF" height="100"/>
  </TimeSeries> 

</View>

<!-- {
  "video": "/static/samples/opossum_snow.mp4",
  "accel_data": "/samples/time-series.csv?time=time&values=accel_x%2Caccel_y&sep=%2C&tf=%H:%m:%d.%f",
  "gyro_data": "/samples/time-series.csv?time=time&values=gyro_x%2Cgyro_y&sep=%2C&tf=%H:%m:%d.%f"
}
-->

Example for time-series-accel.csv for accel_x, accel_y

time,accel_x,accel_y
00:01:01.000000,-0.056646571671882806,2.1066649495524605
00:01:02.000000,-0.6888765232989033,0.35646668995794306
00:01:03.000000,-0.23512086306647553,0.5799351613084716
00:01:04.000000,-0.9314772647682944,-0.5195693066279311
00:01:05.000000,1.321119143958512,-0.622026749003922
00:01:06.000000,0.10592100887528152,0.15477501359739493
00:01:07.000000,-0.6261150686384155,0.5624264458111049
00:01:08.000000,1.0829322997587332,-1.9590268928992862
00:01:09.000000,-1.2267135177322928,-0.4538764395229617
00:01:10.000000,1.6705781810127622,0.38407182850093363

Example for time-series-gyro.csv for gyro_x, gyro_y

time,gyro_x,gyro_y
00:01:01.000000,-0.776563940219835,-1.1115451852904443
00:01:02.000000,0.17111212343134966,-1.377696478819913
00:01:03.000000,-1.168085910547026,-0.8500307427257534
00:01:04.000000,-0.13947878605597916,0.9062482653127198
00:01:05.000000,0.3079887618179474,-1.6722497873634719
00:01:06.000000,-0.3825838786476411,-1.242585234780504
00:01:07.000000,-0.7015245817392025,-1.712515499827561
00:01:08.000000,-0.3437952109000775,-0.9337512501019165
00:01:09.000000,-0.19464021971045084,-0.9653381620475747
00:01:10.000000,-0.29753925483100785,-0.7699832734123578

Advanced Information

Fixes #4892: How can we label video and time series signals while watching them together?
Fixes #5359: Video/audio sync doesn't work

Add sync attribute to TimeSeries tag model to enable group-based synchronization
Compose TimeSeries with SyncableMixin for event broadcasting and handling
Implement sync event handlers for seek, play, and pause events
Add smooth view updates during audio/video playback
Prevent event flooding by suppressing sync events during playback
Maintain view window size during playback
Map media timeline to TimeSeries range (start/end points)

This enables synchronized playback and seeking between Video, Audio, and TimeSeries components when they share the same sync group. Multiple TimeSeries components can also sync with each other, allowing for coordinated view updates across different time series data. For example:

Feature flag

fflag_feat_optic_2125_timeseries_sync=1 label-studio

netlify · 2025-04-15T14:50:34Z

👷 Deploy request for heartex-docs pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	`027418a`

netlify · 2025-04-15T14:50:34Z

👷 Deploy request for label-studio-docs-new-theme pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	`027418a`

netlify · 2025-04-15T14:50:48Z

✅ Deploy Preview for label-studio-storybook canceled.

Name	Link
🔨 Latest commit	`027418a`
🔍 Latest deploy log	https://app.netlify.com/projects/label-studio-storybook/deploys/683a8a544f64eb0008cc0d1a

cloudmark · 2025-05-09T09:19:35Z

@bmartel @makseq updated to include the feature flag. Read for review.

… components Fixes #4892: How can we label video and time series signals while watching them together? Fixes HumanSignal#5359: Video/audio sync doesn't work - Add sync attribute to TimeSeries tag model to enable group-based synchronization - Compose TimeSeries with SyncableMixin for event broadcasting and handling - Implement sync event handlers for seek, play, and pause events - Add smooth view updates during audio/video playback - Prevent event flooding by suppressing sync events during playback - Maintain view window size during playback - Map media timeline to TimeSeries range (start/end points) This enables synchronized playback and seeking between Video, Audio, and TimeSeries components when they share the same sync group. Multiple TimeSeries components can also sync with each other, allowing for coordinated view updates across different time series data. For example: ``` <View> <Video name="video" value="$video" sync="group_a"/> <AudioPlus name="audio" value="$audio" sync="group_a"/> <TimeSeries name="accel_timeseries" value="$accel_data" sync="group_a"> <Channel column="accel_x" strokeColor="#FF0000"/> <Channel column="accel_y" strokeColor="#00FF00"/> </TimeSeries> <TimeSeries name="gyro_timeseries" value="$gyro_data" sync="group_a"> <Channel column="gyro_x" strokeColor="#0000FF"/> <Channel column="gyro_y" strokeColor="#FF00FF"/> </TimeSeries> </View> ```

Add feature flag controlled synchronization between TimeSeries and other media components (audio/video). This allows the TimeSeries component to: - Sync seek operations with other media components - Sync play/pause states - Handle playback speed synchronization - Enable click-to-seek functionality that syncs with other components The feature is controlled by the FF_TIMESERIES_SYNC feature flag, allowing for gradual rollout and testing. This change improves the user experience by enabling coordinated playback and seeking across different media types in the labeling interface.

This commit refactors the time synchronization mechanisms for TimeSeries and Video components to ensure consistent and accurate behavior. The primary goal is to standardize how time is communicated and interpreted during sync events (play, pause, seek). Key changes include: 1. **Standardized Sync Time Unit:** * Sync messages involving time between these components now aim for clarity, with a focus on components sending and expecting time values that can be consistently interpreted. (The ideal is relative seconds, with components converting locally). 2. **TimeSeries Synchronization (`TimeSeries.jsx`):** * **Relative vs. Absolute Time Handling:** `_handleSeek`, `_handlePlay`, and `_handlePause` methods now correctly convert incoming `data.time` (expected as relative seconds) into the TimeSeries' native units. This depends on the `isDate` flag: * If `isDate` is true (using `timeColumn` and `timeFormat`), relative seconds are converted to an absolute epoch millisecond value. * If `isDate` is false (autogenerated indices), the relative second value is used as the target index. * **Outgoing Sync Conversion:** `emitSeekSync` (on scroll/zoom) and `handleMainAreaClick` (on click-to-seek) now convert the TimeSeries' native time (epoch ms or index) into relative seconds before sending the sync message. * **Stability & Echo Prevention:** * Added `isAlive` checks to `_handleSeek`, `_handlePlay`, and `_handlePause` to prevent errors if the model instance is no longer part of the state tree during async operations. * These sync handlers now also check `if (data.initiator === self.name)` and ignore events initiated by the TimeSeries instance itself, preventing processing loops. 3. **Video Synchronization (`Video.js`):** * **Echo Prevention:** Implemented an `isProcessingIncomingSync` flag. This flag is set when `handleSync` is processing an event. The `handleSeek` method (triggered by native video events) now checks this flag and will not propagate a new `triggerSync("seek")` if the seek was a direct result of an incoming sync message. * **Initiator Tracking:** * The `triggerSync` method now consistently adds `initiator: self.name` to all outgoing sync payloads. * `handleSync` and `handleSyncSpeed` now check `if (data.initiator === self.name)` and ignore messages that the Video component itself originated. These changes improve the robustness and predictability of synchronization when TimeSeries and Video components interact, ensuring time values are handled more consistently.

hlomzik · 2025-05-23T11:02:05Z

/git merge

Workflow run
Successfully merged: Already up to date.

netlify · 2025-05-23T11:20:21Z

✅ Deploy Preview for label-studio-playground canceled.

Name	Link
🔨 Latest commit	`027418a`
🔍 Latest deploy log	https://app.netlify.com/projects/label-studio-playground/deploys/683a8a540cbf92000854c3dd

web/libs/editor/src/tags/object/Video/Video.js

hlomzik · 2025-05-25T15:25:34Z

This is awesome job, @cloudmark! I tried it and was impressed, that's exactly why SyncableMixin was introduced, to allow improvements like this!

I see a lot of isAlive() checks, what's the reason for such a precaution? Can we fix the original issue when we are trying render or interact with TimeSeries when they are detached?
What are you fixing in brushstarted()? https://github.com/HumanSignal/label-studio/pull/7376/files?diff=unified&w=1#diff-31822f7cd5c02fc2986d0c6fb4642f4a350a338699574efba86e68e5ad984b39R909
Also I added some questions about how original SyncableMixin worked and your fixes around it

Besides this it's awesome! Looking forward for answers, thank you!

hlomzik · 2025-05-25T16:08:29Z

Also we have to document how does it sync with Audio and Video. I can write a doc, but I need a guidance.

As I understand time from media works as relative delta for TimeSeries if it's time based, right? Questions:

What if it's indices-based?
What if lengths are different?

cloudmark · 2025-05-29T06:53:29Z

This is awesome job, @cloudmark! I tried it and was impressed, that's exactly why SyncableMixin was introduced, to allow improvements like this!

I see a lot of isAlive() checks, what's the reason for such a precaution? Can we fix the original issue when we are trying render or interact with TimeSeries when they are detached?

What are you fixing in brushstarted()? https://github.com/HumanSignal/label-studio/pull/7376/files?diff=unified&w=1#diff-31822f7cd5c02fc2986d0c6fb4642f4a350a338699574efba86e68e5ad984b39R909

Also I added some questions about how original SyncableMixin worked and your fixes around it

Besides this it's awesome! Looking forward for answers, thank you!

I did the isAlive() checks because the component was detaching; I have reverted this now. It could have been when the component was receiving a substantial amount of messages due to the sync bug. I took another look at the brushstarted() and will revert this as well. I'm sure there was an edge case I was trying to fix with this; however, I cannot reproduce it. I created a video to show the SyncableMixin bug; this should be tackled separately, I believe, since it’s not an issue with the sync per se.

cloudmark · 2025-05-29T06:56:16Z

Also we have to document how does it sync with Audio and Video. I can write a doc, but I need a guidance.

As I understand time from media works as relative delta for TimeSeries if it's time based, right? Questions:

What if it's indices-based?

What if lengths are different?

The text was moved to the first PR message

…cing lines of code and optimizing performance.

Tests are successful, they were not triggered, meaning that SyncableMixin catches all cases we checked.

SyncableMixin should block such echo events on its own.

SyncableMixin should block events fired back to origin.

hlomzik · 2025-05-30T22:30:11Z

/git merge

Workflow run
Successfully merged: create mode 100644 web/libs/editor/src/components/TaskSummary/types.ts

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15356923309

github-actions bot added the feat label Apr 15, 2025

cloudmark mentioned this pull request Apr 21, 2025

feat: FIT-173: Audio spectrograms #7400

Merged

makseq changed the title ~~feat: Add sync between TimeSeries, Video, Audio, and other TimeSeries…~~ feat: Add sync between TimeSeries, Video, Audio, and other TimeSeries components Apr 24, 2025

makseq changed the title ~~feat: Add sync between TimeSeries, Video, Audio, and other TimeSeries components~~ feat: Add sync between TimeSeries, Video, Audio Apr 24, 2025

makseq changed the title ~~feat: Add sync between TimeSeries, Video, Audio~~ feat: OPTIC-2125: Add sync between TimeSeries, Video, Audio May 13, 2025

cloudmark requested review from a team, hlomzik, Gondragos and nick-skriabin as code owners May 17, 2025 12:25

cloudmark added 4 commits May 21, 2025 15:58

fix: update the code to handle multiple seek messages.

0e23911

cloudmark force-pushed the feature/timeline-sync branch from f0df35a to 0e23911 Compare May 21, 2025 14:01

Merge remote-tracking branch 'origin/develop' into feature/timeline-sync

25e335c

hlomzik reviewed May 25, 2025

View reviewed changes

web/libs/editor/src/tags/object/Video/Video.js Outdated Show resolved Hide resolved

hlomzik reviewed May 25, 2025

View reviewed changes

web/libs/editor/src/tags/object/Video/Video.js Outdated Show resolved Hide resolved

hlomzik reviewed May 25, 2025

View reviewed changes

web/libs/editor/src/tags/object/Video/Video.js Outdated Show resolved Hide resolved

Merge branch 'HumanSignal:develop' into feature/timeline-sync

0278f1a

fix: refactor TimeSeries and Video components: Simplify logic by redu…

2b6fb9d

…cing lines of code and optimizing performance.

cloudmark and others added 5 commits May 30, 2025 08:11

Merge branch 'HumanSignal:develop' into feature/timeline-sync

f045025

Add logging to test SyncableMixin work with TS

edddddc

Remove logging

674828e

Tests are successful, they were not triggered, meaning that SyncableMixin catches all cases we checked.

Remove excess isProcessingIncomingSync

c2cd77b

SyncableMixin should block such echo events on its own.

Remove excess initiator checks

abb55a6

SyncableMixin should block events fired back to origin.

Merge branch 'develop' into 'feature/timeline-sync'

789bc3f

Workflow run: https://github.com/HumanSignal/label-studio/actions/runs/15356923309

hlomzik approved these changes May 30, 2025

View reviewed changes

makseq and others added 2 commits May 31, 2025 03:38

Update TimeSeries.jsx

7bbcb63

Merge branch 'HumanSignal:develop' into feature/timeline-sync

027418a

makseq enabled auto-merge (squash) June 2, 2025 13:35

makseq merged commit 334fd42 into HumanSignal:develop Jun 2, 2025
52 of 57 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: OPTIC-2125: Add sync between TimeSeries, Video, Audio #7376

feat: OPTIC-2125: Add sync between TimeSeries, Video, Audio #7376

Uh oh!

cloudmark commented Apr 15, 2025 •

edited by makseq

Loading

Uh oh!

netlify bot commented Apr 15, 2025 •

edited

Loading

Uh oh!

netlify bot commented Apr 15, 2025 •

edited

Loading

Uh oh!

netlify bot commented Apr 15, 2025 •

edited

Loading

Uh oh!

cloudmark commented May 9, 2025

Uh oh!

hlomzik commented May 23, 2025 •

edited by robot-ci-heartex

Loading

Uh oh!

netlify bot commented May 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hlomzik commented May 25, 2025

Uh oh!

hlomzik commented May 25, 2025

Uh oh!

cloudmark commented May 29, 2025

Uh oh!

cloudmark commented May 29, 2025 •

edited by makseq

Loading

Uh oh!

hlomzik commented May 30, 2025 •

edited by robot-ci-heartex

Loading

Uh oh!

Uh oh!

Uh oh!

feat: OPTIC-2125: Add sync between TimeSeries, Video, Audio #7376

feat: OPTIC-2125: Add sync between TimeSeries, Video, Audio #7376

Uh oh!

Conversation

cloudmark commented Apr 15, 2025 • edited by makseq Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TimeSeries Synchronization

Overview

Time Units

Configuration

Synchronization Behavior

Length Mismatches

Best Practices

Configurations

Index based TimeSeries + Video + TimeSeriesLabels

Time-based multiple TimeSeries + Audio + Video + TimeSeriesLabels

Advanced Information

Uh oh!

netlify bot commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

👷 Deploy request for heartex-docs pending review.

Uh oh!

netlify bot commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

👷 Deploy request for label-studio-docs-new-theme pending review.

Uh oh!

netlify bot commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for label-studio-storybook canceled.

Uh oh!

cloudmark commented May 9, 2025

Uh oh!

hlomzik commented May 23, 2025 • edited by robot-ci-heartex Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for label-studio-playground canceled.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hlomzik commented May 25, 2025

Uh oh!

hlomzik commented May 25, 2025

Uh oh!

cloudmark commented May 29, 2025

Uh oh!

cloudmark commented May 29, 2025 • edited by makseq Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hlomzik commented May 30, 2025 • edited by robot-ci-heartex Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cloudmark commented Apr 15, 2025 •

edited by makseq

Loading

netlify bot commented Apr 15, 2025 •

edited

Loading

netlify bot commented Apr 15, 2025 •

edited

Loading

netlify bot commented Apr 15, 2025 •

edited

Loading

hlomzik commented May 23, 2025 •

edited by robot-ci-heartex

Loading

netlify bot commented May 23, 2025 •

edited

Loading

cloudmark commented May 29, 2025 •

edited by makseq

Loading

hlomzik commented May 30, 2025 •

edited by robot-ci-heartex

Loading