Skip to content

Conversation

bruno-f-cruz
Copy link
Member

@bruno-f-cruz bruno-f-cruz commented Sep 2, 2025

This PR implements an ffmpeg logger for all cameras. The following caveats should be considered:

  • I tried to touch the workflow as little as possible. A better approach would be to adopt the same pattern for dynamically creating cameras as needed like we do in VrForaging but this would require substantial changes.
  • The codecs are currently hard-coded according to https://allenneuraldynamics.github.io/aind-file-standards/file_formats/behavior_videos/#acquisitionrawprimary-data-format.
  • I did not want to implement them flexibly since I don't have visibility into how the CSV settings are maintained and the currently reserved keywords. If you need it, it should be an easy addition.
  • Despite enabling encoding for all cameras, i can make no guarantees as to whether the hardware will be able to keep up with the 4 cameras currently specified at 500fps each (I highly doubt it)
  • There was an undocumented prefix on top of the root logging path (i.e. /../../behavior-videos) that I have no idea why was necessary. Perhaps the dynamic foraging GUI is considering something else as being the "root path" than what bonsai is? Lmk if this is an issue, and I can easily fix it.
  • The rest of the workflow is still responsible for making sure the triggering only starts AFTER the cameras have started and, ideally, make sure that cameras stop before the recording stops. This will ensure that you get both the start and end of the stream in your data.

Important

This PR requires the following 3rd party dependencies:

alexpiet and others added 22 commits June 13, 2025 10:00
Add "transfer_service_job_type" to Settings and watchdog manifest
@alexpiet
Copy link
Collaborator

alexpiet commented Sep 2, 2025

@bruno-f-cruz can you merge into develop instead of main

@bruno-f-cruz bruno-f-cruz changed the base branch from main to develop September 2, 2025 21:12
@bruno-f-cruz
Copy link
Member Author

bruno-f-cruz commented Sep 2, 2025

@bruno-f-cruz can you merge into develop instead of main

Rebased on developed and retargetted. Sorry, I didn't read the PR instructions. If you want people to target dev, what repos usually do is to make that the "default" branch. (see https://github.com/AllenNeuralDynamics/aind-data-schema for an example)

@rachelstephlee
Copy link
Contributor

Review request:

  1. Check if the functionality is sufficient (right now the codec is hard-coded for instance)
  2. Test with 1 camera at say 200hz (should work)
  3. Test with all cameras you want to use and see if it works (no clue)

@alexpiet
Copy link
Collaborator

alexpiet commented Sep 2, 2025

CSV settings file are maintained by this schema: https://github.com/AllenNeuralDynamics/dynamic-foraging-task/blob/main/src/foraging_gui/settings_model.py

@micahwoodard
Copy link
Collaborator

@XX-Yin would be a better person to review this tahn me since he is more familiar with how the bonsai workflows work here

@micahwoodard micahwoodard requested a review from XX-Yin September 4, 2025 16:21
@XX-Yin
Copy link
Collaborator

XX-Yin commented Sep 4, 2025

@bruno-f-cruz This looks good to me. Can you have some screenshot to highlight the bonsai change? It's difficult for us to review what's changed on the bonsai workflow?

@rachelstephlee
Copy link
Contributor

i had bruno take this screenshot as the changed version:
image

@rachelstephlee
Copy link
Contributor

We're hitting an error:

"Unable to cast object of type
"BOnsai.Expressions.Subscribe.Subject' to type
"Bosai.Design.Visualizers.TableLayoutPanelBuilder'"

We tried rebuilding the packages, but the error persists when we open and start up bonsai. This was tested on the ephys rigs downstairs.
image

@bruno-f-cruz
Copy link
Member Author

This is an error with the layout file. You should delete your local copy of the layout files and try again.

@bruno-f-cruz
Copy link
Member Author

@rachelstephlee @XX-Yin

  • Did you manage to understand why the Python GUI was crashing?
  • Can you document in this PR how the root path that arrives to Bonsai looks like? We should update this PR to reflect that

@alexpiet
Copy link
Collaborator

alexpiet commented Sep 9, 2025

@rachelstephlee @XX-Yin can you share the GUI logfile from the crashes?

@rachelstephlee
Copy link
Contributor

@bruno-f-cruz and I did a debugging session at 446. We're hitting the same problems we saw in the ephys room.

  1. GUI log file during crash doesn't inform us of much:

W10DT714673-C_gui_log_2025-09-10_11-48-30.txt

  1. We did note that even though the python GUI correctly creates new folders C:\behavior_data\446-6-C\0\behavior_0_2025-09-10_11-49-34\fib, the Bonsai seemed to write OSC files to a different location: dynamic_foraging_task/Data/TestPath1... It only saved an OSC file that was empty (OSC_run_1904_02-19T00-00-00.csv

  2. We also noted that the point at which Bonsai stopped receiving information from the GUI was at a node called CameraStartType-- it's unclear where or how this information is sent.

image

Given that this is failing at the handshake between Python GUI and Bonsai, we need someone who can help us figure out that protocol.

@XX-Yin
Copy link
Collaborator

XX-Yin commented Sep 14, 2025

@bruno-f-cruz @rachelstephlee The stuck issue was solved.

In this PR, LoggingRootPath was added after RootPath, which prevented RootPath from being propagated to other variables.

This PR:
image

Updated:
image

Folder structure:
image

@bruno-f-cruz
Copy link
Member Author

Can you explain why this crashes the GUI tho?

@XX-Yin
Copy link
Collaborator

XX-Yin commented Sep 14, 2025

Can you explain why this crashes the GUI tho?

The GUI is waiting for a return signal after logging starts. This return signal should go from RootPath to TOBOnsaiOSC. When SubscribeSubject RootPath is not triggered, the GUI never receives the return signal and ends up stuck.

image

@bruno-f-cruz
Copy link
Member Author

Are you sure the variable is not being propagated vs just a race condition from somewhere else?

I can't seem to reproduce this with a self-contained example:

<?xml version="1.0" encoding="utf-8"?>
<WorkflowBuilder Version="2.8.2"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                 xmlns:rx="clr-namespace:Bonsai.Reactive;assembly=Bonsai.Core"
                 xmlns:io="clr-namespace:Bonsai.IO;assembly=Bonsai.System"
                 xmlns="https://bonsai-rx.org/2018/workflow">
  <Workflow>
    <Nodes>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="StringProperty">
          <Value>Split_</Value>
        </Combinator>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="rx:Take">
          <rx:Count>1</rx:Count>
        </Combinator>
      </Expression>
      <Expression xsi:type="rx:CreateObservable">
        <Workflow>
          <Nodes>
            <Expression xsi:type="WorkflowInput">
              <Name>Source1</Name>
            </Expression>
            <Expression xsi:type="Combinator">
              <Combinator xsi:type="rx:Take">
                <rx:Count>1</rx:Count>
              </Combinator>
            </Expression>
            <Expression xsi:type="rx:AsyncSubject">
              <Name>Foo</Name>
            </Expression>
            <Expression xsi:type="SubscribeSubject">
              <Name>Foo</Name>
            </Expression>
            <Expression xsi:type="rx:AsyncSubject">
              <Name>Bar</Name>
            </Expression>
            <Expression xsi:type="Add">
              <Operand xsi:type="StringProperty">
                <Value>Foo</Value>
              </Operand>
            </Expression>
            <Expression xsi:type="Combinator">
              <Combinator xsi:type="io:WriteLine" />
            </Expression>
            <Expression xsi:type="SubscribeSubject">
              <Name>Bar</Name>
            </Expression>
            <Expression xsi:type="Add">
              <Operand xsi:type="StringProperty">
                <Value>Bar</Value>
              </Operand>
            </Expression>
            <Expression xsi:type="Combinator">
              <Combinator xsi:type="io:WriteLine" />
            </Expression>
          </Nodes>
          <Edges>
            <Edge From="0" To="1" Label="Source1" />
            <Edge From="1" To="2" Label="Source1" />
            <Edge From="3" To="4" Label="Source1" />
            <Edge From="4" To="5" Label="Source1" />
            <Edge From="5" To="6" Label="Source1" />
            <Edge From="7" To="8" Label="Source1" />
            <Edge From="8" To="9" Label="Source1" />
          </Edges>
        </Workflow>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="rx:Switch" />
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="StringProperty">
          <Value>Together_</Value>
        </Combinator>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="rx:Take">
          <rx:Count>1</rx:Count>
        </Combinator>
      </Expression>
      <Expression xsi:type="rx:CreateObservable">
        <Workflow>
          <Nodes>
            <Expression xsi:type="WorkflowInput">
              <Name>Source1</Name>
            </Expression>
            <Expression xsi:type="Combinator">
              <Combinator xsi:type="rx:Take">
                <rx:Count>1</rx:Count>
              </Combinator>
            </Expression>
            <Expression xsi:type="rx:AsyncSubject">
              <Name>Foo</Name>
            </Expression>
            <Expression xsi:type="rx:AsyncSubject">
              <Name>Bar</Name>
            </Expression>
            <Expression xsi:type="SubscribeSubject">
              <Name>Foo</Name>
            </Expression>
            <Expression xsi:type="Add">
              <Operand xsi:type="StringProperty">
                <Value>Foo</Value>
              </Operand>
            </Expression>
            <Expression xsi:type="Combinator">
              <Combinator xsi:type="io:WriteLine" />
            </Expression>
            <Expression xsi:type="SubscribeSubject">
              <Name>Bar</Name>
            </Expression>
            <Expression xsi:type="Add">
              <Operand xsi:type="StringProperty">
                <Value>Bar</Value>
              </Operand>
            </Expression>
            <Expression xsi:type="Combinator">
              <Combinator xsi:type="io:WriteLine" />
            </Expression>
          </Nodes>
          <Edges>
            <Edge From="0" To="1" Label="Source1" />
            <Edge From="1" To="2" Label="Source1" />
            <Edge From="2" To="3" Label="Source1" />
            <Edge From="4" To="5" Label="Source1" />
            <Edge From="5" To="6" Label="Source1" />
            <Edge From="7" To="8" Label="Source1" />
            <Edge From="8" To="9" Label="Source1" />
          </Edges>
        </Workflow>
      </Expression>
      <Expression xsi:type="Combinator">
        <Combinator xsi:type="rx:Switch" />
      </Expression>
    </Nodes>
    <Edges>
      <Edge From="0" To="1" Label="Source1" />
      <Edge From="1" To="2" Label="Source1" />
      <Edge From="2" To="3" Label="Source1" />
      <Edge From="4" To="5" Label="Source1" />
      <Edge From="5" To="6" Label="Source1" />
      <Edge From="6" To="7" Label="Source1" />
    </Edges>
  </Workflow>
</WorkflowBuilder>

If you run this on your PC, what do you get on the command line?

@XX-Yin
Copy link
Collaborator

XX-Yin commented Sep 14, 2025 via email

@alexpiet
Copy link
Collaborator

@bruno-f-cruz By race condition, do you think its this issue: #1521 ?

@bruno-f-cruz
Copy link
Member Author

@bruno-f-cruz By race condition, do you think its this issue: #1521 ?

I don't have a clear idea what it could be tbh. It could be related to that issue or, as XinXin pointed out it could be a bug in bonsai (there were some edge cases of initialization solved in 2.9). Could also be something that I am just not seeing just be looking at the workflow.

All I am finding weird is that I can't reproduce it with a minimal example in my PC.

Let's see what we find, even if it is working I find it dangerous to merge without understanding what this is.

@XX-Yin
Copy link
Collaborator

XX-Yin commented Sep 15, 2025

@rachelstephlee @micahwoodard @bruno-f-cruz

To do after the debugging meeting:

  1. @XX-Yin will update the bonsai workflow to fix the stuck issue.
  2. @XX-Yin will document the issue to let Gonçalo and other people be aware of this issue. Where to post this issue?
  3. Update the layout file. @bruno-f-cruz suggested to discard the layout file and be replaced with the visualizer.
  4. Update the python code to make the "check drop of frame" work for this new video folder structure.
  5. Update the manifest file to trigger the video compression after uploading.

@rachelstephlee
Copy link
Contributor

  1. @rachelstephlee and @XX-Yin will stress test the fix (rachel on 446/447, Xinxin on the ephys rigs in 320) to ensure that this works well. Once we do that, we can approve this PR.

@bruno-f-cruz
Copy link
Member Author

@rachelstephlee @micahwoodard @bruno-f-cruz

To do after the debugging meeting:

  1. @XX-Yin will update the bonsai workflow to fix the stuck issue.
  2. @XX-Yin will document the issue to let Gonçalo and other people be aware of this issue. Where to post this issue?
  3. Update the layout file. @bruno-f-cruz suggested to discard the layout file and be replaced with the visualizer.
  4. Update the python code to make the "check drop of frame" work for this new video folder structure.
  5. Update the manifest file to trigger the video compression after uploading.

Some notes:

  1. I think that if you want to post this as an issue, you should first try to run the workflow in version 2.9. The Bonsai running the task is two years old, it is possible that this got solved meanwhile. As someone who tries to help people with these issues in the Bonsai forum, I would also caution against putting a workflow with hundreds of nodes and expect people to debug it. First, it will be hard to reproduce since people would need to use the GUI and have access to the hardware. Second, when I tried to reproduce the bug using a minimal example, I did not succeed in getting the behavior XinXin describes. This suggests that the behavior is probably the result of an interaction with something else in the workflow.

  2. In order for this to work, you will have to upgrade to Bonsai 2.9 and use the VisualizerWindow operator. This will programmatically trigger the visualizer of any node.

  3. We coded this during one of the Aind.Behavior.Services meetings. It may be easier to just have Bonsai do this check instead of having yet another interface between python and bonsai.

  4. This MAY not be necessary depending n what the default jobs are for your job type. Talk to Jon Young he will likely know about it.

@rachelstephlee
Copy link
Contributor

on 5-- might also want to message @Ahad-Allen as well, because he has been updating the job type for dynamic foraging task. We probably want it to work for the new job types.

@XX-Yin
Copy link
Collaborator

XX-Yin commented Sep 26, 2025

@rachelstephlee Bonsai workflow was updated. We need somebody to update the bonsai visualizer and the python code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants