MSTP's VideoFrame lifetime management model assumes a single consumer

[jan-ivar]

MediaStreamTrackProcessor appears to expect consumers to call VideoFrame.close() manually. Unfortunately, this:

1. Assumes a single consumer per VideoFrame
2. Creates a footgun where consumers may forget to call this method, with potentially devastating OS-level side-effects

Ditto MediaStreamTrackGenerator which calls VideoFrame.close() implicitly, preventing reuse with another sink. It too:

1. Assumes a single consumer per VideoFrame

VideoFrames are immutable objects (with a necessary mutable close() method because we don't want to wait for GC, a pattern that works well for a single owner).

A benefit of immutability is supposed to be shareability, so we should be able to tee a stream:

const [readable1, readable2] = new MediaStreamTrackProcessor({track}).readable.tee();

await Promise.all([
  readable1.pipeThrough(new MyTransform(640)).pipeTo(sink1),
  readable2.pipeThrough(new MyTransform(320)).pipeTo(sink2)
]);

...and this would work, except it breaks if MyTransform calls VideoFrame.close().

IOW, since sources may have multiple sinks, it seems better for sources to own the VideoFrames it doles out, and manage their lifetime.

source -.---> sink1
        `---> sink2

Proposal

• Have MediaStreamTrackGenerator manage lifetime of VideoFrames it passes to immediate sinks.
  • (and any transform stream that wants to pass through a VideoFrame needs to copy or renew it somehow)
• Stop having MediaStreamTrackGenerator call VideoFrame.close() and rely on the producer to do this.

[aboba]

The VideoFrame interface is defined here. The copyTo and clone methods potentially enable support for multiple consumers. The reference counting mechanism is defined here.

[aboba]

I am wondering whether the set of APIs using VideoFrames (media-capture transform, WebCodecs, Canvas, WebGL/WebGPU) has a consistent strategy with respect to memory handling and implicit close() operations.

The argument for MSTG implicitly calling VideoFrame.close() is that converting to a MediaStreamTrack is typically the last step in the pipeline. But can we assume that every VideoFrame output by MSTP will eventually be consumed by an API with an implicit close() operation? For example, do other rendering approaches (e.g. canvas or WebGL/GPU) also include an implicit close()?

WebCodecs encode() or decode() methods do not implicitly call close(). But if the VideoFrame is eventually rendered then it may be taken care of by an implicit close().

[jan-ivar]

@MattiasBuelens raised another problem in whatwg/streams#1155 (comment):

The Streams standard will discard all queued chunks, but then it's up to the garbage collector to clean up any resources associated with these VideoFrames. Can that cause problems if these are not cleaned up soon enough? 🤔

Good observation. This could be solved if the MSTP owned the VideoFrames and was responsible for their cleanup.

[dogben]

@alvestrand has looked into this.

[aboba]

"This could be solved if the MSTP owned the VideoFrames and was responsible for their cleanup."

[BA] MSTP can't be the only owner if there are multiple references (e.g. if VideoFrame.clone() was called). In that case, even an implicit close() won't reclaim the VideoFrame.

[chcunningham]

1. Assumes a single consumer per VideoFrame

I think it's fair to say the current design is not optimized for the tee scenario you outlined. But I wouldn't go so far as to say "assumes a single consumer", since you can always clone() the frame as @aboba points out.

You could workaround the tee() limitation by implementing a custom FrameTee that reads from the original stream, and writes to two output streams, cloning the frames as needed for the second stream.

2. Creates a footgun where consumers may forget to call this method, with potentially devastating OS-level side-effects

It's definitely critical that folks close() their frames. The WC spec attempts to highlight this and Chrome's impl actually console warns if we observe GC for a VideoFrame that isn't already closed.

I'm not sure it's as bad as OS-level side-effects. Chrome's pool of camera frames is currently a fixed size thing, so it is technically possible for one bad tab to ruin the experience for other tabs. We have some basic protections in place, but the long term plan is to simply make the pool dynamic and limit each tab's number of frames such that a tab may harm itself by leaking frames, but it can't harm other tabs or the larger OS. This would follow the same pattern as OOM for general JS... tab is to OOM itself, but it doesn't bring down the system.

Good observation. This could be solved if the MSTP owned the VideoFrames and was responsible for their cleanup.

How would MSTP know when it was safe to close() the frame. Is the idea that you allow a single pass of the main thread to use the frame after its read from the ReadableStream? I'm not immediately sure if that is architecturally (from streams spec pov) feasible. Such a design would also have footguns.

[BA] MSTP can't be the only owner if there are multiple references (e.g. if VideoFrame.clone() was called). In that case, even an implicit close() won't reclaim the VideoFrame.

Agreed. Authors are still on the hook to close() their own clones.

[chcunningham]

I am wondering whether the set of APIs using VideoFrames (media-capture transform, WebCodecs, Canvas, WebGL/WebGPU) has a consistent strategy with respect to memory handling and implicit close() operations.

The argument for MSTG implicitly calling VideoFrame.close() is that converting to a MediaStreamTrack is typically the last step in the pipeline. But can we assume that every VideoFrame output by MSTP will eventually be consumed by an API with an implicit close() operation? For example, do other rendering approaches (e.g. canvas or WebGL/GPU) also include an implicit close()?

WebCodecs, Canvas, and WebGL are all consistent in that they don't automatically close(). Specific WebGPU : WebCodecs integration isn't spec'ed yet (beyond Canvas integration), but I anticipate any WebGPU method that takes VideoFrames would not automatically close them.

Media-capture transform is mixed, with only the MSTG automatically closing for reasons you mentioned earlier. With MSTP not automatically closing, it becomes necessary that MSTG do the auto close() to enable code like processor.readable.pipeThrough(transformer).pipeTo(generator.writable);.

[youennf]

How would MSTP know when it was safe to close() the frame. Is the idea that you allow a single pass of the main thread to use the frame after its read from the ReadableStream? I'm not immediately sure if that is architecturally (from streams spec pov) feasible. Such a design would also have footguns.

A proposed alternative for VideoFrame exposure to MediaStreamTrack is to have a callback which returns a promise. Video Frame is closed when promise is settled.
Translated to WebCodecs, that would mean VideoFrameOutputCallback returning a promise. This would introduce backpressure to WebCodecs which is a nice feature provided by streams.
With that model, there is no need for MSTG to automatically call close.

[guidou]

MediaStreamTrackProcessor appears to expect consumers to call VideoFrame.close() manually. Unfortunately, this:

1. Assumes a single consumer per VideoFrame

VideoFrames can be cloned and a separate clone can be passed to multiple consumers. Each clone should be closed, of course. Clone is an inexpensive operation since it's just adding a reference to an underlying resource.

2. Creates a footgun where consumers may forget to call this method, with potentially devastating OS-level side-effects

What are those devastating side effects?
AFAICT, the only one that has been mentioned is exhausting a shared fixed pool of open underlying video frames, but implementations can limit the number of open underlying frames for a single page in order to prevent resource exhaustion.

Automatic closing by MSTP might help with an application that accidentally forgets to close, but it is not useful against an application that intentionally keeps VideoFrames alive (e.g., by cloning them and not closing the clone). The effects of accidentally forgetting to close are normally very visible, so automatic closing is of very limited utility in practice.

Ditto MediaStreamTrackGenerator which calls VideoFrame.close() implicitly, preventing reuse with another sink. It too:

1. Assumes a single consumer per VideoFrame

MSTG is a consumer of VideoFrame objects and does not have VideoFrame consumers. It closes VideoFrame objects automatically in order to allow piping to work well. Note that MSTG closes the JS VideoFrame objects. The underlying resources may still be alive after closing the VideoFrame, depending on the sinks to which MSTG is connected to.

I'm closing this issue since the premise is that MSTP assumes a single consumer, which is obviously incorrect.
The issue of resource exhaustion is separate and has also been discussed and addressed.