Make audio processors MediaItem aware

[Tolriq]

My app support gapless and everything works fine.

But my app also support Replay Gain that it applies from extracted metadata. The gain is applied from onTracksChanged and this works nicely for the first song (And also when applying for albums since the replay gain is constant).

The problem arise when users want to play random songs with random replay gain values. In that case due to pre buffering for gapless, onTracksChanged is called too late and the replay gain is applied late (For medium quality mp3 up to 2 seconds). This obviously is not nice.

Since they play random tracks there's no need for gapless, and they prefer a small delay for the next song than a wrong volume for a few seconds.

The question is how can I achieve that (disable gapless, so that I can apply the replaygain at the proper moment) in normal usage of exoplayer and media sources and everything.

I'd like to avoid rewrite a complete different path to not use ExoPlayer playlists and handle everything manually.

[tonihei]

Could you point out how exactly you apply the replay gain once you have the metadata? Is this a simple call to setVolume() or something more advanced than that? And also, where exactly do you get the metadata from - is it Format.metadata?

The reason for these questions is that waiting for onTracksChanged may work in practise, but it is strictly speaking also too late because you likely miss a few audio samples at the beginning of the stream. I wonder if whatever needs to be done should be implemented as an AudioProcessor or as an adaptation/wrapper of an existing audio processor. They unfortunately don't see the full Format object, but just a simplified view of it, but we may be able to make some adaptations to set the full Format (including the needed metadata). The advantage is not only that the first start should be better, but also all transitions (gapless/prebuffered or not) can apply the right effect very accurately to each audio sample.

[Tolriq]

Yes I get the RG values from the Format.metadata. (But I could also get them before if necessary).

And I use most of the time an AudioProcessor for that but there's no calls to configure or anything on media change if they are the same codec and everything. (And it's also an issue for example to enable / disable audio processor on demand, currently we can only bypass by doing a simple copy). The other way I use sometimes is to apply an Android DynamicsProcessing at higher level when I also apply some EQ stuff) but this is even less precise.

If there was a way with AudioProcessor then I would always use it but this is not the case currently.

For the first song or when pressing next (so a new prepare) this works perfectly the metadata is extracted before the codec is initialized and applied correctly before first audio sample or at least the flush works soon enough to ensure no wrong sample is played.

But with gapless we are not talking about 1 audio sample but could be up to 2 seconds that is already passed through the processor and the flush is wayy too late to recover. With flac or high res media this is a lot less but with mid quality MP3 this is 2 seconds.

Anyway if there's no way to disable the current gapless behavior, an audio processor solution would be nice.
For this need and some others (like enabling / disabling processor) having a way at the processor level to know when there's a media change would help a lot. (Including a way to disable the processor based on the media for a small performance gain and avoid an useless buffer copy).

[Tolriq]

Bump ? Currently applying the RG via ffmpeg as a secondary solution but this is not efficient at all. A future native solution would be better.

[Samrobbo]

AudioProcessor is the ideal place to handle this replay gain.

It sounds to me like the problem is around the fact that AudioProcessor instances do not get the configure call when the media item changes. This is an intentional design because avoiding configuring/flushing the DefaultAudioSink means there are no audio discontinuities.

@tonihei - You're more familiar with ExoPlayer components. Is there functionality in audio renderers to force the sink to be configured when the media item changes? I can't determine what the trigger is for MediaCodecAudioRenderer/DecoderAudioRenderer to configure the underlying sink is - It all seems based on codec reuse.

Once/if @tonihei confirms this, my recommendation would be for @Tolriq to pass in a custom ForwardingAudioSink (of DefaultAudioSink), and intercept the AudioSink#configure call. Use the Format passed in to check for replay values in the metadata, and pass that information to a custom AudioProcessor. DefaultAudioSink#configure will then reconfigure and flush all AudioProcessor instances, which will make active the new replay value.