FragmentedMp4Muxer: add support for sidx and mfra boxes to enable seekability in fragmented MP4 output

[mmilazzo-telsy]

Use case description

The FragmentedMp4Muxer currently does not write a sidx nor a mfra box in the produced fragmented MP4 files.

As a result, FragmentedMp4Extractor cannot treat these files as seekable (VLC or even the iOS player can play files without these boxes, I don't know how they do it).

This is a significant limitation when the output file is intended for local playback or any use case that requires seeking (e.g. trimming, re-muxing, or progressive download scenarios where the full file is available).

Proposed solution

Add an option to FragmentedMp4Muxer (or enable it by default) to write one or both of the following boxes:

sidx — written after the moov box, before the first moof, providing a compact byte-range index per segment. This is the preferred approach for streaming-friendly files (DASH, etc.) but it would strictly require a seekable file when muxing.
mfra — written at the end of the file, containing a mfro (Movie Fragment Random Access Offset) and one or more tfra (Track Fragment Random Access) boxes. This is simpler to produce since all offsets are known at finalization time and would not require seeking on the outputfile.

Adding either box would allow FragmentedMp4Extractor to recognize the file as seekable and enable full random-access playback via ExoPlayer/Media3.

Alternatives considered

See how VLC allows to seek even if those boxes are not present, maybe it uses more approssimative methods. This would maximize compatibility.

[ychaparov]

Thanks, this is a valid feature request! Assigning to @droid-girl and @shahdDaghash for prioritization.

In the mean time, could you transmux the FragmentedMp4 file into a static Mp4 file using media3-transformer?

Transmuxing would copy the compressed samples into a new, properly seekable output MP4 file. The transmuxing operation should be a lot faster than decoding and encoding.

See https://developer.android.com/media/media3/transformer/transformations#transcode for a bit more details

[mmilazzo-telsy]

I can’t, because the file must be treated while it’s being generated, so I can neither remux it nor seek within it during muxing.

[ychaparov]

Thank you for the details.

Could you confirm -- this is some sort of file system encryption, and not video sample encryption such as DRM Common Encryption (CENC)? ( https://docs.unified-streaming.com/documentation/drm/common-encryption.html / https://developers.google.com/widevine/drm/overview )

I don't completely understand the request. You say:

sidx — written after the moov box, before the first moof, providing a compact byte-range index per segment. This is the preferred approach for streaming-friendly files (DASH, etc.) but it would strictly require a seekable file when muxing

And then:

the file must be encrypted while it’s being generated, so I can neither remux it nor seek within it during muxing.

If you cannot seek within the muxer output file because of the encryption, how would adding sidx and requiring seekable muxer output for fragmented mp4 help?

Is your request to add mfra support in the muxer specifically?

[mmilazzo-telsy]

I'm not referring to DRM, but to other protection mechanisms.
The fact that in my specific case I can't seek is just a limitation of my setup; anyone else without that limitation could still choose to write the sidx.
In my case, having the mfra box would be the preferred solution, and it could also be supported when using a non-seekable output stream.

[ychaparov]

Thank you, this is an important clarification.

sidx is nice to have but requires changing FragmentedMp4Muxer#Builder(WritableByteChannel) into FragmentedMp4Muxer#Builder(SeekableMuxerOutput)

If we were to change FragmentedMp4Muxer to always require seekable muxer output, the specific case could not be supported.

And when SeekableMuxerOutput can be used, the non-fragmented Mp4Muxer would be a more widely supported output format.

@shahdDaghash I recommend treating the sidx part of this request as lower priority than adding mfra support.