Source code

Revision control

Copy as Markdown

Other Tools

# Sample Times in Fragmented MP4
This document explains how `MoofParser` reads a fragmented MP4 file and gives
each sample a decode time and a composition range. It also gives the reason
for each step. The examples use small numbers.
## Plain and fragmented MP4
A plain MP4 file keeps one sample table for each track in `moov/stbl`.
`MP4Metadata` reads that table through mp4parse. The `MP4SampleIndex`
constructor then copies the samples into `MP4SampleIndex::mIndex`.
A fragmented MP4 file splits the samples into fragments. Each fragment is a
`moof` box and an `mdat` box. The `moof` box has a `traf` box for each track.
The `traf` box has a `tfdt` box and one or more `trun` boxes. When the `moov`
box has no samples, `MP4SampleIndex` creates a `MoofParser`. The parser makes
one `Moof` object for each `moof` box, and each `Moof` has its own `mIndex`.
| File type | Where the samples are | Index |
| ---------- | --------------------- | ----------------------------- |
| Plain | `moov/stbl` | `MP4SampleIndex::mIndex` |
| Fragmented | `trun` in each `moof` | `Moof::mIndex`, one per moof |
MSE must use fragmented MP4. In the ISO BMFF byte stream format for MSE, a
media segment is a `moof` box and its `mdat` box.
## Where the parser runs
For file playback, the call chain is:
```
MP4TrackDemuxer::EnsureUpToDateIndex
MP4SampleIndex::UpdateMoofIndex
MoofParser::RebuildFragmentedIndex
Moof::Moof one call for each moof box
Moof::ParseTraf
Moof::ParseTrun
```
`SampleIterator::GetNext` then makes a `MediaRawData` from each sample:
| `MediaRawData` field | Value from the sample |
| -------------------- | ------------------------------ |
| `mTime` | start of `mCompositionRange` |
| `mDuration` | length of `mCompositionRange` |
| `mTimecode` | `mDecodeTime` |
| `mKeyframe` | `mSync` |
For MSE, `TrackBuffersManager::CreateDemuxerforMIMEType` creates an
`MP4Demuxer` on the appended data. The demuxer uses the same path. Then
`TrackBuffersManager::ProcessFrames` runs the MSE coded frame processing on
the samples.
## Step 1: read each trun entry
`Moof::ParseTraf` starts the decode time at the `tfdt` value. If the `traf`
box has no valid `tfdt`, it continues from the decode end of the previous
moof.
`Moof::ParseTrun` then reads one entry for each sample. For each entry, it
does these steps:
1. It sets the decode time (DTS) of the sample to the current decode time.
2. It sets the composition range to `[DTS + ctsOffset, DTS + ctsOffset +
duration)`.
3. It adds the entry duration to the current decode time.
The edit list moves both times by the same amount. The examples have no edit
list.
The `trun` duration is a decode delta: the time from this DTS to the next DTS.
It is not the time that the frame stays on the screen.
## Step 2: decode order and display order
The parser stores the samples in decode order. B-frames make the display order
different from the decode order. The `ctsOffset` of each entry moves its
composition time (PTS) away from its DTS.
For example, a 30 fps H.264 stream has two B-frames between its I- and
P-frames. Its timescale is 15360, so one frame is 512 ticks. The encoder puts
three samples in each moof. The table shows the first two moofs in decode
order.
| moof | type | DTS | PTS |
| ---- | ---- | ---- | ---- |
| 1 | I | 0 | 512 |
| 1 | P | 512 | 2048 |
| 1 | B | 1024 | 1024 |
| 2 | B | 1536 | 1536 |
| 2 | P | 2048 | 3584 |
| 2 | B | 2560 | 2560 |
In display order, moof 1 has the frames at 512, 1024 and 2048. The frame at
1536 is in moof 2. So moof 1 has a gap in its composition times, and the next
moof fills it.
```
PTS 512 1024 1536 2048 2560
|-----|-----|-----|-----|
moof 1 [ I ] [ B ] [ P ]
moof 2 [ B ] [ B ] ...
```
## Step 3: close the composition gaps
After `ParseTrun`, each sample has the composition range from its `trun`
duration. But the `trun` duration is a decode delta, so it can be different
from the display spacing. For example, a stream has a timescale of 90000, so
one frame is 3000 ticks. In decode order, its first `trun` duration is 3000
ticks. Then the durations alternate between 1 and 5999 ticks, and one of them
is 6149. But its composition times are about 3000 ticks apart. With the `trun`
durations only, the sample at 6000 has a duration of 1 tick, and a gap of one
frame follows it.
So `Moof::Moof` sorts the samples of the moof by composition start. Then it
sets the end of each sample to a new value.
### Why not end at the next start
A simple rule sets each end to the start of the next sample in the moof. This
closes every gap in the moof. But it also closes a gap that the next moof
fills. In the B-frame example, the frame at 1024 gets the end 2048, and it
covers the frame at 1536 in moof 2.
| PTS | `trun` end | simple rule end | bounded rule end |
| ---- | ---------- | --------------- | ---------------- |
| 512 | 1024 | 1024 | 1024 |
| 1024 | 1536 | 2048 | 1536 |
| 2048 | 2560 | 2560 | 2560 |
The video sink shows a frame until the clock passes its end. So the frame at
1024 stays on the screen too long. Then the sink drops the frames that start
inside its span.
### The bounded rule
`Moof::Moof` limits each end:
```
end = min(start of the next sample, start + longest duration in this moof)
```
The longest `trun` duration in the moof is the bound. In the B-frame example,
every duration is 512. So the frame at 1024 ends at 1536.
In the example with uneven durations, the longest duration is 6149. The sample
at 6000 has a duration of 1 tick, and the next sample starts at 9001. The gap
is smaller than the bound, so the sample ends at 9001.
### Fallbacks
The rule uses the start of the next sample as the end in two cases:
| Case | Why |
| ------------------------------------- | -------------------------------- |
| The longest duration is 0 | A zero bound gives empty samples |
| `start + longest` is too large | The sum is not a valid time |
### Known limit
One long sample raises the bound for all the samples in its moof. Suppose that
in the B-frame example, the frame at 1024 in moof 1 has a duration of 7680
ticks. The bound becomes 7680, so the gap does not limit the end, and in moof
1 alone the frame at 1024 ends at 2048 again. The next section shows how the
next moof ends it at 1536:
| PTS | end |
| ---- | ---- |
| 512 | 1024 |
| 2048 | 2560 |
| 1024 | 2048 |
### Ends across moofs
ISO BMFF stores no display duration. A sample duration is the step to the next
decode time (ISO/IEC 14496-12, 8.6.1.2), and a sample has a composition time
(8.6.1.3) but no composition duration; only the composition duration of the
last sample is called unclear (8.6.1.4). So a sample is shown until the next
composition time in the track, which can be in a later moof. For example, in
variable frame rate content, the next frame can come after a long display gap,
in the next moof.
Suppose that every sample duration is 512 ticks, and that after the frame at
2048, the next frame comes at 17920, in moof 2. Moof 1 alone gives the frame at
2048 its own duration, so it ends at 2560:
```
PTS 512 1024 1536 2048 2560 17920 18432
moof 1 [ I ] [ B ] [ P ]
moof 2 [ B ] [ P ]
```
When moof 2 is added, the next composition time is known, and the frame at
2048 is extended to it:
```
PTS 512 1024 1536 2048 17920 18432
moof 1 [ I ] [ B ] [ P ......................]
moof 2 [ B ] [ P ]
```
A sample can also come in between later. For example, with one frame per moof
and B-frames, the P-frame comes before the B-frames that are shown before it.
Moof 2 extends the I-frame to the P-frame, and then the B-frames of moofs 3 and
4 start inside the I-frame:
```
PTS 512 1024 1536 2048
moof 1 [ I .............]
moof 2 [ P ]
moof 3 [ B ]
moof 4 [ B ]
```
So when a new sample starts inside an earlier sample, and the earlier sample
ends at the start of the next earlier sample, that end came from the next
sample, and the earlier sample is ended at the new sample. The new samples are
then extended up to that next earlier sample, so the time stays covered:
```
PTS 512 1024 1536 2048
moof 1 [ I ]
moof 2 [ P ]
moof 3 [ B ]
moof 4 [ B ]
```
An earlier sample that ends at no sample start keeps its own duration. Only
samples next to the moof boundary in composition order can change. The search
covers up to 16 samples before the new moof, the largest decoded picture buffer
of H.264 and HEVC. A longer run of non-reference frames can put the next frame
further back, and then the search misses it.
## Step 4: rewrite the decode times
MSE coded frame processing checks each DTS. `TrackBuffersManager::ProcessFrames`
finds a discontinuity in two cases:
- The DTS is smaller than the last DTS.
- The DTS is more than 2 * `mLongestFrameDuration` after the last DTS.
After a discontinuity, the track needs a random access point. So
`ProcessFrames` drops each frame until the next keyframe.
The DTS from `ParseTrun` steps by the `trun` durations. But Step 3 changed the
durations, so the two can be different. So `Moof::Moof` rewrites the DTS of
each sample in the moof. It keeps the decode span of the moof, from the first
DTS to the decode end of the `trun` entries. It spreads that span over the
samples in proportion to their durations:
```
adjust = decode span / sum of the sample durations
DTS(n) = first DTS + (sum of the durations before sample n) * adjust
```
The last DTS is then never above the decode end of the `trun` entries. That
end is the start of the next moof only when the next `tfdt` matches the `trun`
durations.
### Normal case
In moof 1 of the B-frame example, the decode span is 3 * 512 = 1536. The sum
of the durations is also 1536. So `adjust` is 1.0, and the DTS values stay 0,
512 and 1024. The next moof starts at 1536.
### Uneven case
In the example with uneven durations, a moof has a decode span of 69150 ticks.
After Step 3, the sum of the durations is 72150 ticks. So `adjust` is
69150 / 72150, about 0.958. The first samples get these values:
| PTS | end | DTS |
| ---- | ---- | ---- |
| 0 | 3001 | 0 |
| 6000 | 9001 | 2876 |
| 3001 | 6000 | 5752 |
### Why the sum and not the composition span
The composition span of the moof could also be the denominator. But the span
includes the gaps that the next moof fills. In moof 1 of the B-frame example,
the span is 2560 - 512 = 2048. The sum of the durations is 1536.
| Denominator | Value | `adjust` | DTS values | Step into moof 2 |
| ---------------- | ----- | -------- | ------------ | ---------------- |
| Composition span | 2048 | 0.75 | 0, 384, 768 | 768 |
| Sum of durations | 1536 | 1.0 | 0, 512, 1024 | 512 |
With the span, the DTS steps do not follow the sample durations.
## Step 5: the moof time range
`Moof::mTimeRange` is the composition span of the moof: from the first
composition start to the last composition end. It includes the gaps that the
next moof fills. `Moof::FixRounding` moves its end to the start of the next
moof when the gap between them is small. The limit is 35 ms plus one tick for
each sample.
These functions use `mTimeRange`:
| Function | Use |
| ----------------------------------------------- | ----------------------- |
| `MP4SampleIndex::ConvertByteRangesToTimeRanges` | buffered ranges |
| `MP4SampleIndex::GetEvictionOffset` | data that it can remove |
| `MoofParser::GetCompositionRange` | time span of the data |