Skip to content

Add SUPPLEMENTAL-CODECS support in both DASH and HLS #1785

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jan 7, 2025
Merged

Add SUPPLEMENTAL-CODECS support in both DASH and HLS #1785

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
b650119
Merge pull request #10 from androidx/main
ybai001 Sep 20, 2024
79078c3
Add SUPPLEMENTAL-CODECS support in both DASH and HLS
ybai001 Oct 8, 2024
c18c6b3
Remove the modification on Format
ybai001 Oct 14, 2024
e8d29dc
Remove redundant blank line
ybai001 Oct 14, 2024
545d37c
Fix two bugs in the updated code
ybai001 Oct 14, 2024
1ac847a
Update code according to review result
ybai001 Dec 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions libraries/common/src/main/java/androidx/media3/common/Format.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1190,6 +1190,9 @@ public Format withManifestFormatInfo(Format manifestFormat) {
codecs = codecsOfType;
}
}
if (MimeTypes.VIDEO_DOLBY_VISION.equals(sampleMimeType)) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you explain why we need this special DV check here? It looks like the check above (L1185-1192) already achieves the same goal unless there are multiple video codecs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The purpose of this special check is to update codecs with manifest codec info. For my test stream,
manifestFormat.codecs = "dvhe.08.01" while codecs = "hev1.08.01" before this special handling. (So that L1185 is false and corresponding logic is not touched.)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. Does this happen because the extractor doesn't know this is a DV format? For MP4 containers for example I'd assume they have a dvcC or dvvC, which should results in the format.codecs already populated with the expected values.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

During the fmp4 paring, the parser finds the 'dvcc' or 'dvvc' box here, and corresponding codec string is updated in DolbyVisionConfig::parse(). After tracking the history of this file, I finally found this one. The author was Dolby engineer but is Google engineer now. :)

I'm not sure whether there is any side effect on codec selection if I change the logic (e.g. from hev1 to dvhe). My understanding is OK since there is a method to handle the case (A non Dolby Vision licensed device to play Dolby Vision content.) But I didn't do a full testing.

Do you want to change the logic in DolbyVisionConfig::parse()? or using a new pull request to do that? At least, I need to provide a new PR for profile 10 handling in that method.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we assume the dvcc' or 'dvvc' box is always there and we should always be able to tell from the actual media container which Dolby Vision codec we have? If so, I think it's better to fix the logic in DolbyVisionConfig::parse() and not make the change in Format.withManifestFormatInfo.

The reason for that is that withManifestFormatInfo merges information from the actual media files and the manifest. In some cases like labels and language it's generally better to rely on the manifest. In other cases like codecs, it's usually better to rely on the actual media file as it's more likely to get it right (or has more details) than the manifest in case there are any differences. So if we can be certain that the dvcc' or 'dvvc' boxes always exist for a Dolby Vision stream (and we fix our util method to set the right codec strings for all profiles), then it seems better to rely on this logic.

or using a new pull request to do that?

That seems like a separate change and I think it's easier if I continue to merge the current pull request as it is and we can look at the other change separately?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we assume the dvcc' or 'dvvc' box is always there and we should always be able to tell from the actual media container which Dolby Vision codec we have? If so, I think it's better to fix the logic in DolbyVisionConfig::parse() and not make the change in Format.withManifestFormatInfo.

Agree. I think there is always a Dolby Vision box (dvcc or dvvc) to indicate the codec info. I need to double confirm it.

That seems like a separate change and I think it's easier if I continue to merge the current pull request as it is and we can look at the other change separately?

I'll submit a new pull request after this one is merged. Is it fine for you?

Copy link
Contributor Author

@ybai001 ybai001 Dec 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the latest spec today.

• For profiles less than or equal to 7: dvcC
• For profiles 8 to 10: dvvC
• For profile 11 to 19: dvwC
• For profile 20: dvcC
• For profiles greater than 20: dvwC

Currently, we just need to support profile 4/5/7/8/9/10 and potentially support profile 20 so that there is always a 'dvcC' or 'dvvC' box exist.

codecs = Util.getCodecsOfType(manifestFormat.codecs, C.TRACK_TYPE_VIDEO);
}

@Nullable
Metadata metadata =
Expand Down
28 changes: 28 additions & 0 deletions libraries/common/src/main/java/androidx/media3/common/util/Util.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2115,6 +2115,34 @@ public static String getCodecsOfType(@Nullable String codecs, @C.TrackType int t
return builder.length() > 0 ? builder.toString() : null;
}

/**
* Returns a copy of {@code codecs} without the codecs whose track type matches {@code
* trackType}.
*
* @param codecs A codec sequence string, as defined in RFC 6381.
* @param trackType The {@link C.TrackType track type}.
* @return A copy of {@code codecs} without the codecs whose track type matches {@code
* trackType}. If this ends up empty, or {@code codecs} is null, returns null.
*/
@UnstableApi
@Nullable
public static String getCodecsWithoutType(@Nullable String codecs, @C.TrackType int trackType) {
String[] codecArray = splitCodecs(codecs);
if (codecArray.length == 0) {
return null;
}
StringBuilder builder = new StringBuilder();
for (String codec : codecArray) {
if (trackType != MimeTypes.getTrackTypeOfCodec(codec)) {
if (builder.length() > 0) {
builder.append(",");
}
builder.append(codec);
}
}
return builder.length() > 0 ? builder.toString() : null;
}

/**
* Splits a codecs sequence string, as defined in RFC 6381, into individual codec strings.
*
Expand Down
39 changes: 39 additions & 0 deletions ...player_dash/src/main/java/androidx/media3/exoplayer/dash/manifest/DashManifestParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -399,6 +399,8 @@ protected AdaptationSet parseAdaptationSet(

String mimeType = xpp.getAttributeValue(null, "mimeType");
String codecs = xpp.getAttributeValue(null, "codecs");
String supplementalCodecs = xpp.getAttributeValue(null, "scte214:supplementalCodecs");
String supplementalProfiles = xpp.getAttributeValue(null, "scte214:supplementalProfiles");
int width = parseInt(xpp, "width", Format.NO_VALUE);
int height = parseInt(xpp, "height", Format.NO_VALUE);
float frameRate = parseFrameRate(xpp, Format.NO_VALUE);
Expand Down Expand Up @@ -455,6 +457,8 @@ protected AdaptationSet parseAdaptationSet(
!baseUrls.isEmpty() ? baseUrls : parentBaseUrls,
mimeType,
codecs,
supplementalCodecs,
supplementalProfiles,
width,
height,
frameRate,
Expand Down Expand Up @@ -673,6 +677,8 @@ protected RepresentationInfo parseRepresentation(
List<BaseUrl> parentBaseUrls,
@Nullable String adaptationSetMimeType,
@Nullable String adaptationSetCodecs,
@Nullable String adaptationSetSupplementalCodecs,
@Nullable String adaptationSetSupplementalProfiles,
int adaptationSetWidth,
int adaptationSetHeight,
float adaptationSetFrameRate,
Expand All @@ -696,6 +702,10 @@ protected RepresentationInfo parseRepresentation(

String mimeType = parseString(xpp, "mimeType", adaptationSetMimeType);
String codecs = parseString(xpp, "codecs", adaptationSetCodecs);
String supplementalCodecs =
parseString(xpp, "scte214:supplementalCodecs", adaptationSetSupplementalCodecs);
String supplementalProfiles =
parseString(xpp, "scte214:supplementalProfiles", adaptationSetSupplementalProfiles);
int width = parseInt(xpp, "width", adaptationSetWidth);
int height = parseInt(xpp, "height", adaptationSetHeight);
float frameRate = parseFrameRate(xpp, adaptationSetFrameRate);
Expand Down Expand Up @@ -781,6 +791,8 @@ protected RepresentationInfo parseRepresentation(
adaptationSetRoleDescriptors,
adaptationSetAccessibilityDescriptors,
codecs,
supplementalCodecs,
supplementalProfiles,
essentialProperties,
supplementalProperties);
segmentBase = segmentBase != null ? segmentBase : new SingleSegmentBase();
Expand All @@ -797,6 +809,27 @@ protected RepresentationInfo parseRepresentation(
Representation.REVISION_ID_DEFAULT);
}

protected boolean isDolbyVisionFormat(
@Nullable String codecs, @Nullable String supplementalCodecs) {
if (codecs == null) {
return false;
}
if (codecs.startsWith("dvhe") || codecs.startsWith("dvh1")) {
// profile 5
return true;
}

if (supplementalCodecs == null) {
return false;
}

return (supplementalCodecs.startsWith("dvhe") && codecs.startsWith("hev1")) || // profile 8
(supplementalCodecs.startsWith("dvh1") && codecs.startsWith("hvc1")) || // profile 8
(supplementalCodecs.startsWith("dvav") && codecs.startsWith("avc3")) || // profile 9
(supplementalCodecs.startsWith("dva1") && codecs.startsWith("avc1")) || // profile 9
(supplementalCodecs.startsWith("dav1") && codecs.startsWith("av01")); // profile 10
}

protected Format buildFormat(
@Nullable String id,
@Nullable String containerMimeType,
Expand All @@ -810,6 +843,8 @@ protected Format buildFormat(
List<Descriptor> roleDescriptors,
List<Descriptor> accessibilityDescriptors,
@Nullable String codecs,
@Nullable String supplementalCodecs,
@Nullable String supplementalProfiles,
List<Descriptor> essentialProperties,
List<Descriptor> supplementalProperties) {
@Nullable String sampleMimeType = getSampleMimeType(containerMimeType, codecs);
Expand All @@ -819,6 +854,10 @@ protected Format buildFormat(
codecs = MimeTypes.CODEC_E_AC3_JOC;
}
}
if (isDolbyVisionFormat(codecs, supplementalCodecs)) {
sampleMimeType = MimeTypes.VIDEO_DOLBY_VISION;
codecs = supplementalCodecs != null ? supplementalCodecs : codecs;
}
@C.SelectionFlags int selectionFlags = parseSelectionFlagsFromRoleDescriptors(roleDescriptors);
@C.RoleFlags int roleFlags = parseRoleFlagsFromRoleDescriptors(roleDescriptors);
roleFlags |= parseRoleFlagsFromAccessibilityDescriptors(accessibilityDescriptors);
Expand Down
58 changes: 58 additions & 0 deletions ...exoplayer_hls/src/main/java/androidx/media3/exoplayer/hls/playlist/HlsPlaylistParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,10 @@ public static final class DeltaUpdateException extends IOException {}
private static final Pattern REGEX_CLOSED_CAPTIONS = Pattern.compile("CLOSED-CAPTIONS=\"(.+?)\"");
private static final Pattern REGEX_BANDWIDTH = Pattern.compile("[^-]BANDWIDTH=(\\d+)\\b");
private static final Pattern REGEX_CHANNELS = Pattern.compile("CHANNELS=\"(.+?)\"");
// VIDEO-RANGE attribute: https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis-15
private static final Pattern REGEX_VIDEO_RANGE = Pattern.compile("VIDEO-RANGE=(SDR|PQ|HLG)");
private static final Pattern REGEX_CODECS = Pattern.compile("CODECS=\"(.+?)\"");
private static final Pattern REGEX_SUPPLEMENTAL_CODECS = Pattern.compile("SUPPLEMENTAL-CODECS=\"(.+?)\"");
private static final Pattern REGEX_RESOLUTION = Pattern.compile("RESOLUTION=(\\d+x\\d+)");
private static final Pattern REGEX_FRAME_RATE = Pattern.compile("FRAME-RATE=([\\d\\.]+)\\b");
private static final Pattern REGEX_TARGET_DURATION =
Expand Down Expand Up @@ -323,6 +326,40 @@ private static int skipIgnorableWhitespace(BufferedReader reader, boolean skipLi
return c;
}

private static boolean isDolbyVisionFormat(
@Nullable String videoRange,
@Nullable String codecs,
@Nullable String supplementalCodecs,
@Nullable String supplementalProfiles) {
if (codecs == null) {
return false;
}
if (codecs.startsWith("dvhe") || codecs.startsWith("dvh1")) {
// profile 5
return true;
}

if (supplementalCodecs == null) {
return false;
}
// For Dolby Vision, the compatibility brand (i.e. supplemental profiles) and the VIDEO-RANGE
// attribute act as cross-checks. Leaving out either one is incorrect.
if (videoRange == null || supplementalProfiles == null) {
return false;
}
if ((videoRange.equals("PQ") && !supplementalProfiles.equals("db1p")) ||
(videoRange.equals("SDR") && !supplementalProfiles.equals("db2g")) ||
(videoRange.equals("HLG") && !supplementalProfiles.startsWith("db4"))) { // db4g or db4h
return false;
}

return (supplementalCodecs.startsWith("dvhe") && codecs.startsWith("hev1")) || // profile 8
(supplementalCodecs.startsWith("dvh1") && codecs.startsWith("hvc1")) || // profile 8
(supplementalCodecs.startsWith("dvav") && codecs.startsWith("avc3")) || // profile 9
(supplementalCodecs.startsWith("dva1") && codecs.startsWith("avc1")) || // profile 9
(supplementalCodecs.startsWith("dav1") && codecs.startsWith("av01")); // profile 10
}

private static HlsMultivariantPlaylist parseMultivariantPlaylist(
LineIterator iterator, String baseUri) throws IOException {
HashMap<Uri, ArrayList<VariantInfo>> urlToVariantInfos = new HashMap<>();
Expand Down Expand Up @@ -374,7 +411,28 @@ private static HlsMultivariantPlaylist parseMultivariantPlaylist(
int roleFlags = isIFrameOnlyVariant ? C.ROLE_FLAG_TRICK_PLAY : 0;
int peakBitrate = parseIntAttr(line, REGEX_BANDWIDTH);
int averageBitrate = parseOptionalIntAttr(line, REGEX_AVERAGE_BANDWIDTH, -1);
String videoRange = parseOptionalStringAttr(line, REGEX_VIDEO_RANGE, variableDefinitions);
String codecs = parseOptionalStringAttr(line, REGEX_CODECS, variableDefinitions);
String supplementalCodecsStrings =
parseOptionalStringAttr(line, REGEX_SUPPLEMENTAL_CODECS, variableDefinitions);
String supplementalCodecs = null;
String supplementalProfiles = null; // i.e. Compatibility brand
if (supplementalCodecsStrings != null) {
String[] supplementalCodecsString = Util.splitAtFirst(supplementalCodecsStrings, ",");
// TODO: Support more than one element
String[] codecsAndProfiles = Util.split(supplementalCodecsString[0], "/");
supplementalCodecs = codecsAndProfiles[0];
if (codecsAndProfiles.length > 1) {
supplementalProfiles = codecsAndProfiles[1];
}
}
String videoCodecs = Util.getCodecsOfType(codecs, C.TRACK_TYPE_VIDEO);
if (isDolbyVisionFormat(videoRange, videoCodecs, supplementalCodecs, supplementalProfiles)) {
videoCodecs = supplementalCodecs != null ? supplementalCodecs : videoCodecs;
String nonVideoCodecs = Util.getCodecsWithoutType(codecs, C.TRACK_TYPE_VIDEO);
codecs = nonVideoCodecs != null ? videoCodecs + "," + nonVideoCodecs : videoCodecs;
}

String resolutionString =
parseOptionalStringAttr(line, REGEX_RESOLUTION, variableDefinitions);
int width;
Expand Down