video codecs I've gotten working in HLS playback on iPadOS 15 using fragmented MP4 container:
* VP9 video
* H.264 video

codecs I can't get to work even though it would save me some effort:
* VP8 video
* MPEG-4 Visual
* MPEG-2 Visual
* MPEG-1 Visual
* Motion-JPEG

· · Web · 1 · 1 · 1

The great thing about VP9 being that it "just works" once set up correctly. The bad thing being it doesn't work on desktop. The ok thing being that on desktop, you should be able to play it using MSE.

A bad thing being that on an iPhone, MSE won't know it can't play the codecs and so won't fail over automatically to another source. A good thing being you get an error raised from the video element, and can fail over manually

Another alternative is getting some flavor of H.264 into the HLS manifest as a fallback for iPhone. And indeed, it'd be *nice* to use H.264-plus-MP3-in-mp4 to have a low-res fallback that can be referenced in things like social media links (though better yet, supply an iframe player)

This raises the possibility of creating fallback H.264 compatibility content, but that brings us back to the patents/formats/open/closed minefield that pushed us away in the first place.

Worst case: minimal "bad" encoder that avoids using advanced capabilities and doesn't worry too much about bitrate, but gets bits on screen. :)

However doing that right means checking with lawyers a lot.

And it doesn't resolve the open issue -- we didn't get a mandate on creating compatibility files in non-open formats.

I really don't like this flavor of standards, though:

* it's an international standard, but the document itself is copyrighted, can't be shared, and costs a lot to order

That's kinda antithetical to open source even if the patents weren't a problem at all.

But for compatibility output? Probably fine? But the Wikimedia MP4 video decision closed with a vote *against* making compatibility output, and I feel like I'm still bound by that unless we can reframe & restart some discussion.

And that's not something I can do as just one guy, I need to build org buy-in and that takes work and it's not gonna be any time soon.

Best case on linking to videos on social media is if we can get an iframe player, then we can play our silly games with loading up HLS or running the HLS via MSE or giving up and running the WebM decode in WebAssembly and rendering to a canvas, and nobody will need to know which.

Note that HLS is now an RFC so is an open standard, though the ISO BMFF aka fragmented MP4 container is itself one of the dreaded ISO standards, so to actually push any data through your playlists you have to go into non-free-standards land.

Nobody cares about any patents related to it, though, and it's widely implemented in open source software and is well understood. But.... sigh.

But if it's ok to use MP4, why not h.264 if we avoid the scary bits? (other than... lawyering)

There is a second choice for container formats in HLS... which is MPEG-2 Transport Streams, also in ISO land. And I suspect VP9 won't squeeze into that anyway.

When it comes to H.264 license terms, I *think* we'd be able to get away gratis because everything we'd be encoding/distributing would be 'free internet broadcasting', but we'd still have to get the license in theory.

Does it restrict downstream users? Maybe? But if it does, then that makes all h.264 conversions of free files suspect and *shrug* what you gonna do. The original .webm or .ogv or .mpg is right there if you want it.

oh here's another fun one:

In regular playback in a <video>, VP8 in MP4 works in Chrome but not in Firefox

Regarding containers: if I don't care about reaching iPhone/iPad with the switchable-resolution streaming I can use webm/VP9 & webm/Opus & mp4/MP3 tracks and make a single medium-res mp4/H.264+MP3 flat file.

This restricts mp4 container to 'Apple compatibility' usage (the audio for Safari desktop, which doesn't yet grok Opus but can read the WebM VP9 tracks; and the flat file for iOS and piping out in metadata to social media that can't use an iframe player reference)

The most "seamless" option is using MP4 container so iOS "just works":

HLS manifest:
* VP9-in-MP4 video in all resolutions
* H.264-in-MP4 fallback, perhaps limited
* Opus-in-MP4 audio
* MP3-in-MP4 fallback
Use MSE to play HLS on modern desktops.
Flat file fallbacks:
* WebM VP8/Vorbis
* MP4 H.264/MP3

Use the last for linking to random things that want an .mp4 in their link graphs.

Open questions: is it really ok to use h.264, and should we do every resolution or just a few?

A more "pure" option:

DASH manifest:
* VP9-in-WebM video in all resolutions
* Opus-in-WebM audio
* Vorbis-in-WebM fallback for Safari 15

Flat file fallbacks:
* WebM VP8/Vorbis
* MP4 H.264/MP3

This leaves us with only a single "impure" file, without the resolution switching magic. Good? Bad? Who knows.

alternately if I don't do any H.264 or MP4, we can use ogv.js to decode the WebM VP8/Vorbis flat fallback on iPhone Safari, while getting VP9 via MSE/DASH on desktop and iPad with supported hardware codecs

@brion I've lost track, do we know when these patents expire? Simply running out the clock worked for MP3 support...

@legoktm H.264 is some years out (2027 is a number that popped up googling); older codecs aren't consistently supported enough to be worth waiting smaller amounts of time.

It's unclear to me whether AAC-LC (for audio) is clear now; Fedora's shipping a decoder but I don't know if encoding is a separate issue. ;) Hopefully I can get MP3 working consistently here for the audio, since we consider it clean already.

@brion Such standards are painfully common. It's like standards publishing has become a cash cow like music publishing.

Sign in to participate in the conversation
Mastodon for Tech Folks

This Mastodon instance is for people interested in technology. Discussions aren't limited to technology, because tech folks shouldn't be limited to technology either!