video codecs I've gotten working in HLS playback on iPadOS 15 using fragmented MP4 container:
* VP9 video
* H.264 video
codecs I can't get to work even though it would save me some effort:
* VP8 video
* MPEG-4 Visual
* MPEG-2 Visual
* MPEG-1 Visual
The great thing about VP9 being that it "just works" once set up correctly. The bad thing being it doesn't work on desktop. The ok thing being that on desktop, you should be able to play it using MSE.
A bad thing being that on an iPhone, MSE won't know it can't play the codecs and so won't fail over automatically to another source. A good thing being you get an error raised from the video element, and can fail over manually
This raises the possibility of creating fallback H.264 compatibility content, but that brings us back to the patents/formats/open/closed minefield that pushed us away in the first place.
Worst case: minimal "bad" encoder that avoids using advanced capabilities and doesn't worry too much about bitrate, but gets bits on screen. :)
However doing that right means checking with lawyers a lot.
And it doesn't resolve the open issue -- we didn't get a mandate on creating compatibility files in non-open formats.
But for compatibility output? Probably fine? But the Wikimedia MP4 video decision closed with a vote *against* making compatibility output, and I feel like I'm still bound by that unless we can reframe & restart some discussion.
And that's not something I can do as just one guy, I need to build org buy-in and that takes work and it's not gonna be any time soon.
Note that HLS is now an RFC so is an open standard, though the ISO BMFF aka fragmented MP4 container is itself one of the dreaded ISO standards, so to actually push any data through your playlists you have to go into non-free-standards land.
Nobody cares about any patents related to it, though, and it's widely implemented in open source software and is well understood. But.... sigh.
But if it's ok to use MP4, why not h.264 if we avoid the scary bits? (other than... lawyering)
When it comes to H.264 license terms, I *think* we'd be able to get away gratis because everything we'd be encoding/distributing would be 'free internet broadcasting', but we'd still have to get the license in theory.
Does it restrict downstream users? Maybe? But if it does, then that makes all h.264 conversions of free files suspect and *shrug* what you gonna do. The original .webm or .ogv or .mpg is right there if you want it.
Regarding containers: if I don't care about reaching iPhone/iPad with the switchable-resolution streaming I can use webm/VP9 & webm/Opus & mp4/MP3 tracks and make a single medium-res mp4/H.264+MP3 flat file.
This restricts mp4 container to 'Apple compatibility' usage (the audio for Safari desktop, which doesn't yet grok Opus but can read the WebM VP9 tracks; and the flat file for iOS and piping out in metadata to social media that can't use an iframe player reference)
The most "seamless" option is using MP4 container so iOS "just works":
* VP9-in-MP4 video in all resolutions
* H.264-in-MP4 fallback, perhaps limited
* Opus-in-MP4 audio
* MP3-in-MP4 fallback
Use MSE to play HLS on modern desktops.
Flat file fallbacks:
* WebM VP8/Vorbis
* MP4 H.264/MP3
Use the last for linking to random things that want an .mp4 in their link graphs.
Open questions: is it really ok to use h.264, and should we do every resolution or just a few?
@brion I've lost track, do we know when these patents expire? Simply running out the clock worked for MP3 support...
@legoktm H.264 is some years out (2027 is a number that popped up googling); older codecs aren't consistently supported enough to be worth waiting smaller amounts of time.
It's unclear to me whether AAC-LC (for audio) is clear now; Fedora's shipping a decoder but I don't know if encoding is a separate issue. ;) Hopefully I can get MP3 working consistently here for the audio, since we consider it clean already.
@brion Such standards are painfully common. It's like standards publishing has become a cash cow like music publishing.
This Mastodon instance is for people interested in technology. Discussions aren't limited to technology, because tech folks shouldn't be limited to technology either!