Unclaimed project
Are you a maintainer of rx-player? Claim this project to take control of your public changelog and roadmap.
Quick Links:
📖 API documentation
-
⏯ Demo
-
🎓
The v4.3.0 is finally here.
We took some time to release it compared with previous releases due to a lot of other subjects coming our way late last year.
We do have worked on a larger amount of features since the last release, yet many of them are thus still in a review phase for now, and we focused on stability and what we thought were the most awaited features for a (normal-sized!) v4.3.0, especially:
Handling of DASH thumbnail tracks.
To minimize the complexity on your side, our thumbnail-related API takes care of every details about thumbnail fetching, caching and rendering, all you'll have to do is providing a timestamp for the wanted thumbnail and the DOM element on which you want to display the corresponding thumbnail.
We now automatically handle the ManagedMediaSource API (the MSE variant found on iOS devices), along with its startstreaming / endstreaming events, minimizing battery usage on those devices.
We added a new "Representation avoidance" algorithm. The idea is to automatically detect audio and video contents that lead to unexpected decoding issues on the current device and, if detected, to avoid playing them. This follows real issues we had seen on some WebOS and TitanOS devices (respectively mostly LG Smart TVs and mostly Philips Smart TVs).
For now this new behavior is behind an option - though it might become a default feature in the future as explained below.
We added a keySystems[].wantedSessionTypes loadVideo option for very specific DRM-related optimizations. The reasons why applications could need this are somewhat complex and are explained below.
We realized that multiple newer devices (at least the Meta Quest 2 and some other embedded devices) had issues decoding Dolby Digital+ only when it was encrypted.
We had a small issue in our logic detecting codec support in an encrypted context (added in v4.2.0) that made this specific case not handled. Now we should properly avoid any audio or video codec that is unsupported when encrypted.
We fixed an issue preventing playback of some contents with h265 video, for now only seen on the Edge browser if specific updates for H265 support were manually installed and if the RxPlayer was running in "multithreading" mode (which seems rare, but it does happen, especially for some techy customers!).
The root issue may be an Edge Browser bug, but it did reveal an area where we could improve our resilience. We found the components of that issue to be interesting so we added more information on it in this release not!
We brought several improvements for Tizen devices, including fixing two issues that led to infinite rebuffering in specific scenarios.
Tizen devices were known to have a worse experience in general due to the way it implements some media-related aspects from the HTML standard.
We improved multiple minor aspects of the MULTI_THREAD experimental feature, including some bug fixes and performance improvements.
We also brought improvements on some of the RxPlayer's sister projects: most notably, RxPaired (our RxPlayer-specialized remote inspector) is now able to fallback to HTTP POST on devices where WebSocket are not available and README (our documentation generator - which generates the RxPlayer API documentation) has now some improvements linked to search, mainly tree-organized results and a different client-side search library,
ManagedMediaSource on iOS devices [#1562]renderThumbnail and getAvailableThumbnailTracks API [#1496]keySystems[].wantedSessionTypes loadVideo option to also initialize a DRM config for future contents, not just the current one [#1604]experimentalOptions.enableRepresentationAvoidance option to loadVideo to enable our new Representation avoidance mechanism [#1523]onmessageerror being undefined on older devices [#1585]MediaKeys reusage on some devices including desktop browsers [#1615]"persistent-license" MediaKeySession (and not also for "temporary" license) when only a keySystems[].persistentLicenseConfig is communicated [#1604]keySystems[] option changing when reusing a MediaKeySystemAccess with a different keySystems[] configuration [#1616]KEY_UPDATE_ERROR which was mistakenly inheriting the code KEY_LOAD_ERROR [#1670]newAvailablePeriods event [#1643]MediaKeySystemAccess if none is more wanted for the current content [#1591]LOADING and RELOADING attempts may have taken more time than necessary due to a wrong "initial Period prediction", this is fixed [#1628]FREEZING work-arounds by reloading if our initial strategies do not give a result [#1523]hdr information to video Representation [#1583]full on RxPlayer's debug mode [#1625]BufferGarbageCollector buffer removal [#1684]The DASH specifications propose a way to declare a thumbnail track. Thumbnails are pictures representing different timecode of the video content, most often, they are used as preview thumbnails for seeking.
Screenshot: You can see in that screenshot a content being played with a thumbnail acting as a seeking preview on top of a mouse pointer (the thumbnail show a part of the content with a squirrel on screen).
Such "DASH thumbnails" is only one of the many ways to declare thumbnails linked to a video. Historically it was even not the preferred way: for example for VoD contents, an external archive following the BIF format was the most popular way as far as we know.
Schema: Crude representation of a bif file: some metadata at the start then all images concatenated.
However BIF files have a key issue: as an archive containing all images in advance, it doesn't work for live contents as future images are there only generated progressively. This is a problem that DASH thumbnails don't have (because they follow roughly the same syntax than for the linked live video data), and we've recently seen more and more applications generating those, even for VoD contents.
Screenshot: How DASH thumbnails are declared in a DASH MPD. We've here two different sizes possible depending on the thumbnail quality we want.
In the RxPlayer, we ignored those thumbnail tracks, until this release.
The solution we now propose for them tries to be as complete and simple-to-use as possible.
renderThumbnailThe main API to display image thumbnails is the new renderThumbnail method:
rxPlayer
.renderThumbnail({
container: containerElement,
time: 104,
})
.then(() => {
console.log("The thumbnail is now displayed");
});
As you can see in this code example, the idea is to give it just a "container element", which is an HTMLElement (e.g. a <div> tag) you had placed on the screen, and a time for which you want to display the thumbnail, and the RxPlayer will download and render the right thumbnail inside the given HTMLElement.
The renderThumbnail method returns a Promise and has many optimizations and other options. You can get more information in its API documentation.
getAvailableThumbnailTracks methodWe also added a new [getAvailableThumbnailTracks method](https://developers.canal-plus.com/rx-player/doc/api/Thumbnails/getAvailableThumbnailTracks.html], so you can both know whether there is a thumbnail track linked to the current content, but also to allow you to provide multiple thumbnail tracks, e.g. with multiple thumbnail qualities and/or formats, if you have the need to.
Historically, devices running on the iOS operating system (which is mainly used in iPhones and iPads) couldn't depend on the RxPlayer for contents other than in the directfile mode.
This was because the MSE API - the web API allowing to handle media buffering through JavaScript - was not supported by the browsers on those devices.
However Apple, which develops iOS, announced in 2023 that they would add not exactly the MSE API we're used to on iOS browsers on iOS, but nonetheless add a derivative API capable of roughly the same things: the "Managed Media Source" API.
Screenshot: Apple's proposal for the ManagedMediaSource API to the W3C, an organization handling web-related standards.
The idea behind the ManagedMediaSource API was to tweak some details from the historical MSE API so the browser itself can better control the experience:
The browser can indicate to the player that is supposed to stop loading new media data at any given point.
For example, if Safari decides that the RxPlayer has buffered enough data, it can send us an event to indicate that they want us to stop loading new data for some time. As far as we understand it, this mechanism is mainly here to preserve battery life on portable devices by e.g. controlling browser-side when the network can be relied on.
A ManagedMediaSource implementation might also clean-up data in media buffers at any time, while the regular API we were used to could only do so at some key points (like when appending new data). This is actually a welcome change for us, as the browser always had more information on memory usage than the RxPlayer. Knowing when the device's memory might be full has always been a challenge to MSE player developers.
Today, only safari supports that API:
Screenshot: Result of caniuse.com when asking for the support of this new
ManagedMediaSource API
We added support for this API in our v4.3.0 and thus should now support iOS for "dash", "smooth" and all other transports. For browsers that have both the "regular" MSE API and the ManagedMediaSource one (so Safari 17 on macOS - most non-mobile apple devices), the usual MSE API is for now still relied on.
As media playback involves a lot of moving parts (pun intended), we sometimes see playback issues that are only encountered on a particular device and content combination.
Those issues are in the great majority of time linked to issues with the device.
We've recently seen decoding issues on two separate smart TV models and brands where playback could completely stall on some contents, even with available media data in the right lower-level buffers. Those problems - where playback stay stuck in place despite media data being loaded - are what we call "freezing" cases. In the scenario we're speaking of, an interesting characteristic of those cases were that they always happened when a specific media quality began playing.
Video: Playback completely freezes when switching to another media quality. We seem to stay stuck, despite having data in our buffer.
Though we most often report the issue to the device vendors when we encounter those issues, they're often not very responsive on those bug reports. Moreover, we only ever checked on the [small set of] devices we have access to. On the great majority of devices we didn't test on, there's a chance that some of them also have decoding issues on specific video codecs/profiles.
So we decided to work on some logic that would identify those stalling situations, detect if they have a high probability of being linked to a specific video or audio quality, and if it does to just avoid playing that quality for the remaining of the content.
Video: Solution where the RxPlayer detects that the freeze is linked to a video quality switch (here 2000kbps). After some time, it reloads without that video quality (so we fallback here to 1700kbps instead). The detection takes a lot of time to be sure we're not wrongly considering false positives. PS: In this example, our video tag resizes when reloading, but that's just because we're very permissive in our demo page with the element's dimension, your application probably lock in place those dimensions in which case you won't have that effect while reloading.
After presenting the idea internally at Canal+, we saw that some people are a little afraid by those heuristics - especially due to the fact that they are based on an approximation: we here infer that an issue is linked to a specific audio/video quality, it's not based on actual explicit data.
As we understand this fear, this new behavior is for now behind a loadVideo option.
This option is for now marked as "experimental" as the API might change in the near future: the feature might become default (removing the need for that option), or be removed depending on results seen on Canal+ applications:
rxPlayer.loadVideo({
// ...
experimentalOptions: {
// Enable this "Representation avoidance mechanism"
enableRepresentationAvoidance: true,
},
});
Though for the reluctant rest assured that:
We spent a lot of time ensuring that false positives (i.e. the RxPlayer wrongly thinking that it should avoid a quality) almost never happen, or that when they do, it is because there's another, worse, problem preventing smooth content playback anyway.
It is not that big of a change or new territory: Most MSE players (including the RxPlayer) have plenty of other "approximate" heuristics for e.g. adaptive quality selection, gap/discontinuity jumping, handling of errors due to high memory usage of buffers, "freeze" management etc.
Avoiding qualities that the RxPlayer suspects of creating issues will probably be even less noticeable than the effects that each of those other features already had.
Depending on results we see on Canal+' own applications we thus may make this a default-enabled feature if we see clear improvements with no downside. For now, we hear and agree that there's more wisdom in only enabling this for applications that explicitely ask for the behavior through the experimentalOptions.enableRepresentationAvoidance loadVideo option.
"HEVC", also frequently referred to as "h265" is a video compression standard aiming to be more efficient in terms of compression than the ubiquitous "AVC" / "h264" standard most people still rely on today.
Images: Some very anecdotal exemple to illustrate the idea. We see that both images have roughly the same quality here, but the left one is taken from a video encoded with AVC and the right with HEVC - the left one "weighs" 800 kilobits per seconds yet the right one only 400. As such HEVC is more efficient: it can transmit the same video quality with less data / time.
Though it should be noted that this might be an extreme example (source of that image), video compression is a very complex subject and cannot be generalized easily, this was just to illustrate the idea.
It becomes more and more popular for high quality content such as ultra high definition or HDR ("High Dynamic Range" - contents able to provide higher luminosity contrasts) transmitted through the internet - where the high compression ratio HEVC provides is a real advantage.
Though HEVC has several issues: notorious patents-linked ones but also compatibility issues which is what is of interest here: most web browsers today, especially on PC/Mac devices (e.g. smart TVs don't have this issue) do not support it.
There's actually a possibility to be able to play HEVC content on a PC with some efforts.
Microsoft proposes two different applications (confusingly) which may add HEVC support to the Edge browser:
I wrote "may" add support because the rules in which it will lead to HEVC support are confusing and change over time. If you want to know more about the conditions , see the excellent rundown from Ted Mittelstaedt in a "Microsoft answers" page.
So the conclusion is that you may after some head scratching (and perhaps after some 99 cents-paying) end up with HEVC support on Edge and thus be able to play it through the RxPlayer (which auto-detects support) automatically.
...Unless you played in "multithreading" mode.
The "multithreading" mode is an optional (and opt-in) way to load your media contents so we can rely on a "WebWorker" - a web browser's mechanism allowing to parallelize some JavaScript logic with your application. We introduced this feature mainly to improve performance and quality of experience (especially to have a more stable, and hopefully higher, media quality).
schema: crude schema of how the multithreading mode works
The "multithreading" mode is internally complex, in part because some browsers might or might not support some features when running inside a WebWorker. What is especially of interest here is the "MediaSource Extensions" (commonly called just "MSE") API: those are the API allowing to buffer media content. Being able to rely on those API inside the WebWorker brings a lot of advantages. For example, even if your applications perform heavy operations for a long time, media playback will probably be able to continue without stuttering.
MSE inside a WebWorker is mostly only supported by Edge and Chrome today - more exactly only blink does. For other browsers, e.g. Firefox, we are forced to always ask our code running in the "main thread" to call the corresponding API, this is the same context your application runs in so we lose some advantages.
So all that to say that on Edge, in "multithreading" mode, we call most "MSE" API inside the WebWorker.
What's the relation between all that stuff and HEVC you said (probably before adding a "and please just get to the point already" that I will [voluntarily] choose to ignore)?
Well, the RxPlayer relies on an MSE API to know if HEVC is supported on the current device: the MediaSource.isTypeSupported API.
This API was until now always called in main thread, even if MSE could be used inside the WebWorker. This is in opposition with most other MSE API, like those to "push" new video data, which are called inside the WebWorker when we can, as those other API are much more performance-sensitive.
So to briefly summarize: on Edge (and Chrome) in "multithreading" mode, we were asking for HEVC support in the main thread, but pushing HEVC data inside the WebWorker.
Schema: Represents which context does what to check support when loading a content with HEVC video when MSE is available in a WebWorker. The schema is very simplified. For example, the round-trip used to perform the
MediaSource.isTypeSupported check actually transmits more data (more importantly: it indicates the initial tracks to play, an API-side matter) and as such would still be needed even if we didn't need the isTypeSupported call.
We did this for simplicity reasons and didn't think that this was an issue.
But this was before we got reports from people, who were courageous and tech-savy enough to have HEVC support in their Edge browser, having the following error when trying to play contents which have HEVC video available: Failed to execute "addSourceBuffer" on "MediaSource": The type provided "video/mp4; hevc" is unsupported.
Uh oh.
It turns out that HEVC was here supported on Edge when MSE was used in main thread, but not when those same API were used inside our WebWorker. The MediaSource.isTypeSupported API even works as expected here: in the main thread it tells us true but calling the same API inside the WebWorker tells us... <spoiler>false</spoiler>.
We think (and hope) that this is an issue with Edge - as we don't understand why the WebWorker would be more limited here, but it reveals a part where we could be more resilient and logical: we should check for codec support in the same context where we will actually communicate the corresponding data.
So all that to say that's exactly what we did in this release.
Schema: we ended up also calling
MediaSource.isTypeSupported in the WebWorker when possible and relying on that to infer support. Though again this is a very simplified schema and there's technically still a round-trip to the main thread for other things.
Now the Edge case is ""fixed"" as... we do not try to play HEVC in multithreading mode anymore, even if it is supported in main thread. We chose this resolution also because we expect the non-support here to be an Edge issue for what's for now a relatively-rare use case.
keySystems[].wantedSessionTypesThe 4.3.0 release also includes a new option for the keySystems loadVideo option: wantedSessionTypes.
The reason why you might need this new option is to enable some very specific optimizations, explained here.
MediaKeySession cacheThe RxPlayer maintains a "cache" of MediaKeySession objects, a decryption-related abstraction. The idea behind that cache is preventing license requests to a license server (to obtain decryption keys) if the content was already loaded recently (for example for a customer switching between the same channels). This is totally transparent for applications: they profit from our cache without having to configure anything.
The license server is one of the main platform to be sensible to audience spikes for several reasons, so at Canal+ we try to be as optimal as possible on that point: if we can find a way to avoid contacting the license server for some contents, we do it.
Schema: To read chronologically left-to-right. The idea is here that we're switching between two encrypted live channels. The second time we go watch "channel 1", the license is already linked to a MediaKeySession we have cached earlier, so we don't need to ask again for it.
But the "cache" is only kept if the chosen decryption configuration keeps being the same between contents. If we have to change some decryption characteristic (e.g. relying on a different distinctiveIdentifier configuration, different videoCapabilities etc.), we are forced to let go of the previous cache (for the curious readers, the reason behind this is because we rely on restricted browser API to construct that cache. Changing the configuration forces us to reset some of those API).
Schema: Here we change the DRM configuration the second time we play "channel 1". This leads to a cache reset and thus we're forced to perform the license request again.
keySystems optionMoreover, decryption-related configurations wanted by an application ends up being so varied that we progressively ended up providing more and more powerful decryption APIs.
One of the more concrete example of this in our v4.0.0 was our new videoCapabilitiesConfig option, replacing the less capable but simpler videoRobustness option.
Thanks to those new API, applications that needed our cache would specify a complex configuration with the goal of being compatible to all contents the applications planned to play, and not just the current content.
Schema: Applications wanting to exploit the cache at maximum often define their
keySystems option not only for the current content, but for any content they plan to be able to play.
sessionTypesYet, one configuration was missing: the ability to declare a general sessionTypes option.
sessionTypes mainly allows to enable "persistent MediaKeySession", storing decryption keys on the device and persisting them even after restarting the browser.
As such, it can be seen as a longer-term version of the MediaKeySession cache.
Image: Comparison of "persisted" MediaKeySession said and those in our "cache".
Regarding the sessionTypes configuration, previous RxPlayer versions were basically all or nothing:
persistentLicenseConfig option in which case you want persistent sessions for the current content and the RxPlayer will set the corresponding sessionTypes configuration to persistent-licensesessionTypes to its other value: temporary.But this system is not compatible with our caching logic if an application encounters both kinds of contents: contents for which persistent sessions are wanted, other contents for which it is not wanted (it may seem far-fetched, but it does happen, for example due to right-holders contract obligations)? In that case, the fact that the configuration changes in-between those contents completely resets our cache (which would have been especially useful for the contents which do not have persistence).
So mainly for that case, and also perhaps for other cases we don't have in mind yet, we now propose to expose the wantedSessionTypes configuration in our API. It will then be directly used to set the sessionTypes property of the corresponding MediaKeySystemConfiguration.
The idea is that if you know in advance that you might need a configuration handling both type of sessions, you can set it to:
wantedSessionTypes: ["persistent-license", "temporary"]
In any other cases, you probably don't need that new option.
persistentLicenseConfig and wantedSessionTypesIt should be noted that wantedSessionTypes, as a more explicit configuration, takes precedence over what is implied from persistentLicenseConfig:
persistentLicenseConfig is set but wantedSessionTypes doesn't have persistent-license, we won't ask for the support of persistent sessions (we will follow wantedSessionTypes)persistentLicenseConfig is not set but wantedSessionTypes does have persistent-license, we will ask for the support of persistent sessions, even if we won't be able to persist the sessions for the current content.