We recently closed the first half of 2019 and with that it is time to look back and do a quick summary of what the media team has achieved during this 6 months period.
Looking at some stats, we merged 87 Pull Requests, we opened 56 issues, we closed 42 issues and we welcomed 13 new amazing contributors to the media stack.
These are some of the selected A/V playback related H1 acomplishments
Media cache and improved seeking
We significally improved the seeking experience of audio and video files by implementing preloading and buffering support and a media cache.
Basic media controls
The UI is not perfect, among other things, because we still have no way to render a progress or volume bar properly, as that depends on the
input type="range"> layout, which so far is rendered as a simple text box instead of the usual slider with a thumb.
GStreamer backend for MagicLeap
Another great achievement by Xavier Claessens from Collabora has been the GStreamer backend for Magic Leap. The work is not completely done yet, but as you can see on the animation below, he already managed to paint a full screen video on the Magic Leap device.
Hardware accelerated decoding
One of the most wanted features that we have been working on for almost a year and that has recently landed is hardware accelerated decoding.
Thanks to the excellent and constant work from the Igalian Víctor Jáquez, Servo recently gained support for hardware-accelerated media playback, which means lower CPU usage, better battery life and better thermal behaviour, among other goodies.
We only have support on Linux and Android (EGL and Wayland) so far. Support for other platforms is on the roadmap.
The numbers we are getting are already pretty nice. You might not be able to see it clearly on the video, but the renderer CPU time for the non hardware accelerated playback is ~8ms, compared to the ~1ms of CPU time that we get with the accelerated version.
Improved web compatibility of our media elements implementation
We also got a bunch of other smaller features that significantly improved the web compatibility of our media elements.
- ferjm added support for the HTMLMediaElement
- swarnimarun implemented support for the HTMLMediaElement
- jackxbritton implemented the HTMLMediaElement
- Servo got the ability to mute and unmute as well as controlling the volume of audio and video playback thanks to stevesweetney and lucasfantacuci.
- sreeise implemented the AudioTrack, VideoTrack, AudioTrackList and VideoTrackList interfaces.
- georgeroman coded the required changes to allow changing the playback rate of audio and video files.
- georgeroman, again, implemented support for the HTMLMediaElement
- dlrobertson paved the way for timed text tracks support by implementing the basics of the TextTrack API and the HTMLTrackElement interface.
We also got a few additions on the WebAudio land.
- PurpleHairEngineer implemented the StereoPannerNode backend.
- collares implemented the DOM side of the ChannelSplitterNode.
- Akhilesh1996 implemented the AudioParam setValueCurveAtTime function.
- snarasi6 implemented the deprecated setPosition and setOrientation AudioListener methods.
Thanks to jdm’s and Manishearth’s work, Servo has now the foundations of a WebRTC implementation and it is able to perform a 2-way calling with audio and video playback coming from the getUserMedia API.
That’s not all folks! We have exciting plans for the second half of 2019.
On the A/V playback land, we want to:
- Focus on adding hardware accelerated playback on Windows and OSX.
- Add support for fullscreen playback.
- Add support for 360 video.
- Improve the existing media controls by, for instance, implementing a nicer layout for the
<input type="range">element, with a proper slider and a thumb, so we can have progress and volume bars.
For WebAudio there are plans to make some architectural improvements related to the timeline and the graph traversals.
We would also love to work on the MediaElementAudioSourceNode implementation.
For WebRTC, data channels are on the roadmap for the second half.
We currently support the playback of a single stream of audio and video simultaneously, so allowing the playback of multiple simulatenous streams of each type is also something that we would like to get during the following months.
Finally, we have been trying to get Youtube to work on Servo, but it turned out to be a difficult task because of non-media related issues (i.e. layout or web compatibility issues), so we decided to adjust the goal and focus on embedded Youtube support instead.