Centricular

What's new in GStreamer

GUADEC 2015, Göteborg

7 August 2015

Tim Müller <tim@centricular.com>Sebastian Dröge <sebastian@centricular.com>

Centricular

Introduction

Centricular

Who?

● Long-term GStreamer core developers and maintainers since 2005/6

● Founders of Centricular Ltd– Consultancy offering services around GStreamer,

graphics and multimedia related software

Centricular

What is this all about?

● What is GStreamer?● What happened in the last year?

Centricular

GStreamer? What is it?

Centricular

GStreamer

● Pipeline-based multimedia framework● Basic building blocks: elements connected via pads

● Set of libraries with abstract API● Plugins for specific features

– Often wrapping other libraries, e.g. libav/ffmpeg, OpenCV, various codec libraries

Centricular

GStreamer (cont'd)

● Open Source, LGPL v2.1+– Proprietary applications and plugins allowed

● Cross-platform– Linux, embedded Linux, Windows, OSX, iOS, Android, …

– Binary SDKs for Windows, OSX, iOS, Android

– x86, ARM, PowerPC, SPARC, …

● Stable GObject based C API/ABI● Bindings for many languages

– Python, C++, JavaScript, .NET, …

Centricular

GStreamer (cont'd)

● Not a media player or playback library● Not a codec and protocol library● Not a transcoding tool● Not a streaming server

● But can be (and is) used to implement all that

Centricular

GStreamer (cont'd)

● Goals– Flexible and extensible design

– Easy to integrate with other software(in both directions)

● Large, active developer and user community● Large commercial ecosystem

– Users

– Contributors

– Consultancies

Centricular

What happened inthe last year?

Centricular

Releases

● 1.4 released, latest bugfix release 1.4.5– As always with binaries for all non-Linux platforms

– 1.4.6 most likely to finish the release series

● 1.6 really soon now!– Join us after the core days for bug triaging,

bugfixing and getting everything release-ready

Centricular

Better UI toolkit integration

● GTK+ video sink / widget (>= 3.16 with OpenGL)– Works very well with CSS styling, including transparency

and overlays! No need for Clutter anymore

– See Matthew's talk today at 16:30 in Asperö

● Qt QML video sink / QML element (OpenGL)– X11 and Wayland so for

● Should be portable to all platforms, including iOS and Android

● OS X CoreAnimation video sink / CALayer (OpenGL)– Should be portable to iOS with some minor effort

Centricular

GTK+ video sink / widget details

● How does it work? You create a GStreamer video sink as usual– gtksink or gtkglsink

– gtkglsink should be placed into a glsinkbin (“sink” property) if no other GL operations should be done. glsinkbin should be used as the real video sink

– If using playbin, set the gtksink or glsinkbin on the “video-sink” property

● gtk(gl)sink has a “widget” property that gives you a GtkWidget● Place the widget somewhere in your UI and handle it like any

other widget● Anything received by the sink will be automatically rendered in

your UI into the widget

Centricular

GTK+ video sink / widget details

● Bonus: CSS styling, overlaying, transparency, etc. are supported

● Hardware decoders that can output GL things will be zerocopy (e.g. VAAPI)!

Centricular

GstPlayer

● High-level playback API started last GUADEC– https://github.com/sdroege/gst-player

– Planned to be merged into GStreamer soonish (for 1.8)

● For all your media playback needs– All (?) features needed by Totem are implemented

– Should reduce Totem's BaconVideoWidget code

by 90+% together with the GTK+ video sink

● Example GTK+, Android, iOS and command-line applications included– Qt QML application WIP

Centricular

GstPlayer

Centricular

Chrome / Chromium

● GStreamer backend released by Samsung R&D– https://github.com/Samsung/ChromiumGStreamerB

ackend● HTML5 video/audio tag support● Uses GstPlayer, less than 10k SLOC● Hardware accelerated OpenGL rendering● Planned to be merged into Chromium upstream

Centricular

WebKit

● Lots of improvements to the GStreamer backend

● OpenGL accelerated rendering● Media Source Extensions and Encrypted Media

Extensions (ugh!) support● Improved WebAudio support● WebRTC integration started with

OpenWebRTC

Centricular

WebRTC

● New W3C standard, basically “Skype for the browser”. It's going to be big!

● Kurento WebRTC media server– Includes “extractable” GStreamer element for integrating WebRTC sessions into a

pipeline: WebRTCEndPoint

● OpenWebRTC simple WebRTC API– Released by Ericsson R&D last year http://www.openwebrtc.io

● All requires applications integration with Web services, simple nowadays with the GNOME platform– libsoup now has WebSockets support

– json-glib FTW

● Lots of related improvements to GStreamer's RTP stack

Centricular

WebRTC

● Looking for an application to write?● Interested in writing a native GNOME

application for Firefox Hello but don't know how?

● Talk to me, it would be great showcase and fun to implement!

Centricular

pinos aka PulseVideo

● API & service for accessing multimedia streams– Cameras, other devices, streams, …

– Supports compressed streams, also audio

– Multiple users of the same device

● “Give access to your camera to your sandboxed application”– See Alexander Larsson's talk this morning at 10:00

● DBus API for control– fd passing between processes to share media data

Centricular

The 3rd Dimension

● OpenGL support had major improvements and refactoring– Easier integration into other libraries and custom

OpenGL code● GTK+! Qt! SDL! CoreAnimation! Chrome! WebKit!

– Wayland support

– See Matthew's talk today at 16:30 in Asperö

● Support for stereoscopic 3D video

Centricular

Quality Assurance

● Continuous integration with Jenkins– Builds for all supported platforms

– Unit / integration tests, gst-validate on Linux

● Major improvements and new tests for gst-validate– Also run by CI server

– Tests higher-level GStreamer scenarios and runs over a media file library

● GstHarness– Unit test helper for element behaviour tests, including (real-time) timing

● Moar unit tests

Centricular

Assorted

● RTSP server RECORD support to send media from a client to a server

● Live audio/video mixing support● iOS/OSX zerocopy decoding, Android WIP● Improved HTTP Adaptive Streaming support

(DASH, HLS)● splitmuxsink for writing standalone

MP4/WebM/etc in separate files

Centricular

Assorted

● New GStreamer Editing Services backend– No more GNonLin

– Expect PiTiVi to work even better

● Switch from libav back to ffmpeg● New trick-mode API flags: keyframe-only, no audio, etc.● Non-GL Wayland video sink● v4l2 improvements

– dmabuf and renegotiation support

Centricular

Bugzilla & GIT stats

● Bugzilla: +2385 / -1931 bugs, Δ +454 bugs– 1638 open bugs, 902 non-enhancement bugs

– Last year +2136 / -1823, Δ +313 bugs

– Ratio changed from 1.17 to 1.23

● We need to do something about that!● If you're interested in working on GStreamer,

now would be a good time to start. Many open tasks, many for every level of experience

Centricular

Bugzilla & GIT stats

● 220 contributors in the last 12 months, 50 with more than 12 commits in that period– +2% more contributors than in the previous year

● 6521 commits in the last 12 months– -11% compared to previous year

● 74269 new lines of code (+5%)– Excludes comments and whitespace

Centricular

The Future

Centricular

Development Tools

● Tracing subsystem, better debugging facilities● Live pipeline debugging application (GSoC project)

– See Marcin's talk today some time after 17:30 during the interns lightning talks

● Debug log viewer● Continuous integration

– Automated QA and builds

● More usage of new QA tools, more tests– gst-validate, GstHarness

Centricular

Development Resources

● More convenience APIs– Merge GstPlayer

– GstPlayer is there, what else do people need?

● Move to Phabricator– Bug tracking, code review, wiki

● New website? More tutorials?

Centricular

Features

● Improved trick mode support– Incl. HTTP adaptive streaming

● DASH/HLS streaming server– And generic file segmenting

● More cross-platform improvements– Windows support is lacking behind! Android

zerocopy decoding

Centricular

Features

● Decodebin3– better support for reconfiguration and dynamic

scenarios

● Better APIs and support for metadata extraction● Merge various 3rd party plugins, move plugins from

-bad to -good– Android camera source, Kurento WebRTC endpoint,

SCTP

– camerabin (cheese!)

Centricular

Thanks!

Any questions?

Find more information at http://gstreamer.freedesktop.org

Join us at the GStreamer Conference8-9 Oct 2015 – Dublin, Ireland

http://gstreamer.freedesktop.org/conference