GStreamer T-111.5350 Multimedia programming

Mika Ristimäki, Jenny Lahti

History

The project was founded in 1999 by Erik Walthinsen.

Later it was joined by the maintainer of GStreamer, Wim Taymans.

The first corporate sponsor was RidgeRun, that hired Erik Walthinsen to develop methods for embedding GStreamer in smaller devices

The GStreamer logo was designed by Brock A. Frazier, who worked for RidgeRun.

Introduction

GStreamer was created to serve as a host for multimedia applications.

It is written in the C programming language It is a cross-platform product, so it works

e.g. on Linux, Solaris, Mac OS X and Microsoft Windows.

It provides a framework of plugins, for data flow and media type handling.

It is created to handle both audio and video.

With GStreamer’s development framework you can write any type of streaming multimedia application.

License – LGPL

GStreamer is released under LGPL (GNU Lesser General Public License).

LGPL means that the GStremer software itself has copyleft rules which means that you have to keep the software free.

But you are allowed to create your own software with your own copyright rules and link it with GStremer.

For companies like Nokia this license gives a profitable basis for their own software. They can use GStremer as a multimedia framework, but own the copyright to their own software, because it is merely linked to GStreamer.

Applications

GStreamer’s most obvious use is creating a media player on top of it, because it supports many needed formats for this.

GStreamer can also be used to create more complex programs like video or audio editing applications.

We intend to use it to create an Internet radio.

N770, N800 and N810

Nokia’s open source product N770 Internet Tablet and its successors N800 and N810, are palm-sized tablet PC:s running Linux

The tablets use GStreamer as a multimedia framework This enables 3rd party developers to use on-

board DSP without extensive knowledge about its architecture

They support audio (MP3, MPEG4-AAC, WAV, AMR and MP2) and video (MPEG-1, MPEG-4, Real Video, H.263, AVI and 3GP).

They have a Real player and a Flash player

N770, N800 and N810

Lifetrak

A project at University of California, Department for Electrical Engineering by Sasank Reddy and Jeff Mascia.

Lifetrak uses Nokia 770 as a hardware platform for creating a portable music player.

Lifetrak enables a context-aware playlist that automatically selects music based upon the users location, the pace of his movement and the environment

Lifetrak has a simple learning mechanism that can adjust the selection of the music based on the user’s ratings.

Lifetrak

Lifetrak

Lifetrak’s playing functionality is built on top of GStreamer

Lifetrak supports song files in the MP3 format, and in the future also Ogg-Vorbis, WAV, AAC, and MP4 files. GStreamer can handle each of these compression standards

GStreamer is also used for setting the volume, getting the position, and setting the position of the current song.

Mad Hatter’s Cocktail Party A project at Standford University, Palo Alto

Research Center and Robotics Institute in Carnegie Mellon University

Project wants to enhance conference call type mobile communication

The idea is to make communication possible for several small groups in a single audio space and for these groups to be able to regroup when desired.

The communication model is similar to a cocktail party.

Mad Hatter’s Cocktail Party The idea is to analyse the audio, find these

groups and make each listener hear his own group with a higher sound level and dimm out the voices of the rest.

This simulates a situation where you stand closer to the ones you listen to.

This is done by analysing when people speak, assuming that people in the same group mainly speek in turns.

Mad Hatter’s Cocktail Party The server, that analyses the voices, is a

PC running Linux using GStreamer. The server receives audio streame from

each client, analyzes them, applyes audio effect and performs mixing, and finally transmits audio back to the listeners.

Usage

GStreamer is so called “pipelined” multimedia framework Pipe-line consists of sequence of operations

The basic building block of a pipeline is called an element Pipe-line consists of chain of these elements and data

flows through it. For example, a basic mp3 audio decoder

Element 1 Reads mp3 file from hard disk.

Element 2 Decodes the mp3 data to raw audio

Element 3 Sends the raw audio to soundcard

Usage

Every element may also have a different number of pads Sink (input) pad and source (output) pad. Pad is an element’s link to outside world Pads can also be dynamic so that a pad can be

randomly created and/or destroyed. Interconnected elements form a bin

Is also element Same operations that can be done to an

element can be done to a bin also Pipeline is a specialized type of bin so that the

top level bin has to be pipeline

Usage

A bus takes care of internal messaging in GStreamer.

A callback function can be defined that takes care of EOS or other errors messages and acts upon them.

Every pipe-line has a bus by default The developer should create a callback function for the

bus. When the main loop is running the bus is checked for

messages and when a new message is noticed the callback function will be called

Example

Example

Creating elements

Example

Creating elements

Setting filesrc element properties

Example

Creating elements

Setting filesrc element properties

Adding the callback function for bus

Example Adding the callback function for bus

Putting elements to a bin

Creating elements

Setting filesrc element properties

Example

Putting elements to a bin

Creating elements

Setting filesrc element properties

Linking elements together

Adding the callback function for bus

Example Callback funtion for error handling

Function for dynamic pad creation

References

[1] GStreamer Application Development Manual (0.10.12.1), Wim Taymans, Steve Baker, Andy Wingo, Ronald S. Bultje, Stefan Kost

[2] DSP video processing via open-source APIs, EETimes OnLine, Rishi Bhatteracharya, 30.10.2006 http://www.eetimes.com/news/design/showArticle.jhtml?articleID=193401461

[3] The Mad Hatter’s Coktail Party: A Social Mobile Audio Space Supporting Multiple Simultaneous Conversations, Paul M. Aoki, Matthew Romaine, Margaret H. Szymanski, James D. Thornton, Daniel Wilson and Allison Woodruff, Palo Alto Research Center, CCRMA, Dept. of Music Stanford University, Robotics Institute Carnegie Mellon University, Florida USA 5.-10.43.2003, Paper: Design for the Socially Mobile

[4] Lifetrak: Music In Tune With Your Life, Sasank Reddy, Jeff Mascia, NESL, Department of Electrical Engineering University of California, Los Angeles

[5] Multimedia Abstraction Architecture of Cross-platform Applications Based on IMS, Jie Chen, Master Thesis, Helsinki University of Technology / Department of Computer Science and Engineering, 2006

[6] Linux for suits: a first look at the Nokia 770, Doc Searls, 02/2006, Linux Journal, Volume 2006 Issue 142, Publisher: Specialized Systems Consultants, Inc.

[7] Wikipedia, The Free Encyclopedia, (GStreamer, LGPL), http://en.wikipedia.org/wiki/Gstreamer, http://en.wikipedia.org/wiki/GNU_Lesser_General_Public_License