Developing WebRTC applications with advanced media processing

Luis Lopez

lulop@kurento.org

www.kurento.org

WebRTC infrastructures: at the heart of the long tail

Peer-to-Peer WebRTC Application (without media infrastructure)

WebRTC video stream

WebRTC Application based on media infrastructure

media infrastructure

WebRTC media servers: the simplest WebRTC infrastructure

• Media server– Monolithic software

– Based on shared memory

– Scales well vertically

– Can’t scale horizontally

• Media server vendors– Software vendors

– Hardware vendors

WebRTC media server vendors

Common functions of WebRTC media serversTranscoding media server

VP8 H.264

MCU/SFU media server

Recording media server

Evolution of media servers: functionalityWhat common WebRTC Media Servers do:• Transcoding• MCU• Recording

What future MediaServers will do:• Flexible processing• Augmented reality• Blending• Mixing• Analyzing• Etc.

Media ishere

Media goesthere

TranscodingMCU/SFURecording

Current media servers

Media ishere

Rich Mediagoes there

Future media servers

Media Events

Transcoding, MCU/SFU,Recording,

Enrich,Augment, Analyze,

Combine,Transform,Adapt, …

ContextContent

Commands

Why is this important?

Implicit augmentationwith external knowledge

Explicit augmentationwith target information

Why do you communicate? To see other’s face?

WebRTC convergence with TelcoWhy don’t converge with Content?

Kurento Media Server: an open source software WebRTC media server

SendReceiveAnalyze

AugmentEnrich

TransformTranscode

RecordProcess

Replicate

MediaSource

MediaSink

KMS

JavaAPI

JavaScriptAPI

Otherlanguages

Applications define the processingof streams getting through KMS

What does Kurento mean?

Kurento History

2012

•Kurento goes open source LGPL v2.1. Professional services provided by Naevatec(startup)

June’13

•1M€ of investment for providing multimedia capabilities to FI-WARE

Dec’13

•Kurento Featured at WebRTC Expo (Paris)

Feb’14

•3.3M€ of investment for making Kurento an elastic cloud platform (NUBOMEDIA)

April’14

•Kurento featured at IMS World Conference (Barcelone)

July’14

•Kurento featured at DevCon5 (New York)

Sep’14

•500K€ of investment for maintenance of Kurento FI-WARE APIs

Kurento Client API

Sink

SRC Sink

SRC

SRC

Sink

Sink

Media Element

• Provides a specific media

functionality

› Send/receive media

› Process media

› Transform media

• Exchange media through

› Sources

› Sinks

Media pipeline

• Chain of media elements

implementing the desired media

logic.

• The Media API provides the

capability of creating media

pipelines by joining media

elements of the toolbox

Media Element

Sin

k

SRC

Creating applications: the Lego game

Protocols&Codecs

ComputerVision

AugmentedReality

Mul sensoryMul media

MediaRepository

IPTVIntegra on

GroupCommunica ons

Sink%

H pEndpoint

Sink%

SRC%

RtpEndpoint

SRC$

DataChannelEndpoint

Sink%

SRC%

WebRtcEndpoint

Sink%

SRC%

BarCodeReader

Sink%

SRC%

PointerTracker

Sink%

SRC%

FaceDetector

Sink%

SRC%

FaceOverlay

Sink%

SRC%

ChromaFilter

Sink%

SRC%

HeartRateBlender

Sink%

RecorderEndpoint

SRC$

PlayerEndpoint

Sink%

CdnUpload

Sink%

IPTVConnector Mixer

Sink

SRC Sink

SRC

Sink

SRC

Sink

Sink

Application 2

Sink

SRC

Sink

SRC

SinkSink

SRC

Sink

Application 3

Sink

SRC

SRC

Sink

SRC Sink

SinkSRC

Application 1

Toolbox of media elements

Architecting a Kurento application

Ready?

Installing Kurento

• The doc– http://www.kurento.org/docs/current/installation_guide.html

• The video– https://www.youtube.com/watch?v=RGdDpyZAPus

• In shortsudo add-apt-repository ppa:kurento/kurento

wget -O - http://ubuntu.kurento.org/kurento.gpg.key | sudo apt-key add -

sudo apt-get update

sudo apt-get install kurento-media-server

Get a Kurento cloud instance for free

• FI-WARE (http://www.fi-ware.org)– Funded through European Commission research programmes

– PaaS cloud infrastructure offering a bunch of capabilities and enablers for the “Future Internet”

– You can get money developing WebRTC applications with Kurento and FI-WARE• http://www.fi-ware.org/accelerators/

• How to get a Kurento instance in FIWARE in 5 minutes– https://www.youtube.com/watch?v=lFM12pXeWu8

Kurento “Hello World”

Source & demo

https://github.com/Kurento/kurento-tutorial-js/tree/master/kurento-hello-world

Recording and playing

Media Pipeline

WebRTCStreaming

Sin

kSR

C

Sin

k

Mediato

file or URI

Media Pipeline

WebRTCStreaming

Sin

kSR

C

Mediafrom

file or URI

SRC

WebRtcEndpoint

WebRtcEndpoint

RecorderEndpoint

PlayerEndpoint

Source codekurentoClient.create("MediaPipeline", function(error, p) {

if(error) return onError(error);

pipeline = p;

pipeline.create("WebRtcEndpoint", function(error, webRtc){

if(error) return onError(error);

pipeline.create(”RecorderEndpoint”, function(error, recorder){

if(error) return onError(error);

webRtc.connect(recorder, function(error){

if(error) return onError(error);

webRtc.connect(webRtc, function(error){

if(error) return onError(error);

webRtc.processOffer(sdpOffer, …

if(error) return onError(error);

webRtcPeer.processSdpAnswer(sdpAnswer);

});

});

});

});

});

I don’t like callbacks

From that code structureto this code structure

Second try … generators

• https://github.com/Kurento/kurento-tutorial-js/tree/master/kurento-hello-world-recorder-generator

• Transactions very soon …

– By end Q4’14

Advanced media processing

Source & demo

• https://github.com/Kurento/kurento-tutorial-js/tree/master/kurento-magic-mirror

Create your own filter: working with Kurento external modules

WebRTCStreaming

Sin

kSR

C

WebRtcEndpoint

Sin

k

SRC

ArMarkerDetector

Source & demo

• https://github.com/Kurento/kurento-tutorial-js/tree/master/kurento-alvar

Putting it altogether: calls with recording and filtering

Source & demo: let’s switch to Java for fun

• https://github.com/Kurento/kurento-tutorial-java/tree/master/kurento-one2one-call-advanced

Bonus: going beyond the media server vision

WebRTC Application based on media infrastructure

WebRTC Cloud

WebRTC cloud infrastructures

PaaSIaaS SaaSProvider- Computing resourcesDeveloper- Installation- Administration- Security- Application logic

Provider- Development API

Developer- Application logic

Provider- Service

Developer- Nothing to do

Multimedia PaaS: scalability types

Number of concurrent sessions

Nu

mb

er o

f u

sers

per

ses

sio

n

WebRTC PaaS Phone system

TV Broadcasting

CDNs

Media Serverbased Services

“RTC calls”

“Media serverRTC”

“Broadcasting Non-RTC”

“Beyond RTC calls”

The scalability problem in call clouds

CallCallCall Call CallCall

CallCallCall Call CallCall

CallCallCall Call CallCall

The scalability problem beyond call clouds

Media stream

On

e to

MA

NY

WebRTC services beyond the call model

Users in call Monitoring users

Users can dynamically change their role

Nubomedia: Kurento as an elastic media server

Load%Balancer%Func. on%

IaaS%Cloud%Manager%

Applica. on%Server%Func. on%

Broker%Func. on%

Media%Server%Func. on%

Cloud%Orchestrator%

Thank you

Partners

Luis Lopezlulop@kurento.orgwww.kurento.org