kurento cpmx
TRANSCRIPT
![Page 1: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/1.jpg)
Real-time Multimedia Stream Processing Developing rich multimedia applications with Kurento
The Stream oriented GE: Developing rich multimedia applications with Kurento
[email protected] h0p://www.kurento.org
Open APIs for Open Minds
h"ps://join.me/kurentoCPMX
![Page 2: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/2.jpg)
Mul8media infrastructures for the Future Internet
Media is here Media got
there
Media got there
Analyze Transform
Store
Transport
Enrich Augment Adapt
Sensors Context
Events Media is here
2
![Page 3: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/3.jpg)
Enrich, augment, adapt, analyze, transform, store: what’s the problem?
Complexity
3
![Page 4: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/4.jpg)
Future Internet
Mul8media Infrastructure
Simple Development
APIs
Kurento: the equa8on
4
![Page 5: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/5.jpg)
• Interoperable media exchange (mul8plaUorm/mul8protocol) • WebRTC, RTP, HTTP (video tag), etc.
• Process media (Computer vision, augmented reality, media indexing, etc.) • Media and metadata recording and recovery • Transform and adapt media (H.264, H.263, VP8, Ogg, and others) • Media rou8ng and mixing • Etc.
Mul8media infrastructure
• REST API • JavaScript API • Java API
APIs
• LGPL 2.1
Is distributed through a flexible FOSS license
What’s Kurento
5
![Page 6: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/6.jpg)
Kurento Media Server (KMS): the nucleus of Kurento
• KMS is a middleware for media streams – Receives the stream – Process the stream – Issues the stream
Send Receive Analyze Augment Enrich
Transform Transcode Record Process Replicate
Media Source
Media Sink
KMS
6
![Page 7: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/7.jpg)
The Media API: The API for accessing KMS capabili8es
Send Receive Analyze Augment Enrich
Transform Transcode Record Process Replicate
Media Source
Media Sink
KMS
Java Media API
JavaScript Media API
REST API
Applica8ons define the processing of streams geeng through KMS
7
![Page 8: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/8.jpg)
Media API: Media Elements and Media Pipelines
Sink%
SRC% Sink%
SRC%
SRC%
Sink%
Sink%
§ Media Element • Provides a specific media
functionality › Send/receive media › Process media › Transform media
• Exchange media through › Sources › Sinks
§ Media pipeline • Chain of media elements
implementing the desired media logic.
• The Media API provides the capability of creating media pipelines by joining media elements of the toolbox
Media Element
Sink
SRC
8
![Page 9: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/9.jpg)
Media API: trivial example (Java) MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint(“file:///myFile.webm”).build(); H2pGetEndpoint h"pEndpoint = mp.newH"pGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(h"pEndpoint); h"pEndpoint.getUrl(); //URL where the media is made available
Media Pipeline (inside KMS)
H"pGetEndpoint
Media from file or URI
HTTP media streaming Si
nk
SRC
PlayerEndpoint
9
![Page 10: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/10.jpg)
Is that enough? Think about the WWW development model
Process WWW request -‐ DDBB access -‐ Authen8ca8on -‐ XML processing -‐ Etc.
HTTP request: I want this resource
HTTP response: The resource
• Intui8on behind the WWW model – Client asks what it wants
• GET /this_resource HTTP/1.1 – Server side APIs execute the associated processing
You develop this using … • Servlet • JSP • ASP • PHP • Ruby on Rails • CGI • Etc.
10
![Page 11: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/11.jpg)
We need an equivalent model: the role of the Signaling Plane
• We need “something” with the capability of nego8a8ng what, when and how media is exchanged
• A protocol for “saying” – I want “this media” …
• Iden8fica8on of the media to exchange – File in hard-‐drive, IP camera, user, etc.
– in “this way” … • Iden8fica8on of the processing of media
– Augmented, analyzed, etc.
– with “this format” … • Quality
– Codec, screen-‐size, frame-‐rate, etc.
– at “this moment” • Stream control
– Play, stop, start, pause, etc.
11
![Page 12: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/12.jpg)
Don’t get it? think about WWW development again …
Process WWW request -‐ DDBB access -‐ Authen8ca8on -‐ XML processing -‐ Etc.
HTTP request: I want this resource
HTTP response: The resource
Process media request -‐ Media API -‐ DDBB access -‐ Authen8ca8on -‐ XML processing -‐ Etc.
Signaling request: I want this media
Signaling response: The media is here
This is how you create WWW applica8ons
(Servlets, ASP, PHP, Rails, etc.)
This is how you create Kurento enabled applica8ons:
Mul8media RTC is just another feature of your
applica8on
You, as a programmer, create this
12
![Page 13: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/13.jpg)
Abstrac8ng the signaling plane: The Content Handler
KMS
Sink
SRC
Sink
SRC
Sink
SRC
Sink
Media API
REST API (Open API protocol) The Content Handler Equivalent to a Servlet/ASP/PHP script Exposes APIs for specifying: -‐ when receiving “this request”… -‐ … execute “this logic” Developer can use the media API
Code building the media pipeline and execu8ng the applica8on logic the
developer wants
13
![Page 14: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/14.jpg)
Kurento Applica8on Server: The container of Handlers
• Is an extension of Java EE technologies. • Compa8ble with all Java EE Servlet
containers • Hold Handlers
– Your specific applica8on code • Receives signaling requests:
– I want “this media” in this way … • Dispatches request to the appropriate
handler – @annota8ons based mapping
• Generate an answer showing “where and how to find the media” – URL where media is located – Nego8ated SDP
Java EE compaJble container
HTTP Servlet
SIP Servlet
Web services
Kurento REST API
Specific handler implementa8ons
Media API DD.BB.
Kurento ApplicaJon Server (KAS)
Other java APIs.
14
![Page 15: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/15.jpg)
Kurento Architecture: pueng it all together
Kurento Media Server (KMS)
Receive Video
Augmented Reality
Send Video
Computer Vision
Video Playing and Recording
Java EE compaJble container
HTTP Servlet
SIP Servlet
Web services
Kurento REST API
Specific handler implementa8ons
Signaling and WWW traffic
Med
ia
Med
ia
Signaling and WWW traffic
Media API DD.BB.
Kurento ApplicaJon Server (KAS)
Other java APIs.
15
![Page 16: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/16.jpg)
Applica8on execu8on flow Client Code
Applica8on Server (KAS)
Media Server (KMS)
I want this media in this way … (JSON-‐RPC request)
Commands reques8ng the crea8on of a pipeline
What you want is here … (JSON-‐RPC answer)
Media negoJaJon
phase
Media exchange phase
1
2
Specific applica8on logic at the server-‐side
(Content Handler)
Media pipeline creaJon
Media exchange between client and server
16
![Page 17: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/17.jpg)
Content Handler: trivial example //Specify the type of service provided by this handler: Player, Recorder, WebRTC, RTP, etc. @H2pPlayerService(path = "/player”) //Mapping of handler specified in path public class MyPlayerHandler extends H"pPlayerHandler { @Override public void onContentRequest(H"pPlayerSession contentSession) {
//Thie client wants the media this handler provides //Create the pipeline for providing the media
} @Override public void onContentStarted(H"pPlayerSession contentSession) {
//Media started flowing, you can execute addi8onal ac8ons } @Override Public void onSessionTerminated(H"pPlayerSession contentSenssion){
//Media exchange termianted, you can collect your resources }
17
![Page 18: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/18.jpg)
Let’s develop with Kurento • What you need
– A Kurento instance • You can install your own Kurento instance • You can launch a Kurento instance at the FI-‐LAB (FI-‐WARE project)
– h"p://lab.fi-‐ware.org • Geeng help
– FI-‐WARE catalogue • h"p://catalogue.fi-‐ware.org/enablers/stream-‐oriented-‐kurento
– Kurento web site • h"p://www.kurento.org
– Kurento mailing list • h"ps://groups.google.com/forum/#!forum/kurento
– Twi"er • @kurentoms
![Page 19: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/19.jpg)
• FI-‐LAB – Working instance of FI-‐WARE enabling free experimenta8on with
technology – h"p://lab.fi-‐ware.org
• Crea8ng a Kurento instance from an image – Use latest version of Kurento images.
• Crea8ng a Kurento instance using recipes – Use Ubuntu 13.10 clear image – Use latest version of Kurento recipes
• Accessing the Kurento demo at FI-‐LAB – h"p://kurento.lab.fi-‐ware.org:8080/fi-‐lab-‐demo
19
Kurento at the FI-‐LAB
![Page 20: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/20.jpg)
Kurento Hello World: Playing a file with an HTML5 client
Media Pipeline
H"pGetEndpoint
Media from file or URI
HTTP media streaming Si
nk
SRC
PlayerEndpoint
Media API
REST API (Open API protocol)
Handler code I want “this media”
Media is “at this URL”
20
![Page 21: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/21.jpg)
Playing a file with an HTML5 client: Handler code
@H"pPlayerService(path = "/player”) public class MyPlayerHandler extends H"pPlayerHandler {
@Override public void onContentRequest(H"pPlayerSession contentSession) throws Excep8on {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint( "h2p://media.w3.org/2010/05/sintel/trailer.webm").build(); contentSession.setA"ribute("player", playerEndpoint); H2pGetEndpoint h2pEndpoint = mp.newH2pGetEndpoint().terminateOnEOS().build(); playerEndpoint.connect(h2pEndpoint); contentSession.start(h"pEndpoint);
} @Override public void onContentStarted(H"pPlayerSession contentSession) {
PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getA"ribute("player"); playerEndpoint.play();
} }
Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyPlayerHandler.java
21
![Page 22: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/22.jpg)
Playing a file with an HTML5 client: client code <!DOCTYPE html>
<html> <head>
<script src="./js/kws-‐content-‐api.js"></script> <script>
var conn; func8on start() {
var op8ons = { remoteVideoTag : "remoteVideo"
}; conn = new kwsContentApi.KwsContentPlayer("./player", op8ons);
} func8on terminate() {
conn.terminate(); }
</script> </head> <body>
<bu"on onclick="start();">Start</bu"on> <bu"on onclick="terminate();">Terminate</bu"on> <br /> <video id="remoteVideo" autoplay></video>
</body> </html>
Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/player.html
22
![Page 23: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/23.jpg)
Playing a file with an HTML5 client: See the example working
h"ps://www.youtube.com/watch?v=n5BQlhYgGSo
23
![Page 24: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/24.jpg)
Media Pipeline
Adding Computer Vision
H"pGetEndpoint Media from file or URI
HTTP media streaming Si
nk
SRC
PlayerEndpoint
SRC
Sink
JackVaderFilter
Media API
REST API (Open API protocol)
Handler code I want “this media”
Media is “at this URL”
24
![Page 25: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/25.jpg)
Adding Computer Vision: Handler code @H"pPlayerService(path = "/playerWithFilter”) public class MyPlayerHandler extends H"pPlayerHandler {
@Override public void onContentRequest(H"pPlayerSession contentSession) throws Excep8on {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); PlayerEndpoint playerEndpoint = mp.newPlayerEndpoint( "h"p://media.w3.org/2010/05/sintel/trailer.webm").build(); contentSession.setA"ribute("player", playerEndpoint); JackVaderFilter filter = mp.newJackVaderFilter().build(); H"pGetEndpoint h"pEndpoint = mp.newH"pGetEndpoint().terminateOnEOS().build(); filter.connect(h"pEndpoint); playerEndpoint.connect(filter); contentSession.start(h"pEndpoint);
} @Override public void onContentStarted(H"pPlayerSession contentSession) {
PlayerEndpoint playerEndpoint = (PlayerEndpoint) contentSession.getA"ribute("player"); playerEndpoint.play();
} }
Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyPlayerWithFilter.java
25
![Page 26: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/26.jpg)
Adding Computer Vision: Client code <!DOCTYPE html> <html>
<head> <script src="./js/kws-‐content-‐api.js"></script> <script>
var conn; func8on start() {
var op8ons = { remoteVideoTag : "remoteVideo"
}; conn = new kwsContentApi.KwsContentPlayer("./playerWithFilter", op8ons);
} func8on terminate() {
conn.terminate(); }
</script> </head> <body>
<bu"on onclick="start();">Start</bu"on> <bu"on onclick="terminate();">Terminate</bu"on> <br /> <video id="remoteVideo" autoplay></video>
</body> </html>
Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/playerFilter.html
26
![Page 27: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/27.jpg)
Adding Computer Vision: See the example working
h"ps://www.youtube.com/watch?v=yJAQs23eoXw
27
![Page 28: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/28.jpg)
WebRTC loopback
Kurento Media Server
WebRTC Streaming
Media API
JSON-‐RPC API
Handler code I want “this media” (offer SDP)
Media is “here” (answer SDP)
Sink
SRC
28
Kurento ApplicaJon Server
WebRtcEndpoint
Client Browser
![Page 29: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/29.jpg)
WebRTC loopback: Handler code
29
Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyWebRtcLoopback.java
@WebRtcContentService(path = "/webRtcLoopback") public class MyWebRtcLoopback extends WebRtcContentHandler {
@Override public void onContentRequest(WebRtcContentSession contentSession) throws Excep8on {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build(); webRtcEndpoint.connect(webRtcEndpoint); contentSession.start(webRtcEndpoint);
} }
29
![Page 30: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/30.jpg)
WebRTC loopback: client code
30
<!DOCTYPE html> <html>
<head> <script src="./js/kws-‐content-‐api.js"></script> <script src="h"ps://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script> <script>
var conn; func8on start() {
var op8ons = { localVideoTag : "localVideo", remoteVideoTag : "remoteVideo"
}; conn = new kwsContentApi.KwsWebRtcContent("./webRtcLoopback", op8ons); }
func8on terminate() { conn.terminate();
} </script>
</head> <body>
<bu"on id="start" onclick="start();">Start</bu"on> <bu"on id="terminate" onclick="terminate();">Terminate</bu"on> <br /> <video id="localVideo" autoplay></video> <video id="remoteVideo" autoplay></video>
</body> </html>
Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/webrtcLoopback.html
30
![Page 31: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/31.jpg)
WebRTC loopback: see the example working
h"ps://www.youtube.com/watch?v=HaVqO06uuNA
31
![Page 32: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/32.jpg)
WebRTC recording
Media Pipeline
WebRTC Streaming
Media API
REST API (Open API protocol)
Handler code I want “this media (SDP)”
Media is “at here (SDP)”
Sink
SRC
Sink
Media to
file or URI
32
![Page 33: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/33.jpg)
WebRTC recorder: Handler code
33
Source h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyWebRtcRecorder.java Source of handler playing the recorded video: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyPlayerRecording.java
@WebRtcContentService(path = "/webRtcRecorder") public class MyWebRtcRecorder extends WebRtcContentHandler {
@Override public void onContentRequest(WebRtcContentSession contentSession) throws Excep8on {
MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build(); RecorderEndpoint recorderEndpoint = mp.newRecorderEndpoint("file:///tmp/recording").build(); contentSession.setA"ribute("recorder", recorderEndpoint); webRtcEndpoint.connect(webRtcEndpoint); webRtcEndpoint.connect(recorderEndpoint); contentSession.start(webRtcEndpoint);
} @Override public void onContentStarted(WebRtcContentSession contentSession) {
RecorderEndpoint recorderEndpoint = (RecorderEndpoint) contentSession.getA"ribute("recorder"); recorderEndpoint.record();
} }
33
![Page 34: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/34.jpg)
WebRTC recording: client code
34
<!DOCTYPE html> <html>
<head> <script src="./js/kws-‐content-‐api.js"></script> <script src="h"ps://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script> <script>
var conn; func8on start() {
var op8ons = { localVideoTag : "localVideo", remoteVideoTag : "remoteVideo"
}; conn = new kwsContentApi.KwsWebRtcContent("./webRtcRecorder", op8ons); }
func8on terminate() { conn.terminate();
} </script>
</head> <body>
<bu"on id="start" onclick="start();">Start</bu"on> <bu"on id="terminate" onclick="terminate();">Terminate</bu"on> <a href="./playerRecording">Play recorded content</a> <br /> <video id="localVideo" autoplay></video> <video id="remoteVideo" autoplay></video>
</body> </html>
Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/webrtcRecorder.html
34
![Page 35: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/35.jpg)
WebRTC recording: see the example working
h"ps://www.youtube.com/watch?v=ogN81PGkMuE
35
![Page 36: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/36.jpg)
WebRTC one-‐to-‐many (Handler omi"ed for simplicity)
Media Pipeline
Sink
SRC
SRC
Sink
SRC
Sink
SRC
Sink
36
![Page 37: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/37.jpg)
WebRTC one-‐to-‐many: Handler code
Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyWebRtcOneToMany.java
@WebRtcContentService(path = "/webRtcOneToMany") public class MyWebRtcOneToMany extends WebRtcContentHandler {
private WebRtcEndpoint firstWebRtcEndpoint; private String sessionId;
@Override public synchronized void onContentRequest(WebRtcContentSession contentSession) throws Excep8on {
if (firstWebRtcEndpoint == null) { MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); firstWebRtcEndpoint = mp.newWebRtcEndpoint().build(); sessionId = contentSession.getSessionId(); contentSession.releaseOnTerminate(firstWebRtcEndpoint); firstWebRtcEndpoint.connect(firstWebRtcEndpoint); //Loopback contentSession.start(firstWebRtcEndpoint);
} else { MediaPipeline mp = firstWebRtcEndpoint.getMediaPipeline(); WebRtcEndpoint newWebRtcEndpoint = mp.newWebRtcEndpoint().build(); contentSession.releaseOnTerminate(newWebRtcEndpoint); newWebRtcEndpoint.connect(firstWebRtcEndpoint); firstWebRtcEndpoint.connect(newWebRtcEndpoint); //Latest client gives media to the master endpoint contentSession.start(newWebRtcEndpoint);
} }
}
37
![Page 38: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/38.jpg)
WebRTC one-‐to-‐many: client code
38
<!DOCTYPE html> <html>
<head> <script src="./js/kws-‐content-‐api.js"></script> <script src="h"ps://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script> <script>
var conn; func8on start() {
var op8ons = { localVideoTag : "localVideo", remoteVideoTag : "remoteVideo"
}; conn = new kwsContentApi.KwsWebRtcContent("./webRtcOneToMany", op8ons); }
func8on terminate() { conn.terminate();
} </script>
</head> <body>
<bu"on id="start" onclick="start();">Start</bu"on> <bu"on id="terminate" onclick="terminate();">Terminate</bu"on> <br /> <video id="localVideo" autoplay></video> <video id="remoteVideo" autoplay></video>
</body> </html>
Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/webrtcOneToMany.html
38
![Page 39: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/39.jpg)
WebRTC one-‐to-‐many: see the example working
h"ps://www.youtube.com/watch?v=TBkrl3fmHWI
39
![Page 40: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/40.jpg)
WebRTC game (Handler omi"ed for simplicity)
Media Pipeline
Sink
SRC
Sink
SRC
Sink
SRC
40
![Page 41: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/41.jpg)
WebRTC game: Handler code
Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/java/com/kurento/tutorial/MyWebRtcWithFilters.java
@WebRtcContentService(path = "/webRtcWithFilters") public class MyWebRtcWithFilters extends WebRtcContentHandler {
@Override public void onContentRequest(WebRtcContentSession contentSession)throws Excep8on {
final MediaPipeline mp = contentSession.getMediaPipelineFactory().create(); contentSession.releaseOnTerminate(mp); final WebRtcEndpoint webRtcEndpoint = mp.newWebRtcEndpoint().build(); final PointerDetectorFilter pointerDetectorFilter = mp.newPointerDetectorFilter().build(); final FaceOverlayFilter faceOverlayFilter = mp.newFaceOverlayFilter().build(); PointerDetectorWindowMediaParam start = new PointerDetectorWindowMediaParam("start", 100, 100, 280, 380); start.setImage("h"p://ci.kurento.com/imgs/start.png"); pointerDetectorFilter.addWindow(start); pointerDetectorFilter.addWindowInListener(new MediaEventListener<WindowInEvent>() { public void onEvent(WindowInEvent event) {
faceOverlayFilter.setOverlayedImage( "h"p://ci.kurento.com/imgs/mario-‐wings.png", -‐0.35F, -‐1.2F, 1.6F, 1.6F); }
}); webRtcEndpoint.connect(pointerDetectorFilter); pointerDetectorFilter.connect(faceOverlayFilter); faceOverlayFilter.connect(webRtcEndpoint); contentSession.start(webRtcEndpoint);
}
}
41
![Page 42: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/42.jpg)
WebRTC game: client code
42
<!DOCTYPE html> <html>
<head> <script src="./js/kws-‐content-‐api.js"></script> <script src="h"ps://webrtc.googlecode.com/svn/trunk/samples/js/base/adapter.js"></script> <script>
var conn; func8on start() {
var op8ons = { remoteVideoTag : "remoteVideo"
}; conn = new kwsContentApi.KwsWebRtcContent("./webRtcWithFilters", op8ons); }
func8on terminate() { conn.terminate();
} </script>
</head> <body>
<bu"on id="start" onclick="start();">Start</bu"on> <bu"on id="terminate" onclick="terminate();">Terminate</bu"on> <br /> <video id="remoteVideo" autoplay></video>
</body> </html>
Source: h"ps://github.com/Kurento/kurento-‐media-‐framework/blob/develop/kmf-‐samples/kmf-‐tutorial/src/main/webapp/webrtcFilters.html
42
![Page 43: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/43.jpg)
WebRTC game: see the example working
h"ps://www.youtube.com/watch?v=5eJRnwKxgbY
43
![Page 44: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/44.jpg)
Complex applica8on
44
Result: h"ps://www.youtube.com/watch?v=PYCw9-‐4oWWA
Sink
SRC
Sink
Sink
SRC
Sink
SRC
Sink
SRC
Sink
SRC
WebRtcEndpoint
MirrorFilter PointerDetectorFilter
ChromaFilter FaceOverlayFilter
Media Pipeline
• Beyond video-‐conferencing: receiving events from the media stream – Immersive and personalized adver8sing – Games and entertainment
Events (bu"ons)
Game or adver8sement logic
Change face Change background
![Page 45: Kurento cpmx](https://reader034.vdocuments.us/reader034/viewer/2022042518/5569ffd2d8b42a5d758b4dde/html5/thumbnails/45.jpg)
Collabora8ons welcome h"p://www.github.com/kurento
Thank you very much for your a"en8on
Sugges8ons, comments and complains: [email protected]
45