webrtc ... gwt & in-browser computation
DESCRIPTION
Alberto Mancini - Francesca Tosi WebRTC ... GWT & in-browser computation Talk @ Open Source Day 2013TRANSCRIPT
WebRTC... GWT & in-browser computation
Alberto Mancini - [email protected]
Francesca Tosi - [email protected]
WebRTCPlug-in free real-time communication …
WebRTC is a free, open project that enables web browsers with Real-Time Communications (RTC) capabilities via simple Javascript APIs.
source: webrtc.org
WebRTCJust another JS API
or
WebRTC is a new front in the long war for an open and unencumbered webBrendan Eich (Mozilla CTO and inventor of JavaScript)
?
WebRTCJust another JS API
or
WebRTC is a new front in the long war for an open and unencumbered webBrendan Eich (Mozilla CTO and inventor of JavaScript)
?
Once upon a time … browsers
(js-)applications were sandboxed and kept far away from the hosting computer’s resources.
Once upon a time … browsers
(js-)applications were sandboxed and kept far away from the hosting computer’s resources.
Browser Piercing (aka HTML5) a Javascript Application cannot
➔ access the filesystem
➔ open f/d socket connections
➔ use graphics accelerator
➔ span multiple threads
Browser Piercing (aka HTML5) a Javascript Application cannot
➔ access the filesystem → and then FileAPI
➔ open f/d socket connections → and then WebSockets
➔ use graphics accelerator → and then WebGL
➔ span multiple threads → and then WebWorkers
Browser Piercing
a Javascript Application cannot
➔ acquire audio and video
➔ communicate P2P
Browser Piercing
a Javascript Application cannot
➔ acquire audio and video
➔ communicate P2P
WebRTC … secure enough?
“… WebRTC has encryption baked right into it; there's actually no way to send unencrypted media using a WebRTC implementation … both Chrome and Firefox implement.”
WebRTC … secure enough?But ...
“If the developers fail to carefully consider the security implications of their choices, the safeguards mandated by the specification will not be enough to guarantee the security of their WebRTC-based applications and the users.”
WebRTC … secure enough?Do not ...
“... it would be very easy to inadvertently click on something that gave camera or microphone control to someone I don't know and don't care to know.”
Courtney Sato - nerd queen @ConstellationRG
WebRTC … secure enough?See also:WebRTC: APIs, Protocols and Security Considerations - Part 1 / Part 2
WebRTC Security and Confidentiality
Security Considerations for WebRTC
WebRTC Security Architecture
WebRTC across platforms● Chrome (mobile too)● Firefox (mobile too)● Opera● Native Java and Obj-C
WebRTC JS-API
Some code …
WebRTC JS-APIAcquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);
WebRTC JS-APIAcquiring video and audio
navigator.getUserMedia(constraints, successCallback,
errorCallback);
WebRTC JS-APIConstraints
● Controls the contents of the MediaStream● Media type, resolution, frame rate
var constraints = {video: true};
video: { mandatory: { ... }, optional [{... }]}
WebRTC JS-APIConstraints
● Controls the contents of the MediaStream● Media type, resolution, frame rate
var constraints = {video: true};
video: { mandatory: { chromeMediaSource: ‘screen’ }, optional [{... }]}
WebRTC JS-APIAcquiring video and audio
navigator.getUserMedia(constraints,
successCallback,
errorCallback);
WebRTC JS-APIerrorCallback
function errorCallback(error) { console.log("error: ", error);}
;-)
WebRTC JS-APIAcquiring video and audio
navigator.getUserMedia(constraints,
successCallback,errorCallback);
WebRTC JS-APIsuccessCallback
function successCallback(stream) { ...}
stream: MediaStream; a flux of audio- or video-related data.
WebRTC JS-APIsuccessCallback
function successCallback(stream) { var video = ... video.src =
window.URL.createObjectURL(stream);
}
(W3C File API)
WebRTC JS-APIfull sample
var constraints = {video: true};
function successCallback(stream) { var video = document.querySelector("video"); video.src = window.URL.createObjectURL(stream);}
function errorCallback(error) { console.log("navigator.getUserMedia error: ", error);}
navigator.getUserMedia(constraints, successCallback, errorCallback);
WebRTC JS-APIRTCPeerConnection
→getUserMedia
+RTCPeerConnection
←
WebRTC JS-APIRTCDataChannel
Bidirectional communication of arbitrary data
var position = {x: 3.0, y: -2.0};
...var message = {txt: ‘...’, timestamp: ...};
RTCPeerConnection samplepc = new RTCPeerConnection(null);pc.onaddstream = gotRemoteStream;pc.addStream(localStream);pc.createOffer(gotOffer);
function gotOffer(desc) { pc.setLocalDescription(desc); sendOffer(desc);}
function gotAnswer(desc) { pc.setRemoteDescription(desc);}
function gotRemoteStream(e) { attachMediaStream(remoteVideo, e.stream);}
RTCPeerConnection
Live Demo - 2
https://apprtc.appspot.com
RTCPeerConnection
RTCPeerConnection
RTCPeerConnection
STUN
TURN
WebRTC made easy...
● Video chat:○ SimpleWebRTC○ easyRTC○ webRTC.io
● Peer-to-peer data:○ PeerJS○ Sharefest
Our target
Our target
● getUserMedia: gives us real time stream from the camera
● we can draw the <video/> into a <canvas/>
● from the canvas we can extract pixel data getImageData()
and then elaborate.
Computing in the browsergoals- Almost native performance- Manage complex algorithms and applications on the browser
Our target
cam WebRTC <video/> canvas ImageData
UInt8ArrayGWT(NyARToolkit)mv matrix
WebGL OBJ Model Loader3D Shaders
WebGL video
effects
Video
Computing in the browserconstraints- reuse existing code- cross-browser (as much as possible)
Computing in the browserour approach- javascript as a target language (GWT) - hand written optimized pieces of code
GWTGWT Web Toolkit
- java to javascript (optimizing) compiler- emulated jre- widget set
- develop in java and execute in javascript
JooinK’s WebRTC sample
Sample
http://picshare.jooink.com
Demo …
PicShareTecnologies
● WebGL for 3D graphics● WebRTC for MediaSteams● ImgUR for pictures store● AppEngine to store indices
● GWT as the glue
FOSS
gwt-nyartoolkit.googlecode.com
gwt-webgl.googlecode.com
The strategy: compile a java-lib with GWT into javascript
What’s nextChallanges
● real-time on mobile apps● real world recognition: computer vision
(BoofCV)
What’s nextStrategy
● use TypedArrays everywhere ● offload computation to the graphic
accelerator through WebGL● help the JIT compiler/optimizer with asm.js
References
● http://io13webrtc.appspot.com/#1(where we got inspiration and most of the pictures)
● http://www.youtube.com/watch?v=p2HzZkd2A40● http://www.webrtc.org (spec)
References
● http://www.jooink.com
● http://jooink.blogspot.com
● http://www.mokabyte.it
Markers - use at picshare.jooink.com