webrtc live q&a and screen capture session 3
TRANSCRIPT
![Page 1: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/1.jpg)
WebRTC Standards & Implementation Q&A
Amir Zmora TheNewDialTone
Dan Burne3 StandardsPlay
Alex Gouaillard
WebRTC by Dr Alex / Citrix
![Page 2: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/2.jpg)
Session sponsored by
WebRTC.ventures is a custom design and development shop dedicated to building WebRTC based applicaFons for web and mobile. We have built end-‐to-‐end broadcast soluFons for events and entertainment clients, telehealth soluFons for mulFple clients, live support tools, as well as communicaFon tools for a variety of other applicaFons. WebRTC.ventures is a recognized development partner of TokBox and has also built naFve WebRTC soluFons
![Page 3: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/3.jpg)
We use CrowdCast….It’s WebRTC
![Page 4: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/4.jpg)
WebRTCStandards.info
![Page 5: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/5.jpg)
About Us • Amir Zmora • Dan Burnett • Alex Gouaillard
![Page 6: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/6.jpg)
Screen Capture & Screen Sharing with WebRTC
![Page 7: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/7.jpg)
Screen Sharing in WebRTC • Is WebRTC plus Screen Capture
• Screen capture gives you MediaStreamTrack • WebRTC lets you send it
• We will talk about the Screen Capture piece
![Page 8: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/8.jpg)
Security in native apps • If you install it, the app has complete access to your device • So, choosing not to install is the first level of security
![Page 9: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/9.jpg)
Security in the Web model • Visiting a site is the "install" • But visiting a site needs to be safe • So, the Web uses site origin as security
• By default, limited access to the device browser runs on • Also, page has access to JS it loads but no access to JS from other tabs/windows
![Page 10: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/10.jpg)
Problem - API keys in stupid sites
![Page 11: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/11.jpg)
Screen capture breaks web model • Browser controls allow Site A to do a user View-Source on Site B • Normally, user can see B's popped up source but A can't read • But with screen capture, A can read
![Page 12: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/12.jpg)
Nasty scenario • Site A uses WebRTC with user permission to access camera, screen • Site A scrapes screen image to see what other tabs/windows user has open in browser • Site A tracks user's eyes with camera • When user looks away, Site A does view-source on a tab, scrapes the screen, closes
view-source window
![Page 13: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/13.jpg)
WebRTC Screen Capture standard
• http://w3c.github.io/mediacapture-screen-share/
• Still very new
navigator.mediaDevices.getDisplayMedia({ video: true }) .then(stream => { // we have a stream, attach it to a feedback video element videoElement.srcObject = stream; }, error => { console.log("Unable to acquire screen capture", error); });
![Page 14: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/14.jpg)
Protections in the standard • By default no viewing of other tabs or other browser windows, even in other browser apps
(e.g., Chrome app can't see FF browser) • Requirement for explicit, elevated permissions in order to view these since one app could
control what is presented on the others • In practice,
• Permissions will probably be a form of whitelist similar to what FF uses today • Likely no way for WebRTC apps to get exemptions in advance
![Page 15: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/15.jpg)
Screen Sharing with Chrome
![Page 16: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/16.jpg)
Chrome Specific WebRTC Bits
Media Stream Manager (singleton@browser) GUM JS API
(tab/sandbox)
1. Send request
![Page 17: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/17.jpg)
Chrome Specific WebRTC Bits
Media Stream Manager (singleton@browser) GUM JS API
(tab/sandbox)
2. Check if MST is already available
![Page 18: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/18.jpg)
Chrome Specific WebRTC Bits
Media Stream Manager (singleton@browser) GUM JS API
(tab/sandbox)
Security Manager (source, origin)
3. Check rights
2. Check if MST is already available -‐ NO
![Page 19: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/19.jpg)
Chrome Specific WebRTC Bits
Media Stream Manager (singleton@browser)
Audio Capturer
Video Capturer
Security Manager (source, origin)
GUM JS API (tab/sandbox)
4. Ask Corresponding capturer type to start capturing
3. Check rights -‐ OK
![Page 20: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/20.jpg)
Chrome Specific WebRTC Bits
Media Stream Manager (singleton@browser)
Audio Capturer
Video Capturer
Security Manager (source, origin)
A
GUM JS API (tab/sandbox)
4. Ask Corresponding capturer type to create one -‐ OK
V
5. Store the MST
![Page 21: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/21.jpg)
Chrome Specific WebRTC Bits
Media Stream Manager (singleton@browser)
Audio Capturer
Video Capturer
Security Manager (source, origin)
A
GUM JS API (tab/sandbox)
V
6. Trigger callback
Keep feeding frames
![Page 22: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/22.jpg)
Chrome Specific WebRTC Bits
Media Stream Manager (singleton@browser)
Audio Capturer
Video Capturer
Security Manager (source, origin)
A
GUM JS API (tab/sandbox)
V
NOTE 1: second call for same device with same constraints will directly return the MST, that allows to share streams across tabs without blocking
![Page 23: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/23.jpg)
Chrome Specific WebRTC Bits
Media Stream Manager (singleton@browser)
Audio Capturer
Video Capturer
Security Manager (source, origin)
A
GUM JS API (tab/sandbox)
V
NOTE 2: Recently, a second call for the same device but with different constraints (think simulcast) will indeed return a different resoluFon. Before it would return the first resoluFon asked.
![Page 24: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/24.jpg)
Chrome Specific WebRTC Bits
Media Stream Manager (singleton@browser)
Audio Capturer
Video Capturer
Security Manager (source, origin)
A
GUM JS API (tab/sandbox)
V
NOTE 3: Not only this allow to share cams across processes, it allows for global echo cancellaFon (yes, including the key strokes). Before tabs could cross feed.
![Page 25: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/25.jpg)
Chrome Screensharing 2 steps (1)
Media Stream Manager (singleton@browser)
Screen/Windows/Tab Capturer
Security Manager (source, origin)
Screensharing (extension)
1 2
3 4
S 5
![Page 26: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/26.jpg)
Chrome Screensharing 2 steps (2)
Media Stream Manager (singleton@browser)
Screen/Windows/Tab Capturer
S
GUM JS API (tab/sandbox)
With ID
![Page 27: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/27.jpg)
Screen Sharing with Firefox
![Page 28: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/28.jpg)
Firefox • Whitelisting (wiki.mozilla.org/Screensharing)
• Manual • Hardcoded • Extension
![Page 29: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/29.jpg)
Firefox • Whitelisting - Manual
• Manual • Hardcoded • Extension
![Page 30: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/30.jpg)
Firefox • Whitelisting - Manual
• Manual • Hardcoded • Extension
![Page 31: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/31.jpg)
Firefox • Whitelisting (wiki.mozilla.org/Screensharing)
• Hardcoded ⇒ open a bug! ⇒ Attack surface?
![Page 32: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/32.jpg)
Firefox
• webex.com,*.webex.com,ciscospark.com,*.ciscospark.com,projectsquared.com,*.projectsquared.com,
• *.room.co,room.co,
• beta.talky.io,talky.io,
• *.clearslide.com,
• appear.in,*.appear.in,
• tokbox.com,*.tokbox.com, *.opentok.com,
• *.sso.francetelecom.fr,*.si.francetelecom.fr,*.sso.infra.ftgroup,*.multimedia-conference.orange-business.com,*.espacecollaboration.orange-business.com,
• example.com,
• *.mypurecloud.com,*.mypurecloud.com.au,
• spreed.me,*.spreed.me,*.spreed.com,
• air.mozilla.org,
• *.circuit.com,*.yourcircuit.com,circuit.siemens.com,yourcircuit.siemens.com,circuitsandbox.net,*.unify.com,tandi.circuitsandbox.net,
• *.ericsson.net,*.cct.ericsson.net,
• *.conf.meetecho.com,
• meet.jit.si,*.meet.jit.si,
• web.stage.speakeasyapp.net,web.speakeasyapp.net,
• *.hipchat.me,
• *.beta-wspbx.com,*.wspbx.com,
• *.unifiedcloudit.com,
• *.smartboxuc.com,
• *.smartbox-uc.com,
• *.panterranetworks.com,
• pexipdemo.com,
• *.pexipdemo.com,pex.me,*.pex.me,*.rd.pexip.com,
• 1click.io,*.1click.io,
• *.fuze.com,*.fuzemeeting.com,
• *.thinkingphones.com,
• free.gotomeeting.com,g2m.me,*.g2m.me,gotomeeting.com,*.gotomeeting.com,gotowebinar.com,*.gotowebinar.com,gototraining.com,*.gototraining.com,citrix.com,*.citrix.com,expertcity.com,*.expertcity.com,citrixonline.com,*.citrixonline.com,g2m.me,*.g2m.me,gotomeet.me,*.gotomeet.me,gotomeet.at,*.gotomeet.at
![Page 33: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/33.jpg)
Both Firefox and Chrome • FF-Whitelisting – Extension / addOn • Cr – Extension See e.g. here: Blog - https://tokbox.com/developer/guides/screen-sharing/js/
Code - https://github.com/opentok/screensharing-extensions
![Page 34: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/34.jpg)
?
![Page 35: WebRTC Live Q&A and Screen Capture session 3](https://reader031.vdocuments.us/reader031/viewer/2022022413/58ece83b1a28abd44b8b48ff/html5/thumbnails/35.jpg)
Thank You
Amir Zmora TheNewDialTone
Dan Burne3 StandardsPlay
Alex Gouaillard WebRTC by Dr Alex