quality assurance for webrtc services
TRANSCRIPT
Quality Assurance for WebRTCServicesMarch 2016, Kranky Geek India
Tsahi Levent-Levi
People don’t usually test their WebRTC implementationsSeriously
2
And when they do, they test things manually
3
Testing and automation are mandatory and widely usedMeanwhile, on other software fields…
4
How do you go about automating your WebRTCtestbed?7 suggestions for you
5
W. Heath Robinson
• WebRTC changes… monthly
• You have more than a voice callto test
Rely on the browser
6
4041
4243
4445
4647
4849
3536
3738
3940
4142
4344
45
Dec-14 Apr-15 Jul-15 Oct-15 Jan-16 May-16
Chrome
Firefox
Selenium is your friend
1
Rely on the browser… and peer into the future
7
1
Stable What your customers are using
Beta What you should expect in the next release
DevWhere you can complain about breakage – and expect fixes in time
Canary A nightly build of whatever is available at that moment in time
Testing only against the Stable version of browsers means you find out issues only after the service breaks for your customers.
> ffmpeg -i MYFILE.mp4 -pix_fmt yuv420p
> sed -i '0,/C420mpeg2/s//C420/' *.y4m
> google-chrome --use-fake-device-for-media-stream
--use-file-for-fake-video-capture=MYFILE.y4m
See http://testrtc.com/y4m-video-chrome/ for more details
Handle your media feeds
8
and think about the type of content you need2
For some arcane reason (called AGC or something), browsers will loop your fake video but won’t look your fake audio
Loop an input 6 times (=expand 5 minutes to 30)> ffmpeg -f concat -i <(for i in {1..6}; do echo "file '$PWD/INPUT.wav'"; done) –c
copy OUTPUT.wav
> google-chrome --use-fake-device-for-media-stream
--use-file-for-fake-audio-capture=OUTPUT.wav
9
2 Handle your media feedsand make sure your audio recording is suitably long
Take care of the network
10
Use netfilter.org iptables
Network Impairments
• Bitrate limitation• Packet loss• Jitter & latency
Firewall Configuration
• None• Force TURN relay• HTTP only• HTTPS only
Multiple Locations
• Virtualization sucks• Stability & Throughput
is what you need• Explore barebone
options
3
• Caller / Callee
• Caller / Agent
• Caller / Expert
• Meeting point
• Conference
• Webinar
Handle synchronization
11
4
Can you see the problem?
Visualize the results
12
5
Visualize the results
13
Chrome 48 with VP8 Chrome 49 with VP9
5
Are these real network issues with packet losses or bugs in these products and how they report packet loss?
Visualize the results (2)
14
5
• How much time does it take to connect the session?
• Are the media channels you need really there?
• Are you sure there aren’t any “additional” media channels?
• Is the bitrate following your restrictions?
Don’t forget to set your expectations
15
6
You never know when things will become handy…
Here are a few thoughts:
• chrome://webrtc-internals dump
• Browser console logs
• Media recordings
• Machine performance
Collect EVERYTHING
16
7
Thank YouTsahi Levent-Levi
http://bloggeek.me
18