delivering real-time communications for mobile
DESCRIPTION
Hookflash Chief Architect, W3C ORTC Chair, Robin Raymond overview of challenges and solutions to delivering RTC (real-time communications) integrations on Mobile devices.TRANSCRIPT
Delivering Real-Time Communications with Mobile
Presented by Robin Raymond [http://about.me/robinraymond]Chief ArchitectHookflash.com / OpenPeer.org
WebRTC, Mobility, Cloud, and More...
IIT REAL-TIME COMMUNICATIONSConference & Expo Sept 30 - Oct 2, 2014 Chicago
2014-10-01 9:00am WEST: Alumni Lounge
Why should I care about mobility?Digital time spent on mobile apps exceeds the desktop:
Mobile matters
Source: http://www.comscore.com/Insights/Blog/Major-Mobile-Milestones-in-May-Apps-Now-Drive-Half-of-All-Time-Spent-on-Digital
Why should I care about mobility?Mobile is where RTC is used most:
Mobile matters
Source: http://www.comscore.com/Insights/Blog/Major-Mobile-Milestones-in-May-Apps-Now-Drive-Half-of-All-Time-Spent-on-Digital
What's different about mobile RTC?
Mobility differences
A mobile device is always with us…but what makes mobile RTC so different than desktop RTC?
What's different about mobile RTC?
Mobility differences
SlowerCPUs
Limited Battery Life
Mobile + WirelessConnectivity
Small differences. Huge impact.
Limited Storage / Memory
Screen Sizes /
Rotation
CPUs Limitations
CPU limitations
Slower CPU and encoding/decodingThe CPU is limited. Encoding, decoding, capturing and rendering audio/video consume the CPU's resources.
CPU usage also impacts battery life.
CPU limitations
Solution #1: Some assembly requiredFor heavily used encoding, decoding, rendering, scaling and other routines, some assembly code should be expected.
CPU limitations
Expect to have to do this optimization assembly work or find a library that does it for you.
Solution #2: Hardware encoders
Some mobile devices have hardware encoders/decoders either in the CPU or on the camera.
But there are problems:● Encoders/decoders are not ubiquitous● Not always real-time friendly (buffered)● Not always 100% compatible with other encoders/decoders● May only support a limited number of simultaneous streams● Other applications may have the hardware busy
CPU limitations
Solution #3: On-demand high resolutionSend lower resolution/bitrate streams for media not in focus (thus conferencing signalling needs to be made mobile friendly)!
CPU limitations
In-focus. Sent with high resolution/frame rate.
Not-in-focus sent with low resolution / low bitrate.
Battery Life
Battery life
Is there anyone out there?Mobile RTC clients need to listen for incoming RTC connection requests from a remote clients(typically via a server):
Battery life
I AM HERE
OK
Is someone there?Yes, there is...
Maintaining a Connection Consumes Battery Life
Wifi/WWan antennas need a fair amount of power to run.
Battery life
I AM HERE
OK
Keeping this connection active to a server consumes battery life.
Solution #1 - Specialized "push friendly" servers are required
Be sure to use push friendly protocol services for mobile or your communication will break down.
Battery life
When a RTC client application goes to the background the connection to the server will typically be killed. The server will assume the client is gone.
I AM HERE Is someone there?
[X] Not any more...
PUSH TO ME
Solution #1 - Register for push
Most mobile devices offer a "push" service. Register for a push instead of keeping a server connection alive.
Battery life
OK
Temporary connection
REGISTER PUSH
Push service
Solution #1 - Send push upon incoming request
Incoming connections cause a push to happen.
Battery life
Server needs to "hold" incoming connection request while push is delivered.
SEND PUSH
Push service
PUSH NOTICEOK, I
AM BACKIs someone there?
Solution #2 - Application as a service
Ideally have the server ping the client to see if it's alive so the client application can go to "sleep" otherwise.
Battery life
Registering the connection as a "VoIP/RTC" connection and/or registering the application as a background service will allow the connection to be processed when data is received.It's not as efficient as push but it is better.
Is someone there?
Yes, they are here
STILL THERE?
YES
Backgrounding and Messaging
When an application goes to sleep in the background to save battery it can't accept receiving directed messages anymore.
Battery life
Hi Bob!Zzzz...
Backgrounding and Messaging
Protocols that "retry" sending message until they get delivered do not work when applications are sleeping.
Battery life
Hi Bob!
Hi Bob!
Hi Bob!*FAILURE*
Zzzz...
Solution #1: Server Store + ForwardServers need to store messages until the receiving application wakes up.
Battery life
Hi Bob!Zzzz...
Hi Bob!*AWAKE**CONNECTED*
Some time later...
Solution #2: Peer-to-peer SynchronizeMessages are stored locally and synchronized once applications are both awake and connected.
Battery life
- Hi Bob!- How are you?
Zzzz...
Hi Bob!*AWAKE**CONNECTED*
Some time later...
How are you?
Mobility + Wireless Connectivity
Mobility + wireless connectivity
Mobile 4G/3G and IPv4 Address Exhaustion
Mobility + wireless connectivity
Mob
ile P
rovi
der
Fire
wal
l
IP=2.3.4.5
IP=2.3.4.5
IP=2.3.4.5
With millions of new mobile devices, mobile providers are sharing a single IPv4 for many devices. There aren't enough IPv4 addresses for each mobile device online.
Mobile 4G/3G and IPv4 Address Exhaustion
Mobility + wireless connectivity
Mob
ile P
rovi
der
Fire
wal
l
10.0.0.80
5000
7542
32812
2.3.4.5
For efficient usage of IPv4, typically symmetric NAT firewalls are used. Every local port gets mapped to a different firewall port per connection destination.
99.98.97.96
15.16.17.18
Solution #1: Mobile 4G/3G requires TURN relay
Mobility + wireless connectivity
Mob
ile P
rovi
der
Fire
wal
l
10.0.0.80
5000
7542
32812
2.3.4.5
Symmetric NAT firewalls aren't friendly for direct media connections and TURN relays are often required for wwan mobile for IPv4.
99.98.97.96
Media RelayTURN=5.6.7.8
Mobility + wireless connectivity
Solution #2: Mobile 4G/3G and IPv6
Mob
ile P
rovi
der F
ilter
FE80::0202:B3FF:FE1E:
83295000 5000
An inexhaustible supply of IPv6 addresses exist so applications should take advantage of mobile devices and providers that support IPv6!
Mobile provider firewalls don't need to share IPs or do NAT port mappings. They only act as a filter to block bad traffic thus TURN relays can be avoided.
2607:f0d0:1002:51::4
FEC and Wireless Packet Loss
FEC = Forward Error Correction
A/V packets can include forward/backward predictive information so lost packets can be reconstructed.
Mobility + wireless connectivity
But wireless loss is often bursty
What do we do in this situation?
Due to wireless buffering overflow, mobile packet loss often happens in bursts. Single frame FEC does not work well.
Mobility + wireless connectivity
Solution #1 - Scalable Video Codecs
For video, a base layer video frame can be transmitted with enhancement video resolutions/frames. Loss of enhancement frames are non critical.
Mobility + wireless connectivity
Loss of video enhancements frames will decrease quality but the overall video transmission is maintained.
Solution #1 - Scalable Video Codecs
NACK (Negative Acknowledgement) / RTX (Retransmission) / other schemes are used to protect the base layer.
Mobility + wireless connectivity
Lost base layer packets are protected with NACK / RTX and enhancements frames reference protected base layer frame.
Burst audio loss requires a burst capable audio codec and/or multi-packet RTP FEC modes.
In some conditions, retransmitting the same packet multiple times increases the probability of burst packet loss but not always.
Solution #2 - Audio Burst FEC
Mobility + wireless connectivity
Burst audio loss is an ongoing field of study and there are proposals but this is not a heavily standardized area.See also:
http://research.google.com/pubs/pub41611.html
https://github.com/vr000m/payload-1d2d-parity-fec/blob/master/1d2d/draft-singh-payload-1d2d-parity-scheme.xml
http://tools.ietf.org/html/rfc5956
http://tools.ietf.org/html/draft-lennox-payload-ulp-ssrc-mux
Solution #2 - Audio Burst FEC
Mobility + wireless connectivity
Limited storage / memory
Limited Storage / Memory
Recording and Storing Audio/Video
Locally storing raw audio/video frames on a mobile device is not practical due to the limited storage space.
Limited storage / memory
Recording and Storing Audio/Video
Encoding video locally on the mobile device consumes the mobile CPU, meaning less CPU is available for RTC.
Limited storage / memory
Solution #1: Server EncodingRelay all media through a server and have the server do the heavy encoding work.
Limited storage / memory
Increased latency due to relay and increased media cost for hosting.
Solution #2: Local EncodingAudio/Video is encoded locally and optionally upload to a server.
Limited storage / memory
Optional to upload audio/video on a server.
To not impact RTC bandwidth, upload after the video recording is complete and not in real-time.
Solution #2: Local Encoding OffloadedUsing mobile hardware encoders makes more sense here since playback formats are often compatible within the device.
Limited storage / memory
Hardware video encoders work well since they need not be fully real-time and the encoding format need not be compatible with the remote device.
Screen sizes and rotation
Screen Sizes / Rotation
Camera rotation / aspect ratioCamera rotation and multi angle camera (e.g. back vs front) selection affects the capture frame rate, aspect ratio, and scalings scalings.
Screen sizes and rotation
Don't waste bits by needless encoding and sending frame rates and sizes beyond the remote party's capability to render.
Dynamic Sizing of Render
Local and remote mobile parties can both rotate. The capture side need dynamic updates of the render's current output capabilities. The sender should maximize output by not needlessly encoding beyond the capabilities of the render.
Screen sizes and rotation
NOTIFY RENDER SIZE=X,Y
NOTIFY RENDER SIZE=Y,X
Signalling protocol needs to update sender constantly of changing conditions.
Letterboxing / Pillarboxing vs StretchingThe only thing worse than black letterbox / pillarbox bars are stretched images. Don't stretch images!
Screen sizes and rotation
LetterboxPillarbox
Letterboxing / Pillarboxing vs CroppingCropping can work as a solution to fill unused areas but must be done within care and in known use cases.
Screen sizes and rotation
Image loss = OKImage loss = NOT
OK
● Specialized signalling helps save battery via push● Messaging is difficult without store and forward or synchronization● Heavily encoding / decoding optimization is needed● Hardware encoders / decoders aren't ubiquitous across devices● WWAN often requires TURN or IPv6 to work● Scalable Video Codecs with base layer protection help with burst loss● Continuous signalling updates of render capability saves CPU and
bandwidth● Record on a server or passively encode with remaining CPU or via
onboard hardware encoders● Choose image letterbox/pillarbox or clipping, and not stretching
Mobile Wrap Up
Wrap up
Thanks to the IIT RTC Conference and Hookflash for allowing me to present.
http://hookflash.com/http://openpeer.org/
C++ Open Source ORTC Library (mobile friendly)http://ortc.org/
Thank you