delivering real time communications with mobile
DESCRIPTION
Delivering real time communications with mobileTRANSCRIPT
Delivering Real-Time
Communications with MobilePresented by Robin Raymond [http://about.me/robinraymond]
Chief Architect
Hookflash.com / OpenPeer.org
WebRTC, Mobility, Cloud, and More...
IIT REAL-TIME COMMUNICATIONSConference & Expo Sept 30 - Oct 2, 2014 Chicago
2014-10-01 9:00am WEST: Alumni Lounge
Why should I care about mobility?Digital time spent on mobile apps exceeds the desktop:
Mobile matters
Source: http://www.comscore.com/Insights/Blog/Major-Mobile-Milestones-in-May-Apps-Now-Drive-Half-of-All-Time-Spent-on-Digital
Why should I care about mobility?Mobile is where RTC is used most:
Mobile matters
Source: http://www.comscore.com/Insights/Blog/Major-Mobile-Milestones-in-May-Apps-Now-Drive-Half-of-All-Time-Spent-on-Digital
What's different about mobile RTC?
Mobility differences
A mobile device is always with us…
but what makes mobile RTC so different than desktop RTC?
What's different about mobile RTC?
Mobility differences
Slower
CPUsLimited
Battery LifeMobile + Wireless
Connectivity
Small differences. Huge impact.
Limited
Storage /
Memory
Screen
Sizes /
Rotation
CPUs Limitations
CPU limitations
Slower CPU and encoding/decoding
The CPU is limited. Encoding, decoding,
capturing and rendering audio/video consume the
CPU's resources.
CPU usage also impacts battery life.
CPU limitations
Solution #1: Some assembly requiredFor heavily used encoding, decoding, rendering, scaling and
other routines, some assembly code should be expected.
CPU limitations
Expect to have to do this
optimization assembly work or find
a library that does it for you.
Solution #2: Hardware encoders
Some mobile devices have hardware encoders/decoders
either in the CPU or on the camera.
But there are problems:
● Encoders/decoders are not ubiquitous
● Not always real-time friendly (buffered)
● Not always 100% compatible with other encoders/decoders
● May only support a limited number of simultaneous streams
● Other applications may have the hardware busy
CPU limitations
Solution #3: On-demand high resolutionSend lower resolution/bitrate streams for media not in focus (thus
conferencing signalling needs to be made mobile friendly)!
CPU limitations
In-focus. Sent with high
resolution/frame rate.
Not-in-focus sent with low
resolution / low bitrate.
Battery Life
Battery life
Is there anyone out there?
Mobile RTC clients need to listen for incoming RTC
connection requests from a remote clients
(typically via a server):
Battery life
Maintaining a Connection Consumes Battery Life
Wifi/WWan antennas need a fair amount of power to run.
Battery life
Keeping this connection active to a
server consumes battery life.
Solution #1 - Specialized "push friendly" servers
are required
Be sure to use push friendly protocol services for mobile or
your communication will break down.
Battery life
When a RTC client application goes to the background the
connection to the server will typically be killed. The server
will assume the client is gone.
Solution #1 - Register for push
Most mobile devices offer a "push" service. Register for a
push instead of keeping a server connection alive.
Battery life
Temporary connection
Push service
Solution #1 - Send push upon incoming request
Incoming connections cause a push to happen.
Battery life
Server needs to "hold" incoming connection
request while push is delivered.
Push service
Solution #2 - Application as a service
Ideally have the server ping the client to see if it's alive so the
client application can go to "sleep" otherwise.
Battery life
Registering the connection as a "VoIP/RTC" connection and/or
registering the application as a background service will allow
the connection to be processed when data is received.
It's not as efficient as push but it is better.
Backgrounding and Messaging
When an application goes to sleep in the
background to save battery it can't accept
receiving directed messages anymore.
Battery life
Hi Bob!Zzzz...
Backgrounding and Messaging
Protocols that "retry" sending message until they
get delivered do not work when applications are
sleeping.
Battery life
Hi Bob!
Hi Bob!
Hi Bob!*FAILURE*
Zzzz...
Solution #1: Server Store + Forward
Servers need to store messages until the
receiving application wakes up.
Battery life
Zzzz...
*AWAKE*
*CONNECTED*
Some time later...
Solution #2: Peer-to-peer Synchronize
Messages are stored locally and synchronized
once applications are both awake and connected.
Battery life
Zzzz...
*AWAKE*
*CONNECTED*
Some time later...
Mobility + Wireless Connectivity
Mobility + wireless connectivity
Mobile 4G/3G and IPv4 Address Exhaustion
Mobility + wireless connectivity
Mobile
Pro
vid
er
Firew
all
IP=2.3.4.5
IP=2.3.4.5
IP=2.3.4.5
With millions of new mobile devices, mobile providers are sharing a single IPv4 for
many devices. There aren't enough IPv4 addresses for each mobile device online.
Mobile 4G/3G and IPv4 Address Exhaustion
Mobility + wireless connectivity
Mobile
Pro
vid
er
Firew
all
10.0.0.80
5000
7542
32812
2.3.4.5
For efficient usage of IPv4, typically symmetric NAT firewalls are used. Every local
port gets mapped to a different firewall port per connection destination.
99.98.97.96
15.16.17.18
Solution #1: Mobile 4G/3G requires TURN relay
Mobility + wireless connectivity
Mobile
Pro
vid
er
Firew
all
10.0.0.80
5000
7542
32812
2.3.4.5
Symmetric NAT firewalls aren't friendly for direct media connections and TURN
relays are often required for wwan mobile for IPv4.
99.98.97.96
Media Relay
TURN=5.6.7.8
Mobility + wireless
connectivity
Solution #2: Mobile 4G/3G and IPv6
Mobile
Pro
vid
er
Filt
er
FE80::0202:B3F
F:FE1E:8329
5000 5000
An inexhaustible supply of IPv6 addresses exist so applications should take
advantage of mobile devices and providers that support IPv6!
Mobile provider firewalls don't need to share IPs or do NAT port mappings. They
only act as a filter to block bad traffic thus TURN relays can be avoided.
2607:f0d0:1
002:51::4
FEC and Wireless Packet Loss
FEC = Forward Error Correction
A/V packets can include forward/backward predictive
information so lost packets can be reconstructed.
Mobility + wireless connectivity
But wireless loss is often bursty
What do we do in this situation?
Due to wireless buffering overflow, mobile packet loss often
happens in bursts. Single frame FEC does not work well.
Mobility + wireless connectivity
Solution #1 - Scalable Video Codecs
For video, a base layer video frame can be transmitted with
enhancement video resolutions/frames. Loss of enhancement
frames are non critical.
Mobility + wireless connectivity
Loss of video enhancements
frames will decrease quality but
the overall video transmission is
maintained.
Solution #1 - Scalable Video Codecs
NACK (Negative Acknowledgement) / RTX (Retransmission) /
other schemes are used to protect the base layer.
Mobility + wireless connectivity
Lost base layer packets are
protected with NACK / RTX and
enhancements frames reference
protected base layer frame.
Burst audio loss requires a burst capable audio
codec and/or multi-packet RTP FEC modes.
In some conditions, retransmitting the same
packet multiple times increases the probability of
burst packet loss but not always.
Solution #2 - Audio Burst FEC
Mobility + wireless connectivity
Burst audio loss is an ongoing field of study and
there are proposals but this is not a heavily
standardized area.See also:
http://research.google.com/pubs/pub41611.html
https://github.com/vr000m/payload-1d2d-parity-fec/blob/master/1d2d/draft-singh-payload-1d2d-parity-scheme.xml
http://tools.ietf.org/html/rfc5956
http://tools.ietf.org/html/draft-lennox-payload-ulp-ssrc-mux
Solution #2 - Audio Burst FEC
Mobility + wireless connectivity
Limited storage / memory
Limited Storage / Memory
Recording and Storing Audio/Video
Locally storing raw audio/video frames on a
mobile device is not practical due to the limited
storage space.
Limited storage / memory
Recording and Storing Audio/Video
Encoding video locally on the mobile device
consumes the mobile CPU, meaning less CPU is
available for RTC.
Limited storage / memory
Solution #1: Server Encoding
Relay all media through a server and have the
server do the heavy encoding work.
Limited storage / memory
Increased latency
due to relay and
increased media cost
for hosting.
Solution #2: Local Encoding
Audio/Video is encoded locally and optionally
upload to a server.
Limited storage / memory
Optional to upload audio/video on
a server.
To not impact RTC bandwidth,
upload after the video recording is
complete and not in real-time.
Solution #2: Local Encoding OffloadedUsing mobile hardware encoders makes more sense here
since playback formats are often compatible within the device.
Limited storage / memory
Hardware video encoders work well since they
need not be fully real-time and the encoding
format need not be compatible with the remote
device.
Screen sizes and rotation
Screen Sizes / Rotation
Camera rotation / aspect ratio
Camera rotation and multi angle camera (e.g. back vs front) selection affects the
capture frame rate, aspect ratio, and scalings scalings.
Screen sizes and rotation
Don't waste bits by needless encoding and
sending frame rates and sizes beyond the
remote party's capability to render.
Dynamic Sizing of Render
Local and remote mobile parties can both rotate. The capture side need dynamic
updates of the render's current output capabilities. The sender should maximize
output by not needlessly encoding beyond the capabilities of the render.
Screen sizes and rotation
NOTIFY RENDER SIZE=X,Y
NOTIFY RENDER SIZE=Y,X
Signalling protocol
needs to update
sender constantly of
changing conditions.
Letterboxing / Pillarboxing vs Stretching
The only thing worse than black letterbox / pillarbox bars are stretched images.
Don't stretch images!
Screen sizes and rotation
LetterboxPillarbox
Letterboxing / Pillarboxing vs Cropping
Cropping can work as a solution to fill unused areas but must be done within care
and in known use cases.
Screen sizes and rotation
Image loss = OK
Image loss = NOT
OK
● Specialized signalling helps save battery via push
● Messaging is difficult without store and forward or synchronization
● Heavily encoding / decoding optimization is needed
● Hardware encoders / decoders aren't ubiquitous across devices
● WWAN often requires TURN or IPv6 to work
● Scalable Video Codecs with base layer protection help with burst loss
● Continuous signalling updates of render capability saves CPU and
bandwidth
● Record on a server or passively encode with remaining CPU or via
onboard hardware encoders
● Choose image letterbox/pillarbox or clipping, and not stretching
Mobile Wrap Up
Wrap up
Thanks to the IIT RTC Conference and Hookflash for
allowing me to present.
http://hookflash.com/
http://openpeer.org/
C++ Open Source ORTC Library (mobile friendly)
http://ortc.org/
Thank you