delivering real-time communications for mobile

Delivering Real-Time Communications with Mobile

Presented by Robin Raymond [http://about.me/robinraymond]Chief ArchitectHookflash.com / OpenPeer.org

WebRTC, Mobility, Cloud, and More...

IIT REAL-TIME COMMUNICATIONSConference & Expo Sept 30 - Oct 2, 2014 Chicago

2014-10-01 9:00am WEST: Alumni Lounge

Why should I care about mobility?Digital time spent on mobile apps exceeds the desktop:

Mobile matters

Source: http://www.comscore.com/Insights/Blog/Major-Mobile-Milestones-in-May-Apps-Now-Drive-Half-of-All-Time-Spent-on-Digital

Why should I care about mobility?Mobile is where RTC is used most:

Mobile matters

Source: http://www.comscore.com/Insights/Blog/Major-Mobile-Milestones-in-May-Apps-Now-Drive-Half-of-All-Time-Spent-on-Digital

What's different about mobile RTC?

Mobility differences

A mobile device is always with us…but what makes mobile RTC so different than desktop RTC?

What's different about mobile RTC?

Mobility differences

SlowerCPUs

Limited Battery Life

Mobile + WirelessConnectivity

Small differences. Huge impact.

Limited Storage / Memory

Screen Sizes /

Rotation

CPUs Limitations

CPU limitations

Slower CPU and encoding/decodingThe CPU is limited. Encoding, decoding, capturing and rendering audio/video consume the CPU's resources.

CPU usage also impacts battery life.

CPU limitations

Solution #1: Some assembly requiredFor heavily used encoding, decoding, rendering, scaling and other routines, some assembly code should be expected.

CPU limitations

Expect to have to do this optimization assembly work or find a library that does it for you.

Solution #2: Hardware encoders

Some mobile devices have hardware encoders/decoders either in the CPU or on the camera.

But there are problems:● Encoders/decoders are not ubiquitous● Not always real-time friendly (buffered)● Not always 100% compatible with other encoders/decoders● May only support a limited number of simultaneous streams● Other applications may have the hardware busy

CPU limitations

Solution #3: On-demand high resolutionSend lower resolution/bitrate streams for media not in focus (thus conferencing signalling needs to be made mobile friendly)!

CPU limitations

In-focus. Sent with high resolution/frame rate.

Not-in-focus sent with low resolution / low bitrate.

Battery Life

Battery life

Is there anyone out there?Mobile RTC clients need to listen for incoming RTC connection requests from a remote clients(typically via a server):

Battery life

I AM HERE

OK

Is someone there?Yes, there is...

Maintaining a Connection Consumes Battery Life

Wifi/WWan antennas need a fair amount of power to run.

Battery life

I AM HERE

OK

Keeping this connection active to a server consumes battery life.

Solution #1 - Specialized "push friendly" servers are required

Be sure to use push friendly protocol services for mobile or your communication will break down.

Battery life

When a RTC client application goes to the background the connection to the server will typically be killed. The server will assume the client is gone.

I AM HERE Is someone there?

[X] Not any more...

PUSH TO ME

Solution #1 - Register for push

Most mobile devices offer a "push" service. Register for a push instead of keeping a server connection alive.

Battery life

OK

Temporary connection

REGISTER PUSH

Push service

Solution #1 - Send push upon incoming request

Incoming connections cause a push to happen.

Battery life

Server needs to "hold" incoming connection request while push is delivered.

SEND PUSH

Push service

PUSH NOTICEOK, I

AM BACKIs someone there?

Solution #2 - Application as a service

Ideally have the server ping the client to see if it's alive so the client application can go to "sleep" otherwise.

Battery life

Registering the connection as a "VoIP/RTC" connection and/or registering the application as a background service will allow the connection to be processed when data is received.It's not as efficient as push but it is better.

Is someone there?

Yes, they are here

STILL THERE?

YES

Backgrounding and Messaging

When an application goes to sleep in the background to save battery it can't accept receiving directed messages anymore.

Battery life

Hi Bob!Zzzz...

Backgrounding and Messaging

Protocols that "retry" sending message until they get delivered do not work when applications are sleeping.

Battery life

Hi Bob!

Hi Bob!

Hi Bob!*FAILURE*

Zzzz...

Solution #1: Server Store + ForwardServers need to store messages until the receiving application wakes up.

Battery life

Hi Bob!Zzzz...

Hi Bob!*AWAKE**CONNECTED*

Some time later...

Solution #2: Peer-to-peer SynchronizeMessages are stored locally and synchronized once applications are both awake and connected.

Battery life

- Hi Bob!- How are you?

Zzzz...

Hi Bob!*AWAKE**CONNECTED*

Some time later...

How are you?

Mobility + Wireless Connectivity

Mobility + wireless connectivity

Mobile 4G/3G and IPv4 Address Exhaustion

Mobility + wireless connectivity

Mob

ile P

rovi

der

Fire

wal

l

IP=2.3.4.5

IP=2.3.4.5

IP=2.3.4.5

With millions of new mobile devices, mobile providers are sharing a single IPv4 for many devices. There aren't enough IPv4 addresses for each mobile device online.

Mobile 4G/3G and IPv4 Address Exhaustion

Mobility + wireless connectivity

Mob

ile P

rovi

der

Fire

wal

l

10.0.0.80

5000

7542

32812

2.3.4.5

For efficient usage of IPv4, typically symmetric NAT firewalls are used. Every local port gets mapped to a different firewall port per connection destination.

99.98.97.96

15.16.17.18

Solution #1: Mobile 4G/3G requires TURN relay

Mobility + wireless connectivity

Mob

ile P

rovi

der

Fire

wal

l

10.0.0.80

5000

7542

32812

2.3.4.5

Symmetric NAT firewalls aren't friendly for direct media connections and TURN relays are often required for wwan mobile for IPv4.

99.98.97.96

Media RelayTURN=5.6.7.8

Mobility + wireless connectivity

Solution #2: Mobile 4G/3G and IPv6

Mob

ile P

rovi

der F

ilter

FE80::0202:B3FF:FE1E:

83295000 5000

An inexhaustible supply of IPv6 addresses exist so applications should take advantage of mobile devices and providers that support IPv6!

Mobile provider firewalls don't need to share IPs or do NAT port mappings. They only act as a filter to block bad traffic thus TURN relays can be avoided.

2607:f0d0:1002:51::4

FEC and Wireless Packet Loss

FEC = Forward Error Correction

A/V packets can include forward/backward predictive information so lost packets can be reconstructed.

Mobility + wireless connectivity

But wireless loss is often bursty

What do we do in this situation?

Due to wireless buffering overflow, mobile packet loss often happens in bursts. Single frame FEC does not work well.

Mobility + wireless connectivity

Solution #1 - Scalable Video Codecs

For video, a base layer video frame can be transmitted with enhancement video resolutions/frames. Loss of enhancement frames are non critical.

Mobility + wireless connectivity

Loss of video enhancements frames will decrease quality but the overall video transmission is maintained.

Solution #1 - Scalable Video Codecs

NACK (Negative Acknowledgement) / RTX (Retransmission) / other schemes are used to protect the base layer.

Mobility + wireless connectivity

Lost base layer packets are protected with NACK / RTX and enhancements frames reference protected base layer frame.

Burst audio loss requires a burst capable audio codec and/or multi-packet RTP FEC modes.

In some conditions, retransmitting the same packet multiple times increases the probability of burst packet loss but not always.

Solution #2 - Audio Burst FEC

Mobility + wireless connectivity

Burst audio loss is an ongoing field of study and there are proposals but this is not a heavily standardized area.See also:

http://research.google.com/pubs/pub41611.html

https://github.com/vr000m/payload-1d2d-parity-fec/blob/master/1d2d/draft-singh-payload-1d2d-parity-scheme.xml

http://tools.ietf.org/html/rfc5956

http://tools.ietf.org/html/draft-lennox-payload-ulp-ssrc-mux

Solution #2 - Audio Burst FEC

Mobility + wireless connectivity

Limited storage / memory

Limited Storage / Memory

Recording and Storing Audio/Video

Locally storing raw audio/video frames on a mobile device is not practical due to the limited storage space.

Limited storage / memory

Recording and Storing Audio/Video

Encoding video locally on the mobile device consumes the mobile CPU, meaning less CPU is available for RTC.

Limited storage / memory

Solution #1: Server EncodingRelay all media through a server and have the server do the heavy encoding work.

Limited storage / memory

Increased latency due to relay and increased media cost for hosting.

Solution #2: Local EncodingAudio/Video is encoded locally and optionally upload to a server.

Limited storage / memory

Optional to upload audio/video on a server.

To not impact RTC bandwidth, upload after the video recording is complete and not in real-time.

Solution #2: Local Encoding OffloadedUsing mobile hardware encoders makes more sense here since playback formats are often compatible within the device.

Limited storage / memory

Hardware video encoders work well since they need not be fully real-time and the encoding format need not be compatible with the remote device.

Screen sizes and rotation

Screen Sizes / Rotation

Camera rotation / aspect ratioCamera rotation and multi angle camera (e.g. back vs front) selection affects the capture frame rate, aspect ratio, and scalings scalings.

Screen sizes and rotation

Don't waste bits by needless encoding and sending frame rates and sizes beyond the remote party's capability to render.

Dynamic Sizing of Render

Local and remote mobile parties can both rotate. The capture side need dynamic updates of the render's current output capabilities. The sender should maximize output by not needlessly encoding beyond the capabilities of the render.

Screen sizes and rotation

NOTIFY RENDER SIZE=X,Y

NOTIFY RENDER SIZE=Y,X

Signalling protocol needs to update sender constantly of changing conditions.

Letterboxing / Pillarboxing vs StretchingThe only thing worse than black letterbox / pillarbox bars are stretched images. Don't stretch images!

Screen sizes and rotation

LetterboxPillarbox

Letterboxing / Pillarboxing vs CroppingCropping can work as a solution to fill unused areas but must be done within care and in known use cases.

Screen sizes and rotation

Image loss = OKImage loss = NOT

OK

● Specialized signalling helps save battery via push● Messaging is difficult without store and forward or synchronization● Heavily encoding / decoding optimization is needed● Hardware encoders / decoders aren't ubiquitous across devices● WWAN often requires TURN or IPv6 to work● Scalable Video Codecs with base layer protection help with burst loss● Continuous signalling updates of render capability saves CPU and

bandwidth● Record on a server or passively encode with remaining CPU or via

onboard hardware encoders● Choose image letterbox/pillarbox or clipping, and not stretching

Mobile Wrap Up

Wrap up

Thanks to the IIT RTC Conference and Hookflash for allowing me to present.

http://hookflash.com/http://openpeer.org/

C++ Open Source ORTC Library (mobile friendly)http://ortc.org/

Thank you