POI360Panoramic Mobile Video Telephony over LTE Cellular Networks

Xiufeng Xie

University of Michigan-Ann Arbor

Xinyu Zhang

University of California San Diego

CoNEXT 2017

Background: 360° Video for VR

Sphere view Panoramic frame360° camera

time

360° video for VR

30FPS

360° Video + Video Telephony = Interactive VR!

Mobility Coverage

Challenges & Solution Spaces

Huge VR Traffic Load Calls for Compression

• 360° frame High VR stream bitrate: ▪ 10~20Mbps for 4K MP4 format

▪ Exceed LTE UL (5Mbps)/DL (12Mbps) bandwidth

• Compression based on region of interest (ROI)Human eye can only see part of 360°

Quality

Spatial position

Region-of-Interest (ROI)

Compress unseen parts

Challenge 1: Compression Fails over LTE

• Does not matter if RTT < VR frame interval (e.g., 33ms for 30fps)▪ Typical wireline network✓

• LTE has unstable RTT (5~500ms) depending on traffic & channel

High quality

Lowquality

Lowquality

Compressed frame

High quality

Lowquality

Lowquality

t

ROI

Update ROI knowledge

ROI change ROI quality recover

Lower ROI quality for one RTT

VR stream compressed with new ROI

User-perceived VR quality always fluctuates over LTE

ROI Prediction?

• Predict the ROI by reviewer’s motion?▪ Oculus measurements [1]:

• Avg. head angular speed: 60 Τ° 𝑠

• Avg. head angular acceleration: 500 Τ° 𝑠2

• Head can stop rotation within 120ms

▪ Typical end-to-end LTE video latency can be more than 500ms

Prediction: 120msNeed: 500ms

[1] S.M.LaValle, A.Yershova, M.Katsev, and M.Antonov, “Head Tracking for The Oculus Rift,” in Robotics and Automation (ICRA), 2014 IEEE International Conference on, 2014.

ROI prediction does not work on LTE networks!

Solution: Adaptive Compression

• Responsive ROI update Aggressive▪ Maximize the user-perceived quality

• Irresponsive ROI update Conservative▪ Guarantee the stability of VR quality

Conservative

Aggressive

Smooth quality drop

Sharp quality dropV

ideo

qu

alit

y

Spatial position

ROI center

…

Adaptive compression

Many ways to redistribute the quality

Challenge 2: Irresponsive Rate Control

• Insufficient VR rate control responsiveness

Sluggish loop: large RTT over LTE

Request suitable rate

Measure network-layer statistics

Network

Conventional video rate control

VR users: sensitive to video freezes in immersive environment

LTE network: highly dynamic bandwidth

VR stream LTE uplink

Solution: Cellular Link-Informed Adaptation• Cellular link info as congestion indicator

▪ LTE uplink: typical bottleneck for mobile VR telephony

▪ Diagnostic interface: status of UL firmware buffer

Uplink congestion control based on UL buffer status

Network

End-to-end congestion control

Shortcut: shorter adaptation path better responsiveness

Challenge 3: UL Bandwidth Underutilization

• Existing rate control: unaware of this unique feature▪ Buffer left empty (0 throughput) for 40% of

time! ▪ UL throughput << available bandwidth

Video data UL throughput

LTE uplink resource scheduling:UL throughput depends on its own buffer level

LTE UL firmware buffer

Before UL congestion, higher buffer level higher uplink rate

Solution: Adapt to UL Buffer Level

• Learn relation between UL throughput & buffer level

• Push firmware buffer level to the “sweet” region▪ Sweet region: maximize throughput without congestion

• Buffer level too high: slow down traffic to avoid congestion

• Buffer level too low: speed up traffic to exploit bandwidth

POI360 System Design

Design Overview

360° Cam

Firmware Buffer

Buffer level

Buffer Aware Rate Control

RTP traffic

ROIAdaptive Spatial Compression

Compressed VR stream

Viewer

Cellular uplinkSender

Adaptive Spatial Compression

• Adapt compression mode ▪ Balance ROI quality and stability of ROI

quality

• Design:• Switch mode following ROI update responsiveness

• Responsiveness metric: T3-T1 (duration of lower ROI quality)

T2: sender updates ROI knowledgeT3: ROI quality recovered

Conservative

AggressiveVideo quality

Spatial position

T1: ROI change

Buffer Aware Rate Control

▪ Cross-layer design• Learn buffer’s “sweet” region

• PHY buffer level too high reduce RTP & video bitrate

• PHY buffer level too low increase RTP & video bitrate

Video bitrate Application layer

H.264 Encoding

Packet Pacer

RTP bitrate Transport layer

Compressed frame

PHY bitrate

UL Firmware Buffer

Physical layer

Rate Control

PHY buffer level

Implementation

Live stream 360° video

VR player

LTE phone

Client’s ROI

QXDM

Diag. interface

Evaluation

Micro-benchmark Evaluation

• Validate VR compression design

• Benchmark algorithm: ▪ CMU--Conduit (crop & send ROI)

▪ Facebook--Pyramid encoding

ROI quality (PSNR)

Video-frame-level delay

ROI quality stability

11~13dB of improvement

Reduce delay by 15%

Reduce variation by 5X

Micro-benchmark Evaluation

• Validate our UL buffer-based rate control design▪ Compare with Google Congestion Control (GCC, default rate control of

Google Hangouts & Facebook Messenger)

▪ Our rate control FBCC keeps UL buffer level in the “sweet” region (green) for most of the time

Low usage High usage Overuse (saturation)

System-Level Test

• Test POI360 system under various network conditions▪ Different LTE background traffic load

▪ Different physical channel quality

▪ Different mobility level

• Performance metrics▪ Smoothness

• Video freezing ratio

▪ Quality• Frame-level PSNR

• Mean Opinion Score(MOS)

Different Background Traffic Load• Light LTE background traffic load (early morning)

▪ 1% video freeze

• Heavy LTE background traffic load (noon)▪ 4% video freeze & 2dB PSNR drop

▪ Majority of the frames have either excellent or good quality

PSNR & Video freezing ratio MOS

Different Physical Channel Quality

• Test at places with different channel quality▪ Weak (-115dB RSS); Moderate (-82dB RSS); Strong (-73dB RSS)

▪ Better channel: higher PSNR & MOS, less video freezes

▪ Even the weak channel achieves <3% video freezes

PSNR & Video freezing ratio MOS

Different Mobility Level

• Test under 3 different mobility levels▪ Slow (15mph); urban driving (30mph); highway (50mph)

▪ PSNR & MOS drop with higher mobility. But still have good or excellent quality even under 50mph mobility

▪ More freezes with high mobility: 1% for slow driving, 7% for urban driving. Comparable to legacy non-360 LTE video chat

PSNR & Video freezing ratio MOS

POI360 Summary

• Unique challenges when 360° VR video meets LTE▪ Long RTT of LTE breaks spatial VR compression

▪ Heavy VR traffic load

▪ Low cellular bandwidth utilization

• POI360: the first adaptive 360° VR compression ▪ Adapt compression strategy based on network condition

▪ Achieve balance between traffic load & smoothness

▪ Leverage cellular statistics to enable responsive rate control

• Other works in cellular network-informed mobile applications

* “Accelerating Mobile Web Loading Using Cellular Link Information”,

Xiufeng Xie, Xinyu Zhang, Shilin Zhu, ACM MobiSys’17

* “piStream: Physical Layer Informed Adaptive Video Streaming Over LTE”,

Xiufeng Xie, Xinyu Zhang, Swarun Kumar, Li Erran Li, ACM MobiCom’15

Thank you!

Q & A