timecard : controlling user-perceived delays in server-based mobile applications

Timecard:Controlling User-Perceived Delays in

Server-Based Mobile Applications

MIT, Microsoft Research

Lenin Ravindranath, Jitendra Padhye, Ratul Mahajan, Hari Balakrishnan

Cloud Services

Mobile Apps

Response Time MattersUsers are impatient

User interaction Rendering

User-perceived delay

Tightly control

App App

ServerUplink Downlink

User interaction Rendering

Server processing delay

App processing delay

Appprocessing delayRequest

transfer delayResponse

transfer delay

User-perceived delay

Tightly control

ServerUplink Downlink

Server delayControl

App AppUser interaction Rendering

Server processing delay

User-perceived delay

Tightly control

Control Server Delay

Request Response

Deadline

Server processing

Controlling the Server Delay

• Trade-off quality of result for processing time– More time to process, better quality results

Server

Request Response

Deadline

Controlling the Server Delay

WorkerWorker

Worker

Worker

Worker

Server

Request Response

Deadline

Controlling the Server Delay

WorkerWorker

Worker

Worker

Worker

Better result

Server

Deadline

Controlling the Server Delay

Server

Server processing

• Servers keep fixed deadlines– No consideration about external delays– Assume constant external delays

Assumed external delay

Assumed external delay

Device type Data size

TCP stateTCP state

Device type

Significant Variability in External Delays

App App

Server

User interaction Rendering

Server processing

Reading sensors

Radio state

Network(3G, WiFi, LTE, ..)

RTT, tput

Highly variable[AppInsight – OSDI ’12]

Significant Variability in External Delays

Server

• Client with high external delays– Poor end-to-end response time

• Client with low external delays– Do not produce the best quality result

Deadline

Significant Variability in External Delays

Server

Servers should adapt to external delaysto control the user-perceived delay

Significant Variability in External Delays

Server

Deadline

Significant Variability in External Delays

Server

Adapt

Significant Variability in External Delays

Server

User interaction Rendering

End-to-end deadline

Adapt

App App

Timecard

App App

Server

Timecard

App App

Server

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Time elapsed since user interaction

Predicted downlink & app processing delay

Adapt

Timecard

App App

Server

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Desired end-to-end delay

Adapt Processing Time

Timecard

App App

Server

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Adapt Processing Time

Trade-off on quality of the result

Desired end-to-end delay

Timecard

App App

Server

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Adapt Response

Desired end-to-end delay

Timecard

App App

Server

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Adapt Response

Desired end-to-end delay

Timecard

App App

Server

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Better quality result

Desired end-to-end delay

Timecard

App App

Server

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Time elapsed since user interaction

Predicted downlink & app processing delay

Challenges

Server

App App

Challenges

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

Server

App

UI dispatcherWeb callback

AppHighly asynchronous

Server Threads

Challenges

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

Spawn workers

Send response

UI dispatcherWeb callback

Server

App App

Request handler

Transaction

Highly asynchronous

Challenges

Server

App App

GetElapsedTime();

No single reference clock

Variable network delaysand processing delays

PredictRemainingTime(responseSize);

Transaction

Highly asynchronous

Timecard

Server

App App

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Transaction

Transaction Tracking

TimeSync Delay Predictors

Timecard

App

App Instrument

er

InstrumentedApp

Service

Developer

Timecard.dll

GetElapsedTime();PredictRemainingTime

(responseSize);

Desired end-to-end delay

config

App Store

Timecard

Server

App App

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Transaction

Transaction Tracking

TimeSync Delay Predictors

Timecard

Server

App App

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Transaction

Transaction Tracking

TimeSync Delay Predictors

Server Threads

Transaction Tracking

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

Spawn workers

Send response

UI dispatcherWeb callback

Server

App App

Request handler

Transaction

Server Threads

Transaction Tracking

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

Spawn workers

Send response

UI dispatcherWeb callback

Track across thread boundaries o at the app and at the server

Server Threads

Transaction Tracking

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

Spawn workers

Send response

UI dispatcherWeb callback

Server

App App

Request handler

Track between app and server

Server Threads

Transaction Tracking

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

Spawn workers

Send response

UI dispatcherWeb callback

Server

App App

Request handler

Transaction

Server Threads

Transaction Tracking

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

Spawn workers

Send response

UI dispatcherWeb callback

Server

App App

Request handler

TC

TC

TC

TC

TC

TC

TC

TC

TC

TC

Transaction context (TC) attached to every thread– Carry along timestamps, transaction information

Server Threads

Transaction Tracking

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

Spawn workers

Send response

UI dispatcherWeb callback

Server

App App

Request handler

TC

TC

TC

TC

• Event handler – create new Transaction context (TC)– Timestamp and attach to thread

• Asynchronous call – Pass TC from current thread to callback thread– Pass TC by detouring callbacks [AppInsight – OSDI ‘12]

Server Threads

Transaction Tracking

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

Spawn workers

Send response

UI dispatcherWeb callback

Server

App App

Request handler

TC

TC

TC

TC

Server Threads

Transaction Tracking

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

Spawn workers

Send response

UI dispatcherWeb callback

Server

App App

Request handler

TC

TC

TC

TC

TC

HTTP[“x-Timecard-Request”]

TC

• Web Request – pass TC from current thread to server– Encode TC in a special HTTP header

• Request handler at server – Parse TC and attach to server thread

Server Threads

Transaction Tracking

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

Spawn workers

Send response

UI dispatcherWeb callback

Server

App App

Request handler

TC

TC

TC

TC

TC

TC

Server Threads

Transaction Tracking

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

Spawn workers

Send response

UI dispatcherWeb callback

Server

App App

Request handler

TC

TC

TC

TC

TC

TC

TC

TC

TC

TC

TC

GetElapsedTime();

Timecard

Server

App App

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Transaction

Transaction Tracking

TimeSync Delay Predictors

TimeSync

GPS

• Sync with Cell Tower– Off my several seconds to minutes

• GPS– Does not work indoors

• Probing

Probe

TimeSync

Instrument app to automatically send probes

Probe

Symmetric Asymmetric

Clock offsetClock drift

Smartphone clock Server clock

TimeSync

Probe

Radio wakeup delay

with large extra delay

Active

IdleIdle

Network Radio States

TimeSync

Queuing delay

with large extra delay

Probe

Network traffic

Active

Idle

Active

TimeSync

Network busy

Radio idle

> 100ms error

min RTT

TimeSync

Optimal time to send the probes Radio is active No network traffic from the phone

Need to be radio-aware and network-aware

Probe

TimeSync

Server

App App

TimeSync

UI Thread

Background Thread

Background Thread

Thread start

GPS start

Web requestGPS callback

Event handler

App

Probe

TimeSync

~5ms error

Network busy

Radio idleTimecard

min RTT

Timecard

App App

Server

GetElapsedTime();

Time elapsed since user interaction

Predicting Remaining Time

App App

Server

PredictRemainingTime(responseSize);

Downlink delay App processing delay

Predicting Downlink Delay

Response sizeLatencyThroughput

Loss rate

TCP window state• Most transfers are short– Median – 3KB– 90% percentile – 37KB

Analysis of 4000 apps

• Cellular networks– High-bandwidth, rare-loss,

high-latency,

• 99% transfers over HTTP– TCP window state matters -> multiple RTTs

Predicting Downlink Delay

TCP Window

1 extra RTT

Predicting Downlink Delay

Response sizeLatency

TCP parameters

Determined by RTT Number of round trips

Predicting Downlink Delay

• Use recent estimate of RTT– Use time sync probes

Read TCP window state

• Read TCP window state at serverMiddlebo

x• Do not work with middleboxes

– Split-TCP connection

• TCP window state at the middlebox matters– No easy way to estimate

TCP window state

Build an empirical data-driven model

RTT

Number of round trips

Predicting Downlink DelayPredictRemainingTime(responseSize);

Learn

• Decision tree model– Response size– Recent RTT– Network provider– Bytes already transferred in

the TCP connection o Proxy for TCP window state at the

middlebox

Middlebox

Downlink Delay Predictor

• 20 users + controlled experiments– 1 month, two cities

• 250,000 downloads– 1 KB to 500KB

• AT&T, T-Mobile, Sprint, Verizon, Wi-Fi– 3G, 4G (HSPA+), LTE

• Indoors, outdoors, static, walking, driving

Predicting Downlink Delay

• Split the data into training and testing

Wi-Fi Median error - 12 ms, 90th percentile - 31ms

Cellular Median error - 17 ms, 90th percentile - 86 ms

Predicting App Processing Delay

App App

Server

PredictRemainingTime(responseSize);

Predicting Remaining Time

App App

Server

PredictRemainingTime(responseSize);

Processing Delay Predictor

Downlink Delay Predictor

Timecard

App App

Server

GetElapsedTime(); PredictRemainingTi

me(responseSize);

D

e rd = D – e – r

Desired end-to-end delay

• Mobile ads service• Twitter analysis service

Deployment

Mobile Ads ServiceContextual ads to mobile apps [Mobisys ’13]

Mobile Ads Service

Fetch and process ads for keywords

Extract keywords Render adSend keywordsBest Ad

Mobile Ads Service

Extract keywords Render adSend keywordsAd

Ad providerAd provider

Ad provider

Ad provider

Ad provider

A

Keywords

Mobile Ads Service

Extract keywords Render adSend keywordsAd

Mobile Ads Service

Extract keywords Render adSend keywordsAd

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Adapt Processing Time

Mobile Ads Service• ~200,000 transactions• Significant variability in external delay

Within 50ms of the target delay 90% of the time

With Timecard

Without Timecard

Target delay

• Mobile ads service• Twitter analysis service

Deployment

Twitter Analysis Service

Fetch from Twitter and analyze

Get keyword Parse and renderSend keywordScore and tweets

Twitter Analysis Service

Fetch from Twitter and analyze

Get keyword Parse and renderSend keywordScore and tweets

Twitter Analysis ServiceGetElapsedTime(

); PredictRemainingTime(responseSize);

Adapt Processing Time

Adapt Response

Get keyword Parse and renderSend keywordScore and tweets

10KB to 50 KB

Twitter Analysis Service• ~150,000 transactions• Adapt server processing time in steps of 150ms

With Timecard

Without Timecard

Target delay

Tightly control end-to-end delay with Timecard

• 0.1% runtime overhead

• Less than 1% memory overhead

• Less than 1% network overhead

• Negligible battery overhead

Timecard Overhead

Limitations

• Bootstrapping– Training the downlink and app processing delay predictors– Learn app processing delay for every transaction type– Turn off adaptation till enough data is collected

• Complex transactions– Multiple parallel or serial requests– Elapsed time will work!– Need to model complex interactions or need developer help

Timecard

App App

Server

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Adapt Processing Time

Adapt Response

End-to-end deadline

Timecard

App App

Server

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Adapt Resources Used

End-to-end deadline

Timecard

App App

Server

GetElapsedTime(); PredictRemainingTi

me(responseSize);Request Prioritization

End-to-end deadline

Timecard

GetElapsedTime();

PredictRemainingTime(responseSize);

Deadlines

Scheduler

• Controlling end-to-end delays– Elapsed time– Prediction on remaining time

• Can be applied in several context

Take-Away

Adapt quality of the paper

SOSP Deadline

GetElapsedTime();

PredictRemainingTime(paper);

Timecard

App App

Server

GetElapsedTime(); PredictRemainingTi

me(responseSize);

Time elapsed since user interaction

Predicted downlink & app processing delay

Backup

Predicting App Processing Delay

AppResponse size

• Correlated with response size

Analysis of AppInsight data 1653 types of transactions

Response size (KB)

Proc

essin

g de

lay

(ms)

Predicting App Processing Delay

AppResponse size• Parsing delay

– Correlated with data size• Rendering delay– Correlated with data size

• Dependent on device type– Processing speed

• Correlated with response size

Analysis of AppInsight data 1653 types of transactions

Predicting App Processing Delay

App

PredictRemainingTime(responseSize);

Processing Delay Predictor

Learn

• Decision tree model– Response size– Device type

Analysis of AppInsight data 1653 types of transactions

Median error - 8ms90th percentile error - 100ms