Building a Performance Lab for Mobile Apps in a Day

Ashray Mathur Venkatesh Katari

Susie Xia

Salesforce Mobile Apps

Mobile CRM (Hybrid)

Chatter (iOS & Android)

Touch (HTML5 & Hybrid)

Mobile Performance Challenges

iOS Android

War Stories

Mobile Web

Mobile Performance Challenges

Mobile Performance - The Challenges

http://www.html5rocks.com/en/tutorials/internals/howbrowserswork/

Mobile Performance - The Challenges

Mobile Performance - The Challenges

Mobile Performance - The Challenges

Build your own mobile performance lab !

Setting Baselines and Targets

Mobile Usability Engineering 101

Delay User Reaction

< 1s Feels Instant

1 - 3s App is working

3s+ Mental context switch

10s+ I'll come back later ...

Breaking the 1s mobile barrier : http://bit.ly/mobile-barrier

Mobile Web Apps

Key Performance Metrics

Page Load Times

Server / Client Metrics

Network Metrics

Nav Timing Metrics

App  Instrumenta-on  

Mobile  Infrastructure  

Mobile  Automa-on  

3 Essential Steps for Building a Performance Lab

•  WebDriver is the standard for browser automation o  Extensively used for functional testing o  Libraries in almost every language o  W3C working draft

•  Plain functional test, nothing special

o  Device Driver o  iOS – iWebDriver, iOS Driver o  Android - AndroidDriver, ChromeDriver

Mobile Web Automation

•  Don't reinvent the wheel !

App Instrumentation : boomerang.js

•  Steps:  •  Git:    

•  h9ps://github.com/lognormal/boomerang/  

•  Include:  

•  Ini-alize:  

BOOMR.init({      beacon_url:  "h9p://beacons.yoursite.com/path/to/beacon.gif",      site_domain:  "yoursite.com”  

});  

<script  src="/javascript/boomerang.js"  type="text/javascript"></script>  

Documentation - http://lognormal.github.io/boomerang/doc/

App Instrumentation : boomerang.js

•  Page  Load  Times  •  Bandwidth  and  Latency  •  Perceived  Performance  

•  Page  Components  Load  Time  

 

•  Custom  Measurements  BOOMR.plugins.RT.startTimer(“t_-mer”);  

 //some  foo  BOOMR.plugins.RT.endTimer(“t_-mer”);    

BOOMR.plugins.RT.startTimer("t_head");!...!BOOMR.plugins.RT.endTimer(“t_head”).startTimer(“t_body”);!!!  

BOOMR.page_ready();!  

Mobile Infrastructure

•  Webdriver for interactive scenarios o  Multi page flows

•  No Webdriver within instrumented intervals o  Click -> Wait -> Measure

•  Real device performance metrics o  Beacon to server / logging o  Webdriver to collect json

•  Aggregate over multiple iterations ! •  Simulate Network

o  Bandwidth/Latency/Packet Loss •  Visualize Data •  Test -> Measure -> Analyze -> Fix -> Repeat

Testing Methodology

Continuous Integration

Simulator Based CI

Code Checkin Build/deploy Warmup Launch Simulator and run tests

Simulator Based CI

iOS

Automation

UI Automation Framework

Frank Testing Framework

iOS-Driver (selenium based!!)

Logging

Time spent in server + network

Total time spent on client Side

Time to first actionable screen

Caching metrics: Time spent in storing/retrieving data, size of cache, cache eviction counter.

Data processing metrics: Time spent in serializing/de-serializing JSON response, massaging data etc.

Time spent in Controller/data layer, massaging the data etc

My precious….

Stress /longevity testing

-Apple script to generate memory warnings <git repo>

-UI Automation script to continuously go over all the most used scenarios

-Monkey script <git repo>

Stress/longevity testing

Simulator longevity tests with synthetic memory warnings using

apple script

Monkey Script! It goes berserk on your UI!

Repeated user scenarios test on physical device

Instruments –iOS Profiling

Memory Leaks !

CPU monitor

Profiling

Android

Build an Android lab that can collect real device performance metrics in a fully automatic way!

With Selendroid + iWebDriver/ChromeDriver

Web

Native

Heap Usage on Android

•  Dalvik runtime is garbage collected

•  Heap size limit is increasing on devices o  Device specific o  16MB, 48MB, 64MB... o  android:largeHeap="true"

•  We are good, right?

Never tell a woman that she will get a larger wardrobe…

GC Log Message is NOT enough!

Heap Dumps •  Get a heap dump:

o  Dump hprof File in DDMS o  android.os.Debug.dumpHprofData()

•  Convert hprof file to standard format o  hprof-conv android.hprof standard.hprof

•  Analyze hprof file o  Eclipse Memory Analyzer (MAT)

War Stories

Perceived Performance

Do Things Early or Late!

•  Early: o  Important element o  Viewpoint content o  Start actions early

•  Late: o  Async cleanup

Preloading Login

Fool your browser for faster speed.

Fool your browser for faster speed

•  Request Javascript as images while user typing password!

•  Observation: o  Typing password on mobile devices takes time!

•  Fact in Browser: o  Images, CSS are downloaded in parallel o  Javascript is downloaded single-threaded

Mobile Lab Evolution

What the heck is mobile

performance?

Can I try with stop clock?

Haha, I have custom

instrumentation.

I am building an automated

mobile lab.

Automated everything, off to get a BEER !

Takeaways

•  Mobile Performance is hard, but exciting

•  Decide what and how to measure your apps

•  Consider the app stability

•  Set up automation, get rid of stopclock

Future Series

•  Building a Performance Lab for Mobile Apps in a Day o  Tutorial Session at Velocity New York, 3:30pm Oct 14 o  http://velocityconf.com/velocityny2013/public/schedule/detail/30135

•  Don't Let Me Wait: Client-side Optimization o  Workshop Session at Grace Hopper Conference, 5:15pm Oct 3 o  http://gracehopper.org/2013/schedule-at-a-glance/10-3/

Booth #215