how to reliably measure and optimize graphics performance of your android games
TRANSCRIPT
11 August 2015
Ville-Veikko HelppiTechnical Product [email protected]
How to Reliably Measure and Optimize Graphics Performance of Your Android Games
W E B I N A R
Sharif SakrDirector of Business [email protected]
Agenda• Testdroid Statistics from Q2'2015• Testing Mobile Games – Why and
How?• Optimize Graphics Performance
with Help of Gamebench - Introduction
• Demonstration• Q&A
iOS 8.0
iOS 8.0.2
iOS 8.1
iOS 8.1.1
iOS 8.2
iOS 8 ALL
KitKat 4.4
KitKat 4.4.2
KitKat 4.4.3
KitKat 4.4.4
KitKat ALL
Lollipop 5.0
Lollipop 5.0.1
Lollipop 5.0.2
Lollipop ALL
Android 'M'
0 5 10 15 20 25 30 35
Failed test runs. Percentage (%).
TREND
TREND
TREND
How Many Devices is
Enough?
~90%market coverage can
be achieved with
128 devices
~20%market coverage can
be achieved with
14 devices
US Market
30 Android devices= ~2/3 market
Global Market
80 Android devices= ~1/2 market
Correct behaviour across platforms and browsers
Integration with web back-ends
Typically need to fully utilize HW (CPU+GPU)
Resource (e.g. battery) consumption
OpenGL ES 2/3
Performance!
Functionality and usability
Screen orientations, connectivity, user profiles
Robustness
Robustness and security!
Brand
Compliances, verification with back-
ends and data
Different 'Verticals'
OS versions
ChipsetsCPU + GPU
Tens of OEMs
MemoryDisplays
(resolutions, physical hw)
OEM mods
Other hardware (connectivityc
alibration)
Relation to other software
Where Test Automation Can
Help
Why Real Devices Are Must-
to-Have• Emulators cannot help you to test...
• User Experience • Usability• Hardware• Software• Infrastructure
0 % = the percentage of your app users that use emulator to run your app!
Why Testing Games
Matter?• Graphics performance
capability is the most important thing for mobile games when forming UX
• Too many games fail to work properly with low-end/mid-range devices due hardware performance
• The real understanding of game's performance capability can be only figured out by testing on real devices
UnderstandingHow quickly gamer assimilates what the game is about, how to fully play with it, what are the functionalities.
UsabilityThe main User Interface, menus, design
and how usable those are
Playing ExperienceIntuition to follow the game and go
forward with its instructions and game-play
Monetization
Are gamers willing to buy – invest – in game
Types of Game Testing• Functionality of the game-play, featuresFunctional Testing• Done always when new features /regressions are
includedRegression Testing
• How game runs on different configurationsCompatibility Testing
• Different languages, geo-focused materialsLocalization Testing• Endurance test to determine if system can handle
the loadSoak Testing
• Measures the capacity of the system Stress Testing• Simplest form of performance testing, measures
how system handles certain loadsLoad Testing• Isolation of the environment (e.g. from network)
to see how game worksHermetic Testing
Feature-based testing
Performance testing
End-user testing
Gameflow -
Performance• Compared to regular app testing, mobile game
testing may seem to be more challenging• e.g. OpenGL ES content not action-identifiable
• Graphics and UI – Identical look&feel across all devices and variants
• Usability controls in the context of game• Navigation/game-play, Progress, Delays
between "game-stages"• Animations, Fonts, Other graphics assets• Screen Resolution and Orientation
• Studies: Majority of users leave game due sluggish, general slowness or other bad performance behavior
• It's very important to understand how game behaves on different devices, under different circumstances
Performance Makes +/-
UX
Performance Testing• Performance, Load, Stress, Spike, Soak
• Measure the real user experience
• Execute tests with all possible devices (SW&HW)
• Discover the impact of server loads, app performance (CPU/GPU utilization)
Components in Game
Testing
App Middleware
Platform Connectivity/Network
Back-end
GAME
Gam
e
En
gin
e
Android
iOS
...
API&
services
Server
Database
Starts at Design Level
App Middleware
Platform Connectivity/Network
Back-end
GAME
Gam
e
En
gin
e
Android
iOS
...
API&
services
Server
Database
Game Design, GFX Assets,Files, Optimization
Performance Testing
w/HW
App Middleware
Platform Connectivity/Network
Back-end
GAME
Gam
e
En
gin
e
Android
iOS
...
API&
services
Server
Database
Real Devices and Platform
Compatibility Testing
App Middleware
Platform Connectivity/Network
Back-end
GAME
Gam
e
En
gin
e
Android
iOS
...
API&
services
Server
Database
Variations
Reference Testing
App Middleware
Platform Connectivity/Network
Back-end
GAME
Gam
e
En
gin
e
Android
iOS
...
API&
services
Server
Database
Back-End Performance
App Middleware
Platform Connectivity/Network
Back-end
GAME
Gam
e
En
gin
e
Android
iOS
...
API&
services
Server
Database
Game with Back-End Integration / Google Play Services
Thousands of developers and
enthusiasts
OEM and Chip makers
Developers – Games, Apps, VR
Journalists
Testing / QA
40k+ downloads in less than 1 yr since launch,
>250 users any day
Market traction
App and Game Developers
Chip makers OEM
Pre-release phase
In market / production
Consumers
Journalists and
Enthusiasts
Used to certify performance
before release
Used by retailers to match consumers’ usage profiles
to mobile devices
Helps with device and app performance
reviews
Who are the people behind GameBench
•Sri Kannan Iyer: CEO & Founder
(London)•Over 15 years experience in the mobile •ARM, MediaTek, Analog Devices and Intel•MSEE (UT Austin), MBA (Oxford)•4 patents, 1 book chapter, speaker at several conferences
•Karthik Hariharakrishnan: CTO & Founder (Bristol)•Over 10 years in mobile engineering•ARM, Motorola, National Instruments•MSEE (University of Illinois), 3 patents
Six employees based in Bristol & London
Get in touch!