testdroid: build a large scale in-house test lab for mobile apps

webinar

Building a Large-Scale In-House Test Lab for Mobile Apps

4 June 2014Ville-Veikko Helppi

Technical Product [email protected]

mailto:[email protected]

© Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved.

2

Agenda• Intro & Benefits: In-House Mobile App Testing• Challenges w/Apps, Devices, Testing & Market • Infrastructure – both Hardware & Software• Tips & Tricks for Building In-House Lab• Testdroid Update• Q&A

Agenda


3


Agenda


4

Manual vs. Automation

Smaller coverage, More money burnt & time wasted, Error-

prone

Large coverage, quickly

completed, Less money & time wasted, Exact

results.


5

Prerequisites• Testing environment as authentic as possible– Real devices <-> emulators/simulators

• Test coverage needs to be adequate– Software is not identical in all Android/iOS devices– Hardware is even more fragmented landscape

• Surrounding testing conditions (e.g. network) • Try not to build in repetitive, manual processes


6

In-House Mobile App Testing

• Continuous Testing with real devices• Cloud/Out-of-Premises vs. In-House• Improve app ratings & your brand reputation• Operational costs vs. Lost revenue?• Consequences of not-being-tested-enough?


7

Reasons to Build In-House Labs

• Pre-Market devices or other hardware• Corporate security policies • Availability and usage level• Special setups


8

Core Benefits of CI and Testing

• Version control and build revision• Build automation, instant execution & testing• All types of testing & integrations enabled• Frequent commits, Code consolidation -> Fast builds• CI/Testing is the best fit for mobile agile process


9


Agenda


10

Why Apps Fail?


11

EXAMPLE: Android devices

Picture by OpenSignal 2013 - http://opensignal.com/reports/fragmentation-2013/

http://opensignal.com/reports/fragmentation-2013/



12

EXAMPLE: Android OEMs





13

EXAMPLE: Android OS





14

OS Versions

ICS JB KK

17.7M test runs, 288 unique Android devices, 50+

configurations


15

Display Resolution

17.7M test runs, 288 unique Android devices, 50+

configurations


16

Memory


17

Coverage? IS THAT ALL YOU NEED?

Picture by Double Encore - http://www.doubleencore.com/2014/05/android-user-data-2014/

http://www.doubleencore.com/2014/05/android-user-data-2014/

http://www.doubleencore.com/2014/05/android-user-data-2014/


18

Data, results & setup information were gather between 1 January – 31 March 2014 (Q1’2014)

- 17.7M test/device runs- 288 unique devices- Thousands of users

AVAILABLE FOR

DOWNLOADwww.testdroid.com

Research

http://www.testdroid.com/


19

Global Coverage


20

Robust/Unstable – Q1’14The Most Robust Devices

Samsung Infuse 4G SGH-I997Lenovo A60Dell Streak 7 M02M001HTC ChaCha A810eCoolpad 7295Samsung Galaxy Europa GT-I5500Samsung Acclaim SCH-R880Motorola Motosmart XT389Sharp Aquos Phone Zeta SH-02ESamsung Galaxy mini GT-S5570

The Most Unstable Devices

Samsung Galaxy S IV Active SGH-I537 Sony Xperia J ST26i Samsung Galaxy S IV Altius SCH-R970 Samsung Galaxy S IV Altius SGH-M919Samsung Galaxy Note 8.0 GT-N5110 Samsung Galaxy S IV Altius SGH-I337 Samsung Galaxy Mega GT-I9205 HTC One M7 4.3 Asus Transformer Pad TF300TL Sony Xperia Acro S LT26w

99.805 %99.797%99.794%99.785%99.783%99.780%99.778%99.775%99.771%99.770%

29.722%36.376%39.007%41.455%43.284%44.737%45.536%45.794%47.689%50.576 %


21


Agenda


22

Hardware – Top 3 Things

• Device Control Servers– Hard disks– RAM– Energy Efficiency

• USB Hubs• WiFi Infrastructure


23

Software – Top 3 Things

• Testdroid Enterprise

• Continuous Integration – e.g. Jenkins CI

• Other agile tools


24

How Testdroid Helps?

• Speed up dev & testing cycles significantly

• Agile development & integration with mobile practices

• Continuous Testing


25

Simple Setup

Device clusters

Device Control Server(s)

Cluster #1 Cluster #2 Cluster #3

USB Hubs, Cables

Cluster #4 Cluster #5


26

DevelopersSource code

Repository

Build failed

Polling

NotifyingSuccessful build

QA & Testers

Buildin

g

Error reporting

Test Cases

Application

Rele

ase

Com

mit

Users / Consumers


27


Agenda


28

1


29

2


30

3


31

4


32

5


33


Agenda


34

Testdroid Update


35

Testdroid Update


36

Stay tuned!

A Step-by-step guide how to build an in-house test lab for mobile apps.

Will be available at www.testdroid.com

COMING SOON!

Coming Soon!



37

Deployment Options


38

Only $999!

For Trial Purpose!

We’re giving a 30-day trial license with Mac Mini.

In worst case, you got a Mac Mini with $999 – and it the best scenario you’ll have a product that will save you a lot of money, time and results as a better application!


39


Agenda


40

www.testdroid.com

THANK YOU!


testdroid: build a large scale in-house test lab for mobile apps

Technology

bitbar technologies

testing conditions

testing market infrastructure

market devices

android devices picture

testing version control

types of testing integrations

androidios devices hardware