mobile testing and its part in testing tools
DESCRIPTION
Mobile testing and mobile test automation tools case (iOS case with Android and Windows Phone roadmaps), testing tool survey results analyzed, consideration about tool and cloud trends in general, but reflected with mobile testing tools.TRANSCRIPT
Mobile testing and its part in testing tools
Kari Kakkonen, March 25, 2014, ASTQB Software Testing Conference, San Francisco
© Copyright Knowit Oy 2013 | Confidential | Version 1.0
Kari Kakkonen, Knowit
• Roles • Knowit, Director, Quality and Competences & Lead
Consultant • Finnish Software Testing Board FiSTB, chairman • ISTQB international exam and expert working group member • Finnish Software Measurement Association FiSMA ry board
member 2004-2007 • Chairman of research project STX, Lappeenranta University
of Technology • Achievements
• Testing promoter since 1996 • Tietoviikko-magazine 100 most influential IT-persons listing
2010 • Presentations in national and international conferences • Candidate for Tester of the Year 2010 (TestausOSY – FAST) • TestausOSY/FAST founding member
• Education • ISTQB Advanced certified • SPICE provisionary assessor certified • M.SC, Helsinki University of Technology / Aalto-university • Marketing studies, University of Wisconsin-Madison
• Domain background • Embedded • Public sector • Pension insurance • Training provider • Telecom • Finance • Manufacturing
• Services • Training ISTQB Advanced and Foundation + Knowit Testing
Professional • Service management • Test management • Test assessments (TPI Next etc.) and test development • Agile testing services
© Knowit Oy / SECRET (Internal slide)
Twitter: @kkakkonen
LinkedIn: fi.linkedin.com/in/karikakkonen/
110
145 154
189
215 215 220
2007 2008 2009 2010 2011 2012 2013
Trusted ICT Services Partner in Nordics
Experts in 1800 Sweden, Finland, Norway,
Denmark, Russia and Estonia
Knowit AB is listed on the Nordic Exchange in Stockholm
Gro
up
re
ve
nu
e (
ME
UR
)
Case description
© Copyright Knowit Oy 2013 | Confidential | Version 1.0 4
Client
• Worldwide company with many products
• Top of the line in their business
• Consumer products (B2C)
• Established software testing, including automation
• Now wanted their first mobile app
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 5
Three perspectives of mobile solutions
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 6
Technology • Responsive web
• Mobile optimized web
• Native application
• Hybrid application
Target group • Consumers (B2C, B2B2C)
• Retailers and suppliers (B2B)
• Stakeholders
• Own personnel
Benefits • Growth in sales
• Customer loyalty
• Savings
Client choices
in bold
Mobile app scope
• To enable users to access the service also from mobile devices
• not only through the established web service
• Utilize mobile device’s location services heavily
• First platform - iOS
• For anonymity the actual functionality is here called just “activity”
• Supporting functionalities in a bit more detail, meaningful things for testing here
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 7
Expected challenges
• Connectivity to web service, integral part of the service
• Usability – will the vast user base like it?
• Can we automate?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 8
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 9
Functionalities of the application
Location • Activity
• Data
• Map
• Graphs
• Camera
Profile • Latest info
• Following
• Fans
• Shouts
• Edit Profile
Personal • Latest info
• Calendar
Other • Following
• Shouts
• Settings
Our team
• Compact scrum-team
• 2 developers in Sweden
• Project manager, 2 testers and 1 designer in Finland
• All participating in daily scrum meetings
• Customer committed to be in close contact with the team on daily basis
• Customer geographically dispersed
• Remote connection within team and to customer (Lync, Skype)
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 10
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 11
We piloted automation
Knowit test automation process used in the mobile project
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 12
Preparation & piloting Implementation Testing
Te
stin
g
Recording and
scripting
Test execution
New tests
Script maintenance
Reporting
Feasibility
study
Piloting
POC
Imp
lem
en
tatio
n
Training
1.version of
software
under test
© Copyright Knowit Oy 2013 | Confidential | Version 1.0
Feasibility study
Phase Description
• Benefits of automation in this project?
• Discussions about test automation and tools with customer
Proof Of
Concept
(PoC)
• Verifying that the test automation tool is compatible with application
under test
• Identifying parts that are impossible/no benefit to automate
• Installation of test tool to machines
• Placing the hooks to software
• Automating 1-2 test cases
• Investment calculations and workload estimations
• Scripting test cases
• Running test cases in development environment
• New versions of sofware under test in regular intervals
Piloting
• Execution and maintenance of test cases
• Scripting of new test cases
• Reporting
Implementation
Testing
Tool options
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 14
Ava
ilab
le t
oo
ls
MonkeyTalk
Sikuli
Testing with Frank
Xcode
Selenium
NativeDriver
Robot Framework
Junit
eggPlant for mobile
Silk4mobile
Pilo
ted
Robot Framework
Silk4mobile
Xcode
Chosen
Silk4mobile
Xcode
Criteria for selection
• Target operating system iOS
• Several iOS options (iPad, iPad mini, iPod, iPhone)
• Other operating systems in the future (Android, Windows)
• Cost-effectiveness
• Ease of installation and learning
• Existing experiences of use
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 15
Silk4mobile
• Supports all mentioned mobile platforms
• Knowit Silk4Mobile know-how
• Straightforward implementation
• Requires hooks in the software or a jailbreak
• Borland is a Knowit partner
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 16
Xcode
• Default Apple development environment
• Includes test automation environment
• Automation tests run with emulator
• Operational testing in emulator
• Used in Knowit as development and unit test environment
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 17
Project implementation
© Copyright Knowit Oy 2013 | Confidential | Version 1.0 18
Testing process
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 19
Development
Developers’ tests
Test team
• Smoke test
• UI testing
• Field testing
• Automation
Beta testing
Feedback
• Internal testing
• Beta testing
Early challenges
• Challenges of automation
• Tried out Silk4Mobile (Borland)
• Requires hooks within the app
• Part of iOS test automation requires ‘jailbreaking’
• iOS7
• Apple updated its operating system at the same time with this project -> retesting
• Other challenges
• Application needed a lot of field testing in true environment
• A lot of outdoor use
• The web service also updated during the project
• Web service interfaces were modified with changes to app behavior
• Downtimes in web service
• Testing consisted of interaction between app, location information from phone, and web service 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 20
Test planning
• Division of testing tasks between testers and beta testers
• Testers continuously validated the correct functionality of the app features
• Beta testers were required to validate real life scenarios
• Levels of use
• Basic use: activities and following the web service
• Heavy use: more activities, following the web services, shares own information to social media, likes
and comments of friends
• Benchmarking: does app + web service work with same ease and content as web service only?
• Use of exploratory testing
© Copyright Knowit Oy 2013 | Confidential | Version 1.0
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 22
Test management - ChiliProject…
• User story - > test scenarios (different variety) - > field tests
• Incidents, bugs and other issues reported back to Chili
• Roadmap • Sprints • Features • Bugs • Status • Priority • Target version • Files • Reports
Implementation of tests B
asic
user
Indoor
Outdoor City location
Forest
On the water Sailing
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 23
Sp
ecia
l ca
se
s
Network is lost
At sea
In a forest
In the city Connection is lost
Web service is down Data is transferred after connection is restored
Beta testing – field testing
• After smoke tests app moved to beta testing stage
• Purpose of beta testing
• To get the actual end-user experience
• Real life use cases
• Achievements / what was good?
• Wide coverage
• Use cases : active and basic users, like-minded group of active users discussing with others online and at different events, continuously growing number activities, usage in different countries, various programs, all iPhone models plus iPad and iPod…
• Validation testing at the end of the project
• A group of test users, about 10-20 persons who were not involved in the project in any way, but had previously used the web service, from heavy to basic users in different countries
• Using mobile app normally and giving feedback
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 24
Summary
• Challenging start
• Challenging test environment in mobile testing
• Mobile test automation can be tricky, settled to unit test automation only
• Demanding usage environment in this project – field testing was a success
• Despite the challenges, comprehensive and versatile testing
• A total of 140 functionalities and user stories tested
• 445+ issues were reported to Chili, fixed and verified
• Especially happy customer and successful stable application
• Based on this project the customer ordered a new mobile application development and testing from
Knowit
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 25
Test tool survey reflections
© Copyright Knowit Oy 2012 | Luottamuksellinen
Mobiles roll…
Number of smartphones worldwide:
Q3 / 2012 1 000 000 000
2015 2 000 000 000
Source: Strategy Analytics
Test automation options
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 28
Mobile testing
Emulator
Remote device testing
• Devices as a service
Device specific
automation
Mobile testing software
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 29
Open source
• Robot framework
• Robotium
• JUnit / OCUnit
• Appium
• Calabash
• MonkeyTalk
• Sikuli
• Testing with Frankk
Commercial
• Borland Silk4Mobile
• Squish – Frologic
• Keynote DeviceAnywhere
• Testdroid
• Appurify
• EggPlant for mobile
The survey
© Copyright Knowit Oy 2013 | Confidential | Version 1.0 30
Background
• Research target: Software testing tools usage in Finland
• Research is a part of Master Degree of Business Informatics thesis for Metropolia in co-
operation with Knowit Oy
• Used research method was quantitative method with web-form survey
• Survey was sent to IT- and testing professionals in Finland
• Knowit customer contacts
• TestausOSY society
• Soliditet register of Bisnode Finland
• Survey was done in February 2013
• You could choose many options in most questions
© Copyright Knowit Oy 2013 | Confidential | Version 1.0
Respondent’s background
• Response rate was 39 % i.e. total 107 responses
• 64 % was men and 36 % women
• Age distribution; 84 % age between 31 to 50 years
• Education; highly educated, 86 % have university level education
• 87 % of respondents from South of Finland and 10 % from West of Finland
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 32
Profession/role in the company
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 33
38%
4% 6% 5% 5%
13%
3% 1%
3% 2% 3% 1% 1% 1%
12%
4%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Most of the respondents were test
managers/ test coordinators and system
testers
Operating area of the company
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 34
The largest operating areas were
Information and telecommunication,
Finance and insurance business and
Industry
Other service 12 %
Science or technical area 7 %
Finance and insurance business
21 %
Information and telecommunication
35 %
Retail business 2 %
Industry 16 %
International organization and
commission 1 %
Business area unknown
2 %
Mining 1 %
Administration and support
1 % Medical 2 %
Size of the company
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 35
2%
7% 5%
24%
14%
45%
1%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
1 - 10 51 – 100 11 - 50 101 – 500 501 - 1000 I don't know Over 1001employees
The respondents were mostly from
large companies
Current life cycle model in the project
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 36
5%
67%
13% 7%
21% 29%
50%
24%
11% 5%
8%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Current projects life cycle models
were Scrum, V-model and Iterative
model
Future life cycle model
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 37
18%
25%
4% 2% 5%
8% 7%
16%
40%
9%
30%
0%5%
10%15%20%25%30%35%40%45%
Scrum seems to be future
trend, Kanban and Lean are
following, V-model use is
going down
Which employees of your company are using software testing tools?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 38
72% 74%
89%
19% 24% 28% 25%
3% 9%
64%
6%
80%
49%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Testing tools are used by testing
professionals and developers
What kind of training do you have for software testing?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 39
5% 1%
25%
2% 1% 0% 0% 0% 7% 12% 15% 1% 2%
3% 6%
49%
4% 1% 4% 0% 0%
27% 24% 18%
3%
14%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Men
Women
74 % of respondents have ISTQB/ISEB Foundation
course, next most common are tool specific
courses and scrum master course.
What kind of testing tools are in use?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 40
88%
60% 48%
4% 0%
20%
40%
60%
80%
100%
Commercialsoftware
testing tools
Open Sourcetesting tools
Proprietarytools
Other
Tools from many sources are used
side by side
Which way the testing tools are acquired?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 41
40%
6% 3% 4%
25% 22%
0%5%
10%15%20%25%30%35%40%45%
Bought Cloudservice
Other way Rented OpenSource
Coded bycompany
itself
Normally testing tools are acquired
by buying. Next most common is
open source usage and self coding
According to this research cloud
services are not yet a popular
acquisition method..
Which way the testing tools are acquired by company size? (number of persons in the company)
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 42
1
1
2
1 - 10
5
5
2
2 1
11 - 50
5
4 2
2
2 1 Open
Source
Bought
12
22 3
13
2 3 101 - 500
6
14 2
7
1 1 501 - 1000
27
46 6
24
1 5
over 1000
51 – 100
What test management tools do you have in the company?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 43
1%
14%
9%
2%
54%
3% 2% 5%
2%
47%
1% 1% 1% 5%
1%
9%
0%
10%
20%
30%
40%
50%
60%
Most popular Test
management tools were HP:
QC, Atlassian: Jira and
TestLink
Test management- vs office-tools
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 44
1%
54%
9%
47%
0% 1% 1% 1% 1% 0% 0% 0% 5% 9% 1% 2% 0% 0% 5% 3% 2% 2% 1%
13%
8%
7%
2% 1% 1% 1%
1%
0%
10%
20%
30%
40%
50%
60%
70%
No
Yes
Also QC and Jira users are
utilizing often office tools (spread
sheets, word processors)
What incident management tools do you have in the company?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 45
41%
11%
3% 5%
3%
12%
25%
5%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Atlassian:JIRA
IBMRational
ClearQuest
Polarion Proprietarytools
HP: QualityCenter
Bugzilla(Open
source)
Other Mantis
Atlassian: Jira is clearly the largest
in this category and second open
source tool Bugzilla – QC is
surprisingly very little used for
incident management.
What test execution tools do you have in the company?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 46
33%
7% 3% 4% 2% 1% 2% 2% 2% 2% 1% 3% 4% 5%
21%
2% 4%
29%
4% 4%
45%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
Open source tool Selenium was
most popular, next HP:QTP and open
source tool Robot Framework. There
are a large number of different tools
in use in this category.
What test design tools do you have in the company?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 47
1% 1% 1% 2% 2%
46%
2% 3%
8%
1% 4% 4% 2%
0%5%
10%15%20%25%30%35%40%45%50%
Major test design tool was HP: QC, note also
MBT tools and test technique tools (CTE,
allpairs)
What test harness/ unit test framework tools do you have in the company?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 48
27%
5%
17%
2% 3% 1%
12%
7% 7%
2%
7%
2%
0%
5%
10%
15%
20%
25%
30%
These tools are mostly used by
developers. Most used tools in this
category were CppUnit, Junit and .TEST
What specialized unit test frameworks tools do you have in the company?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 49
2% 2%
4%
1%
6%
1%
2%
5%
1%
0%
1%
2%
3%
4%
5%
6%
Specialized unit test frame work tools are
less used but most popular is JBehave
What performance/ load/ stress testing tools do you have in the company?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 50
1% 3%
7%
1%
4% 4%
7%
22%
7%
12%
4%
24%
0%
5%
10%
15%
20%
25%
30%
Most popular in this category
were open source tool Jmeter
and HP Performance Center.
Overall usage of all software testing tools
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 51
88
4 4 12
4 4
98
32
21
7 3 4 6
24 15
27
5
26
5 5 2
29
45
15
3 5 9
3
31
12
0
20
40
60
80
100
120
% o
f re
sp
on
de
nts
98 % have QC in use or have used and 88 % have
JIRA in different testing phases.
Purpose of software testing tools?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 52
79% 76%
80%
29%
48% 50%
77%
30%
4%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Reporting Testexecution
Planning testcases
Generatingtest cases
Requirementmanagement
Defecttracking
Defectcontrol and
management
Reviews Other
Typical testing activities have the lead in
this category, i.e. planning test cases,
reporting and test execution.
Do you use ‘office tools’ in testing (Spreadsheet, word processor etc.)?
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 53
35%
1%
17%
12%
1% 1%
10%
19%
1% 3%
0%
5%
10%
15%
20%
25%
30%
35%
40%
Reporting Requirementmanagement
Test data Test design Defect controland
management
Testmanagement
Test execution Documentation Design Calculation
• Yes, office tools in use 80 %
• Which functions are you using them for?
Office tools are used to same
functions as the software
testing tools
Testing tool markets directions
• Open source is gaining momentum, in midsize companies
• Big (commercial) test tool suites work for big companies
• Cloud (software testing tool as a service) is just emerging, but a growing trend
• Specialized tools, like mobile test automation, emerge all the time
• Are not first considered testing tools (or testers are not the users)
• Are soon swallowed by big test suite providers
• Integrated tool sets for multiple platforms are seen as most useful
• Support e.g. all mobile platforms and all browsers
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 54
Recommendations
• Pay attention to test tool life cycle – does it have future?
• Open source – the user and developer base
• Commercial – company size, market shares
• Consider tool integrations
• Mix and match open source and commercial
• Integrate tools to work together – e.g. Jira – QC – Robot Framework
• Look for good tools
• Superb usability
• Excellent technical flexibility
• Even small vendors grow soon with superb tools
• Word of mouth is big player (Finnish Robot Framework phenomena)
21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 55
”There are plenty of software testing tools. Expertise is needed to select effective testing tools to suitable areas of testing processes”
© Copyright Knowit Oy 2013 | Confidential | Version 1.0 56