mobile testing and its part in testing tools

Mobile testing and its part in testing tools

Kari Kakkonen, March 25, 2014, ASTQB Software Testing Conference, San Francisco

© Copyright Knowit Oy 2013 | Confidential | Version 1.0

Kari Kakkonen, Knowit

• Roles • Knowit, Director, Quality and Competences & Lead

Consultant • Finnish Software Testing Board FiSTB, chairman • ISTQB international exam and expert working group member • Finnish Software Measurement Association FiSMA ry board

member 2004-2007 • Chairman of research project STX, Lappeenranta University

of Technology • Achievements

• Testing promoter since 1996 • Tietoviikko-magazine 100 most influential IT-persons listing

2010 • Presentations in national and international conferences • Candidate for Tester of the Year 2010 (TestausOSY – FAST) • TestausOSY/FAST founding member

• Education • ISTQB Advanced certified • SPICE provisionary assessor certified • M.SC, Helsinki University of Technology / Aalto-university • Marketing studies, University of Wisconsin-Madison

• Domain background • Embedded • Public sector • Pension insurance • Training provider • Telecom • Finance • Manufacturing

• Services • Training ISTQB Advanced and Foundation + Knowit Testing

Professional • Service management • Test management • Test assessments (TPI Next etc.) and test development • Agile testing services

© Knowit Oy / SECRET (Internal slide)

Twitter: @kkakkonen

LinkedIn: fi.linkedin.com/in/karikakkonen/

110

145 154

189

215 215 220

2007 2008 2009 2010 2011 2012 2013

Trusted ICT Services Partner in Nordics

Experts in 1800 Sweden, Finland, Norway,

Denmark, Russia and Estonia

Knowit AB is listed on the Nordic Exchange in Stockholm

Gro

up

re

ve

nu

e (

ME

UR

)

Case description

© Copyright Knowit Oy 2013 | Confidential | Version 1.0 4

Client

• Worldwide company with many products

• Top of the line in their business

• Consumer products (B2C)

• Established software testing, including automation

• Now wanted their first mobile app

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 5

Three perspectives of mobile solutions

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 6

Technology • Responsive web

• Mobile optimized web

• Native application

• Hybrid application

Target group • Consumers (B2C, B2B2C)

• Retailers and suppliers (B2B)

• Stakeholders

• Own personnel

Benefits • Growth in sales

• Customer loyalty

• Savings

Client choices

in bold

Mobile app scope

• To enable users to access the service also from mobile devices

• not only through the established web service

• Utilize mobile device’s location services heavily

• First platform - iOS

• For anonymity the actual functionality is here called just “activity”

• Supporting functionalities in a bit more detail, meaningful things for testing here

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 7

Expected challenges

• Connectivity to web service, integral part of the service

• Usability – will the vast user base like it?

• Can we automate?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 8

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 9

Functionalities of the application

Location • Activity

• Data

• Map

• Graphs

• Camera

Profile • Latest info

• Following

• Fans

• Shouts

• Edit Profile

Personal • Latest info

• Calendar

Other • Following

• Shouts

• Settings

Our team

• Compact scrum-team

• 2 developers in Sweden

• Project manager, 2 testers and 1 designer in Finland

• All participating in daily scrum meetings

• Customer committed to be in close contact with the team on daily basis

• Customer geographically dispersed

• Remote connection within team and to customer (Lync, Skype)

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 10

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 11

We piloted automation

Knowit test automation process used in the mobile project

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 12

Preparation & piloting Implementation Testing

Te

stin

g

Recording and

scripting

Test execution

New tests

Script maintenance

Reporting

Feasibility

study

Piloting

POC

Imp

lem

en

tatio

n

Training

1.version of

software

under test

© Copyright Knowit Oy 2013 | Confidential | Version 1.0

Feasibility study

Phase Description

• Benefits of automation in this project?

• Discussions about test automation and tools with customer

Proof Of

Concept

(PoC)

• Verifying that the test automation tool is compatible with application

under test

• Identifying parts that are impossible/no benefit to automate

• Installation of test tool to machines

• Placing the hooks to software

• Automating 1-2 test cases

• Investment calculations and workload estimations

• Scripting test cases

• Running test cases in development environment

• New versions of sofware under test in regular intervals

Piloting

• Execution and maintenance of test cases

• Scripting of new test cases

• Reporting

Implementation

Testing

Tool options

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 14

Ava

ilab

le t

oo

ls

MonkeyTalk

Sikuli

Testing with Frank

Xcode

Selenium

NativeDriver

Robot Framework

Junit

eggPlant for mobile

Silk4mobile

Pilo

ted

Robot Framework

Silk4mobile

Xcode

Chosen

Silk4mobile

Xcode

Criteria for selection

• Target operating system iOS

• Several iOS options (iPad, iPad mini, iPod, iPhone)

• Other operating systems in the future (Android, Windows)

• Cost-effectiveness

• Ease of installation and learning

• Existing experiences of use

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 15

Silk4mobile

• Supports all mentioned mobile platforms

• Knowit Silk4Mobile know-how

• Straightforward implementation

• Requires hooks in the software or a jailbreak

• Borland is a Knowit partner

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 16

Xcode

• Default Apple development environment

• Includes test automation environment

• Automation tests run with emulator

• Operational testing in emulator

• Used in Knowit as development and unit test environment

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 17

Project implementation

© Copyright Knowit Oy 2013 | Confidential | Version 1.0 18

Testing process

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 19

Development

Developers’ tests

Test team

• Smoke test

• UI testing

• Field testing

• Automation

Beta testing

Feedback

• Internal testing

• Beta testing

Early challenges

• Challenges of automation

• Tried out Silk4Mobile (Borland)

• Requires hooks within the app

• Part of iOS test automation requires ‘jailbreaking’

• iOS7

• Apple updated its operating system at the same time with this project -> retesting

• Other challenges

• Application needed a lot of field testing in true environment

• A lot of outdoor use

• The web service also updated during the project

• Web service interfaces were modified with changes to app behavior

• Downtimes in web service

• Testing consisted of interaction between app, location information from phone, and web service 21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 20

Test planning

• Division of testing tasks between testers and beta testers

• Testers continuously validated the correct functionality of the app features

• Beta testers were required to validate real life scenarios

• Levels of use

• Basic use: activities and following the web service

• Heavy use: more activities, following the web services, shares own information to social media, likes

and comments of friends

• Benchmarking: does app + web service work with same ease and content as web service only?

• Use of exploratory testing

© Copyright Knowit Oy 2013 | Confidential | Version 1.0

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 22

Test management - ChiliProject…

• User story - > test scenarios (different variety) - > field tests

• Incidents, bugs and other issues reported back to Chili

• Roadmap • Sprints • Features • Bugs • Status • Priority • Target version • Files • Reports

Implementation of tests B

asic

user

Indoor

Outdoor City location

Forest

On the water Sailing

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 23

Sp

ecia

l ca

se

s

Network is lost

At sea

In a forest

In the city Connection is lost

Web service is down Data is transferred after connection is restored

Beta testing – field testing

• After smoke tests app moved to beta testing stage

• Purpose of beta testing

• To get the actual end-user experience

• Real life use cases

• Achievements / what was good?

• Wide coverage

• Use cases : active and basic users, like-minded group of active users discussing with others online and at different events, continuously growing number activities, usage in different countries, various programs, all iPhone models plus iPad and iPod…

• Validation testing at the end of the project

• A group of test users, about 10-20 persons who were not involved in the project in any way, but had previously used the web service, from heavy to basic users in different countries

• Using mobile app normally and giving feedback

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 24

Summary

• Challenging start

• Challenging test environment in mobile testing

• Mobile test automation can be tricky, settled to unit test automation only

• Demanding usage environment in this project – field testing was a success

• Despite the challenges, comprehensive and versatile testing

• A total of 140 functionalities and user stories tested

• 445+ issues were reported to Chili, fixed and verified

• Especially happy customer and successful stable application

• Based on this project the customer ordered a new mobile application development and testing from

Knowit

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 25

Test tool survey reflections

© Copyright Knowit Oy 2012 | Luottamuksellinen

Mobiles roll…

Number of smartphones worldwide:

Q3 / 2012 1 000 000 000

2015 2 000 000 000

Source: Strategy Analytics

Test automation options

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 28

Mobile testing

Emulator

Remote device testing

• Devices as a service

Device specific

automation

Mobile testing software

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 29

Open source

• Robot framework

• Robotium

• JUnit / OCUnit

• Appium

• Calabash

• MonkeyTalk

• Sikuli

• Testing with Frankk

Commercial

• Borland Silk4Mobile

• Squish – Frologic

• Keynote DeviceAnywhere

• Testdroid

• Appurify

• EggPlant for mobile

The survey

© Copyright Knowit Oy 2013 | Confidential | Version 1.0 30

Background

• Research target: Software testing tools usage in Finland

• Research is a part of Master Degree of Business Informatics thesis for Metropolia in co-

operation with Knowit Oy

• Used research method was quantitative method with web-form survey

• Survey was sent to IT- and testing professionals in Finland

• Knowit customer contacts

• TestausOSY society

• Soliditet register of Bisnode Finland

• Survey was done in February 2013

• You could choose many options in most questions

© Copyright Knowit Oy 2013 | Confidential | Version 1.0

Respondent’s background

• Response rate was 39 % i.e. total 107 responses

• 64 % was men and 36 % women

• Age distribution; 84 % age between 31 to 50 years

• Education; highly educated, 86 % have university level education

• 87 % of respondents from South of Finland and 10 % from West of Finland

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 32

Profession/role in the company

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 33

38%

4% 6% 5% 5%

13%

3% 1%

3% 2% 3% 1% 1% 1%

12%

4%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Most of the respondents were test

managers/ test coordinators and system

testers

Operating area of the company

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 34

The largest operating areas were

Information and telecommunication,

Finance and insurance business and

Industry

Other service 12 %

Science or technical area 7 %

Finance and insurance business

21 %

Information and telecommunication

35 %

Retail business 2 %

Industry 16 %

International organization and

commission 1 %

Business area unknown

2 %

Mining 1 %

Administration and support

1 % Medical 2 %

Size of the company

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 35

2%

7% 5%

24%

14%

45%

1%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

1 - 10 51 – 100 11 - 50 101 – 500 501 - 1000 I don't know Over 1001employees

The respondents were mostly from

large companies

Current life cycle model in the project

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 36

5%

67%

13% 7%

21% 29%

50%

24%

11% 5%

8%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Current projects life cycle models

were Scrum, V-model and Iterative

model

Future life cycle model

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 37

18%

25%

4% 2% 5%

8% 7%

16%

40%

9%

30%

0%5%

10%15%20%25%30%35%40%45%

Scrum seems to be future

trend, Kanban and Lean are

following, V-model use is

going down

Which employees of your company are using software testing tools?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 38

72% 74%

89%

19% 24% 28% 25%

3% 9%

64%

6%

80%

49%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Testing tools are used by testing

professionals and developers

What kind of training do you have for software testing?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 39

5% 1%

25%

2% 1% 0% 0% 0% 7% 12% 15% 1% 2%

3% 6%

49%

4% 1% 4% 0% 0%

27% 24% 18%

3%

14%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Men

Women

74 % of respondents have ISTQB/ISEB Foundation

course, next most common are tool specific

courses and scrum master course.

What kind of testing tools are in use?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 40

88%

60% 48%

4% 0%

20%

40%

60%

80%

100%

Commercialsoftware

testing tools

Open Sourcetesting tools

Proprietarytools

Other

Tools from many sources are used

side by side

Which way the testing tools are acquired?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 41

40%

6% 3% 4%

25% 22%

0%5%

10%15%20%25%30%35%40%45%

Bought Cloudservice

Other way Rented OpenSource

Coded bycompany

itself

Normally testing tools are acquired

by buying. Next most common is

open source usage and self coding

According to this research cloud

services are not yet a popular

acquisition method..

Which way the testing tools are acquired by company size? (number of persons in the company)

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 42

1

1

2

1 - 10

5

5

2

2 1

11 - 50

5

4 2

2

2 1 Open

Source

Bought

12

22 3

13

2 3 101 - 500

6

14 2

7

1 1 501 - 1000

27

46 6

24

1 5

over 1000

51 – 100

What test management tools do you have in the company?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 43

1%

14%

9%

2%

54%

3% 2% 5%

2%

47%

1% 1% 1% 5%

1%

9%

0%

10%

20%

30%

40%

50%

60%

Most popular Test

management tools were HP:

QC, Atlassian: Jira and

TestLink

Test management- vs office-tools

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 44

1%

54%

9%

47%

0% 1% 1% 1% 1% 0% 0% 0% 5% 9% 1% 2% 0% 0% 5% 3% 2% 2% 1%

13%

8%

7%

2% 1% 1% 1%

1%

0%

10%

20%

30%

40%

50%

60%

70%

No

Yes

Also QC and Jira users are

utilizing often office tools (spread

sheets, word processors)

What incident management tools do you have in the company?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 45

41%

11%

3% 5%

3%

12%

25%

5%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

Atlassian:JIRA

IBMRational

ClearQuest

Polarion Proprietarytools

HP: QualityCenter

Bugzilla(Open

source)

Other Mantis

Atlassian: Jira is clearly the largest

in this category and second open

source tool Bugzilla – QC is

surprisingly very little used for

incident management.

What test execution tools do you have in the company?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 46

33%

7% 3% 4% 2% 1% 2% 2% 2% 2% 1% 3% 4% 5%

21%

2% 4%

29%

4% 4%

45%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

Open source tool Selenium was

most popular, next HP:QTP and open

source tool Robot Framework. There

are a large number of different tools

in use in this category.

What test design tools do you have in the company?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 47

1% 1% 1% 2% 2%

46%

2% 3%

8%

1% 4% 4% 2%

0%5%

10%15%20%25%30%35%40%45%50%

Major test design tool was HP: QC, note also

MBT tools and test technique tools (CTE,

allpairs)

What test harness/ unit test framework tools do you have in the company?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 48

27%

5%

17%

2% 3% 1%

12%

7% 7%

2%

7%

2%

0%

5%

10%

15%

20%

25%

30%

These tools are mostly used by

developers. Most used tools in this

category were CppUnit, Junit and .TEST

What specialized unit test frameworks tools do you have in the company?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 49

2% 2%

4%

1%

6%

1%

2%

5%

1%

0%

1%

2%

3%

4%

5%

6%

Specialized unit test frame work tools are

less used but most popular is JBehave

What performance/ load/ stress testing tools do you have in the company?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 50

1% 3%

7%

1%

4% 4%

7%

22%

7%

12%

4%

24%

0%

5%

10%

15%

20%

25%

30%

Most popular in this category

were open source tool Jmeter

and HP Performance Center.

Overall usage of all software testing tools

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 51

88

4 4 12

4 4

98

32

21

7 3 4 6

24 15

27

5

26

5 5 2

29

45

15

3 5 9

3

31

12

0

20

40

60

80

100

120

% o

f re

sp

on

de

nts

98 % have QC in use or have used and 88 % have

JIRA in different testing phases.

Purpose of software testing tools?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 52

79% 76%

80%

29%

48% 50%

77%

30%

4%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Reporting Testexecution

Planning testcases

Generatingtest cases

Requirementmanagement

Defecttracking

Defectcontrol and

management

Reviews Other

Typical testing activities have the lead in

this category, i.e. planning test cases,

reporting and test execution.

Do you use ‘office tools’ in testing (Spreadsheet, word processor etc.)?

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 53

35%

1%

17%

12%

1% 1%

10%

19%

1% 3%

0%

5%

10%

15%

20%

25%

30%

35%

40%

Reporting Requirementmanagement

Test data Test design Defect controland

management

Testmanagement

Test execution Documentation Design Calculation

• Yes, office tools in use 80 %

• Which functions are you using them for?

Office tools are used to same

functions as the software

testing tools

Testing tool markets directions

• Open source is gaining momentum, in midsize companies

• Big (commercial) test tool suites work for big companies

• Cloud (software testing tool as a service) is just emerging, but a growing trend

• Specialized tools, like mobile test automation, emerge all the time

• Are not first considered testing tools (or testers are not the users)

• Are soon swallowed by big test suite providers

• Integrated tool sets for multiple platforms are seen as most useful

• Support e.g. all mobile platforms and all browsers

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 54

Recommendations

• Pay attention to test tool life cycle – does it have future?

• Open source – the user and developer base

• Commercial – company size, market shares

• Consider tool integrations

• Mix and match open source and commercial

• Integrate tools to work together – e.g. Jira – QC – Robot Framework

• Look for good tools

• Superb usability

• Excellent technical flexibility

• Even small vendors grow soon with superb tools

• Word of mouth is big player (Finnish Robot Framework phenomena)

21.2.2014 © Copyright Knowit Oy 2013 | Confidential | Version 1.0 55

”There are plenty of software testing tools. Expertise is needed to select effective testing tools to suitable areas of testing processes”

© Copyright Knowit Oy 2013 | Confidential | Version 1.0 56

Questions?

© Copyright Knowit Oy 2013 | Confidential | Version 1.0

@kkakkonen or kari.kakkonen@knowit.fi