Mukul R. PrasadSoftware Systems Innovation group

Fujitsu Labs. Of AmericaSunnyvale, CA

A Grey-Box Approach for Automated

GUI-Model Generation of Mobile Applications

Wei YangDepartment of Computer Science,

North Carolina State University, Raleigh, North Carolina

Tao XieDepartment of Computer Science,

North Carolina State University, Raleigh, North Carolina

Growing popularity of Smartphone

2

Growing demand for high quality apps

Automated Testing Tools!

Solution?

3

GUI model

• GUI model is essential for developing automated testing tools– What does it represent?– Why is such a model useful/important for

testing?

4

Sample App: Simple TippyTipper

Used to calculate tip amount for a meal

MENU MENU

5

GUI Model for Simple TippyTipper

1 2

3

5a

4

5c 5d

5b

00

10

01

Calculate

Menu Menu

About

Settings

a1

a1

a2a2

a1: Toggle exclude tax rate option.a2: Toggle round up option.

11

（ 1） （ 2） （ 4）

（ 3） （ 5）（ 3）

6

Applications of GUI Model

• Classical Model-based testing applications– Automatic (system-level) test-case generation– Use as a driver for symbolic execution (SE) based testing

• More recent “crawl-and-compare” applications– Cross-platform compatibility testing of an application– Regression testing across different versions of the

application

7

 Related Work

• GUITAR: GUI Ripping for Windows, Java-based GUI (desktop) apps, Atif et al.

8

Click 1 Click

2Click

5Click

4

Click 3

Click 6

Click 9

Click 8

Click 7

Click 0

Click Calculat

e

Click DEL

Click CLEAR

1 2

3

5a

4

5c 5d

5b

00

10

01

Calculate

Menu Menu

AboutSettings

a1

a1

a2a2

a1: Toggle exclude tax rate option.a2: Toggle round up option.

11

A simplified event-flow model of simple tippytipper. Our GUI model of simple tippytipper.

 Related Work(Cont.)

• Concolic Testing (SE) of Android Applications, Saswat et al.

• Random Testing of Android Applications, Cuixiong et al.

• Crawljax: Crawling of AJAX Web Applications, Ali et al.

• AndroidRipper: Stress Testing of Android Applications, Domenico et al

9

ORBIT GUI Crawler: Overview

10

Automated Analysis

(including source code)

Android Apps

GUI Model

ORBIT GUI Crawler: Overview

11

Android AUT

Action inference Dynamic Crawler

GUI ModelORBITAction

Mapping

(source code)

Proposed GUI Model

• Visual Observable State– Inspired by the UI-design principles espoused by

the Android team. – Composition of the state

• Model– A finite-state machine over visual observable

states with the user actions constituting the transitions between these states

12

GUI Model for Simple TippyTipper

1 2

3

5a

4

5c 5d

5b

00

10

01

Calculate

Menu Menu

About

Settings

a1

a1

a2a2

a1: Toggle exclude tax rate option.a2: Toggle round up option.

11

（ 1） （ 2） （ 4）

（ 3） （ 5）（ 3）

13

Illustration: State

14

Two different states of Screen # 5b & 5d

These two states differ in the state of this widget

（ 5b） （ 5d）

Action Inference

R.Id.java

View btn_delete = findViewById(R.id.btn_delete);

Btn_delete.setOnClickListener(new onClickListener() { public void onClick(View v) { removeBillAmount(); FlurryAgent.onEvent(“Delete Button”); } });

Btn_delete.setOnClickListener(new onLongClickListener() { public void onLongClick(View v) { clearBillAmount(); return true; } });

……

TippyTipper.java

Inference: Widget btn_delete with Id = 0x7f0000a supports actions click and longClick

• Approach:– Locate all instances of event handler definitions in the code (sinks)– Associate each with the corresponding widget instantiations (sources)

15

Simple TippyTipper: Actions Found

• ScreenActionMapping– (net.mandaria.tippytipper.activities.TippyTipper, Menu)– (net.mandaria.tippytipper.activities.Total, Menu)

• WidgetActionMapping – (R.id.btn_delete, click)– (R.id.btn_delete, longclick)– (R.id.btn_clear, click)– (R.id.btn_clear, longclick)

16

MENUBACK

Activity Total

Dynamic Crawling: Illustration

17

Activity TippyTipper

MENU

Activity Setting

SETTINGS

BACK

Toggle

Toggle

Toggle

Toggle

Toggle

Calculate

Toggle

BACK

ToggleToggle

Tool implementation

FwdCrawl Algorithm

Robotium

Android Runtime

Dynamic Crawler

Action Detector

WALA

Intent Passing Logic

Sub-CallGraph

Partial Connected Call Graph

Inference Algorithm

Action MappingAndroid

AUT

sour

ce c

ode

ORBIT

GUI Modelde

ploy

18

Research question

• RQ1: Is the proposed GUI crawling algorithm more efficient than a standard depth-first state-traversal algorithm?

• RQ2: Are the widget and screen actions inferred by static analysis effective in enhancing the behavior covered by the generated model?

• RQ3: Can our tool generate a higher-quality model, more efficiently, compared to other state-of-the-art techniques?

19

Subjects used in the evaluation

Subject #LOC

#A

Category Purpose

TippyTipper 2238 5 Tool Dining tip calculator

OpenManager 1595 6 Business File manager for Android

Notepad 322 3 Productivity Note creation and management

TomDroid 3711 3 Business Online note-reading

Aarddict 4518 4 Books Aard Dictionary for Android

HelloAUT 234 1 Entertainment

Color shapes

ContactManager

497 2 Productivity Contacts manager

ToDoManager 323 2 Productivity Create and manage task lists

20

• Our evaluation is based on subjects drawn from existing related tools• We avoid bias by including all subjects used to evaluate

Android GUITAR and GUI Ripper in previous work.

Crawling algorithm comparison

Subjects FwdCrawl DFS

Time(sec)

Cov. (%)

#Fwd

Back

Time(sec)

Cov. (%)

#Fwd

Back

TippyTipper 198 78 61 15 512 82 134 52

OpenManager 480 63 92 18 822 56 209 29

Notepad2 102 82 25 4 147 83 39 12

Notepad0 80 78 18 2 75 71 15 2

TomDroid 340 70 78 23 459 58 61 8

AardDict 173 65 15 2 397 60 20 8

HelloAUT 156 86 46 0 278 85 61 0

ContactManager

125 91 20 1 137 92 22 2

ToDoManager 178 75 60 2 294 74 84 4

21

• DFS takes 70% more time to traverse all 9 subjects together

Validity of action inference

Subjects #clicks

#longClicks

#menu #States

Coverage(%)

C C+I

C C+I C C+I

C C+I

C C+I

TippyTipper 21 55 - 2 - 4 3 9 47 78

OpenManager 50 67 - 19 - 4 10 20 39 63

Notepad2 2 13 - 3 - 9 2 7 39 82

Notepad0 0 8 - 1 - 9 0 7 14 78

TomDroid 3 52 - 0 - 26 2 9 36 70

AardDict 4 15 - 0 - 7 3 7 43 64

HelloAUT 15 34 - 0 - 12 4 8 53 86

ContactManager

20 20 - 0 - 0 5 5 92 92

ToDoManager 60 60 - 0 - 0 7 7 76 7622

• Firing non-click actions during crawling increases the coverage by 34% on average.

• Our crawler is 32%-75% faster while constructing a 5%-140% more complete model than Android GUITAR and Android GUI Ripper.

Comparison with other toolsSubjects Monkey Android

GUITARAndroid GUI

RipperORBIT

Time(sec)

Cov.(%)

Time (sec)

Cov. (%)

Time (sec)

Cov.(%)

Time (sec)

Cov.(%)

TippyTipper 83 41 322 47 - - 198 78

OpenManager 90 29 - - - - 480 63

Notepad2 127 60 - - - - 102 82

Notepad0 122 59 - - - - 80 78

TomDroid 69 46 - - 529 40 340 70

AardDict 124 51 - - 694 27 173 65

HelloAUT 98 71 117 51 - - 156 86

ContactManager

90 53 247 61 - - 125 91

ToDoManager 115 71 194 71 - - 178 75

23

Conclusions & Future Work

• Crawling algorithm– Incorporate the context of event call to prune the order of the

sequence.• GUI model

– Explore the possibility to generate a complete specification or test oracle by using our driver and partial specification.

• Selection of subjects– Inter-application communication of Android application.

• Manual effort

24

Thanks!

*This project was supported in part by NSF grants CCF-0845272, CCF-0915400, CNF-0958235, CNS-1160603, and an NSA Science of Security Lablet Grant.