Automated Concolic Testing

of Smartphone Apps

Saswat AnandStanford Univ.

Mayur NaikGeorgia Tech.

Hongseok YangUniv. of Oxford

Mary Jean HarroldGeorgia Tech.

Motivation

Motivation

Problems with Smartphone Apps

Automatically generate test inputs for bounded exhaustive testing of

smartphone apps

Problem

Test Inputs for Apps

• Whole-program testing• Test input is a sequence of events e1, e2…,

en

• Types of events: a tap on the screen, change in geo-location, arrival of a SMS message, etc.

Bounded Exhaustive Testing of Apps

S, the set of all event sequences* s.t. each sequence

takes a unique path

*of bounded-length

Set of covered

branchesGoal: cover

these

1. Generate individual events2. Generate sequences of events

Two subproblems

Generating Individual Events

• An event is associated with datao X & Y coordinates of a tap evento geo-location of a change-in-geo-location evento content of an incoming SMS evento etc.

• Data determine which program path is taken

Challenge: Generate the “right” data for events

Example: Music Player App

Play Pause

Stop Eject

Rewind Skip

Example: Music Player App

public void onClick(View target) { if (target == play) startService(new Intent(ACTION_PLAY)); else if (target == pause) startService(new Intent(ACTION_PAUSE)); else if (target == skip) startService(new Intent(ACTION_SKIP)); else if (target == rewind) startService(new Intent(ACTION_REWIND)); else if (target == stop) startService(new Intent(ACTION_STOP)); else if (target == eject) showUrlDialog();}

tap(136, 351)

Example: Music Player App

public void onClick(View target) { if (target == play) startService(new Intent(ACTION_PLAY)); else if (target == pause) startService(new Intent(ACTION_PAUSE)); else if (target == skip) startService(new Intent(ACTION_SKIP)); else if (target == rewind) startService(new Intent(ACTION_REWIND)); else if (target == stop) startService(new Intent(ACTION_STOP)); else if (target == eject) showUrlDialog();}

tap(248, 351)

Example: Music Player App

public void onClick(View target) { if (target == play) startService(new Intent(ACTION_PLAY)); else if (target == pause) startService(new Intent(ACTION_PAUSE)); else if (target == skip) startService(new Intent(ACTION_SKIP)); else if (target == rewind) startService(new Intent(ACTION_REWIND)); else if (target == stop) startService(new Intent(ACTION_STOP)); else if (target == eject) showUrlDialog();}

tap(360, 351)

Example: Music Player App

public void onClick(View target) { if (target == play) startService(new Intent(ACTION_PLAY)); else if (target == pause) startService(new Intent(ACTION_PAUSE)); else if (target == skip) startService(new Intent(ACTION_SKIP)); else if (target == rewind) startService(new Intent(ACTION_REWIND)); else if (target == stop) startService(new Intent(ACTION_STOP)); else if (target == eject) showUrlDialog();}

tap(24, 351)

Example: Music Player App

public void onClick(View target) { if (target == play) startService(new Intent(ACTION_PLAY)); else if (target == pause) startService(new Intent(ACTION_PAUSE)); else if (target == skip) startService(new Intent(ACTION_SKIP)); else if (target == rewind) startService(new Intent(ACTION_REWIND)); else if (target == stop) startService(new Intent(ACTION_STOP)); else if (target == eject) showUrlDialog();}tap(136, 493)

Example: Music Player App

public void onClick(View target) { if (target == play) startService(new Intent(ACTION_PLAY)); else if (target == pause) startService(new Intent(ACTION_PAUSE)); else if (target == skip) startService(new Intent(ACTION_SKIP)); else if (target == rewind) startService(new Intent(ACTION_REWIND)); else if (target == stop) startService(new Intent(ACTION_STOP)); else if (target == eject) showUrlDialog();}tap(305, 544)

Existing alternatives• Random Testing

oCannot perform systematic/exhaustive testing

• Platform-specific tools (e.g., hierarchy viewer in Android)oLimited to GUI EventsoCannot handle third-party GUI widgets

Generating Individual Events

Generating Individual Events

Our solutionUse concolic execution to generate data

associated with events

FT

FT

tap(int x, int y){1 if (x>2 && x<4){2 if (y>1 && y<3)3 W1_clicked();4 else5 W2_clicked();6 }else7 W3_clicked(); }

Generating Individual Tap Events

1

72

3 5

x>2 && x<4

y>1 && y<3

Generating Individual Tap Events

tap(1, 5)

FT

FT1

72

3 5

x>2 && x<4

y>1 && y<3

Generating Individual Tap Events

tap(1, 5)

F1 !(x>2 && x<4)

W3_clicked()

FT

FT1

72

3 5

x>2 && x<4

y>1 && y<3

(x>2 && x<4)

tap(3, 5)

(x>2 && x<4)

Generating Individual Tap Events

tap(1, 5)

T1 (x>2 && x<4)F2 !(y>1 && y<3) W2_clicked()

tap(3, 5)

FT

FT1

72

3 5

x>2 && x<4

y>1 && y<3

(x>2 && x<4)(y>1 && y<3)

tap(3, 2)

Generating Individual Tap Events

tap(1, 5)

(x>2 && x<4)(y>1 && y<3)

tap(3, 5) tap(3, 2)

T1 (x>2 && x<4)T2 (y>1 && y<3) W1_clicked()

FT

FT1

72

3 5

x>2 && x<4

y>1 && y<3

Example: Music Player App

❖

❖

❖

❖ ❖❖

❖ ❖

❖❖

❖

1. Generate individual events2. Generate sequences of events

Two subproblems

Generating Sequences of Events

Concatenate individual events generated by concolic execution.

Baseline Algorithm

Set of covered

branches

S, Set of all event sequences s.t. each sequence takes a

unique pathBaseline

algorithm

Goal: cover these

Number of sequences generated for Music Player app by baseline algorithm

Baseline Algorithm Suffers from Path Explosion

1 2 3 40

5000

10000

15000

20000

25000

ACTEve Algorithm

ACTEve: Automated Concolic Testing of Event-driven programs

ACTEve Algorithm

Set of covered

branches

R s.t. R S⊆

S, Set of all event sequences s.t. each sequence takes a

unique pathBaseline

algorithm

ACTEve algorithm

Goal: cover these

ACTEve is relatively sound

Path Subsumption

¿ γ ,C>¿Maps memory location to values (symbolic or concrete)

Path constraint

Program state in concolic execution

Program entry

Path Path

¿ γ1 ,𝐶1>¿ ¿ γ2 ,𝐶2>¿

subsumes

Path Subsumption

Note - memory map – path constraint

Program entry

Path Path

¿ γ1 ,𝐶1>¿ ¿ γ2 ,𝐶2>¿

subsumes

Path Subsumption

Note - memory map – path constraint

- Don’t generate test corresponding to any path that is an extension of - Only generate tests corresponding to paths that are extension of

• Checking path subsumption is very expensive in generalo Constraint implication checkoMatching memory map

• But, path subsumption can be checked cheaply in special caseso Read-only eventso Events whose mutual ordering does not mattero etc.

Path Subsumption

event is does not write to any memory location.

Read-only Events

Program Entry

corresponds to

corresponds to

Path executed for event sequence

is subsumed by q

❖

❖

❖

❖❖ ❖❖❖

❖ ❖❖

Read-only Events

Read-only events are represented as ❖

ACTEve System Architecture

Empirical Study

• Apply ACTEve and baseline algorithmso event sequences of length up to 4o 16 concurrently running emulators o time budget of 12 hours

• Measured three metricso running timeo number of feasible pathso number of satisfiability checks

Empirical Results

Future Work

Widget Explosion

1. Concolic execution to generate individual events

2. ACTEve: an efficient algorithm for bounded exhaustive testing of event-driven programs o Requires only a small fraction (5-36%) of time

compared to baseline algorithm

3. Implementation for Android

Main Contributions

Backup slides

¿ γ 1 ,𝐶1>¿

¿ γ 2 ,𝐶1∧𝐶′>¿

1. because does not write to any memory location.

Read-only Events

Program Entry

corresponds to event sequence

corresponds to in

Path executed for input event sequence

Output of Android’s “Hierarchy Viewer” tool

A Solution: Use Platform-specific Knowlege

A Solution: Use Platform-specific Knowlege

void onTouchEvent(MotionEvent e) { int rawX = (int) e.getX(); int rawY = (int) e.getY(); int x = (rawX – MARGIN) / SIZE; int y = (rawY – MARGIN) / SIZE; if (x >= 0 && x < 3 && y >= 0 & y < 3) { int cell = x + 3 * y; … }

Output of Android’s “Hierarchy Viewer” tool

Program Entry

Path

{is feasible} {is feasible}

Path

Program Entry

Covered branches

Covered branches

same program location

Path Subsumption

Program Entry

Path

{is feasible} {is feasible}

Path

Program Entry

Covered branches

Covered branches

same program location

if we explore all paths that extends , then no need to explore any path that extends because no additional branch coverage will be obtained.

Path Subsumption

Path constraint when PAUSE button is tapped on

Example: Music Player App