exploiting the saturation effect in automatic random testing of android applications

Migrating Legacy Spreadsheets-based Systems to Web MVC Architecture: an Industrial Case Study

Exploiting the Saturation Effect in AutomaticRandom Testing of Android ApplicationsDomenico Amalfitano,Nicola Amatucci,Anna Rita Fasolino, Porfirio Tramontana Emily Kowalczyk,Atif M. Memon

MOBILESoft 2015 - 2nd ACM International Conference on Mobile Software Engineering and Systems

1

Context and MotivationsContext Automatic GUI Random Testing of Mobile Applications.

MotivationTo formalize a stopping criterion for random testing.MOBILESoft 2015 Florence, Italy May 162

Exploiting the Saturation Effect in Automatic Random Testing of Android Applications

GUI Random Testing TechniquesA.K.A. Monkey Fuzz Testing Techniques.Usually exploited for stress testing.Sequences of random events are sent to the application under test (AUT).Generally only user events (i.e. mouse or keyboard events).Fully automated.MOBILESoft 2015 Florence, Italy May 163


MFT ToolsTools implementing MFT techniques.Easy to use growing popularity for testing mobile applications.

4MOBILESoft 2015 Florence, Italy May 16https://github.com/reverse-unina/AndroidRipper


The Problem When the execution of an MFT tool must be stopped?The stop condition is:able to affect the effectiveness and the cost of automatic testing processes.even more important with random testing techniques, affected by problems of reliability and efficiency of the test adequacy.Covered Statements Percentage usually exploited to measure the test adequacy of testing techniques.

MOBILESoft 2015 Florence, Italy May 165


Mostly Used Stop Criteria Predefined testing time or number of fired events.Low #Events (1) Low Reliability, (2) Low EffectivenessNumber of fired EventsCovered Statement PercentageMOBILESoft 2015 Florence, Italy May 166


Mostly Used Stop CriteriaFix the testing time or the number of fired events.Too many Events Low EfficiencyNumber of fired Events7Covered Statement PercentageMOBILESoft 2015 Florence, Italy May 16


The proposed solution A testing process that automatically stops its execution at a point representing an optimal trade-off between effectiveness and cost.A point were test adequacy is maximized and no testing effort is wasted.It exploits two typical phenomena of random testing.

8MOBILESoft 2015 Florence, Italy May 16


First phenomenon1The existence of a saturation effect in MFT.A well-known phenomenon where the test adequacy tends to a specific value as the amount of test execution increases. There is a point at which no more progress is made towards maximizing a test adequacy criterion. It is called saturation point. We propose it as the optimal termination point.

91- E. Sherman, M. B. Dwyer, and S. Elbaum, Saturation-based testing of concurrent programs, SIGSOFT 09MOBILESoft 2015 Florence, Italy May 16


Second phenomenon2 The predictability property of MFTThe difference between the test adequacies reached by two random test cases tend to be definitely zero.In other words, two test cases tend to reach the same test adequacy (i.e., same found faults or same set of executed LOCs )102- A. Arcuri, M. Z. Iqbal, and L. Briand, Random testing: Theoretical results and practical implications, IEEE Trans. Softw. Eng.MOBILESoft 2015 Florence, Italy May 16


The proposed solutionThe process exploits the definition of predictability to reach a potential saturation point.The determination of the potential saturation point is based on the simultaneous execution of several random testing sessions. The testing process is stopped when the difference in code coverage is zero.When the sessions have executed the same code.11MOBILESoft 2015 Florence, Italy May 16


Graphical interpretation of the solution12MOBILESoft 2015 Florence, Italy May 16Saturation Effect Saturation PointPredictability differences in code coverage tend definitely to zeroUnreliability Zone


Differences in code coverageCovered Statements Percentage (CSP) as test adequacy.All the sessions have actually covered the same statements of the AUT all the sessions reach a test adequacy value equals to the Cumulative Coverage Statement Percentage (CSCP).13MOBILESoft 2015 Florence, Italy May 16

CSP = 40%

CSP = 40%

CSCP = 70%Sessions have covered different code


Differences in code coverageCovered Statements Percentage (CSP) as test adequacy.All the sessions have actually covered the same statements of the AUT all the sessions reach a test adequacy value equals to the Cumulative Coverage Statement Percentage (CSCP).14MOBILESoft 2015 Florence, Italy May 16

CSP = 55%CSP = 55%CSCP = 55%Sessions have covered the same code


The algorithmMobilesoft 2015 Florence, Italy May 16

Launch of k testing sessionsWhile the k sessions do not reach the same coverageEach session fires a new random eventThe termination condition is evaluated at predefined sampling steps

15


TerP and TerLThe algorithm produces two outputs:Termination Point (TerP): the minimum number of events at which the termination condition is verified.Termination Level (TerL): the coverage statement percentage reached by all the k testing sessions.Mobilesoft 2015 Florence, Italy May 16(TerP = 3700, TerL = 85%)16Number of fired EventsCovered Statement Percentage


Testing infrastructureImplemented for executing the testing process.It targets the Android mobile platform.

Mobilesoft 2015 Florence, Italy May 1617


Experimentation RQsExperiment aimed at answering the following research questions:RQ 1: Is the proposed testing process able to reach the Saturation at the termination point?RQ 2 : Which are the main factors able to affect the effectiveness of the proposed testing process at the termination point?Mobilesoft 2015 Florence, Italy May 1618

Why TerL 100% ?


Experimentation - Subjects18 open source Android applications were selected.

19


Experimentation - MetricsA set o metrics were proposed to assess the Saturation effect at the termination point.RePRS it is the Residual Percentage of Reachable Statements by the process at the termination point.If RePRS = 0 (or approximately equal to zero) then all the reachable code of the application has been actually covered.

Executed Code

Not Executed Code20MOBILESoft 2015 Florence, Italy May 16


Experimentation - MetricsA set o metrics were proposed to assess the Saturation effect at the termination point.RePRS it is the Residual Percentage of Reachable Statements by the process at the termination point.If RePRS = 0 (or approximately equal to zero) then all the reachable code of the application has been actually covered.

Executed Code

Unreachable CodeReachable Code21MOBILESoft 2015 Florence, Italy May 16


Experimental ProcedureAndroidRipper was exploited as MFT tool.It was configured for:sending events on the GUI widgets having at least a registered listener, for emulating the pressure of the back and openMenu buttons and the device rotation;filling in the EditText widgets with random numeric values;firing consecutive events with a delay of 500 ms;For each AUT:a specific initial precondition was defined;k=12 testing sessions were executed.All the testing sessions were executed on Android Virtual Devices (AVD) having the same configuration.

22MOBILESoft 2015 Florence, Italy May 16


ResultsTo answer the RQs the code statements left uncovered by the testing process were manually analyzed.Mobilesoft 2015 Florence, Italy May 16

23


Discussion of the resultsRQ 1 answer: in all the cases the proposed testing process was able to reach the Saturation at the termination point.RQ 2 answer: the main factors affecting the effectiveness of the proposed testing processes were:the preconditions of the AUT,the configurations of the testing platform, the limitations and the configuration of the MFT tool, and the existence of unreachable statements in the source code of the AUT.

Mobilesoft 2015 Florence, Italy May 1624


Lesson learned We were able to derive an alternative and more efficient stop condition.The data suggested that the process could be stopped, without loss of code coverage, as soon as the coverage of one of the sessions reached the cumulative one.Mobilesoft 2015 Florence, Italy May 1625

Dont wait that the slowest session covers the same code executed by the other ones.

The process terminates when the fastest session covers also all the code executed by the other ones.

what it means in practice


Lesson learnedTo validate this intuition, we evaluated the new Termination points and Termination Levels.The reduction rate varies between 24.48% and 93.58%.The same TerL values were obtained.Mobilesoft 2015 Florence, Italy May 1626


ConclusionsA fully automatic MFT process that is able to find a cost-effective termination point was presented.The process is supported by a software infrastructure we developed for testing Android mobile applications. The validity of the approach was shown by an experiment.27Mobilesoft 2015 Florence, Italy May 16Future WorksImprove the experimentation in order to extend the validity of the proposed approach in mobile context.Extend the validity of the approach in other contexts, i.e. desktop applications.


Thank you !!!Mobilesoft 2015 Florence, Italy May 1628
