random testing
DESCRIPTION
Random Testing. Tor Stålhane Jonas G. Brustad. What is random testing. The principle of random testing is simple and can be described as follows: For each input parameter, generate a random but legal value. Apply the full set of inputs to the SUT Register the result and go b ack to step 1. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/1.jpg)
Random Testing
Tor StålhaneJonas G. Brustad
![Page 2: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/2.jpg)
What is random testing
The principle of random testing is simple and can be described as follows:1.For each input parameter, generate a random but legal value.2.Apply the full set of inputs to the SUT3.Register the result and go back to step 1.
![Page 3: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/3.jpg)
Chen’s observation
Inputs that are close to each others in the input domain tends to go through the same path.
Thus, in order to find most of the errors, we should spread the test cases as much as possible.
This approach is called Adaptive Random Testing. We will look at four approaches:• Partition Adaptive Random Testing• Basic Random Testing – RT • Basic Adaptive Random Testing – ART• Mirror Adaptive Random Testing – MART
![Page 4: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/4.jpg)
Block failure pattern
![Page 5: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/5.jpg)
Strip failure pattern
![Page 6: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/6.jpg)
Point failure pattern
![Page 7: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/7.jpg)
Some notation • D: input domain size• n: number of test cases• m: number of tests that fail• F = D/m. Note that– Large F => small m => few errors detected– Small F => large m => many errors detected
• = 1/F – failure rate • Frel = Fobs *
![Page 8: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/8.jpg)
Partition Adaptive Random Testing
Tor StålhaneJonas G. Brustad
![Page 9: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/9.jpg)
ART by random partitioning – 1 Algorithm for a two-dimensional case1.Start with C = {(Xmin, Ymin), (Xmax, Ymax)}
2.Draw a random point in C = (X1, Y1). This will split C into four regions – R1, R2, R3 and R4. Select T = max area {R1, R2, R3, R4}See next slide.
![Page 10: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/10.jpg)
ART by random partitioning – 2 (Xmax, Ymax)
{(Xmin, Ymin),
(X1, Y1)
R1 T = R2
R3 R4
Select a test (X1, Y1) in T. If it is a failure, report the failure and stop.Otherwise, split T in the same way as we split C – see nextslide
![Page 11: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/11.jpg)
ART by random partitioning – 3
Select a test (X2, Y2) in T. If it is a failure, report the failure and stop.Otherwise, repeatthe process
(Xmax, Ymax)
{(Xmin, Ymin),
(X1, Y1)
R1 T
R3 R4
(X2, Y2)
![Page 12: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/12.jpg)
ART by bisection – 1 Algorithm for a two-dimensional case1.Start with C = {(Xmin, Ymin), (Xmax, Ymax)}
2.Draw a random test in C – (X1, Y1). If it fails we are finished. Otherwise split C in two equal parts – see next slide.
![Page 13: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/13.jpg)
ART by bisection – 2 (Xmax, Ymax)
{(Xmin, Ymin),
(X1, Y1)
Select a test (X2, Y2) in the untested half of C. If it is a failure, report the failure and stop.Otherwise, split C again – see nextslide
(X2, Y2)
![Page 14: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/14.jpg)
ART by bisection – 3
Select a test (X2, Y2) in T. If it is a failure report, the failure and stop.Otherwise, repeatthe process for eachpart that we havenot tested – (X3, Y3) and (X4, Y4)
(Xmax, Ymax)
{(Xmin, Ymin),
(X1, Y1)
(X2, Y2)
(X4, Y4)
(X3, Y3)
![Page 15: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/15.jpg)
The exclusion factor – 1
All types of Adaptive Random Testing (ART) can be improved by introducing the exclusion factor, usually denoted by f. This factor will force the new tests away from the tests that have already been run.The optimal factor value will vary, depending on the failure rate and on the failure pattern – block, strip or point.
![Page 16: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/16.jpg)
The exclusion factor – 2
![Page 17: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/17.jpg)
The exclusion factor – 3
Based on this, we have chosen f = 0.4 as the best value.
![Page 18: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/18.jpg)
Comparisons
![Page 19: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/19.jpg)
Basic Adaptive Random Testing
Tor StålhaneJonas G. Brustad
![Page 20: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/20.jpg)
Fixed Size Candidate Set - FSCS
![Page 21: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/21.jpg)
Max distance – 1
Let a and b be two n-dimensional inputs {a1, a2, ..., an} and {b1, b2, ..., bn}.
m
iii babadist
1
2)(),(
E.g. in a two-dimensional space we have the twoparameters a ={1, 2} and b = {2, 5}. Then we havedist(a, b) = sqrt(1 + 9) = 3.16
![Page 22: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/22.jpg)
Max distance – 2 Let T and C be two disjoint sets T = {t1, t2, ...tn} is the set of executed testsC = {c1, c2, ...ck} is the candidate set.Find the ch that satisfies:
),(min),(min},,...2,1{ 11 ijniih
ni tcdisttcdistkjfor
This criterion will spread the test cases evenly by finding the largest minimum distance betweenthe next test case – selected from C – and the already executed test cases in T.
![Page 23: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/23.jpg)
Max distance algorithm
![Page 24: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/24.jpg)
A small example – 1 We have two data sets:•T = {(1, 1), (3,4)} – already executed tests•C = {(1, 2), (3, 1)} – candidate test set
3
2
1
1
2 3
t1c2
c1
4 t2
![Page 25: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/25.jpg)
A small example – 2
Using the max distance algorithm we get:•j = 1 => (c1, t1) dist = 1.0, (c1, t2) dist = 2.8
•j = 2 => (c2, t1) dist = 2.0, (c2, t2) dist = 3.0
•min(dist) = 1.0 and the first distance larger than min(dist) is 2.0 => Next test is c2 = (3, 1)
![Page 26: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/26.jpg)
Test comparison – 1
Defect types seeded:• AOR: Arithmetic Operator Replacement• ROR: Relational Operator Replacement• SVR: Scalar Variable Replacement• CR: Constant Replacement
![Page 27: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/27.jpg)
Results with RT
1
![Page 28: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/28.jpg)
Results with ART (a) and FSCS (r)
N
![Page 29: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/29.jpg)
Test comparison – 2
r
ar
F
FFI
![Page 30: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/30.jpg)
Mirror Adaptive Random Testing
Tor StålhaneJonas G. Brustad
![Page 31: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/31.jpg)
The problem with ART
All versions of ART require a large amount of computations due to the distance calculations and comparisons.The MART – Mirror ART – is simpler and requires less computation.
![Page 32: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/32.jpg)
The MART procedure The procedure has four steps:1.Partition the input domain into m disjoint subdomains. One is chosen as source subdomain. The rest are mirror subdomains 2.Apply the D-ART process to generate the next test case from the source subdomain. Execute this test case and quit if we find a defect.3.Apply the mirror function to the test case from step 2 to generate a test case for each mirror subdomain. Execute the test cases in sequential order and stop when we find a defect.4.Repeat steps 2 and 3 until finding the first failure or until reaching the stopping condition.
![Page 33: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/33.jpg)
Mirror partitioning Below we see several ways to create mirror partitions:X2Y1 => X is bisected, y is unchangedX2Y2 => both X and Y are bisectedX4Y2 => X is split into four parts, Y is bisectedX4Y1 => X is split into four parts, Y is unchanged
![Page 34: Random Testing](https://reader035.vdocuments.us/reader035/viewer/2022062408/56813daf550346895da77697/html5/thumbnails/34.jpg)
The D-ART process
),(min),(min},,...2,1{ 11 irniij
ni EcdistEcdistkrfor
1. Set E to be the empty set2. Select a random test case from the input domain
and execute it. If no failure, add the test case to E, otherwise stop.
3. Construct C = {c1, c2,…, ck}, where all ci are randomly selected and E and C are disjoint.
4. Let n = |E| and select cj so that
5. Repeat steps 3 ad 4 until first defect is found