test de conformité - laboratoire de recherce en...

Test de Conformité

Burkhart WolffUniversité Paris-Saclay

Test de Conformité

Burkhart WolffUniversité Paris-Saclay

05/12/17 B. Wolff, Test de Conformance 2

Software Testing can be formal too

• « We know less about the theory of testing, which we do often, than about the theory of program proving, which we do seldom »

Goodenough J. B., Gerhart S., IEEE Transactions on Software Engineering, 1975


Software Testing can be formal too

• « We know less about the theory of testing, which we do often, than about the theory of program proving, which we do seldom »

Goodenough J. B., Gerhart S., IEEE Transactions on Software Engineering, 1975


Relevance

Why is it important to get software right?

? ? ?


Relevance

Why is it important to get software right?

? ? ?


Relevance

• Since information technology becomes more pervasive, the risks become more important– Reliability, Safety and Security becomes more critical :

• transport systems (Cars, Métros, TGV), aviation controls, aerospace, ...

• critical industriel processes, nuclear power plants, weapons• medical technologies: tele-surgery, radiation control…• critical telecommunication infrastuctures and networks, • electronic commerce (SAP)


Relevance


• transport systems (Cars, Métros, TGV), aviation controls, aerospace, ...

• critical industriel processes, nuclear power plants, weapons• medical technologies: tele-surgery, radiation control…• critical telecommunication infrastuctures and networks, • electronic commerce (SAP)


Relevance


• transport systems (Cars, Métros, TGV), aviation controls, aerospace, …

• critical industriel processes, nuclear power plants, weapons• medical technologies: tele-surgery, radiation control…• critical telecommunication infrastuctures and networks, • electronic commerce (SAP, ATOS)

This should be a sufficient reason, but actually, it isn't.


Relevance


• transport systems (Cars, Métros, TGV), aviation controls, aerospace, …

• critical industriel processes, nuclear power plants, weapons• medical technologies: tele-surgery, radiation control…• critical telecommunication infrastuctures and networks, • electronic commerce (SAP, ATOS)

This should be a sufficient reason, but actually, it isn't.


Relevance

• The more likely reason is:

it is so expensive if you don't !!!

50 % of the overall costs were spent for test and verification in large software projects ... So, if the development of MS Vista cost 8 billion $ ...


Relevance

• The more likely reason is:

it is so expensive if you don't !!!

50 % of the overall costs were spent for test and verification in large software projects ... So, if the development of MS Vista cost 8 billion $ ...


Relevance in SE Processes

• Another reason is:

We want to build more complex systems, and validation and verification techniques are a limiting factor!

We simply can't do it without !



• Another reason is:

We want to build more complex systems, and validation and verification techniques are a limiting factor!

We simply can't do it without !


Relevance

RequirementAnalysis

Design

Coding Phase

Integration

Deployment

!

!


Relevance

RequirementAnalysis

Design

Coding Phase

Integration

Deployment

!

!



RequirementAnalysis

Design

Coding Phase

Integration

Deployment

!

!

RequirementAnalysis

Design

Coding Phase

Integration

Deployment

!

!



RequirementAnalysis

Design

Coding Phase

Integration

Deployment

!

!

RequirementAnalysis

Design

Coding Phase

Integration

Deployment

!

!



• Yet another reason is:– In an industrial setting, you might be

bound by laws to provide reasonable(i.e. formal) documents …

See Microsoft Windows Server Monopoly Case...



• Yet another reason is:– In an industrial setting, you might be

bound by laws to provide reasonable(i.e. formal) documents …

See Microsoft Windows Server Monopoly Case...



• Example in industrial Practice: eg.W Grieskamp:„Microsoft's Protocol Document Program: A Succes-Story for Model-Based Testing“. TestCom/Fates 09– 222 protocols/technical documents tested– 22,847 pages studied and converted

into formal requirements– 250 man years effort

• 250 test engineers in Hyderabad• 100 test engineers in Beijing



• Example in industrial Practice: eg.W Grieskamp:„Microsoft's Protocol Document Program: A Succes-Story for Model-Based Testing“. TestCom/Fates 09– 222 protocols/technical documents tested– 22,847 pages studied and converted

into formal requirements– 250 man years effort

• 250 test engineers in Hyderabad• 100 test engineers in Beijing


Validation and Verification

• Validation : – Does the system meet the clients requirements ? – Will the performance be sufficient ?– Will the usability be sufficient ?







Do we build the right system ?




Do we build the right system ?



• Verification:

Does the system meet the specification ?

Do we build the system right ? Is it « correct » ?



• Verification:

Does the system meet the specification ?

Do we build the system right ? Is it « correct » ?


How to do Validation ?

• Mesuring customer satisfaction ...(well, that's afterwards, and its difficult)

• Interviews, inspections (again post-hoc)• How to validate a system early?

– early prototypes, including performance analysis ...

– mock-ups (fonctionnality, ergonomics,…)– Test and Animation on the basis of formal

specifications (e.g., à la UML; Matlab/Simulink, ...)05/12/17 B. Wolff, Test de Conformance 15

How to do Validation ?

• Mesuring customer satisfaction ...(well, that's afterwards, and its difficult)

• Interviews, inspections (again post-hoc)• How to validate a system early?

– early prototypes, including performance analysis ...

– mock-ups (fonctionnality, ergonomics,…)– Test and Animation on the basis of formal

specifications (e.g., à la UML; Matlab/Simulink, ...)


How to do Verification ?

• Test and

Deductive Verification (Proof)

on the basis of formal specifications (e.g., à la B, ACSL, JML, Spec#, UML/OCL !) against programs ...



• Test and

Deductive Verification (Proof)

on the basis of formal specifications (e.g., à la B, ACSL, JML, Spec#, UML/OCL !) against programs ...



• Deductive Verification (Proof) ... has a number of advantages:

• very high confidence level gained• can be applied to (some) programs

that are not testable

... and drawbacks:• expensive• proof engineers are a rare resource



• Deductive Verification (Proof) ... has a number of advantages:

• very high confidence level gained• can be applied to (some) programs

that are not testable

... and drawbacks:• expensive• proof engineers are a rare resource



• In this course, we address

Testing

(using modeling and using automated proof techniques ...)



• In this course, we address

Testing

(using modeling and using automated proof techniques ...)


Limits of testing ?

• Systematic, Model-based Testing... has a lot of advantages– an approximation to deductive verification– usually easier and less expensive – easier to integrate in a conventional process– can better cover aspects of a validation

... and drawbacks: – Difficult to apply to a number of types of faults

low-level OS implementations, memory allocation, garbage collectionmemory virtualization, ... crypt-algorithms, ...non-deterministic programs withno control over the non-determinism.


Limits of testing ?

• Systematic, Model-based Testing... has a lot of advantages– an approximation to deductive verification– usually easier and less expensive – easier to integrate in a conventional process– can better cover aspects of a validation

... and drawbacks: – Difficult to apply to a number of types of faults

low-level OS implementations, memory allocation, garbage collectionmemory virtualization, ... crypt-algorithms, ...non-deterministic programs withno control over the non-determinism.


A Philosophical Position Statement :

Test vs. Proof• Note:

Some researcher consider test as opposite to formal proof! Reasons:– “A test can only reveal the presence of bugs,

but not their absence” (Dijkstra, v. Dalen)– ... these researchers referred to unsystematic tests ...

(which are, admittedly, still quite common in SE practice)




Some researcher consider test as opposite to formal proof! Reasons:– “A test can only reveal the presence of bugs,

but not their absence” (Dijkstra, v. Dalen)– ... these researchers referred to unsystematic tests ...

(which are, admittedly, still quite common in SE practice)




We consider (systematic!) test more as an approximation to formal proof. Reasons:– The nature of the approximation can be

made formally precise (via explicit test-hypothesis ...)– both techniques, model-based tests and formal

verification, share a lot of technologies ...– even full-blown proof attempts may profit from

testing,since it can help to debug specs early and cost-effectively




We consider (systematic!) test more as an approximation to formal proof. Reasons:– The nature of the approximation can be

made formally precise (via explicit test-hypothesis ...)– both techniques, model-based tests and formal

verification, share a lot of technologies ...– even full-blown proof attempts may profit from

testing,since it can help to debug specs early and cost-effectively


Test in the SE Process

! General challenges for test-based verification:

" How to select test-data ? To which purpose ?

" How to focus verification activities?Where to verify formally, and where to test, and when did we test enough?

Note: The quality of a test does not increasenecessarily by the number of test-cases !

" Automation ? Tools ?05/12/17 B. Wolff, Test de Conformance 22

Test in the SE Process

! General challenges for test-based verification:

" How to select test-data ? To which purpose ?

" How to focus verification activities?Where to verify formally, and where to test, and when did we test enough?

Note: The quality of a test does not increasenecessarily by the number of test-cases !

" Automation ? Tools ?


Some empirical data ...

• Size of Software ?– Peugeot 607 : 2 Mb embedded software– Windows 90: 10 Mb. LOC source, Win2000: 30 Mb.– Kernel Hyper V: 50000 LOC. (Highly complex, concurrent C)– Noyau RedHat 7.1 (2002) : ~2.4 M. LOC, XWindow ~1.8,Mozilla

~2.1 M.– Space Shuttle (and its environment) : ~50 MLOC

• Reminder: Development Cost ?– Percentage of «Coding» ? 15 - 20 %– Trend (?): Code is more and more generated (CASE Tools)– Proportion of Validation et Verification ? ~20% / ~20%


Some empirical data ...

• Size of Software ?– Peugeot 607 : 2 Mb embedded software– Windows 90: 10 Mb. LOC source, Win2000: 30 Mb.– Kernel Hyper V: 50000 LOC. (Highly complex, concurrent C)– Noyau RedHat 7.1 (2002) : ~2.4 M. LOC, XWindow ~1.8,Mozilla

~2.1 M.– Space Shuttle (and its environment) : ~50 MLOC

• Reminder: Development Cost ?– Percentage of «Coding» ? 15 - 20 %– Trend (?): Code is more and more generated (CASE Tools)– Proportion of Validation et Verification ? ~20% / ~20%


Verification Costs

• costs ? 35 - 50 % of the global effort ?

• all “real” (large) software has remaining bugs …

• The cost of bug ?– the cost to reveal and fix it …

or:the cost of a legal battle it may cause...

or the potential damage to the image (difficult to evaluate, but veeeery real)

or costs as a result to come later on the market05/12/17 B. Wolff, Test de Conformance 24

Verification Costs


• all “real” (large) software has remaining bugs …

• The cost of bug ?– the cost to reveal and fix it …

or:the cost of a legal battle it may cause...

or the potential damage to the image (difficult to evaluate, but veeeery real)

or costs as a result to come later on the market


Verification Costs


• on the other side – you can't test infinitely, and deductive verificationis again 10 times more costly than thoroughly testing !


Verification Costs


• on the other side – you can't test infinitely, and deductive verificationis again 10 times more costly than thoroughly testing !


Verification Costs

• Some Conclusion: – verification is vitally important,

and also critical activity in the development process

– to do it cost-effectively, it requires• a lot of expertise on products and processes• a lot of knowledge over methods,

tools, and tool chains ...


Verification Costs

• Some Conclusion: – verification is vitally important,

and also critical activity in the development process

– to do it cost-effectively, it requires• a lot of expertise on products and processes• a lot of knowledge over methods,

tools, and tool chains ...


General Aims of Testmethods

• Main objective in a process: – finding bugs early– adaption to changes of the model (“agility”)– gaining confidence in the system ...

• A partial answer to this need systematic test:– process programs and specifications

and to compute a set of test-cases under controlled conditions, perhaps in a loop with the evolving system

– ideally: testing is complete if a certain criteria,the adequacy criteria is reached.


General Aims of Testmethods

• Main objective in a process: – finding bugs early– adaption to changes of the model (“agility”)– gaining confidence in the system ...

• A partial answer to this need systematic test:– process programs and specifications

and to compute a set of test-cases under controlled conditions, perhaps in a loop with the evolving system

– ideally: testing is complete if a certain criteria,the adequacy criteria is reached.


A Taxonomy on Test Methods• A taxonomy on types of tests

– Static Test / Dynamic (Runtime) Test – Structural Test / Functional Test– For one Unit / a Sequence of – [Statistic Tests]

• Functional Test (based on a Spec)– Dynamic Unit Tests, Static Unit Tests, – Unit and Sequence TMs

• Structural Tests (based on Prog and Spec)– Dynamic Unit Tests, Static Unit Tests, – Unit and Sequence Test Methods


A Taxonomy on Test Methods• A taxonomy on types of tests

– Static Test / Dynamic (Runtime) Test – Structural Test / Functional Test– For one Unit / a Sequence of – [Statistic Tests]

• Functional Test (based on a Spec)– Dynamic Unit Tests, Static Unit Tests, – Unit and Sequence TMs

• Structural Tests (based on Prog and Spec)– Dynamic Unit Tests, Static Unit Tests, – Unit and Sequence Test Methods


A Taxonomy on Test Methods

• A taxonomy on types of tests– static: running a program before deployment on data

carefully constructed by the analyst (in a test environment)analyse the result on the basis of all componentsworking on some classes of executions symbolically= representing infinitely many executions

– dynamic: running the programme (or component)after deployment, on “real data” as imposed by the application domain

– experiment with the real behaviour– essentially used for post-hoc analysis and debugging


A Taxonomy on Test Methods

• A taxonomy on types of tests– static: running a program before deployment on data

carefully constructed by the analyst (in a test environment)analyse the result on the basis of all componentsworking on some classes of executions symbolically= representing infinitely many executions

– dynamic: running the programme (or component)after deployment, on “real data” as imposed by the application domain

– experiment with the real behaviour– essentially used for post-hoc analysis and debugging


Taxonomy: Unit / Sequence / Reactive Tests

• A taxonomy on dimension of functional (black-box) tests– unit: testing of a local component (function, module),

typically only one step of the underlying state.(In functional programs, thats essentially all whatyou have to do!)

– sequence: testing of a local component (function, module), but typically sequences of executions, which typically depend on internal state

– reactive sequence: testing components by sequences of steps, but these sequences represent communication where later parts in the sequence depend on what has been earlier communicated


Taxonomy: Unit / Sequence / Reactive Tests

• A taxonomy on dimension of functional (black-box) tests– unit: testing of a local component (function, module),

typically only one step of the underlying state.(In functional programs, thats essentially all whatyou have to do!)

– sequence: testing of a local component (function, module), but typically sequences of executions, which typically depend on internal state

– reactive sequence: testing components by sequences of steps, but these sequences represent communication where later parts in the sequence depend on what has been earlier communicated


Taxonomy: Functional / Structural Test

• A taxonomy on basis of tests– functional: (also: black-box tests). Tests were

generated on a specification of the component, the test focusses on input output behaviour.

– structural: (also: white-box tests). Tests were generated on the basis of the structure or the program, i.e. using control-flow, data-flow paths or by using symbolic executions.

– both: (also: grey-box testing, in some sense also fuzz-testing, mutation-testing).


Taxonomy: Functional / Structural Test

• A taxonomy on basis of tests– functional: (also: black-box tests). Tests were

generated on a specification of the component, the test focusses on input output behaviour.

– structural: (also: white-box tests). Tests were generated on the basis of the structure or the program, i.e. using control-flow, data-flow paths or by using symbolic executions.

– both: (also: grey-box testing, in some sense also fuzz-testing, mutation-testing).

05/12/17 B. Wolff - Conformance Tests 32

Context:Where are we ?

Reminder: Taxonomy

functional tests= specification- based test= black-box test

structural tests= specification+ program-based test= white-box test

dynamic

static

unit

sequence

reactive sequence

05/12/17 B. Wolff - Conformance Tests 32

Context:Where are we ?

Reminder: Taxonomy

functional tests= specification- based test= black-box test

structural tests= specification+ program-based test= white-box test

dynamic

static

unit

sequence

reactive sequence


Functional Unit Test

• We got the spec, but not the program, which is considered a black box:

input output???

we focus on what the program should do !!!


Functional Unit Test

• We got the spec, but not the program, which is considered a black box:

input output???

we focus on what the program should do !!!

05/12/17 B. Wolff - GLA - System Test 34

Functional Unit Test : An Example

The (informal) specification:

Read a “Triangle Object” (with three sides of integral type),and test if it is isoscele, equilateral, or (default) arbitrary.

Each length should be positive.

Give a specification, and develop a test set ...



The (informal) specification:

Read a “Triangle Object” (with three sides of integral type),and test if it is isoscele, equilateral, or (default) arbitrary.

Each length should be positive.

Give a specification, and develop a test set ...



The specification in UML/MOAL:

Trianglesa, b, c: Integer

- mk(Integer,Integer,Integer):Triangle- is_Triangle(): {equ (*equilateral*), iso (*isosceles*), arb (*arbitrary*)}



The specification in UML/MOAL:





We add the constraints of the analysis:



inv 0<a 0<b 0<c∧ ∧

inv c≤a+b a∧ ≤b+c b∧ ≤c+a

operation t.is_Triangle(): post t.a=t.b t.b=t.c result=equ∧ ⟶ post (t.a≠t.b t.b≠t.c) ∨ ∧ (t.a=t.b t.b=t.c t.a=t.c)) result=iso∨ ∨ ⟶post (t.a≠t.b t.b≠t.c t.a≠t.c)) result=arb∨ ∨ ⟶



We add the constraints of the analysis:



inv 0<a 0<b 0<c∧ ∧

inv c≤a+b a∧ ≤b+c b∧ ≤c+a

operation t.is_Triangle(): post t.a=t.b t.b=t.c result=equ∧ ⟶ post (t.a≠t.b t.b≠t.c) ∨ ∧ (t.a=t.b t.b=t.c t.a=t.c)) result=iso∨ ∨ ⟶post (t.a≠t.b t.b≠t.c t.a≠t.c)) result=arb∨ ∨ ⟶

05/12/17 B. Wolff - GLA - Black Box Test 37

Revision: Boolean Logic + Some Basic Rules

• ¬(a ∧ b)=¬ a ¬ b ∨ (* deMorgan1 *)• ¬(a b)=¬ a ∨ ∧ ¬ b (* deMorgan2 *)• a ∧ (b c) = (a ∨ ∧ b) (a ∨ ∧ c)• ¬(¬ a) = a• a ∧ b = b ∧ a; a b = b a∨ ∨

• a ∧ (b ∧ c) = (a ∧ b) ∧ c• a (b c) = (a b) c∨ ∨ ∨ ∨

• a b = (¬ a) b⟶ ∨• (a=b ∧ P(a)) = P(b) (* one point rule *)• let x = E in C(x) = C(E) (* let elimination *)• if c then C else D = (c ∧ C) (¬ c ∨ ∧ D)

= (c C) ⟶ ∧ (¬ c D)⟶05/12/17 B. Wolff - GLA - Black Box Test 37

Revision: Boolean Logic + Some Basic Rules

• ¬(a ∧ b)=¬ a ¬ b ∨ (* deMorgan1 *)• ¬(a b)=¬ a ∨ ∧ ¬ b (* deMorgan2 *)• a ∧ (b c) = (a ∨ ∧ b) (a ∨ ∧ c)• ¬(¬ a) = a• a ∧ b = b ∧ a; a b = b a∨ ∨

• a ∧ (b ∧ c) = (a ∧ b) ∧ c• a (b c) = (a b) c∨ ∨ ∨ ∨

• a b = (¬ a) b⟶ ∨• (a=b ∧ P(a)) = P(b) (* one point rule *)• let x = E in C(x) = C(E) (* let elimination *)• if c then C else D = (c ∧ C) (¬ c ∨ ∧ D)

= (c C) ⟶ ∧ (¬ c D)⟶


Functional Dynamic Unit Test : An Example

• We can already use our specification for a particular testtechnique, which is according to our taxonomy a – functional – dynamic– unit-test

• We compile class invariants:

Class A: inv l1 : C1, ...,

inv ln : Cn




• We compile class invariants:

Class A: inv l1 : C1, ...,

inv ln : Cn




• ... to assertion checking code for constructors orobject update functions (with assert as in assert.h in standard lib of C, for example, ...)

A(a1, ..., a

n) = assert(C1); ... ;

assert(Cn); ConstructorCode(a1, ..., an)




• ... to assertion checking code for constructors orobject update functions (with assert as in assert.h in standard lib of C, for example, ...)

A(a1, ..., a

n) = assert(C1); ... ;

assert(Cn); ConstructorCode(a1, ..., an)




• We compile pre-postconditions of operation constructs:

method m(a1, ..., a

n):

pre P(a1, ..., a)

post Q(a1, ..., a,result)




• We compile pre-postconditions of operation constructs:

method m(a1, ..., a

n):

pre P(a1, ..., a)

post Q(a1, ..., a,result)




• to pre- and postludes with assertions

method m(a1, ..., a

n):

assert(P(a1, ..., a))

result = m_core_code(a1, ..., a

n)

assert(Q(a1, ..., a,result))




• to pre- and postludes with assertions

method m(a1, ..., a

n):

assert(P(a1, ..., a))

result = m_core_code(a1, ..., a

n)

assert(Q(a1, ..., a,result))


Functional Dynamic Unit Test : Summary


• test-execution : – on real user data, – in the shipped system– post hoc, so we learn about

an error when it already happened at the customer ...05/12/17 B. Wolff, Test de Conformance 42

Functional Dynamic Unit Test : Summary


• test-execution : – on real user data, – in the shipped system– post hoc, so we learn about

an error when it already happened at the customer ...


Functional Unit-Test

• Can we do better ?– earlier, during development time ?– can we do it like a static check,

like a type-checker ?– can we do it systematically ?


Functional Unit-Test

• Can we do better ?– earlier, during development time ?– can we do it like a static check,

like a type-checker ?– can we do it systematically ?


Test-Data Generation“by hand“

• Consider the test specification (the “Test Case”):

mk(x,y,z).isTriangle() X

i.e. for which input (x,y,z) should an implementation of our contract yield which X ?

Note that we define mk(0,0,0) to invalid,as well as all other invalid triangles ...



• Consider the test specification (the “Test Case”):

mk(x,y,z).isTriangle() X

i.e. for which input (x,y,z) should an implementation of our contract yield which X ?

Note that we define mk(0,0,0) to invalid,as well as all other invalid triangles ...



• an arbitrary valid triangle: (3, 4, 5)

• an equilateral triangle: (5, 5, 5)

• an isoscele triangle and its permutations :(6, 6, 7), (7, 6, 6), (6, 7, 6)

• impossible triangles and their permutations :(1, 2, 4), (4, 1, 2), (2, 4, 1) -- x + y > z(1, 2, 3), (2, 4, 2), (5, 3, 2) -- x + y = z (necessary?)

• a zero length : (0, 5, 4), (4, 0, 5),

• Would we have to consider negative values?05/12/17 B. Wolff - GLA - Black Box Test 45


• an arbitrary valid triangle: (3, 4, 5)

• an equilateral triangle: (5, 5, 5)

• an isoscele triangle and its permutations :(6, 6, 7), (7, 6, 6), (6, 7, 6)

• impossible triangles and their permutations :(1, 2, 4), (4, 1, 2), (2, 4, 1) -- x + y > z(1, 2, 3), (2, 4, 2), (5, 3, 2) -- x + y = z (necessary?)

• a zero length : (0, 5, 4), (4, 0, 5),

• Would we have to consider negative values?


Test-Data Generation- a first summary -

• Ouf, is there a systematic and automatic way to compute all these tests ?

• Can we avoid hand-written test-scripts ?Avoid the task to maintain them ?

• And the question remains:

When did we tested „enough“ ?









When did we tested „enough“ ?









What criterion could tell usthat we tested „enough“ ?






What criterion could tell usthat we tested „enough“ ?


Static Specification-based Unit Test

• How can we generate the testing set before deployment, at coding time,even at design-time ?







• Idea: Random-Data Generation. (as in QuickCheck, and many reimplemts)– we randomly generate input,– filter those which do not satisfy pre– run the code, and check the output with post.




• Idea: Random-Data Generation. (as in QuickCheck, and many reimplemts)– we randomly generate input,– filter those which do not satisfy pre– run the code, and check the output with post.




• Problem with Random-Data Generation: inefficient/impractical if invariants or preconditions complex.




• Problem with Random-Data Generation: inefficient/impractical if invariants or preconditions complex.




• Example : test the following Java - Property:

0 <= abs(x) (where x : int)

How likely is it to find the error ? 05/12/17 B. Wolff, Test de Conformance 51



• Example : test the following Java - Property:

0 <= abs(x) (where x : int)

How likely is it to find the error ?


Difficulties with Dynamic Unit Tests so far

• And what answer gives us Random-based testing on the question:

When do we have tested enough ? When is our test “adequate” ?

– In software standards, we have to decide on adequacy criteria in advance. This can be:

• criteria on the coverage of the spec of the program

• criteria on statistical models and anerror model A


Difficulties with Dynamic Unit Tests so far

• And what answer gives us Random-based testing on the question:

When do we have tested enough ? When is our test “adequate” ?

– In software standards, we have to decide on adequacy criteria in advance. This can be:

• criteria on the coverage of the spec of the program

• criteria on statistical models and anerror model A


Test-Data Generation

• Another Method (Going back to Dick/Faivre93 and Gaudel95)

... also called the DNF Method.

Idea: We unfold the entire spec („the functional model“)and convert it into DNF.



• Another Method (Going back to Dick/Faivre93 and Gaudel95)

... also called the DNF Method.

Idea: We unfold the entire spec („the functional model“)and convert it into DNF.



• Recall the test specification:

mk(x,y,z).isTriangle() r



• Recall the test specification:

mk(x,y,z).isTriangle() r



• Recall the test specification:mk(x,y,z).isTriangle() r

invTriangle(σ) ∧ preisTriangle(mk(x,y,z))(σ) ∧

invTriangle(σ') ∧ postisTriangle(mk(x,y,z),r)(σ,σ')

Some Facts:– this program does not change the state,

it follows σ σ ’ hence invTriangle(σ) = invTriangle(σ’)

– mk(x,y,z) Triangle ∈ (σ) follows that: 0<x 0<y 0<z∧ ∧ x∧ ≤y+z y∧ ≤x+z z∧ ≤x+y ( inv)






Some Facts:– this program does not change the state,

it follows σ σ ’ hence invTriangle(σ) = invTriangle(σ’)

– mk(x,y,z) Triangle ∈ (σ) follows that: 0<x 0<y 0<z∧ ∧ x∧ ≤y+z y∧ ≤x+z z∧ ≤x+y ( inv)






Some Facts:– arb≠equ≠iso

– postisTriangle(mk(x,y,z),r)(σ σ ) can be simplified to:

(x=y y=z r=equ∧ ⟶ ) ∧ ((x≠y y≠z) (x=y y=z x=z) r=iso∨ ∧ ∨ ∨ ⟶ ) ∧ ((x≠y y≠z x≠z) r=arb∧ ∧ ⟶ ) 05/12/17 B. Wolff - GLA - Black Box Test 56





Some Facts:– arb≠equ≠iso

– postisTriangle(mk(x,y,z),r)(σ σ ) can be simplified to:

(x=y y=z r=equ∧ ⟶ ) ∧ ((x≠y y≠z) (x=y y=z x=z) r=iso∨ ∧ ∨ ∨ ⟶ ) ∧ ((x≠y y≠z x≠z) r=arb∧ ∧ ⟶ )






⟹ (* the discussed facts *)

inv ∧(x=y y=z r=equ∧ ⟶ ) ∧((x≠y y≠z) (x=y y=z x=z) r=iso∨ ∧ ∨ ∨ ⟶ ) ∧(x≠y y≠z x≠z r=arb∧ ∧ ⟶ )






⟹ (* the discussed facts *)

inv ∧(x=y y=z r=equ∧ ⟶ ) ∧((x≠y y≠z) (x=y y=z x=z) r=iso∨ ∧ ∨ ∨ ⟶ ) ∧(x≠y y≠z x≠z r=arb∧ ∧ ⟶ )

Some Facts:• arb≠equ≠iso • postisTriangle(mk(x,y,z),r)(s,s)

can be simplified to:

(x=y y=z r=equ) ∧ ⟶ ∧ ((x≠y y≠z) (x=y y=z x=z) r=iso) ∨ ∧ ∨ ∨ ⟶ ∧

((x≠y y≠z x≠z) r=arb) ∧ ∧ ⟶

Some Facts:• arb≠equ≠iso • postisTriangle(mk(x,y,z),r)(s,s)

can be simplified to:

(x=y y=z r=equ) ∧ ⟶ ∧ ((x≠y y≠z) (x=y y=z x=z) r=iso) ∨ ∧ ∨ ∨ ⟶ ∧

((x≠y y≠z x≠z) r=arb) ∧ ∧ ⟶



• Recall the test specification:inv ∧ (x=y y=z r=equ∧ ⟶ ) ∧

((x≠y y≠z) (x=y y=z x=z) r=iso∨ ∧ ∨ ∨ ⟶ ) ∧ (x≠y y≠z x≠z r=arb∧ ∧ ⟶ )

(* elimination ⟶ , deMorgan*)

inv ∧(x≠y y≠z r=equ∨ ∨ ) ∧((x=y y=z) (x≠y y≠z x≠z) r=iso∧ ∨ ∧ ∧ ∨ ) ∧(x=y y=z x=z r=arb∨ ∨ ∨ )



• Recall the test specification:inv ∧ (x=y y=z r=equ∧ ⟶ ) ∧

((x≠y y≠z) (x=y y=z x=z) r=iso∨ ∧ ∨ ∨ ⟶ ) ∧ (x≠y y≠z x≠z r=arb∧ ∧ ⟶ )

(* elimination ⟶ , deMorgan*)

inv ∧(x≠y y≠z r=equ∨ ∨ ) ∧((x=y y=z) (x≠y y≠z x≠z) r=iso∧ ∨ ∧ ∧ ∨ ) ∧(x=y y=z x=z r=arb∨ ∨ ∨ )



• This first part of the calculation could be called

PURIFICATION

We eliminate UML, object-orientation, MOAL etcppand reduce it to the pure logical core ...

Now, under which precise conditions do we have – r = iso– r = arb– r = equ ???




PURIFICATION


Now, under which precise conditions do we have – r = iso– r = arb– r = equ ???




PURIFICATION


Can we transform the spec into the form– A1 ∧ ... ∧ Ai ∧ r = iso

– B1 ∧ ... ∧ Bk ∧ r = arb

– C1 ∧ ... ∧ Cl ∧ r = equ ???05/12/17 B. Wolff - GLA - Black Box Test 61



PURIFICATION


Can we transform the spec into the form– A1 ∧ ... ∧ Ai ∧ r = iso

– B1 ∧ ... ∧ Bk ∧ r = arb

– C1 ∧ ... ∧ Cl ∧ r = equ ???




PURIFICATION


Can we transform the spec into a

Disjunctive Normal Form (DNF) ?05/12/17 B. Wolff - GLA - Black Box Test 62



PURIFICATION


Can we transform the spec into a

Disjunctive Normal Form (DNF) ?


Excursion

• Generalized Distribution Laws:

(A1 ∨ A2) ∧ (B1 ∨ B2) = (A1 ∧ (B1 ∨ B2)) ∨ (A2 ∧ (B1 ∨ B2))

= (A1 ∧ B1) ∨ (A2 ∧ B1) ∨ (A1 ∧ B2) ∨ (A2 ∧ B2)

(A1 ∨ A2 ∨ A3) ∧ (B1 ∨ B2 ∨ B3) ∧ (C1 ∨ C2 ∨ C3)

= ...= (A1 ∧ B1 ∧ C1) ∨ (A1 ∧ B1 ∧ C2) ∨ (A1 ∧ B1 ∧ C3) ∨ (A2 ∧ B1 ∧ C1) ∨ (A2 ∧ B1 ∧ C2) ∨ (A2 ∧ B1 ∧ C3) ∨ ...

(A1 ∧ B3 ∧ C3) ∨ (A2 ∧ B3 ∧ C3) ∨ (A3 ∧ B3 ∧ C3)


Excursion

• Generalized Distribution Laws:

(A1 ∨ A2) ∧ (B1 ∨ B2) = (A1 ∧ (B1 ∨ B2)) ∨ (A2 ∧ (B1 ∨ B2))

= (A1 ∧ B1) ∨ (A2 ∧ B1) ∨ (A1 ∧ B2) ∨ (A2 ∧ B2)

(A1 ∨ A2 ∨ A3) ∧ (B1 ∨ B2 ∨ B3) ∧ (C1 ∨ C2 ∨ C3)

= ...= (A1 ∧ B1 ∧ C1) ∨ (A1 ∧ B1 ∧ C2) ∨ (A1 ∧ B1 ∧ C3) ∨ (A2 ∧ B1 ∧ C1) ∨ (A2 ∧ B1 ∧ C2) ∨ (A2 ∧ B1 ∧ C3) ∨ ...

(A1 ∧ B3 ∧ C3) ∨ (A2 ∧ B3 ∧ C3) ∨ (A3 ∧ B3 ∧ C3)



• Recall the test specification:...

inv ∧(x≠y y≠z r=equ∨ ∨ ) ∧(x=y y=z x=z r=arb∨ ∨ ∨ ) ∧((x=y y=z) (x≠y y≠z x≠z) r=iso∧ ∨ ∧ ∧ ∨ )

(* generalized distribution 2nd/3rd line *) inv ∧((x≠y ∧ x=y) (∨ x≠y y=z) (∧ ∨ x≠y x=z) (∧ ∨ x≠y r=arb)∧ ) ∨((y≠z ∧ x=y) (y≠z y=z) (y≠z x=z) (y≠z r=arb)∨ ∧ ∨ ∧ ∨ ∧ ) ∨((r=equ x=y) (r=equ y=z) (r=equ x=z) (r=equ r=arb)∧ ∨ ∧ ∨ ∧ ∨ ∧ ) ∨((x=y y=z) (x≠y y≠z x≠z) r=iso∧ ∨ ∧ ∧ ∨ )





(* generalized distribution 2nd/3rd line *) inv ∧((x≠y ∧ x=y) (∨ x≠y y=z) (∧ ∨ x≠y x=z) (∧ ∨ x≠y r=arb)∧ ) ∨((y≠z ∧ x=y) (y≠z y=z) (y≠z x=z) (y≠z r=arb)∨ ∧ ∨ ∧ ∨ ∧ ) ∨((r=equ x=y) (r=equ y=z) (r=equ x=z) (r=equ r=arb)∧ ∨ ∧ ∨ ∧ ∨ ∧ ) ∨((x=y y=z) (x≠y y≠z x≠z) r=iso∧ ∨ ∧ ∧ ∨ )





(* elimination contradictions *) inv ∧((x≠y ∧ x=y) (∨ x≠y y=z) (∧ ∨ x≠y x=z) (∧ ∨ x≠y r=arb)∧ ∨ (y≠z ∧ x=y)∨(y≠z y=z)∧ (y≠z x=z) (y≠z r=arb)∨ ∧ ∨ ∧ ∨ (r=equ x=y) (r=equ y=z) (r=equ x=z)∧ ∨ ∧ ∨ ∧ ∨(r=equ r=arb)∧ ) ∨((x=y y=z) (x≠y y≠z x≠z) r=iso∧ ∨ ∧ ∧ ∨ )





(* elimination contradictions *) inv ∧((x≠y ∧ x=y) (∨ x≠y y=z) (∧ ∨ x≠y x=z) (∧ ∨ x≠y r=arb)∧ ∨ (y≠z ∧ x=y)∨(y≠z y=z)∧ (y≠z x=z) (y≠z r=arb)∨ ∧ ∨ ∧ ∨ (r=equ x=y) (r=equ y=z) (r=equ x=z)∧ ∨ ∧ ∨ ∧ ∨(r=equ r=arb)∧ ) ∨((x=y y=z) (x≠y y≠z x≠z) r=iso∧ ∨ ∧ ∧ ∨ )




(* elimination contradictions *) inv ∧((x≠y y=z) (∧ ∨ x≠y x=z) (∧ ∨ x≠y r=arb)∧ ∨ (y≠z ∧ x=y) (y≠z x=z) (y≠z r=arb)∨ ∧ ∨ ∧ ∨ (r=equ x=y) (r=equ y=z) (r=equ x=z)∧ ∨ ∧ ∨ ∧ ) ∧((x=y y=z) (x≠y y≠z x≠z) r=iso∧ ∨ ∧ ∧ ∨ )




(* elimination contradictions *) inv ∧((x≠y y=z) (∧ ∨ x≠y x=z) (∧ ∨ x≠y r=arb)∧ ∨ (y≠z ∧ x=y) (y≠z x=z) (y≠z r=arb)∨ ∧ ∨ ∧ ∨ (r=equ x=y) (r=equ y=z) (r=equ x=z)∧ ∨ ∧ ∨ ∧ ) ∧((x=y y=z) (x≠y y≠z x≠z) r=iso∧ ∨ ∧ ∧ ∨ )



• (* generalized distribution 2nd/3rd ((9 * 3 = 27 cases !)*) inv ∧((x≠y y=z x=y y=z) (∧ ∧ ∧ ∨ x≠y x=z ∧ ∧ x=y y=z) (∧ ∨ x≠y r=arb x=y y=z)∧ ∧ ∧ ∨ (y≠z x=y x=y y=z) (y≠z x=z∧ ∧ ∧ ∨ ∧ ∧ x=y y=z) (y≠z r=arb x=y y=z)∧ ∨ ∧ ∧ ∧ ∨ (r=equ x=y x=y y=z) (r=equ∧ ∧ ∧ ∨ ∧ y=z x=y y=z) (r=equ x=z x=y y=z)∧ ∧ ∨ ∧ ∧ ∧ ) ∨((x≠y y=z x≠y y≠z x≠z) (∧ ∧ ∧ ∧ ∨ x≠y x=z x≠y y≠z x≠z) (∧ ∧ ∧ ∧ ∨ x≠y r=arb ∧ ∧x≠y y≠z x≠z) (y≠z x=y x≠y y≠z x≠z) (y≠z x=z x≠y y≠z ∧ ∧ ∨ ∧ ∧ ∧ ∧ ∨ ∧ ∧ ∧ ∧x≠z) (y≠z r=arb x≠y y≠z x≠z) (r=equ x=y x≠y y≠z x≠z) (r=∨ ∧ ∧ ∧ ∧ ∨ ∧ ∧ ∧ ∧ ∨equ y=z x≠y y≠z x≠z) (r=equ x=z x≠y y≠z x≠z)∧ ∧ ∧ ∧ ∨ ∧ ∧ ∧ ∧ ) ∨((x≠y y=z r=iso) (∧ ∧ ∨ x≠y x=z r=iso) (∧ ∧ ∨ x≠y r=arb r=iso)∧ ∧ (y≠z x=y r=iso) (y≠z x=z r=iso) (y≠z r=arb r=iso)∨ ∧ ∧ ∨ ∧ ∧ ∨ ∧ ∧ ∨ (r=equ x=y r=iso) (r=equ y=z r=iso) (r=equ x=z r=iso)∧ ∧ ∨ ∧ ∧ ∨ ∧ ∧ )



• (* generalized distribution 2nd/3rd ((9 * 3 = 27 cases !)*) inv ∧((x≠y y=z x=y y=z) (∧ ∧ ∧ ∨ x≠y x=z ∧ ∧ x=y y=z) (∧ ∨ x≠y r=arb x=y y=z)∧ ∧ ∧ ∨ (y≠z x=y x=y y=z) (y≠z x=z∧ ∧ ∧ ∨ ∧ ∧ x=y y=z) (y≠z r=arb x=y y=z)∧ ∨ ∧ ∧ ∧ ∨ (r=equ x=y x=y y=z) (r=equ∧ ∧ ∧ ∨ ∧ y=z x=y y=z) (r=equ x=z x=y y=z)∧ ∧ ∨ ∧ ∧ ∧ ) ∨((x≠y y=z x≠y y≠z x≠z) (∧ ∧ ∧ ∧ ∨ x≠y x=z x≠y y≠z x≠z) (∧ ∧ ∧ ∧ ∨ x≠y r=arb ∧ ∧x≠y y≠z x≠z) (y≠z x=y x≠y y≠z x≠z) (y≠z x=z x≠y y≠z ∧ ∧ ∨ ∧ ∧ ∧ ∧ ∨ ∧ ∧ ∧ ∧x≠z) (y≠z r=arb x≠y y≠z x≠z) (r=equ x=y x≠y y≠z x≠z) (r=∨ ∧ ∧ ∧ ∧ ∨ ∧ ∧ ∧ ∧ ∨equ y=z x≠y y≠z x≠z) (r=equ x=z x≠y y≠z x≠z)∧ ∧ ∧ ∧ ∨ ∧ ∧ ∧ ∧ ) ∨((x≠y y=z r=iso) (∧ ∧ ∨ x≠y x=z r=iso) (∧ ∧ ∨ x≠y r=arb r=iso)∧ ∧ (y≠z x=y r=iso) (y≠z x=z r=iso) (y≠z r=arb r=iso)∨ ∧ ∧ ∨ ∧ ∧ ∨ ∧ ∧ ∨ (r=equ x=y r=iso) (r=equ y=z r=iso) (r=equ x=z r=iso)∧ ∧ ∨ ∧ ∧ ∨ ∧ ∧ )



• (* elimination of the contradictions and redundancies *) inv ∧((x≠y y=z x=y y=z)∧ ∧ ∧ ∨(x≠y x=z∧ ∧ x=y y=z)∧ ∨(x≠y r=arb x=y y=z)∧ ∧ ∧ ∨ (y≠z x=y x=y y=z)∧ ∧ ∧ ∨(y≠z x=z∧ ∧ x=y y=z)∧ ∨(y≠z r=arb x=y y=z)∧ ∧ ∧ ∨ (r=equ x=y x=y y=z)∧ ∧ ∧ ∨(r=equ∧ y=z x=y y=z)∧ ∧ ∨(r=equ x=z x=y y=z)∧ ∧ ∧ ) ∨((x≠y y=z x≠y y≠z x≠z)∧ ∧ ∧ ∧ ∨(x≠y x=z x≠y y≠z x≠z)∧ ∧ ∧ ∧ (∨ x≠y r=arb ∧ ∧x≠y∧y≠z x≠z)∧ ∨(y≠z x=y x≠y y≠z x≠z)∧ ∧ ∧ ∧ ∨(y≠z x=z x≠y y≠z ∧ ∧ ∧ ∧x≠z)∨(y≠z r=arb x≠y y≠z x≠z)∧ ∧ ∧ ∧ ∨(r=equ x=y x≠y y≠z x≠z)∧ ∧ ∧ ∧ ∨(r=equ y=z x≠y y≠z x≠z)∧ ∧ ∧ ∧ ∨(r=equ x=z x≠y y≠z x≠z)∧ ∧ ∧ ∧ ) ∨((x≠y y=z r=iso) (∧ ∧ ∨ x≠y x=z r=iso)∧ ∧ ∨(x≠y r=arb r=iso)∧ ∧ (y≠z x=y r=iso) (y≠z x=z r=iso)∨ ∧ ∧ ∨ ∧ ∧ ∨(y≠z r=arb r=iso)∧ ∧ ∨ (r=equ x=y r=iso)∧ ∧ ∨(r=equ y=z r=iso)∧ ∧ ∨(r=equ x=z r=iso)∧ ∧ )



• (* elimination of the contradictions and redundancies *) inv ∧((x≠y y=z x=y y=z)∧ ∧ ∧ ∨(x≠y x=z∧ ∧ x=y y=z)∧ ∨(x≠y r=arb x=y y=z)∧ ∧ ∧ ∨ (y≠z x=y x=y y=z)∧ ∧ ∧ ∨(y≠z x=z∧ ∧ x=y y=z)∧ ∨(y≠z r=arb x=y y=z)∧ ∧ ∧ ∨ (r=equ x=y x=y y=z)∧ ∧ ∧ ∨(r=equ∧ y=z x=y y=z)∧ ∧ ∨(r=equ x=z x=y y=z)∧ ∧ ∧ ) ∨((x≠y y=z x≠y y≠z x≠z)∧ ∧ ∧ ∧ ∨(x≠y x=z x≠y y≠z x≠z)∧ ∧ ∧ ∧ (∨ x≠y r=arb ∧ ∧x≠y∧y≠z x≠z)∧ ∨(y≠z x=y x≠y y≠z x≠z)∧ ∧ ∧ ∧ ∨(y≠z x=z x≠y y≠z ∧ ∧ ∧ ∧x≠z)∨(y≠z r=arb x≠y y≠z x≠z)∧ ∧ ∧ ∧ ∨(r=equ x=y x≠y y≠z x≠z)∧ ∧ ∧ ∧ ∨(r=equ y=z x≠y y≠z x≠z)∧ ∧ ∧ ∧ ∨(r=equ x=z x≠y y≠z x≠z)∧ ∧ ∧ ∧ ) ∨((x≠y y=z r=iso) (∧ ∧ ∨ x≠y x=z r=iso)∧ ∧ ∨(x≠y r=arb r=iso)∧ ∧ (y≠z x=y r=iso) (y≠z x=z r=iso)∨ ∧ ∧ ∨ ∧ ∧ ∨(y≠z r=arb r=iso)∧ ∧ ∨ (r=equ x=y r=iso)∧ ∧ ∨(r=equ y=z r=iso)∧ ∧ ∨(r=equ x=z r=iso)∧ ∧ )



• (* cleanup, distribution *) (inv x=y x=y y=z r=equ) ∧ ∧ ∧ ∧ ∨ (1) (inv ∧ x≠y y≠z x≠z r=arb ) ∧ ∧ ∧ ∨ (2) (inv ∧ x≠y y=z r=iso) ∧ ∧ ∨ (3) (inv ∧ x≠y x=z r=iso) ∧ ∧ ∨ (4) (inv y≠z x=y r=iso) ∧ ∧ ∧ ∨ (5) (inv y≠z x=z r=iso)∧ ∧ ∧ (6)

• Test-Case-Construction by DNF Method yields six abstract test casesrelating input x y z to output r

• Note: In general, output r is not necessarily uniquely defined as in our example ... The spec can be non-deterministic admitting several results.



• (* cleanup, distribution *) (inv x=y x=y y=z r=equ) ∧ ∧ ∧ ∧ ∨ (1) (inv ∧ x≠y y≠z x≠z r=arb ) ∧ ∧ ∧ ∨ (2) (inv ∧ x≠y y=z r=iso) ∧ ∧ ∨ (3) (inv ∧ x≠y x=z r=iso) ∧ ∧ ∨ (4) (inv y≠z x=y r=iso) ∧ ∧ ∧ ∨ (5) (inv y≠z x=z r=iso)∧ ∧ ∧ (6)

• Test-Case-Construction by DNF Method yields six abstract test casesrelating input x y z to output r

• Note: In general, output r is not necessarily uniquely defined as in our example ... The spec can be non-deterministic admitting several results.



• Test-Data-Selection:For each abstract test-case, we construct one concrete test, by choosing values that makethe abstract test case true (« that satisfies theabstract test case »)

case x y z result(1) 3 3 3 equ(2) 3 4 6 arb(3) 4 5 5 iso(4) 5 4 5 iso(5) 5 5 4 iso(6) 4 3 4 iso



• Test-Data-Selection:For each abstract test-case, we construct one concrete test, by choosing values that makethe abstract test case true (« that satisfies theabstract test case »)

case x y z result(1) 3 3 3 equ(2) 3 4 6 arb(3) 4 5 5 iso(4) 5 4 5 iso(5) 5 5 4 iso(6) 4 3 4 iso



• A First Summary on the Test-Generation Method:– PHASE I: Stripping the Domain-Language

(UML-MOAL) away, “purification” – PHASE II: Abstract Test Case Construction by

“DNF computation”– PHASE III: Constraint Resolution

(by solvers like CVC4 or Z3)“Test Data Selection”

– COVERAGE CRITERION:DNF - coverage of the Spec; for each abstract test-case one concrete test-input is constructed. (ISO/IEC/IEEE 29119 calls this: Equivalence class testing)



• A First Summary on the Test-Generation Method:– PHASE I: Stripping the Domain-Language

(UML-MOAL) away, “purification” – PHASE II: Abstract Test Case Construction by

“DNF computation”– PHASE III: Constraint Resolution

(by solvers like CVC4 or Z3)“Test Data Selection”

– COVERAGE CRITERION:DNF - coverage of the Spec; for each abstract test-case one concrete test-input is constructed. (ISO/IEC/IEEE 29119 calls this: Equivalence class testing)



• Variants:• PHASE III: Borderline analysis.

Principle: we replace in our DNF inequalities by „the closest values that make the spec true“

x≠y x = y + 1 x = y - 1 ↦ ∨

x ≤ y ↦ x = y x < y∨x < y x = y - 1 etc.↦

– ... and recompute the DNF. In general, this gives a much finer mesh ...



• Variants:• PHASE III: Borderline analysis.

Principle: we replace in our DNF inequalities by „the closest values that make the spec true“

x≠y x = y + 1 x = y - 1 ↦ ∨

x ≤ y ↦ x = y x < y∨x < y x = y - 1 etc.↦

– ... and recompute the DNF. In general, this gives a much finer mesh ...



• Variants:– PHASE I: Test for exceptional behaviour.

We negate the precondition and to DNF generation on the precondition only.

Test objectives could be:

• should raise an exception if public• should not diverge



• Variants:– PHASE I: Test for exceptional behaviour.

We negate the precondition and to DNF generation on the precondition only.

Test objectives could be:

• should raise an exception if public• should not diverge



• Problem:

How to handle Recursion ?– on the level of data types ?– on the level of recursive operations ?

(More to come)



• Problem:

How to handle Recursion ?– on the level of data types ?– on the level of recursive operations ?

(More to come)



•

Remark: During Coding phase, when the Spec does not change, the test-data-selection can be repeated easily creating always different test sets ...

This can help to detect programmersthat just program “against test-sets”.



•

Remark: During Coding phase, when the Spec does not change, the test-data-selection can be repeated easily creating always different test sets ...

This can help to detect programmersthat just program “against test-sets”.



• Now, we converted the entire specification into a Disjunctive Normal Form (DNF)

• Each Conjoint in the DNF is a Test-Case• Test Data is constructed by picking one

instance of variables that makes the conjoint true !

• Test-Hypothesis applied here: Uniformity hypothesis



• Now, we converted the entire specification into a Disjunctive Normal Form (DNF)

• Each Conjoint in the DNF is a Test-Case• Test Data is constructed by picking one

instance of variables that makes the conjoint true !

• Test-Hypothesis applied here: Uniformity hypothesis



General Scheme of Uniformity hypothesis:

Formally: ( x. class x TestSpec(sut,x)) ( x. class x TestSpec(sut,x))

if there is a data in a test-class for which the system under test sut satisfies the test specification, sut will always satisfy it for data in this class.





if there is a data in a test-class for which the system under test sut satisfies the test specification, sut will always satisfy it for data in this class.





This is a coverage criterion ! Coverage of “Equivalence Classes” (ISO/IEC/IEEE 29119 for software testing)





This is a coverage criterion ! Coverage of “Equivalence Classes” (ISO/IEC/IEEE 29119 for software testing)


Summary

• Testing is a vital, cost-intensive technique in SE Development

• There is a variety of techniaques -we suggest a taxonomy for orientation

• Active Tests require Models andTest-Generation Techniques

• We presented a first method:


Summary

• Testing is a vital, cost-intensive technique in SE Development

• There is a variety of techniaques -we suggest a taxonomy for orientation

• Active Tests require Models andTest-Generation Techniques

• We presented a first method:



• DNF Generation Method: (DNF Partition Adequacy)All test cases result from DNF normalization of the TestSpec ... All Test-Cases must be covered by a test.Test data are constructed via application of Uniformity Hypothesis for the test case.

• The process can be automated by logic inferencesand SMT-solvers, that can indeed be practically usedfor test-case generation (e.g. Z3, Alt-Ergo, ...).

• Surprisingly, Testing requires Theorem-Proving !!!



• DNF Generation Method: (DNF Partition Adequacy)All test cases result from DNF normalization of the TestSpec ... All Test-Cases must be covered by a test.Test data are constructed via application of Uniformity Hypothesis for the test case.

• The process can be automated by logic inferencesand SMT-solvers, that can indeed be practically usedfor test-case generation (e.g. Z3, Alt-Ergo, ...).

• Surprisingly, Testing requires Theorem-Proving !!!

test de conformité - laboratoire de recherce en...

Documents