lsr test purposes: adapting the notion of specification to testing yves ledru, l. du bousquet, p....
TRANSCRIPT
![Page 1: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/1.jpg)
LSR Test purposes:adapting the notion of specification to testing
Yves Ledru,
L. du Bousquet, P. Bontron,
O. Maury, C. Oriat, M.-L. PotetLSR/IMAG
Grenoble, France
ASE2001, Nov 27-29 2001, San Diego
![Page 2: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/2.jpg)
LSR
Conformance testing• Given a black box
• And its specification
• Test cases compare the black box to its specification
X = sqrt(X)*sqrt(X)andsqrt(X) >= 0
?sqrt(4)
!2
accept
sqrt
![Page 3: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/3.jpg)
LSR
Complex test cases
Unfortunately, complex applications
often result in complex test cases!
?Money_Transfer(1000000€)
!OK
accept
e-Bank
Money_transfer
![Page 4: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/4.jpg)
LSR
Specifying test cases• It then makes sense
to specify these complex test cases
• « I want a test case that will execute the Money_transfer procedure successfully! »
• It is the notion of test purpose!
?Money_Transfer(1000000€)
!OK
accept
![Page 5: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/5.jpg)
LSR
The COTE project• COTE:
– conformance testing of components vs UML specifications
– Softeam, France Telecom, Gemplus, IRISA, LSR/IMAG
– Languages and tools to express and exploit test purposes and test cases
– (compilers from test cases to target technologies)
• This work comes from a brainstorming activity around the notion of test purpose within the COTE project
![Page 6: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/6.jpg)
LSR
The notion of test purposeTest purpose :
Description of a precise goal of thetest case, in terms of exercising a particular execution path or verifying the compliance with a specific requirement
[ISO IEC 9646-1 …conformance testing methodology and framework…]
• This notion is used:– As a structuring element in industrial test suites– In some tools based on formal specifications
(SAMSTAG, TGV, TorX)
![Page 7: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/7.jpg)
LSR
Test purposes in TGV• TGV is a tool developed jointly by IRISA
(Rennes) and Vérimag (Grenoble)
• Test synthesis tool
• Given– A specification of the system under test– A test purpose
It generates one corresponding test case.
![Page 8: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/8.jpg)
LSR
Example : coffee machine
?coin(1)
?coin(1)
?coin(2)
Pay 1 or 2 coins
?sugar
Ask for sugar (optional)
?tea?tea
?coffee
?coffeeAsk for tea or coffee
!coffee
!coffee!tea
!tea
!sugar
!sugar
Get the beverage
Loop
![Page 9: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/9.jpg)
LSR
TGVTGV
?coin(1)
?coin(1)
?coin(2)
?sugar
?tea?tea
?coffee
?coffee
!coffee
!coffee!tea
!tea
!sugar
!sugar
Dynamic specification
?coin(1)
?coin(1)
?coin(2)
?sugar
?tea?tea
?coffee
?coffee
!coffee
!coffee!tea
!tea
!sugar
!sugar
A sample synthesis
!coffee
accept
Test purpose
?coin(2)
?coffee
!coffee
Test case
![Page 10: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/10.jpg)
LSR
Test purposes as specification
• A test purpose (P) is an abstraction of test cases
• In TGV, a partial sequence of the test cases
• A test purpose specifies a set of test cases
!coffee
accept?coin(2)
?coffee
!coffee
?coin(1)
?coin(1)
?sugar
?tea?coffee
!coffee
?coin(1)
?coin(1)
?coin(2)
?sugar
?tea
?coffee
!coffee
!tea
!sugar
![Page 11: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/11.jpg)
LSR
Benefits of abstraction
• Capture the essence of the test(here test the delivery of coffee)
• Shorter to state than the full test case(clerical completion left to the tool)
• Robust to evolutions of the dynamic specification(test cases are affected more often than test purposes, e.g. pay 3 instead of 2)
!coffee
accept
![Page 12: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/12.jpg)
LSR
Relations between test purposes (P),
test cases (C) and specifications (S)
![Page 13: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/13.jpg)
LSR
Weakly_refines(C,P)• A test case « refines » the test
purpose(if it includes the partial sequence)
• But several irrelevant test cases refine the test purpose also. Weak refinement!
!coffee
accept
?coin(2)
?coffee
!coffee
test case not conforming to the specification
!coffee
![Page 14: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/14.jpg)
LSR
refines(C,P,S)• The refinement takes place in the
context of the specification
!coffee
accept?coin(2)
?coffee
!coffee
?coin(1)
?coin(1)
?coin(2)
?sugar
?tea?tea
?coffee
?coffee
!coffee
!coffee!tea
!tea
!sugar
!sugar
refines
![Page 15: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/15.jpg)
LSR
weakly_refines vs refines• Refines is a ternary relation
• Weakly_refines is defined asweakly_refines(C,P) S refines(C,P,S)
refines
weakly_refinesTest case(C)
Specification(S)
Test Purpose(P)
![Page 16: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/16.jpg)
LSR
Semantics for refines(C,P,S)
• Depending on the languages used for C, P, and S, a semantics can be defined for refines(C,P,S)
• Semantics of refines in TGV: C is one of the paths of the synchronous product of P and S
• See also, in the paper:semantics of a test refinement relation in the context of the B method
![Page 17: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/17.jpg)
LSR
Semantics of weakly_refines
• As a consequence:weakly_refines(C,P) S refines(C,P,S)
gives a semantics to weakly_refines on top of the semantics of refines
![Page 18: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/18.jpg)
LSR
conforms(C,S)
conforms(C,S) P refines(C,P,S)
• Conforms checks that the test case corresponds to the specification
refines
Test case(C)
Specification(S)
Test Purpose(P)
conforms
![Page 19: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/19.jpg)
LSR
consistent(P,S)
consistent(P,S) C refines(C,P,S)
• Consistent checks if the test purpose is compatible with the specification
refines
cons
isten
t
Test case(C)
Specification(S)
Test Purpose(P)
![Page 20: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/20.jpg)
LSR
Summary
• If refines(C,P,S) has a semantics,it gives a semantics to the binary relations!
• In some cases, it is possible to systematically construct the third element of the relation
refines
weakly_refines
cons
isten
t
Test case(C)
Specification(S)
Test Purpose(P)
conforms
![Page 21: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/21.jpg)
LSR
Test specificationvs
software specification
![Page 22: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/22.jpg)
LSR
refinesSWProgram
(M)
SoftwareSpecification
(SS)
refines
Test case(C)
Specification(S)
Test Purpose(P)
![Page 23: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/23.jpg)
LSR
Automated Software Engineering
• Formal specs as a basis for software tools:– Verification of specification consistency – Program synthesis– Program verification
![Page 24: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/24.jpg)
LSR
Specification consistencyoften demonstrated by proving:
M refinesSW(M,SS)(there exists a model which refines the specification)
corresponds toconsistent(P,S) C refines(C,P,S)
(a test purpose makes sense if there exists a test case)
Tool
Detection of inconsistentpurposes(e.g. !sugar comes after !coffee)avoids useless searches
!coffee
accept
reject
?coin(*)
!sugar
![Page 25: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/25.jpg)
LSR
SynthesisMajor theme at ASE
synthesisSW(SS) M
corresponds totest synthesis
e.g., TGV can be seen as a function
synthesisTGV(P,S) C
Good news for tools
The search domain is restricted by S(more decidable?, less complex?)
![Page 26: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/26.jpg)
LSR
Verificationchecks refinesSW(M,SS)
(program M is a valid refinement of specification SS)
corresponds to3 possible test case verifications– refines(C,P,S)
C is a refinement of P in the context of S– weakly_refines(C,P)
C is a refinement of P in some context– conforms(C,S)
C is a test case for S
![Page 27: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/27.jpg)
LSR
Tools for test verification• Synthesis involves searching a huge
space of solutions
• If verification is cheaper than synthesis,it provides a natural complement.
• Two examples:– Evolution of specifications– Test case retrieval
![Page 28: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/28.jpg)
LSR
Evolution of specifications• Evolution of S into S’
– Given a set of (Ci,Pi) for S– Verify refines(Ci,Pi,S’) for each (Ci,Pi)– When it fails, verify conforms(Ci,S’)– A second failure reveals regression of S’
• Example:No need to recompute existing test cases if
the coffee machinealso serves hot chocolate
?tea?coffee
?choc
![Page 29: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/29.jpg)
LSR
Test case retrievalGiven a large set of test cases
– Retrieve a test that weakly_refines a new test purpose
– Minimize the set by identifying test cases that weakly_refine several test purposes
!coffee
accept
?coin(2)
?coffee
!coffeeweakly_refines
weakly_refines
?coin(2)
accept
![Page 30: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/30.jpg)
LSR
Related workand
conclusion
![Page 31: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/31.jpg)
LSR
Test selection criteria
• 1st category: domain independent– Coverage criteria– Fault models
(mutation testing)– Analysis rules,
strategies– Random testing
• 2nd category: exploits domain knowledge– Test purposes
(based on reqts)– Operational profiles
• Must be consistent with the reference
Synthesis tool
Test selection criterion(aka test requirement)
Test case(s)Reference
(specification, program)
![Page 32: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/32.jpg)
LSR
Conclusion• Test purposes are a specification of test
cases!
• 4 relations between C, P, and S– Not specific to a given tool or language– Not specific to an application domain
(TGV: reactive systems, B: transf. systems)
• Tools: from ASE to Automated Test Eng.– Based on these 4 relations– Test verification complements test synthesis– Applications to specification evolution, test
case retrieval,…
![Page 33: LSR Test purposes: adapting the notion of specification to testing Yves Ledru, L. du Bousquet, P. Bontron, O. Maury, C. Oriat, M.-L. Potet LSR/IMAG Grenoble,](https://reader035.vdocuments.us/reader035/viewer/2022070415/5697c00d1a28abf838cc9951/html5/thumbnails/33.jpg)
LSR
Interactive Coffee Crafting