École de technologie supÉrieure – montrÉal
TRANSCRIPT
![Page 1: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/1.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
ScenarioScenario--Based BlackBased Black--Box Testing Box Testing in COSMICin COSMIC--FFPFFP
Manar Abu Talib, Olga Ormandjieva, Alain Abran, Luigi Buglione
2nd Software Measurement European Forum 2005
![Page 2: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/2.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
AgendaØ IntroductionØ COSMIC-FFP Measurement MethodØ Scenario-Based TestingØ Using Entropy for assigning priorities to
test casesØ Discussion and Next Steps
2
![Page 3: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/3.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
F At SMEF 2004, papers were presented on : – Functional sizing with COSMIC-FFP – Functional Complexity measurement with Entropy
concepts.
– Both types of measurement methods share significant similarities & some differencesF Presented at IWSM 2004 in Berlin
Introduction
3
![Page 4: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/4.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
F Functional complexity and Entropy are playing a significant role in testing – in general
F Introducing Entropy in software testing strategies could be of interest.
F In this presentation we investigate a contribution of COSMIC-FFP in introducing testing strategies and in leveraging the benefits of using Entropy concepts to assign priorities to test cases.
4
Introduction
![Page 5: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/5.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
Context 1F Software Engineering: A discipline for the systematic production and maintenance of large and complex software systems
F Software Measurement: a mechanism to provide feedback on software quality
5
![Page 6: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/6.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
1980 1985 1990 1995 2000
Allan Albrecht
FPA
IFPUG 4.0
IFPUG 4.1
MkIIFPA
MkIIFPA 1.3
NESMA
COSMICCOSMIC--FFP FFP –– ISO 19761ISO 19761
COSMIC-FFP Measurement Method
6
![Page 7: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/7.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
COSMIC-FFP Measurement Method
or
Engineered
Devices
Stor
age
Har
dwar
e
SOFTWARE
ENTRIES
EXITS
« Frontend »USERS
READS
WRITES
« Backend »
EXITS
ENTRIES
I/O
Har
dw
are
BO
UN
DA
RY
7
![Page 8: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/8.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
Context 2
F Testing: Software testing consists of the dynamicverification of the behaviour of a program on a finiteset of test cases, suitably selected from the usually infinite executions domain, against the specified expected behaviour
(SWEBOK definition – ISO TR 19759)
8
![Page 9: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/9.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
« Expected » means:
Expected Output
User Expectations, Specification, anticipated behaviour requirements
or reasonable expectations Program Under Test
Observed Output
Are they conformed?
Fault is detected Conformed
No Yes
Test Set
Oracle Methodology 9
![Page 10: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/10.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
Context 2
Types of Testing:Black-box:
F test cases are generated and executed from the specification of the required functionality at defined interfaces
White-box:F test suite is generated from the implemented structures
Gray-box: F the modular structure of the implementation is known but not the
details of the programs within each component
10
![Page 11: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/11.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
F Scenario based testing is a typical black box testing methodology at the system level
F Scenarios depict the sequence of executions of the system, and the test cases can be derived from the use-case model and its corresponding UML diagrams
Scenario-Based Testing
11
![Page 12: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/12.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
Related Work on Scenario-Based Testing
Use cases
Scenarios Thin-Threads
Test cases
Use Case: is defined as a collection of related scenarios that describes actors and operations in a system
Scenario: a specific sequence of actions and interactions between actors and the system
Thin Thread: is a minimum usage scenario in a software system. The execution of a thin thread demonstrates that a method performs a specified function
12
![Page 13: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/13.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
COSMIC-FFP and Scenarios
Class attributeData attributeClassData groupOperation (message)Data movementUse caseFunctional processActorUserUse case diagramBoundaryRelated UML ConceptsCOSMIC-FFP Concepts
COSMIC-FFP Concepts and their related UML concepts
13
![Page 14: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/14.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
ScenarioScenario--Based Testing with COSMICBased Testing with COSMIC--FFPFFPØ Goals:Ø Reduce the number of test cases while keeping the
highest fault coverage within the budgetary constraints
Ø Fault coverage: The power of a test cases generation technique to detect faults in an implementation
14
![Page 15: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/15.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
Our Testing ApproachOur Testing Approach
VTest Selection Domain
STCGenerated Test Set
Test Case Generation Algorithm
Metric-Based Test Set Partition Algorithm
Equivalence Classes TS1, ..., TSi, ...
Optimal Set of Test Cases
Test Cases Selection Algorithm
15
![Page 16: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/16.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
Test Case GenerationTest Case Generation
The Procedure for
Generation of the Test cases
Use-CaseModel
Each Scenariois a Test Case
Ø A test case is mapping the scenarios to a sequence of events in time (or data movements in COSMIC-FFP)
16
![Page 17: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/17.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
MetricMetric--Based Test Case Partitioning AlgorithmBased Test Case Partitioning Algorithm
Precondition :{ STC ≠ ∅ ∧ εmax > 0 ∧ ∀ i • TSi = ∅}
-sets εmax (predefined value based on experimental work)While STC ≠ ∅{
- selects the longest test case t in set STC-sets ε to 0
- moves t from the set STC to the set TSiWhile ( STC ≠ ∅ ∧ ( ε < εmax) {
-chooses a test case t in set STC whose distance to the set TSi is minimum in order to put similar test cases into one equivalence class-sets ε to the distance between t and TSi-if ε < εmax then moves t from STC to set TSi
} // equivalence class TSi is createdi=i+1;}
17
![Page 18: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/18.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
F Distance between two test cases t1 and t2 is:
td ( t1, t2) = similarity (t1 , t2) * dissimilarity (t1 , t2) * e ^ l
Where:F l = -(length (t1)/length (ts))F The similarity (t1 , t2) = 2 (- length (LCP ( t1, t2) ) where LCP is the
longest common prefix of the two test cases. The range of the similarity measure is between 0 and 1
F The dissimilarity measure between two test cases t1 and t2 is calculated as the number of elementary transformations that are minimally needed to transform the string (t1/LCP ( t1, t2)) into the string (t2/ (LCP ( t1, t2))
Distance between Test CasesDistance between Test Cases
18
![Page 19: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/19.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
F The distance between the following two test cases: e1.e2.e3.e4.e1 and e1.e1.e1.e2.e1 = ½ * 3 * e ^ -1
F The distance formula td ( t1, t2) indicates that the greater the distance you have between two test cases the more they will differ from each other
Distance between Test Cases (Example)Distance between Test Cases (Example)
19
![Page 20: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/20.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
Ø In our approach, the Entropy-based functional complexity measure (FC) is used to prioritize the test cases.
Ø The formula that is used to calculate the functional complexity is as follows:
Ø More and diverse functionality of the system would lead to bigger portion of the system involved in that usage.
Ø The entropy calculated on a sequence of events abstracting a scenario quantifies the average information interchange for a given usage of the system.
Priority of Test CasesPriority of Test Cases
)/(log)/( - 1
2 NEfNEfFCn
iii∑
=
=
)/(log)/( - 1
2 NEfNEfFCn
iii∑
=
=
20
![Page 21: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/21.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
F FC is calculated on a sequence of events: abstracting a scenario quantifies the average information interchange for a given usage of the system– Example: FC of sequence (e1.e2.e3.e4.e1) for e1 has a higher
average information content than sequence (e1.e1.e1.e2.e1)
– It is then more complex than another and it will have greater priority than the second test case F Eg. more system usage
Functional Complexity (FC) of a Test CaseFunctional Complexity (FC) of a Test Case
21
![Page 22: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/22.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
For all non-empty equivalent classes TSi
Step 1. Choose the highest priority test case from the equivalence class TSi;Step 2. Add the chosen test case to the Optimal Set and remove it from the equivalence class TSi;Step 3. Increase the total testing cost in CIf (The total testing cost exceeds a given budget Cmax)
End the algorithm
End For
Test Selection AlgorithmTest Selection Algorithm
22
![Page 23: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/23.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
F COSMIC-FFP method was extended for testing purposes by combining the functions measured by the COSMIC-FFP measurement procedure with the black box testing strategy
F Applicability of using Entropy measurement with COSMIC-
FFP for assigning priorities to test cases
ConclusionConclusion
23
![Page 24: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/24.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
Future Work DirectionsFuture Work Directions
Ø Apply this testing strategy into case studiesØ Will facilitate implementation in industry
Ø Entropy is used in Reliability Estimation:Ø Investigate the use of COSMIC-FFP for the ‘reliability
estimation’ of the software
24
![Page 25: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/25.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
References1. [1] ABRAN, A., DESHARNAIS, J.-M., OLIGNY, S., ST-PIERRE, D. AND SYMONS, C., COSMIC Implementation guide to ISO 19761,
École de technologie supérieure - Université du Québec, Montréal. 2003, URL: http://www.cosmicon.com2. [2] ABRAN, A., ORMANDJIEVA, O. AND TALIB, M.A. Functional Size and Information Theory-Based Functional Complexity
Measures: Exploratory study of related concepts using COSMIC-FFP measurement method as a case study, 14th International Workshop of Software Measurement (IWSM-MetriKon 2004), Springer-Verlag, Konigs Wusterhausen, Germany, 2004, pp. 457-471.
3. [3] ALAGAR, V.S., CHEN, M., ORMANDJIEVA, O. AND ZHENG, M., Automated Generation of Test Suits from Formal Specifications of Real Time Reactive Systems. Submitted to a Journal, 2004.
4. [4] BAI, X., PENG, L.C. AND LI, H., An Approach to Generate Thin Threads from UML Diagrams. Technical Report, TR-03-12, Software Engineering Research Group, School of Computer and Information Science, Edit Cowan University, 2002.
5. [5] BAI, X., TSAI, W.T., FENG, K. AND L.YU, Scenario-based Modeling and Its Applications to Object Oriented Analysis Design and Testing, Proceedings of IEEE Workshop on Object-Oriented Real-Time Dependable Systems (WORDS 2002), San Diego (USA), 7-9 January 2002, pp. 253-270
6. [6] BEIZER, B. Software Testing Techniques. Van Nostrand Reinhold, 2/e, 1990, ISBN 18503288037. [7] BERTOLINO, A. Knowledge Area Description of Software Testing Guide to the SWEBOK. URL: http://www.swebok.org , 2004.8. [8] CHOW, T.S. Testing Software Design Modeled by Finite State Machines, IEEE Transactions on Software Engineering, Vol.
SE-4, No.3, 1978, pp.178-1879. [9] DAVIS J.S. & LEBLANC R.J., A Study of the Applicability of Complexity Measures, IEEE Transactions on Software
Engineering, Vol. 14 No.9, September 1988, pp.1366-137210. [10] EN-NOUAARY, A., DSSOULI, R. & KHENDEK, F., Timed Wp-Method: Testing Real Time Systems, . IEEE Transactions on
Software Engineering, Vol. 28 No.11, November 2002, pp.1023-103811. [11] ISO/IEC19761. Software Engineering - COSMIC-FFP - A functional size measurement method, International Organization for
Standardization - ISO, Geneva. 2003.12. [12] JENNER, M., Automation of Counting of Functional Size Using COSMIC-FFP in UML, in Proceedings of the 12th International
Workshop on Software Measurement (IWSM 2002), Magdeburg (Germany), October 2002, pp.43-51.13. [13] LARMAN, C. Applying UML and Patterns, An introduction to Object Oriented Analysis and Design and the Unified Process.
Prentice-Hall, 2001, ISBN 013092569114. [14] WEISS, S.N. AND WEYUKER, E.J., An Extended Domain-Bases Model of Software Reliability, IEEE Transaction on Software
Engineering, Vol.14 No.10, October 1988, pp.1512-152415. [15] PHILIPPE KRUCHTEN, The Rational Unified Process : An Introduction, 2nd Edition, Addison-Wesley Pub Co, 2000, ISBN:
020170710
25
![Page 26: ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL](https://reader034.vdocuments.us/reader034/viewer/2022051716/589efb781a28abb8498c338f/html5/thumbnails/26.jpg)
ÉCOLE DE TECHNOLOGIE SUPÉRIEURE – MONTRÉAL - CANADA
CONCORDIA UNIVERSITY &
Thank You !
Questions?
26