code and functional coverage tutorial - ibm and functional coverage tutorial may '99 shmuel ur...

Code and Functional Coverage Tutorial

May '99

Shmuel UrShmuel UrIBM Research Laboratory in HaifaIBM Research Laboratory in Haifa

Explaining the System....Explaining the System....Explaining the System....Explaining the System....

1-2

Outline Outline Outline Outline

Introduction to CoverageIntroduction to Coverage

Code Coverage Code Coverage

RegressionRegression

Functional CoverageFunctional Coverage

Class ExerciseClass Exercise

Automatic Coverage Directed GenerationAutomatic Coverage Directed Generation

Comparing Functional Coverage to Code CoverageComparing Functional Coverage to Code Coverage

Conclusions Conclusions

What is Coverage?What is Coverage?What is Coverage?What is Coverage?

3-4

Testing is based on samples Testing is based on samples

Cannot run all possible testsCannot run all possible tests

Need to know that all areas of design are tested Need to know that all areas of design are tested

A test suite is not measured by its size A test suite is not measured by its size

Solution: Coverage AnalysisSolution: Coverage Analysis

Why Coverage?Why Coverage?Why Coverage?Why Coverage?

CoverageCoverage is any metric of completeness with respect to is any metric of completeness with respect to a test selection criterion.a test selection criterion.

The main ideas behind coverage:The main ideas behind coverage:

Systematically create a list of tasks (the testing Systematically create a list of tasks (the testing requirements)requirements)Check that each task is covered during the testingCheck that each task is covered during the testing

Definition of Coverage Definition of Coverage Definition of Coverage Definition of Coverage

5-6

Coverage TaskCoverage Task - a Boolean function on a trace - a Boolean function on a trace

Statement 534 was executedStatement 534 was executed

A plane landed on Friday the 13thA plane landed on Friday the 13th

Coverage ModelCoverage Model - a cohesive set of coverage tasks - a cohesive set of coverage tasks

Execution of each statement in a programExecution of each statement in a program

A plane landedA plane landed every day of the year every day of the year

IIIlegal TaskIlegal Task - a coverage task that should not occur - a coverage task that should not occur

Task Coverage List (TCL)Task Coverage List (TCL) - a list of all the legal tasks - a list of all the legal tasks

Other DefinitionsOther DefinitionsOther DefinitionsOther Definitions

Coverage GoalsCoverage GoalsCoverage GoalsCoverage Goals

Measure the "quality" of a set of testsMeasure the "quality" of a set of tests

Supplement test specifications by pointing to Supplement test specifications by pointing to untested areasuntested areas

Help create regression suitesHelp create regression suites

Provide a stopping criteria for unit testingProvide a stopping criteria for unit testing

7-8

Risks of Using CoverageRisks of Using CoverageRisks of Using CoverageRisks of Using CoverageUsing coverage without commitment to using the Using coverage without commitment to using the resultsresultsGenerating simple tests to cover specific uncovered Generating simple tests to cover specific uncovered taskstasks

The painted wall analogy:The painted wall analogy:

Some coverage models are ill-suited to deal with Some coverage models are ill-suited to deal with common problemscommon problems

Missing codeMissing codeLow coverage goalsLow coverage goals

Automatic Test Generation and CoverageAutomatic Test Generation and CoverageAutomatic Test Generation and CoverageAutomatic Test Generation and Coverage

Main goal is to improve biasingMain goal is to improve biasing

Working with feedback (closed loop)Working with feedback (closed loop)

Avoiding the risk of specific biasingAvoiding the risk of specific biasing

9-10



Code CoverageCode Coverage






ConclusionsConclusions

Provides a quantitative measurement of the testing Provides a quantitative measurement of the testing efforteffort

Can assist in directing the tester's future effortsCan assist in directing the tester's future efforts

Can demonstrate redundancy in test casesCan demonstrate redundancy in test cases

Can be used as entry/exit criteria between test phasesCan be used as entry/exit criteria between test phases

Studies have demonstrated a direct correlation between Studies have demonstrated a direct correlation between increased code coverage and defect detectionincreased code coverage and defect detection

What is the Value of Code Coverage?What is the Value of Code Coverage?What is the Value of Code Coverage?What is the Value of Code Coverage?

11-12

Types of Code Coverage ModelsTypes of Code Coverage ModelsTypes of Code Coverage ModelsTypes of Code Coverage Models

Control flowControl flowCheck that the control flow of the program has been fully Check that the control flow of the program has been fully exercisedexercised

Data flowData flowModels that look at the flow of data in, and between, Models that look at the flow of data in, and between, programsprograms

MutationMutationModels that check directly for common bugsModels that check directly for common bugs

Visual coverage modelsVisual coverage modelsProgram types - OO, concurrent, distributiveProgram types - OO, concurrent, distributive

Control Flow ModelsControl Flow ModelsControl Flow ModelsControl Flow Models

Routine (function entry)Routine (function entry)

Function callFunction call

Function returnFunction return

Statement (block)Statement (block)

BranchBranch

Multi-conditionMulti-condition

LoopLoop

13-14

C-Use (Computational use)C-Use (Computational use)

P-Use (Predicate variable use)P-Use (Predicate variable use)

All UsesAll Uses

Data Flow ModelsData Flow ModelsData Flow ModelsData Flow Models

Weak MutationWeak Mutation

Strong MutationStrong Mutation

Examples:Examples:<= ==<= ==> >=> >=& &&& &&

Mutation (Relational Operation) CoverageMutation (Relational Operation) CoverageMutation (Relational Operation) CoverageMutation (Relational Operation) Coverage

15-16

Hierarchy of Coverage ModelsHierarchy of Coverage ModelsHierarchy of Coverage ModelsHierarchy of Coverage Models

Function EntryFunction Entry

Function ReturnFunction Return

BlockBlock

Function CallFunction Call

DecisionDecision

P-UseP-Use

All-UsesAll-Uses

C-UseC-Use

Hardware ModelsHardware ModelsHardware ModelsHardware Models

State machine coverage - states, transitionsState machine coverage - states, transitions

All values for signalsAll values for signals

Stuck AtStuck At

17-18

What is What is What is What is xxxxSuds (from Bellcore)?Suds (from Bellcore)?Suds (from Bellcore)?Suds (from Bellcore)?

Principally a code coverage tool suitePrincipally a code coverage tool suite

Parses C and C++ source codeParses C and C++ source code

Runs on AIX, Solaris, Linux, WinNTRuns on AIX, Solaris, Linux, WinNT

xxATACATAC - a test coverage measurement, management & test - a test coverage measurement, management & test creation toolcreation toolxxVueVue - supports software maintenance by visualizing code - supports software maintenance by visualizing code featuresfeaturesxxSliceSlice - a dynamic program slicing debugger that graphically - a dynamic program slicing debugger that graphically localizes bugslocalizes bugsxxProfProf - an execution count-based profiler - an execution count-based profilerxxDiffDiff - displays differences between two files in a more - displays differences between two files in a more understandable way understandable way xFindxFind - uses date seeds to locate Y2K sensitivities in code - uses date seeds to locate Y2K sensitivities in codexRegressxRegress - assists in the selection of a minimum set of tests - assists in the selection of a minimum set of tests to cover changed code to cover changed code

xSuds Tool Suite ComponentsxSuds Tool Suite ComponentsxSuds Tool Suite ComponentsxSuds Tool Suite Components

19-20

How Does How Does How Does How Does xxxxSuds Work?Suds Work?Suds Work?Suds Work?

21-22

What is Missing from What is Missing from What is Missing from What is Missing from xxxxSuds?Suds?Suds?Suds?

Signing on coverage tasksSigning on coverage tasks

Removing covered coverage tasks (to improve Removing covered coverage tasks (to improve performance)performance)

Coverage modelsCoverage models

Operating systemsOperating systems

Programming languagesProgramming languages

Creating Your Own Code Coverage ToolCreating Your Own Code Coverage ToolCreating Your Own Code Coverage ToolCreating Your Own Code Coverage Tool

Avoid if possible:Avoid if possible:It is a lot of workIt is a lot of workIt is bug proneIt is bug proneIt is slowIt is slow

When creating such a tool, consider:When creating such a tool, consider:InstrumentationInstrumentationTracesTracesOff-line vs. on-line task updatingOff-line vs. on-line task updatingReducing the size of tracesReducing the size of tracesIntegrating informationIntegrating information

25-26

Test Coverage for Visual Programs Test Coverage for Visual Programs Test Coverage for Visual Programs Test Coverage for Visual Programs

Visual elements to be covered:Visual elements to be covered:ConnectionsConnectionsEventsEventsActionsActionsAttributesAttributes

Path Coverage - First ViewPath Coverage - First ViewPath Coverage - First ViewPath Coverage - First View

27-28

Path Coverage - Second ViewPath Coverage - Second ViewPath Coverage - Second ViewPath Coverage - Second View

A TEST!!! A TEST!!! A TEST!!! A TEST!!! From Cem Kaner "Software negligence and testing coverage"From Cem Kaner "Software negligence and testing coverage"From Cem Kaner "Software negligence and testing coverage"From Cem Kaner "Software negligence and testing coverage"

A program fails in the field and someone dies. At the A program fails in the field and someone dies. At the trial, the QA Manager takes the stand and the trial, the QA Manager takes the stand and the plaintiff's lawyer states three facts:plaintiff's lawyer states three facts:

Failure occurred when the program reached a specific line of code.Failure occurred when the program reached a specific line of code.This line had never been tested, therefore the bug had not been This line had never been tested, therefore the bug had not been found before the product was released.found before the product was released.Had the QA used the coverage monitor and tested to complete Had the QA used the coverage monitor and tested to complete coverage, this bug would have been found and the victim would be coverage, this bug would have been found and the victim would be alive today.alive today.

The plaintiff's lawyer therefore proved that the The plaintiff's lawyer therefore proved that the company was negligent and that the victim's family company was negligent and that the victim's family would win the lawsuit.would win the lawsuit.

29-30










A A regression suiteregression suite is a set of tests that are run on the is a set of tests that are run on the application after every software or data change, and application after every software or data change, and every night or every weekend, in order to check that no every night or every weekend, in order to check that no new bugs have been introduced.new bugs have been introduced.

This is vital because every bug fix, on the average, This is vital because every bug fix, on the average, introduces one fifth of a bugintroduces one fifth of a bug..

RegressionRegressionRegressionRegression

31-32

Contradictory RequirementsContradictory RequirementsContradictory RequirementsContradictory Requirements

A regression suite must be: A regression suite must be:

Comprehensive so that it is likely to catch all the Comprehensive so that it is likely to catch all the !@#$% bugs introduced !@#$% bugs introduced

Small so that it can economically be executed many Small so that it can economically be executed many times times

How can we make our regression suite small and How can we make our regression suite small and comprehensive?comprehensive?

Let S = {1 ... M}Let S = {1 ... M}

Let T = {TLet T = {T11 ... T... Tnn} be a set of subset of S} be a set of subset of S

The set cover problem:The set cover problem:

Find the smallest subset of T that covers SFind the smallest subset of T that covers S

The set cover problem is NP-C, however, there are a The set cover problem is NP-C, however, there are a number of good algorithms for it.number of good algorithms for it.

The Set Cover Problem The Set Cover Problem The Set Cover Problem The Set Cover Problem

33-34

Find all the tests that contain unique tasks Find all the tests that contain unique tasks Loop Loop

Remove all the tasks covered by selected testsRemove all the tasks covered by selected tests Choose the test that covers most remaining tasks Choose the test that covers most remaining tasks

Implementation issues:Implementation issues: Input type (sparse or dense)Input type (sparse or dense) Frequency of use Frequency of use Real data vs. random data Real data vs. random data

The Greedy HeuristicThe Greedy HeuristicThe Greedy HeuristicThe Greedy Heuristic

Compaction StrategiesCompaction StrategiesCompaction StrategiesCompaction Strategies

SimpleSimple - if a test introduces a new task, choose it. - if a test introduces a new task, choose it.

Merge NMerge N - add every set of N new tests to the - add every set of N new tests to the regression suite, and compact them into a new regression suite, and compact them into a new regression suite. (Size may decrease!)regression suite. (Size may decrease!)

35-36

Insight on Regression Suite CompactionInsight on Regression Suite CompactionInsight on Regression Suite CompactionInsight on Regression Suite Compaction

An improvement of fifty percent is to be expected in An improvement of fifty percent is to be expected in MergeMerge over over Simple.Simple.

Real information does not behave in the same way as Real information does not behave in the same way as randomly generated information.randomly generated information.

The real resource limitation is space.The real resource limitation is space.

Incremental compaction is useful.Incremental compaction is useful.

The size of the regression suite does not change by The size of the regression suite does not change by much, once the set is fully covered.much, once the set is fully covered.










37-38

Functional CoverageFunctional CoverageFunctional CoverageFunctional Coverage

Summary of ATAC on a Number of TestsSummary of ATAC on a Number of TestsSummary of ATAC on a Number of TestsSummary of ATAC on a Number of Tests

39-40

Did every agency asked for every type of Did every agency asked for every type of ticket? ticket?

Did the agencies ask for service at the same Did the agencies ask for service at the same time?time?

How many times did Lion wait more than 10 How many times did Lion wait more than 10 minutes?minutes?

Is the system fair?Is the system fair?

Coverage Questions Not Answered by Coverage Questions Not Answered by Coverage Questions Not Answered by Coverage Questions Not Answered by Code Coverage ToolsCode Coverage ToolsCode Coverage ToolsCode Coverage Tools

Example of a Simple Functional Example of a Simple Functional Example of a Simple Functional Example of a Simple Functional Coverage ModelCoverage ModelCoverage ModelCoverage Model

A model is composed of the following parts:A model is composed of the following parts:A semantic description (story) of the functional coverage A semantic description (story) of the functional coverage model which contains attributesmodel which contains attributes

Has every agency {Agency name} Has every agency {Agency name} asked for every type of show {Service type} asked for every type of show {Service type}

A tuple of attributes <Agency name, Service type>A tuple of attributes <Agency name, Service type>Possible values for each attribute (e.g., Agency name: Possible values for each attribute (e.g., Agency name: Disney, Fox, STAR, LION)Disney, Fox, STAR, LION)Restrictions on the cross-productRestrictions on the cross-product

41-42

What are Attributes Made Out Of?What are Attributes Made Out Of?What are Attributes Made Out Of?What are Attributes Made Out Of?

Functional coverage attributes may come from any Functional coverage attributes may come from any relevant source.relevant source.Our favorite sources include:Our favorite sources include:

InputsInputsOutputsOutputsProgram variablesProgram variablesFunctions on program variablesFunctions on program variablesHigh level descriptions not mapped to variablesHigh level descriptions not mapped to variables

"Homemade" Functional Coverage "Homemade" Functional Coverage "Homemade" Functional Coverage "Homemade" Functional Coverage ImplementationImplementationImplementationImplementation

Instrument your code with the following statement:Instrument your code with the following statement:

print("Agency name = %d Service type = %d \n",agency,service);print("Agency name = %d Service type = %d \n",agency,service);

Execute > traceExecute > trace

cat *.trace | sort -unique > outputcat *.trace | sort -unique > output

What are the problems?What are the problems?

43-44

Simple Functional Coverage -Simple Functional Coverage -Simple Functional Coverage -Simple Functional Coverage -Attributes Attributes Attributes Attributes

Simple Functional Coverage - Simple Functional Coverage - Simple Functional Coverage - Simple Functional Coverage - Cases Cases Cases Cases

45-46

/* example of call: traceFunc("showbiz",agency,service); *//* example of call: traceFunc("showbiz",agency,service); */#include <stdio.h>#include <stdio.h>#include <sys/types.h>#include <sys/types.h>#include <sys/errno.h>#include <sys/errno.h>#include <sys/time.h>#include <sys/time.h>

FILE *trace_file_out;FILE *trace_file_out;

void traceFunc(char *modelname,char *agency,int service) {void traceFunc(char *modelname,char *agency,int service) { struct timeval val; struct timeval val; struct timezone zone; struct timezone zone;

char filename[100]; char filename[100]; static int called = 0; static int called = 0; if ( called == 0 ) { if ( called == 0 ) { gettimeofday(&val,&zone); gettimeofday(&val,&zone); sprintf(filename,"%s_trace_%d",modelname,val.tv_sec); sprintf(filename,"%s_trace_%d",modelname,val.tv_sec); trace_file_out = fopen(filename,"w"); trace_file_out = fopen(filename,"w"); } } called==1; called==1; fprintf(trace_file_out,"%s %d \n",agency,service); fprintf(trace_file_out,"%s %d \n",agency,service); return; return;}}

Simple Functional Coverage - Simple Functional Coverage - Simple Functional Coverage - Simple Functional Coverage - FunctionsFunctionsFunctionsFunctions

Elementary Report Generation Screen for Elementary Report Generation Screen for Elementary Report Generation Screen for Elementary Report Generation Screen for the Ticket Examplethe Ticket Examplethe Ticket Examplethe Ticket Example

47-48

A Basic Functional Coverage ReportA Basic Functional Coverage ReportA Basic Functional Coverage ReportA Basic Functional Coverage Report

More Functional Models for the Simple More Functional Models for the Simple More Functional Models for the Simple More Functional Models for the Simple ExampleExampleExampleExample

1.1. <Agency name, Reply><Agency name, Reply>

2.2. <Agency name, Reply, Service><Agency name, Reply, Service>

3.3. <Agency name, Reply, Service, Time><Agency name, Reply, Service, Time>

4.4. <Agency1 calling, Agency2 calling, Agency3 calling, <Agency1 calling, Agency2 calling, Agency3 calling, Agency4 calling>Agency4 calling>

5.5. <Agency1 reply, Agency2 reply, Agency3 reply, <Agency1 reply, Agency2 reply, Agency3 reply, Agency4 reply>Agency4 reply>

6.6. <Service, Service><Service, Service>

7.7. Round Robin Bits Round Robin Bits

49-50

Arbitration DataArbitration DataArbitration DataArbitration Data

Improvement Over TimeImprovement Over TimeImprovement Over TimeImprovement Over Time

51-52

How Restrictions are CreatedHow Restrictions are CreatedHow Restrictions are CreatedHow Restrictions are Created

The designer creates a list of restrictions for a model.The designer creates a list of restrictions for a model.

Coverage feedback modifies the restrictions.Coverage feedback modifies the restrictions.

Sub-models are used to economically check and Sub-models are used to economically check and refine the restrictions. refine the restrictions.

Restrictions are Self-correctingRestrictions are Self-correctingRestrictions are Self-correctingRestrictions are Self-correcting

53-54

Restrictions are Created HierarchicallyRestrictions are Created HierarchicallyRestrictions are Created HierarchicallyRestrictions are Created Hierarchically

Car Assembly Line ExampleCar Assembly Line ExampleCar Assembly Line ExampleCar Assembly Line Example

55-56

Simple Coverage ModelsSimple Coverage ModelsSimple Coverage ModelsSimple Coverage Models

< Car type, engine size, color>< Car type, engine size, color>

< Car type, total work time, total elapse time, day > < Car type, total work time, total elapse time, day >

< # cars in line, # of workers in line>< # cars in line, # of workers in line>

White Box Model - White Box Model - White Box Model - White Box Model - Cars Location on the Assembly LineCars Location on the Assembly LineCars Location on the Assembly LineCars Location on the Assembly Line

(CL1, CT1, CL2, CT2, CL3, CT3, CL4, CT4, CL5, CL6, (CL1, CT1, CL2, CT2, CL3, CT3, CL4, CT4, CL5, CL6, CL7, Problem)CL7, Problem)

Problem: (0 = none, 1 = full stop, 2 = electrical shortage)Problem: (0 = none, 1 = full stop, 2 = electrical shortage)So there are: 10So there are: 10..99..88..66..55..55..33..22..22..22..22..3 = 15,552,000 tasks ?3 = 15,552,000 tasks ?!!

Car Location Car Type1 0-9 Accord-Civic2 0-8 Accord-Civic3 0-7 Accord-Civic4 0-5 Accord-Civic5 0-4 Accord6 0-4 Accord7 0-2 Accord

57-58

Model RestrictionsModel RestrictionsModel RestrictionsModel Restrictions

CL1 >= CL2 >= CL3 >= CL4 >= CL5 >= CL6 >= CL7CL1 >= CL2 >= CL3 >= CL4 >= CL5 >= CL6 >= CL7 No car may pass another in the assembly line No car may pass another in the assembly line

If CT2 == Civic then CL2 > 4 or CL2 = 0 If CT2 == Civic then CL2 > 4 or CL2 = 0 If CT3 == Civic then CL3 > 4 or CL3 = 0 If CT3 == Civic then CL3 > 4 or CL3 = 0 If CT4 == Civic then CL4 > 4 or CL4 = 0 If CT4 == Civic then CL4 > 4 or CL4 = 0

Civic may only be at stages 5 to 9 for painting Civic may only be at stages 5 to 9 for painting

More Model RestrictionsMore Model RestrictionsMore Model RestrictionsMore Model Restrictions

Only one car can be at stages 3 and 7 Only one car can be at stages 3 and 7

Only one car can be at stages 5 and 6Only one car can be at stages 5 and 6

Only one car can be at stages 6 and 7Only one car can be at stages 6 and 7

Only one car can be at stages 1, 2 and 3Only one car can be at stages 1, 2 and 3

Only two Accords can be at stages 4, 5 and 6Only two Accords can be at stages 4, 5 and 6

Only 1415 legal tasks!!!Only 1415 legal tasks!!!

59-60

Restriction ImplementationRestriction ImplementationRestriction ImplementationRestriction Implementation

Simulation OutputsSimulation OutputsSimulation OutputsSimulation Outputs

61-62

Simulation OutputsSimulation OutputsSimulation OutputsSimulation Outputs

Feedback and Bugs FoundFeedback and Bugs FoundFeedback and Bugs FoundFeedback and Bugs Found

Not creating enough of the right kind of scenariosNot creating enough of the right kind of scenarios

Performance related bugsPerformance related bugs

Illegal scenarios foundIllegal scenarios found

63-64

Functional Coverage Models Functional Coverage Models Functional Coverage Models Functional Coverage Models

Archetypes Archetypes

PicturePicture

Black BoxBlack Box - based on interface details - based on interface details

White BoxWhite Box - based on internal details - based on internal details

Broken BoxBroken Box - combined internal and external - combined internal and external information information

Model Classification Based on Information Model Classification Based on Information Model Classification Based on Information Model Classification Based on Information SourcesSourcesSourcesSources

65-66

Model Classification Based on Temporal Model Classification Based on Temporal Model Classification Based on Temporal Model Classification Based on Temporal PropertiesPropertiesPropertiesProperties

SnapshotSnapshot - - a single point in timea single point in timeCould be abstract. For example, if two routines are Could be abstract. For example, if two routines are called sequentially, a model containing the values of called sequentially, a model containing the values of variables from both routines is considered a snapshot variables from both routines is considered a snapshot model even though the variables are not "alive" at the model even though the variables are not "alive" at the same time.same time.

TemporalTemporal - - a coverage model that checks scenariosa coverage model that checks scenarios Usually the type where event 1 happens first, then Usually the type where event 1 happens first, then event 2 , and then event 3. Coverage is measured on event 2 , and then event 3. Coverage is measured on the events (event 1 could have a few legal types), on the events (event 1 could have a few legal types), on time between events, and on the way the events time between events, and on the way the events interact.interact.

Special Functional Coverage ModelsSpecial Functional Coverage ModelsSpecial Functional Coverage ModelsSpecial Functional Coverage Models

StressStress - Coverage model that checks how many things - Coverage model that checks how many things happen within a short interval of time, or at the same happen within a short interval of time, or at the same time. Also a model that checks that resources have time. Also a model that checks that resources have been used to their full capacity.been used to their full capacity.

Statistically orientedStatistically oriented - Coverage model used to - Coverage model used to measure statistics such as how many times each task measure statistics such as how many times each task has been covered. The main motivation is usually has been covered. The main motivation is usually performance. (Do we need a stack that large?) performance. (Do we need a stack that large?)

67-68

Invent Your Own Model Invent Your Own Model Invent Your Own Model Invent Your Own Model

Ticket office Car assembly

Stress Statistically oriented

Snapshot Temporal

Black Box

White Box

Broken Box

Invent Your Own Model - cont.Invent Your Own Model - cont.Invent Your Own Model - cont.Invent Your Own Model - cont.

69-70

Model HierarchyModel HierarchyModel HierarchyModel Hierarchy

Building a model hierarchically, out of sub-models, is Building a model hierarchically, out of sub-models, is desirable for a number of reasonsdesirable for a number of reasons

Errors can be found earlierErrors can be found earlierImplementation is cleaner and fasterImplementation is cleaner and fasterRestrictions are more easily debuggedRestrictions are more easily debugged

Sub-models should be made out of semantically Sub-models should be made out of semantically related variablesrelated variablesHigh level models can be reused many timesHigh level models can be reused many times

Comet - COverage MEasurement ToolComet - COverage MEasurement ToolComet - COverage MEasurement ToolComet - COverage MEasurement Tool

A generic functional coverage tool for use in a heavy A generic functional coverage tool for use in a heavy industrial settingindustrial settingDevelopment leaders: Amir Eliaz and Raanan GrinwaldDevelopment leaders: Amir Eliaz and Raanan Grinwald

71-72

Overview of CometOverview of CometOverview of CometOverview of Comet

Measures coverage of a given set of event traces over Measures coverage of a given set of event traces over user-defined modelsuser-defined models

Based on a relational database as the repository for Based on a relational database as the repository for coverage datacoverage data

Outputs various types of coverage reportsOutputs various types of coverage reports

Relational Database

System OverviewSystem OverviewSystem OverviewSystem Overview

COMET

Trace

Regression SuitesModel

Reports

73-74

Comet Tool ComponentsComet Tool ComponentsComet Tool ComponentsComet Tool Components

GUI componentsGUI componentsModel Definition Model Definition Coverage Measurement Control Coverage Measurement Control Coverage Report EngineCoverage Report Engine

Insertion and measurement of coverage data enginesInsertion and measurement of coverage data engines

Preparing the DomainPreparing the DomainPreparing the DomainPreparing the Domain

The Comet domain is a set of attributes with which The Comet domain is a set of attributes with which coverage data is expressed coverage data is expressed

Based on the domain attributes, Comet creates Based on the domain attributes, Comet creates database tables in the designated databasedatabase tables in the designated database

Event trace for storing tracesEvent trace for storing traces

Trace table for storing trace history informationTrace table for storing trace history information

... ...

75-76

Populating the DatabasePopulating the DatabasePopulating the DatabasePopulating the Database

The The trace analyzertrace analyzer is written, for every domain, to is written, for every domain, to extract the attribute values out of the raw data. extract the attribute values out of the raw data. The resulting The resulting tracestraces are inserted into the database. are inserted into the database. Every line in the trace is an event.Every line in the trace is an event.

.tst file.tst file trace filetrace file DB2

Trace File ExampleTrace File ExampleTrace File ExampleTrace File Example

** HOST ppc051.haifa.ibm.com** FILE default.tst eliaz** DOMAIN FP** TRACE 0** ATTR_NAMES cluster, processor, count, mnemonic, opcode, s1_sign, s1_fraction, s1_bin_frac,s1_exponent, s1_type, s2_sign, s2_fraction, s2_bin_frac, s2_exponent, s2_type, s3_sign, s3_fraction, s3_bin_frac, s3_exponent, s3_type, t_sign, t_fraction, t_bin_frac, t_exponent, t_type, fpscr_before, round_mode, fpscr_after

0,1,1,"fsub","11111111111000010010100000101000","-","3202530140641068","1011011000001010111100100111011010100101011100101100",-973,"-Norm","-","1132007388740218","0100000001011000111000000010010010011111101001111010",-978,"-Norm","none","none","none",-9999,"none","-","3026417421387608","1010110000001000001010110111010110000000011101011000",-973,"-Norm","00000000000001110110000000010101","Round_Toward_Zero","10000010000000101000000000010101"

...

77-78

Functional Coverage Models in the Functional Coverage Models in the Functional Coverage Models in the Functional Coverage Models in the Domain ParadigmDomain ParadigmDomain ParadigmDomain Paradigm

Comet models are composed of: Comet models are composed of:

A semantic descriptionA semantic description

A query on the traceA query on the trace

A tuple of attributesA tuple of attributes

Model attributes are not necessarily domain Model attributes are not necessarily domain attributesattributes

RestrictionsRestrictions

Using SQL to mark some of the tasks as illegalUsing SQL to mark some of the tasks as illegal

Model Definition DialogModel Definition DialogModel Definition DialogModel Definition Dialog

79-80

Coverage Measurement ControlCoverage Measurement ControlCoverage Measurement ControlCoverage Measurement Control

Monitoring the coverage processMonitoring the coverage process

Controlling the insertion of traces into the database Controlling the insertion of traces into the database and which TCLs to measure them againstand which TCLs to measure them against

Controlling various aspects of the measurement, Controlling various aspects of the measurement, such as collecting illegal task information, optimizing, such as collecting illegal task information, optimizing, etc.etc.

Coverage Measurement Dialog Coverage Measurement Dialog Coverage Measurement Dialog Coverage Measurement Dialog

81-82

Coverage Report Engine ToolCoverage Report Engine ToolCoverage Report Engine ToolCoverage Report Engine Tool

Querying the coverage data and getting specific cuts, Querying the coverage data and getting specific cuts, views, statistics, graphsviews, statistics, graphs

Examples (in the FP domain):Examples (in the FP domain):Did we see all FP instructions?Did we see all FP instructions?Did we see all rounding modes?Did we see all rounding modes?Source and Target operand typesSource and Target operand typesExponent difference for addition and subtractionExponent difference for addition and subtractionUncovered subspaces (holes in the coverage)Uncovered subspaces (holes in the coverage)

CRE Application - Basic DialogCRE Application - Basic DialogCRE Application - Basic DialogCRE Application - Basic Dialog

83-84

Basic Report ResultsBasic Report ResultsBasic Report ResultsBasic Report Results

CRE Application - Progress Graph CRE Application - Progress Graph CRE Application - Progress Graph CRE Application - Progress Graph

85-86

CRE Application - 2D ScatterCRE Application - 2D ScatterCRE Application - 2D ScatterCRE Application - 2D Scatter

Scatter Report ResultsScatter Report ResultsScatter Report ResultsScatter Report Results

87-88

CRE Application - SummariesCRE Application - SummariesCRE Application - SummariesCRE Application - Summaries

Comet Report Engine - Summary ReportComet Report Engine - Summary ReportComet Report Engine - Summary ReportComet Report Engine - Summary Report

89-90



















91-92

Two types of Coverage Directed Two types of Coverage Directed Two types of Coverage Directed Two types of Coverage Directed GenerationGenerationGenerationGeneration

Based on functional coverageBased on functional coverageCreate a functional coverage modelCreate a functional coverage modelGenerate all the tasksGenerate all the tasksTransform the model into part of a test planTransform the model into part of a test planTransform the tasks into testsTransform the tasks into tests

Based on an abstract modelBased on an abstract modelDescribe the application using a formal model (automata)Describe the application using a formal model (automata)Derive coverage tasks from the formal modelDerive coverage tasks from the formal modelDerive tests from the coverage tasksDerive tests from the coverage tasksExpected results are built-inExpected results are built-in

Usually easier to apply the first method, some tools Usually easier to apply the first method, some tools are availableare available

The ProblemThe ProblemThe ProblemThe Problem

The test effort is a software project that requires The test effort is a software project that requires design and implementation.design and implementation.

High level test design documents are required for High level test design documents are required for effective reviews and maintainability.effective reviews and maintainability.

Because of a lean test budget, the complete test Because of a lean test budget, the complete test design cycle is not implemented and quality is design cycle is not implemented and quality is hindered.hindered.

93-94

A Solution for API TestingA Solution for API TestingA Solution for API TestingA Solution for API Testing

Functional models provide a simple language that Functional models provide a simple language that enables enables high levelhigh level descriptiondescription of the test design. of the test design.

This level of description is easily reviewed and This level of description is easily reviewed and maintained, thus increasing qualitmaintained, thus increasing quality.y.

Low level test documents are produced automatically. Low level test documents are produced automatically. Test implementation is straightforward, using low Test implementation is straightforward, using low level test documents. Automatic generation of tests level test documents. Automatic generation of tests is also possible.is also possible.Time saved by the automation is used to define Time saved by the automation is used to define stronger (e.g., time dependent) models. stronger (e.g., time dependent) models.

An ExampleAn ExampleAn ExampleAn Example

A file system being tested has migrated to support A file system being tested has migrated to support files that are bigger than 2 GB:files that are bigger than 2 GB:

There are two types of There are two types of file openfile open: open and open64: open and open64

The functional model is defined as: The functional model is defined as:

<file open, file size> file open = {open, open64} file size = {less than 2 GB, more than 2 GB}

95-96

Test AutomationTest AutomationTest AutomationTest Automation

The attribute 'The attribute 'open fileopen file' is mapped to the class method:' is mapped to the class method:

void open(StringBuffer apiName){ System.out.println( apiName+ "(\"file1\", O_RDWR, S_IRWXU)"); }

Test AutomationTest AutomationTest AutomationTest Automation

open("open64") produces the call:open("open64") produces the call: open64("file1", O_RDWR, S_IRWXU);open64("file1", O_RDWR, S_IRWXU);

The tool produces the following tuples to be tested: The tool produces the following tuples to be tested: (open, less than 2 GB), (open64, less than 2 GB), etc.(open, less than 2 GB), (open64, less than 2 GB), etc.

When automatically producing tests from the tuples, When automatically producing tests from the tuples, open64 is mapped to the method callopen64 is mapped to the method call open("open64") thusopen("open64") thus producing the required code producing the required code

97-98

Model Based Automatic Testing -Model Based Automatic Testing -Model Based Automatic Testing -Model Based Automatic Testing -GOTCHA Testing MethodologyGOTCHA Testing MethodologyGOTCHA Testing MethodologyGOTCHA Testing Methodology

Input is a functional specification Input is a functional specification

Used to produce the model Used to produce the model

Generates abstract test case suite involving:Generates abstract test case suite involving:Sequences of rules and states (assertions and post conditions)

Runs testsRuns tests

Has the following benefits:Has the following benefits:One hundred percent coverage of modelEarly defect prevention:

InconsistenciesAmbiguityIncompleteness

TCBeans/GOTCHA Software Testing TCBeans/GOTCHA Software Testing TCBeans/GOTCHA Software Testing TCBeans/GOTCHA Software Testing Methodology - Test DesignMethodology - Test DesignMethodology - Test DesignMethodology - Test Design

Functional Specification

GOTCHA Engine

GOTCHA Abstract Test

SuiteGOTCHA Behavior

Model

Early Defect Discovery:

InconsistencyAmbiguity

Incompleteness

TCBeans Test

Object(s)

100% Model

Coverage

99-100

TCBeans/GOTCHA Software Testing TCBeans/GOTCHA Software Testing TCBeans/GOTCHA Software Testing TCBeans/GOTCHA Software Testing Methodology - Test ExecutionMethodology - Test ExecutionMethodology - Test ExecutionMethodology - Test Execution

GOTCHA Abstract Test

Suite

TCBeans GOTCHA Factory

Application Under Test

Test Results

TCBeans Test

Object(s)

Start/Stop TestCustomize Test

ObjectsMonitor Results

Defect Discovery/Resolution

Model Adjustment

The GOTCHA Definition LanguageThe GOTCHA Definition LanguageThe GOTCHA Definition LanguageThe GOTCHA Definition Language

Used to define a finite state machineStatesTransitionsStart/FinalInvariant

p1 p2

p1 p2

p1 p2

p1 p2

p1 p2

p1 p2

p1 p2p1 p2

p1 p2

101-102

GOTCHA Suite GenerationGOTCHA Suite GenerationGOTCHA Suite GenerationGOTCHA Suite Generation

IIIInput nput GOTCHA Definition Language modelGOTCHA Definition Language modelCoverage model - projected state/transitionCoverage model - projected state/transitionDefinition of start and final states for testsDefinition of start and final states for tests

OutputOutputTest suite guaranteed to cover all coverable tasksTest suite guaranteed to cover all coverable tasksList of reachable, but uncoverable, tasksList of reachable, but uncoverable, tasks

TestTestExecution path from a start state to a final stateExecution path from a start state to a final state

By-productBy-productViolated invariance encounteredViolated invariance encountered

Model Based Automatic Testing -Model Based Automatic Testing -Model Based Automatic Testing -Model Based Automatic Testing -TC Beans GOTCHA Test Case FactoryTC Beans GOTCHA Test Case FactoryTC Beans GOTCHA Test Case FactoryTC Beans GOTCHA Test Case Factory

JavaBeans Test Case ComponentJavaBeans Test Case ComponentInput: GOTCHA abstract test suite (text)Input: GOTCHA abstract test suite (text)Output: concrete test case execution and resuOutput: concrete test case execution and resulltsts

No additNo additional ional source filessource filesNo explicit compile and link phaseNo explicit compile and link phaseOne hundred percent coverage of model One hundred percent coverage of model

Target and Expected object (JavaBeans)Target and Expected object (JavaBeans)Rules = target method invocationRules = target method invocationStates = expected property settingStates = expected property setting

Process:Process:Parse test suiteParse test suiteExecuteExecuteCompare target and expected propertiesCompare target and expected properties

Interactive customization with BDKInteractive customization with BDK

103-104

Model Based Automatic Testing -Model Based Automatic Testing -Model Based Automatic Testing -Model Based Automatic Testing -A Use Case ScenarioA Use Case ScenarioA Use Case ScenarioA Use Case Scenario

1.1. Tester reviews functional specification.Tester reviews functional specification.2.2. Tester codes a behavior model in GOTCHA.Tester codes a behavior model in GOTCHA.3.3. Tester uncovers defects in specification.Tester uncovers defects in specification.4.4. Tester runs GOTCHA to create an abstract test suite.Tester runs GOTCHA to create an abstract test suite.5.5. Tester codes a Java class (JavaBeans): Tester codes a Java class (JavaBeans):

Wrapper methods to application methodsWrapper methods to application methodsWrapper properties to application state variablesWrapper properties to application state variablesStatic properties for interactive customizationStatic properties for interactive customization

6.6. Tester runs TCBeans GOTCHA Test Case Factory.Tester runs TCBeans GOTCHA Test Case Factory.7.7. Tester customizes target properties.Tester customizes target properties.8.8. Tester runs test, uncovers lots of bugs, celebrates.Tester runs test, uncovers lots of bugs, celebrates.9.9. Tester analyzes structural coverage, adjusts GOTCHA Tester analyzes structural coverage, adjusts GOTCHA

model.model.

GOTCHA Hardware Test StudyGOTCHA Hardware Test StudyGOTCHA Hardware Test StudyGOTCHA Hardware Test Study

Tests

0

50

100

150

200

250

300

350

Tra

nsiti

ons

ST Hand Tests

GOTCHA Transition Tests

Reachable Transitions

Cumulative Transition Coverage

105-106










Availability Availability Availability Availability Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage

Code coverageCode coverageMany tools are available for most operating Many tools are available for most operating systems/languages systems/languages

Tools are hard to find or are very expensive for custom Tools are hard to find or are very expensive for custom made/embedded environmentsmade/embedded environmentsTools are hard to implement Tools are hard to implement

Functional coverageFunctional coverageLimited functionality tool available (Comet)Limited functionality tool available (Comet)

Works in any environmentWorks in any environment Preparing a simple tool or script is easy Preparing a simple tool or script is easy

107-108

Code coverageCode coverageUsers can benefit from the tools almost immediatelyUsers can benefit from the tools almost immediately

The tools are easy to use; usually plug and playThe tools are easy to use; usually plug and play

Functional coverageFunctional coverageNeed some time to learn how to define and implement modelsNeed some time to learn how to define and implement models

Tools are not as easy to useTools are not as easy to use

Need to have at least one person in the organization with a Need to have at least one person in the organization with a working knowledge of functional coverage methodologyworking knowledge of functional coverage methodology

Takes about 3 days to acquire knowledgeTakes about 3 days to acquire knowledge

Learning CurveLearning CurveLearning CurveLearning CurveCode Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage

Code coverageCode coverage Cost of tools is usually lowCost of tools is usually low Almost no education cost Almost no education cost Slowdown of 10 - 100 percent Slowdown of 10 - 100 percent

Functional coverageFunctional coverage Higher cost to develop tools and modelsHigher cost to develop tools and models Higher education costs Higher education costs Slowdown of a few percent (variable in # of models) Slowdown of a few percent (variable in # of models)

CostCostCostCostCode Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage

109-110

Code coverageCode coverageUsed by many projects and in many standardsUsed by many projects and in many standardsPapers on correlation between good coverage at unit level Papers on correlation between good coverage at unit level and the number of integration bugsand the number of integration bugsRefinements in tools as a result of experience Refinements in tools as a result of experience

Functional coverageFunctional coverageNew field, not a lot of experience New field, not a lot of experience Harder to transfer experience from one project to anotherHarder to transfer experience from one project to another

Experience Experience Experience Experience Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage

So Why Use Functional Coverage?So Why Use Functional Coverage?So Why Use Functional Coverage?So Why Use Functional Coverage?

F C

111-112

Focus Focus Focus Focus Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage

Code coverageCode coverage Uniformly spread over the entire programUniformly spread over the entire program

Hard-coded coverage model hierarchy Hard-coded coverage model hierarchy

All models are local All models are local

Functional coverageFunctional coverage Risk-drivenRisk-driven

Models can be local or global Models can be local or global

Adapting to Testing ResourcesAdapting to Testing ResourcesAdapting to Testing ResourcesAdapting to Testing ResourcesCode Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage

Code coverageCode coverage Hard to adapt; hard-coded modelsHard to adapt; hard-coded models

Hard to adapt; very primitive focusing optionsHard to adapt; very primitive focusing options

Functional coverageFunctional coverage Easy to adaptEasy to adapt

Can use the right number of models with the desired Can use the right number of models with the desired granularitygranularity

Built-in model hierarchyBuilt-in model hierarchy

113-114

Impact on ProjectImpact on ProjectImpact on ProjectImpact on ProjectCode Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage Code Coverage vs. Functional Coverage

Code coverageCode coverageImpacts testing process and testing requirementsImpacts testing process and testing requirements

Functional coverageFunctional coverage Impacts the design itself from early design stages Impacts the design itself from early design stages

Enables better understanding of programs and Enables better understanding of programs and environmentsenvironments

Helps with test specifications and requirements Helps with test specifications and requirements










115-116

Which Coverage Method to Use?Which Coverage Method to Use?Which Coverage Method to Use?Which Coverage Method to Use?

Functional coverageFunctional coverage High risk areasHigh risk areas

Complex, error prone areas Complex, error prone areas

Changes to existing code (maintenance) Changes to existing code (maintenance)

Used when cheaper than manual test inspection Used when cheaper than manual test inspection

Used for global coverage models Used for global coverage models

Code coverageCode coverage On all code that is written, given sufficient resourcesOn all code that is written, given sufficient resources

As a criterion for finishing unit testing As a criterion for finishing unit testing

Conclusions Conclusions Conclusions Conclusions

Code coverage is easier to use, cheaper, and more Code coverage is easier to use, cheaper, and more available than functional coverage.available than functional coverage.

Given the resources, code coverage should be used Given the resources, code coverage should be used on the entire code base.on the entire code base.

Functional coverage is an excellent way to improve Functional coverage is an excellent way to improve testing of risky, error prone areas.testing of risky, error prone areas.

The quality of functional coverage depends directly The quality of functional coverage depends directly on the quality of the created functional coverage on the quality of the created functional coverage modelsmodels

117-118

Conclusions - cont.Conclusions - cont.Conclusions - cont.Conclusions - cont.

Functional coverage can help at all design stages. Functional coverage can help at all design stages. Functional coverage models should be created at the Functional coverage models should be created at the high level design stage.high level design stage.

Combining code based coverage and functional Combining code based coverage and functional coverage can lead to high quality testing.coverage can lead to high quality testing.

Any Questions?Any Questions?Any Questions?Any Questions?

119-120

code and functional coverage tutorial - ibm and functional coverage tutorial may '99 shmuel ur...

Documents