guidelines for using test automation framework measures · with measures and analysis techniques...

Copyright © 2003, Software Productivity Consortium, NFP. All rights reserved. SPC-2003056-MC Version 1.0, July 2003

Guidelines for Using Test Automation Framework Measures Mark Blackburn, Ph.D.

(Software Productivity Consortium)

1 INTRODUCTION ................................................................................................................................ 1

1.1 Scope ........................................................................................................................................... 1 1.2 Audience and Benefits................................................................................................................. 1 1.3 Organization of Technical Note .................................................................................................. 1

2 CONTEXT FOR TAF MEASUREMENT......................................................................................... 1 2.1 Definitions ................................................................................................................................... 1 2.2 Member Experience Using TAF.................................................................................................. 2 2.3 Background.................................................................................................................................. 2

2.3.1 Test Automation Framework .................................................................................................. 2 2.3.2 Measurement Framework........................................................................................................ 3 2.3.3 Role of Measurement in Project Management ........................................................................ 4

3 TAF METHOD AND TOOL OVERVIEW ....................................................................................... 5 4 MEASUREMENT FRAMEWORK DETAILS................................................................................. 6

4.1 Information Needs ....................................................................................................................... 6 4.2 Measurement and Analysis Process ............................................................................................ 7 4.3 Information Product and Measurement Construct....................................................................... 7 4.4 Tabular Representation of an Indicator ....................................................................................... 8

5 TAF MEASUREMENT ....................................................................................................................... 9 5.1 Attributes ..................................................................................................................................... 9 5.2 Base Measures ............................................................................................................................. 9 5.3 Derived Measures ...................................................................................................................... 10 5.4 Indicators ................................................................................................................................... 10 5.5 Example Information Product ................................................................................................... 11

6 TAF MEASUREMENT FOR PROJECT INFORMATION NEEDS ........................................... 12 6.1 Fundamental TAF Measures ..................................................................................................... 12 6.2 Estimated Completion of Requirement Modeling Indicator...................................................... 14 6.3 Indicators for Object Mappings................................................................................................. 16 6.4 Estimating the Number of Object Mappings ............................................................................. 17 6.5 Combined Estimate-to-Complete .............................................................................................. 18

6.5.1 Estimating Project Duration .................................................................................................. 18 6.5.2 Estimated Weeks to Project Completion............................................................................... 18

7 SUMMARY......................................................................................................................................... 20 7.1 Process Summary ...................................................................................................................... 20 7.2 Future Work............................................................................................................................... 21 7.3 Acknowledgments ..................................................................................................................... 21 7.4 For More Information................................................................................................................ 22

8 REFERENCES ................................................................................................................................... 22

Software Productivity Consortium NFP, Inc. SPC-2003056-MC Page 1 of 25 Version 1.0, July 2003

Members would like the Software Productivity Consortium (the Consortium) to provide support for effectively measuring progress and productivity associated with applying the Test Automation Framework (TAF) method and associated tools. This deliverable provides members with measures and analysis techniques focused primarily on project measurement when applying the TAF method.

1 Introduction Several Software Productivity Consortium (Consortium) members have been using the Test Automation Framework (TAF) for model-based test automation. TAF is a method and associated toolset to support model development, model analysis, and model-based test automation. TAF helps reduce cost, provide early identification of requirement defects, and improve test coverage (Kelly et al. 2001; Busser, Blackburn, and Nauman 2001; Statezni 2000; Statezni 2001). Project managers would like to better understand how measures derived from the artifacts developed through model-based analysis and testing can be used to help in project, product, and process measurement, where such measures can be used to support decision making within their organizations.

1.1 Scope This technical note includes guidance on defining, collecting, and analyzing measures derived from using TAF. These measures and their use are described in the context of the Consortium measurement framework. Although it is believed that these measures can be used for product and process measurement, this version of the guideline will focus on project measures related to project planning, scheduling, and performance. Specifically, the technical note describes how to use historical measures to predict project duration and usage of real-time project data to predict the completion of an ongoing project.

1.2 Audience and Benefits The technical note is applicable to managers, project leads, software developers, and test engineers who are responsible for managing, planning, and estimating project effort and duration. The technical note assumes that the readers are familiar with TAF and are either using TAF or plan to use TAF in the future. References to additional information on TAF are provided in Section 7.4.

1.3 Organization of Technical Note Section 2 provides some context by introducing two TAF definitions, with some example scenarios, that are necessary for understanding the key measurement attributes associated with TAF. In addition, Section 2 provides some background information on TAF and the Consortium measurement framework. Section 3 provides an overview of how TAF is used and how the developed artifacts are related to attributes that support the measurement process. Section 4 provides an overview of the measurement framework. Section 5 defines fundamental units of measure derived from TAF artifacts. Section 6 describes how TAF measures support project decision-making information needs. Section 7 provides a summary and describes potential future work.

2 Context for TAF Measurement

2.1 Definitions The measurable attributes that can be quantified when using TAF relate to refining and modeling requirements and developing object mappings to support test generation and automated test execution. These attributes include the following:

• Domain Convergence Paths (DCP). The developer constructs a model of requirements (or implementation-derived requirements in TAF) that is translated into a primitive set of DCPs. A DCP can be thought of as a requirement thread. The number of DCPs for a project or product is


a base measure for quantifying requirements. A DCP is also referred to as a test path because there are minimally two tests for every DCP. Requirement contradictions or inconsistencies (defects) also are directly related to a DCP. Therefore, a DCP is a fundamental unit of measure representing the functional requirement size for a project or product.

• Object Mappings. Object mappings define the relationship between modeled variables and the implementation interfaces. Developers and testers must construct object mappings for variables described in the modeled requirements. Object mappings relate the modeled variables to the implementation interfaces (e.g., application programming interfaces [APIs], variables, and object attributes).

2.2 Member Experience Using TAF Consortium members who use TAF identified trends when the total number of DCPs and object mappings were plotted on a weekly basis. Figure 1 provides a conceptual perspective of observed data representing the rate at which DCPs and object mappings were produced for a project. This information, with some interpretation, could be used to support project decision making. For example, suppose that a block of textual requirements (e.g., a paragraph of text associated with a SHALL Header, or a requirement paragraph), typically results in 10 DCPs, and there are 100 requirement blocks, then this means that there are approximately 1,000 DCPs to produce. Recording the rates of production for a given team supports predictions such as the estimated project completion date.

Unitspertime

DCPs Object mappings

Figure 1. Conceptual Trends in TAF Measurements

It also was observed that graphs of the collected measures showed flat spots, as reflected in Figure 1, making it apparent that little or no progress was being made during certain weeks. There are many reasons why this could occur, for example, modelers may be waiting for requirements that are not defined, people may be confused, and not asking questions to help resolve misunderstanding, people may have been directed to work on another project, or people may be away from the office (e.g., travel, vacation, sick leave). The continuous availability and visibility of this information could help managers and project leaders resolve problems early, reducing risk of schedule slips.

2.3 Background

2.3.1 Test Automation Framework

Consortium members have been applying the TAF since 1996 on applications in various domains, including critical applications for aerospace (Mars Polar Lander) (Blackburn et al. 2001), medical devices, flight navigation, guidance, autopilots, display systems, flight management and control laws, engine controls, and airborne traffic and collision avoidance. TAF also has been applied to noncritical applications such as databases, client-server, web-based, automotive, and telecommunications. The related test driver generation has been developed for many languages (e.g., C, C++, Java, Ada, Perl, PL/I, Structured Query Language [SQL], and Extensible Markup Language [XML]) as well as proprietary languages, commercial off-the-shelf (COTS) test injection products (e.g., DynaComm, WinRunner), and test environments. Most users of the approach have reduced their verification/test effort by 50% (Kelly et al. 2001; Safford 2000).


TAF integrates various government and commercially available model development and test generation tools to support defect prevention and automated testing of systems and software. TAF supports modeling methods that focus on representing requirements, like the Software Cost Reduction (SCR) method, as well as methods that focus on representing design-level information, like MathWork’s Simulink and Stateflow or National Instrument’s MATRIXx, which supports control system modeling for automotive and aircraft systems. Through the use of model translation, requirement-based or design-based models are converted to test specifications. T-VEC is the test generation component of TAF that uses the test specification to produce tests. T-VEC supports test vector generation, test driver generation, requirement test coverage analysis, and test results checking and reporting. Test vectors include inputs and expected outputs with requirement-to-test traceability information. The object mappings and the test vectors are inputs to the test driver generator, which produces test drivers that are executed against the implemented system during test execution.

2.3.2 Measurement Framework

The Consortium is developing a five-volume measurement framework series to explain the measurement principals with examples to describe how the framework can be used to support project, product, process, and enterprise measurement. This technical note uses material extracted from the Integrated Measurement Series, Volume 1: Measurement and Analysis Guidebook (MacIver and Card 2001). Figure 2 represents the following four components of the measurement framework:

1. A technical or management decision-making process or individual

2. The information needs of the decision maker

3. A measurement planning and analysis process that supports the collection and analysis of relevant data

4. An information product resulting from the analysis that satisfies the needs of the decision maker

3. Measurement andAnalysis Process

1. DecisionProcess

2. InformationNeeds

4. InformationProduct

3. Measurement andAnalysis Process

1. DecisionProcess

2. InformationNeeds

4. InformationProduct

Figure 2. Measurement Framework

The essential purpose of a measurement program is to provide information that assists decision makers in selecting successful courses of action. The decisions that are made can be organized into four types or classes, often mapped to hierarchical levels within an organization (see Figure 3).


Business performanceEnterprise Business performanceEnterpriseEnterprise

Efficiency and effectiveness ofproduction processesProcess Efficiency and effectiveness ofproduction processesProcessProcess

Delivery of product within budgetand schedule parametersProject Delivery of product within budgetand schedule parametersProjectProject

Technical and quality andattributes of productProduct Technical and quality andattributes of productProductProduct

Figure 3. Types of Decision-Making Processes

The key discriminator between the levels is the focus of concern or type of responsibility of the decision maker rather than the specific organizational position. The focuses of concern are as follows:

• Enterprise Level. These decisions concern the long-term competitiveness of the organization. At the enterprise level, decisions are made to invest in products and services to offer in a market or to invest in improvements that enhance organizational performance.

• Process Level. These decisions focus on the efficiency and effectiveness of the organization's means of accomplishing work. Processes consist of people, methods, and technology (e.g., tools that support the process).

• Project Level. These decisions involve the realization of a specific product or service for a specific customer or class of customers. At the project level, decisions are made to allocate budget and resources to product or service teams to deliver specified capabilities.

• Product Level. These decisions deal with the technical aspects of the product or service. The product team seeks to maximize quality and functionality (key components of customer satisfaction) within the envelope of budget and schedule provided by the project.

As indicated in the descriptions, these areas of concern are interrelated. Measurement and analysis at all these levels often depend on many of the same data sources. However, the level of detail and the manner in which the data is analyzed or reported for different purposes may vary. To optimally satisfy information needs at all levels, the nature of the information needs, the process used to satisfy them, and the resulting information product must be considered, too.

2.3.3 Role of Measurement in Project Management

As defined by Card et al. (2003), the role of the project manager is very broad. Consequently, measurement cannot completely address all the responsibilities of a project manager. Five fundamental roles of the project manager supported by measurement are:

• Developing estimates that serve as the basis for detailed project planning

• Establishing a detailed project plan, including budget and schedule

• Determining the status of the project relative to the plan

• Reacting to discrepancies between planned and actual performance

• Reporting the status of the project to executive management and customers


3 TAF Method and Tool Overview The most typical application of TAF is a process based on interface-driven requirements modeling, which is applicable to most systems. With this approach, test engineers work in parallel with developers to stabilize interfaces, refine requirements, and build models to support iterative testing and development. The following outlines the process, as depicted in Figure 4:

1. Working from whatever requirements artifacts are available, testers create a verification model1 in a table-based format2 using a tool based on the SCR method (Heitmeyer, Jeffords, and Labaw 1996), such as the SCRtool or T-VEC Tabular Modeler (TTM). Simplistically, each table represents an output, specifying the relationship between input values and resulting output values. T-VEC automatically checks for inconsistencies in the model. The tester interacts with the requirements engineers or analysts to validate the model as a complete and correct interpretation of the requirements.

2. The tester maps the variables (inputs and outputs) of the verification model to the interfaces of the system in object mappings. The nature of these interfaces depends on the level of testing performed. At the system level, the interfaces may include graphical user interface widgets, database APIs, or hardware interfaces. At the lowest level, they can include class interfaces or library APIs. The tester uses these object mappings with a test driver schema to support automated test driver generation. The tester works with the designers to ensure the validity of the mappings from model to implementation.

3. Prior to test generation, T-VEC compiles the model into DCPs that represent the requirement threads of the model. The T-VEC tool generates an optimal set of test vectors by analyzing the input space for each DCP. These test vectors include test inputs and expected test outputs, as well as model-to-test traceability for each DCP.

4. T-VEC generates the corresponding test drivers using the object mappings and schema. A schema is created once for each test environment. The schema defines the algorithmic pattern to carry out the execution of the test cases. The test driver executes in the target or host environment. The test drivers typically are designed as an automated test script that sets up the test inputs enumerated in each test vector, invokes the unit under test, and captures the results.

5. Finally, T-VEC analyzes the test results. It compares the actual test results to the expected results and highlights any discrepancies in a summary report.

1 “… a verification model specifies behavioral requirements in terms of the interfaces for the system under test.

This is in contrast to a “pure” requirements model, which specifies the requirements in terms of logical entities representing the environment of the system under test. Verification modeling from the interfaces is analogous to the way a test engineer develops tests in terms of the specific interfaces of the system under test.” (Blackburn et al. 2002)

2 Although analysts and designers commonly develop models based on state machines or other notations, testers typically find it easier to learn and develop requirements for test in the form of tables. See Blackburn et al. (2001) and Kelly et al. (2001) for more information.


Global init;Forall testsinit target;set inputs;execute SUT;get outputs;store output;

endforall

Test Vector Generator

Test Driver Generator

TAF ModelTranslator

Test Driver

Model

Test Environment

Test scripts generated from test driver object

mappings and schema, and generated tests

execute in testenvironment to drive the unit under test

Tester builds model to capture required

behavior

Object mappings relate model variable to implementation

interfaces, and test driver schema (created once) defines a pattern

for generating test scripts

T-VEC uses model to generate tests for each DCP

Tester

RequirementsAnalyst

Designer

Designspec

Requirementsspecification

Test ResultAnalyzer

TestAnalysis

Test results compared

against expected

results

Schema

ObjectMappings

Figure 4. Typical TAF Process

Although it is common to start the process with poorly defined requirements, inputs to the process can include requirement specifications, user documentation, interface control documents, API documents, previous designs, and old test scripts. The key prerequisite for test design, whether done manually or through automated test selection and generation, is defining the relationships between input and output values. As with any approach to test design, TAF requires two primary inputs:

• Functional requirements, which describe the required behavior of the unit, component, or subsystem under test (a requirement specification)

• Interface definitions, which describe the inputs and outputs, their data types and range information, and the facilities for interacting with the unit under test

The system or software design must permit testers to create valid test driver templates, from which T-VEC generates working test drivers to run test scripts in the test environment. This means that interfaces must be available for injecting test inputs (including setting internal state as needed), executing the system under test, and observing the test results.

4 Measurement Framework Details

4.1 Information Needs The information needs of the decision makers relate to their goals and the obstacles to achieving those goals. To have confidence in achieving goals, project leaders must manage the key obstacles to achievement. Consider the following three types of obstacles:

• Risks. Things that may go wrong.

• Problems. Things that are fairly certain to go wrong or that have already gone wrong.

• Lack of Information. Situations where decisions or plans are made without an adequate quantitative basis.


Risks, problems, and lack of information may arise at any decision-making level. Information needs result from these goals and obstacles. Information needs provide the basis for selecting the measures and analysis techniques incorporated into the information product, which is the final result of the measurement process.

A measurement initiative may begin by addressing the information needs of one specific level of decision making, but this report focuses specifically on project management even though information needs of the different levels can be related. Information needs at one level may require measurement activity at another level. Information needs may be classified in terms of the type of information (goal or obstacle) to which they relate. While a measurement program can be initiated with a focus on any level of decision making, anticipating the information needs of other levels makes it easier to evolve a measurement program to address a wider range of issues over time.

4.2 Measurement and Analysis Process The measurement and analysis process provides the mechanisms for identifying and addressing information needs of all types and levels. This process consists of a set of activities that are generally applicable, regardless of the specific information needs of any situation. It addresses both the selection of appropriate measures to satisfy the information needs and the collection and analysis of the data. A process must be developed that relies on four iterative activities: establish, plan, perform, and evaluate. Together, plan and perform are referred to as the core measurement process because these are the activities that interact most intensely with measurement users. In this particular case, the measures are provided inherently in the development process.

4.3 Information Product and Measurement Construct Information products make up the primary output of the measurement process. The generic structure of a measurement construct can be defined in terms of the information model, as shown in Figure 5. The information model shows how the attributes of specific software and systems entities are related to the information needs of the measurement user.

InterpretationInterpretationInterpretationEstimate or evaluation that provides a basis for decisionmaking

Indicator

Model

DerivedMeasure

Value resulting from applying the algorithm to two or more measures

Algorithm combining measures and decision criteria

DerivedMeasure

Operations mapping an attribute to a scaleMethod

BaseMeasure

Function Algorithm combining two or more base measures

Value resulting from applying the method to one attribute

AttributeAttributeProperty relevant to information needs Entities

Information NeedsInformation

Product Information NeedsInformation

Product

Source: Adapted from ISO/IEC 15939, Software Measurement Process Framework by Joe Seppy

Method

BaseMeasure

Method

BaseMeasure

Method

BaseMeasure

Figure 5. Measurement Construct


Key elements of the information model are as follows:

• Attribute. An attribute is a property or characteristic of a process or product that is the subject of measurement. For example, DCPs and object mappings are attributes of the TAF process.

• Base Measure. A base measure or data primitive is a quantification of a single attribute. This is obtained by applying a measurement method. For example, the number of DCPs of the project or product.

• Derived Measure. A derived measure combines two or more values of base measures using a mathematical function. For example, the DCP weekly production rate is the total number of DCPs divided by the number of weeks.

• Indicator. An indicator can be a base or derived measure or a combination of such measures that are associated with decision criteria by means of a mathematical or heuristic model. Indicators often are presented in graphic or chart form, like the one shown in Figure 1.

• Information Product. An information product consists of one or more indicators with corresponding interpretations. The information product forms the basis for action by the decision maker. For example, if you know the number of DCPs per requirement, the total number of requirements, and a production rate of the DCPs, you can estimate the duration of a project.

The information model suggests that standardization should focus at the level of base measures. Then these measures can be combined in many different ways to form indicators that satisfy specific information needs. As long as the base measures are collected consistently, the needs of all levels of decision making can be satisfied—although each user might analyze and interpret the data in a different way for a different purpose.

4.4 Tabular Representation of an Indicator Indicators can be defined using a table with the following set of fields. See Table 3 for an example.

• Indicator Name. A unique, descriptive name. This is generally the title of the chart, graph, etc. (e.g., Trend of Weekly Totals for Defects Found versus Fixed).

• Information Need. A description of the information need.

• Measures Used. A list of all derived measures and their associated base measures used by this indicator.

• Function(s). The manner in which the base measures are calculated into derived measures, if appropriate (e.g., show formula for standard return on investment [ROI] calculation for cost/benefit analysis).

• Analysis Model. Describes the model used to construct the indicator (e.g., regression analysis).

• Decision Criteria. Relevant information and interpretation guidance to assist in decision making (e.g., threshold values or trigger points, control limits, or confidence limits).

• Graphical Representation. The type of chart or manner in which the indicator will be presented (e.g., line chart).


5 TAF Measurement This section works bottom-up from the information model of the measurement construct (see Figure 5) to describe the relationship of two key TAF attributes to various information products supporting project measurement. Section 6 describes information models for related information products to illustrate how TAF measures can be used to support project measurement.

5.1 Attributes As described in Section 3, the two primary attributes derived from produced artifacts to support model-based test generation are:

• DCPs

• Object mappings

There are other attributes that could be used to produce derived measures with DCPs and object mappings. Some of these related attributes include the following:

• Requirements.

• Variables. A variable is defined in the model. There are three types of variables: inputs, terms, and outputs. Input and output variables are associated one-to-one with object mappings.

5.2 Base Measures A base measure is a quantification of a single attribute obtained from some method or operation. There are several possible base measures that can be obtained through operations associated with the TAF attributes. Some key base measures are defined in Table 1.

Table 1. TAF Base Measures

Base Measure Explanation Number of DCPs DCPs produced during a particular measurement period (e.g., DCPs produced

within a week)

Number of variables Input and output variables produced during a particular measurement period that require a corresponding object mapping

Number of object mappings Object mappings specified during a particular measurement period

Total number of DCPs Sum of the DCPs measured from the start of the project

Total number of variables Sum of input and output variables measured from the start of the project

Total number of object mappings Sum of specified object mappings measured from the start of the project

Total number of requirements Sum of total requirements being modeled using TAF for the project

To better understand the progress being made during project development, it is necessary to measure the DCPs, object mappings, and variables produced each week (these measures could be tracked daily, monthly, or yearly). Although it is idealistic to think that the total number of requirements will remain constant for a project, this seldom happens, so the total number of requirements for a project also should be monitored weekly. Fortunately, the total number of DCPs is generated as part of the status report for a project as shown in Figure 6. Similarly, you can record the number of variables and object mappings produced each week. At the publication of this report, the tools do not produce the number of variables and object mappings automatically.


Total Numberof DCPs

Total Numberof DCPs in

each subsystem

Figure 6. Total Test Paths (DCPs) Generated in a Report

5.3 Derived Measures A derived measure combines two or more base measures using a mathematical function. For example, the requirements per DCP is a derived measure relating the number of requirements to modeled-derived DCPs. If a project manager can estimate the total number of requirements that must be modeled, it would be possible to predict the total number of DCPs for the project. If the rate of DCP production per week were known, the scheduled end-date could be predicted. This report assumes that the measurement period is weekly. Table 2 identifies some derived measures.

Table 2. TAF Derived Measures

Derived Measure Explanation DCP rate The average number of DCPs produced per week

DCP rate (current week) The average number of DCPs produced per week at some week, where current week is a week number starting from the beginning of the project, which current week = 0

DCP density The number of DCPs per requirement

Object mapping rate The average number of object mappings specified per week

Object mapping rate (current week) The average number of object mappings specified per week at some week

Variables per DCP The number of input and output variables per DCP

These types of derived measures can be associated with an individual, a specific project, a project team, a product, a product family, a business unit, or an entire corporation. Project teams as well as project team members will have different DCP rates and object mapping rates. These rates can vary based on the modeling skills of the team members, but they also can be affected by requirement rework due to poorly documented requirements, unknown requirements, or poorly defined interfaces.

5.4 Indicators An indicator is a base or derived measure or a combination of such measures that are associated with decision criteria by means of a mathematical or heuristic model. Figure 1 is an indicator that shows the


rate of production of DCPs related to the rate of production of object mappings. This indicator helps make the slope of the active periods, as well as the inactive periods (i.e., flat spots), obvious.

5.5 Example Information Product The following example illustrates the use of the measurement construct. Assume that project decision makers want to compare the DCP density of some current project to previously captured data because it is believed that a DCP density in a certain range provides optimal requirement-to-test traceability. Assume the following:

• Base measures for DCPs and requirements have been collected.

• Number of requirement headers is the base measure for number of requirements, where a requirement header is associated with a body of requirement text.

• Data collected from prior projects estimates the number of DCPs per requirement.

Starting from the bottom of Figure 7 the two attributes are DCPs and requirements. Assume that, for some sample project, there were 223 DCPs developed as models from 21 requirements. Dividing the number of DCPs by the number of requirements produces a derived measure of 10.6 DCPs per requirement (i.e., DCP density). Compare this derived data to historical data, where the density was 16.3 DCPs per requirement. A possible interpretation is that the requirement traceability accuracy for the current project is better than the organizational average.

10.6 < Organizational average of 16.3 DCPs per requirement Indicator

Model

DerivedMeasure

Compare DCP density to organizational data

InterpretationCompare DCP density to organizational average tounderstand granularityof text requirements

InformationProduct

Indicator summarized forrequirement documents

DerivedMeasure

Function Divide (DCP per requirement)

10.6 DCPs per requirement

AttributeAttributeEntities

Countproducedby TAF

Method

BaseMeasure

223

DCPs

Countproducedby TAF

Method

BaseMeasure

223

Countproducedby TAF

Method

BaseMeasure

223

Countproducedby TAF

Method

BaseMeasure

223

DCPs

Method

BaseMeasure

Count only requirement headers

21

RequirementsTAFData

Information NeedsRequirement traceabilityaccuracy

Figure 7. Measurement Construct Example

Table 3 provides a tabular representation of the measurement instrument that is used for this example. This tabular representation is used throughout the remainder of this document. To assess the current project, some historical DCP density data is required to judge the optimality of the requirement traceability accuracy. If the DCP density is 1, then the requirements are one-to-one with the tests. This may be optimal from a traceability point-of-view, but the overhead in writing requirements may be very


high. Therefore, the decision criteria for this indicator should identify an acceptable range of values as is illustrated in Table 3. For example, an acceptable DCP density must be in the range of 10 to 18. Based on the decision criteria, the current project, with a DCP density of 10.6, has a high requirement traceability accuracy that is within the limits of the acceptable organizational range. If the DCP density for a given project is too high, it might be difficult to trace the requirements to particular tests, and it might be recommended that the requirement be further decomposed and rewritten as small blocks of text.

Table 3. Tabular Representation of Measurement Construct Example

Indicator Name Requirement Traceability Accuracy

Information Need Requirement traceability accuracy ensures that there is subjective traceability from a requirement to the modeled requirements (and test generated from the model). If the DCP density is lower, the requirements are more fine-grained, and the traceability to the requirement is generally more accurate.

Derived Measure(s) DCP density – DCPs per requirement

Base Measures Total DCPs Total requirements

Function(s) Total DCPs/Total Requirements

Analysis Model Relational comparison

Decision Criteria A DCP density in the range of 10 to 18 is acceptable.

Graphical Representation None

6 TAF Measurement for Project Information Needs This section illustrates the use of TAF measurement to address project measurement information needs. The estimated project duration and estimated project completion date are usually key project information needs. This information can be affected by requirements creep and design and implementation rework, but this can be factored into the measurement process. For purposes of this document, assume that the number of requirements for a project is known and remains constant throughout the project.

To estimate the project duration requires accounting for the time to model the requirements and specify the object mappings to all modeled variables. Once this is done, tests are automatically generated and can be executed automatically. All tests must pass. It is also desirable to have requirement and model reviews, but during a project, these typically happen continuously as the models are being developed. Therefore, this time is factored into the recorded data. In addition, the recommended process of interface-driven model-based testing also forces the object mapping development to proceed in parallel as reflected in Figure 1. When all requirements are modeled, and all object mappings are created, the test execution can be performed. This requires a complete implementation also. Therefore, the actual end of these tasks follows very closely in time with the completion of the project.

6.1 Fundamental TAF Measures This section presents several graphs of base or derived measures that are used in the indicators described at the end of this section. The line graphs in this section plot the project week along the x-axis. The y-axis varies depending on the interpretation of the data or indicator.

The first graph, shown in Figure 8 shows the weekly number of DCPs. This type of chart is useful because it accentuates the variances in progress for each week. For example, the dips in progress (or DCPs) for weeks 3, 6, and 11 should trigger a project lead’s interest. This information should be analyzed to better understand why little progress was being made during these weeks.


Weekly Number of DCPs

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13

Current Project Week

DC

Ps

Figure 8. Weekly Number of DCPs

Figure 9 shows the cumulative number of DCPs plotted each project week. This provides a general trend of the production of modeled requirements, much like the graph shown in Figure 1.

Cumulative DCP Count

050

100150200250300350400450500

1 2 3 4 5 6 7 8 9 10 11 12 13


DCPs

Figure 9. Cumulative DCP Count

The project DCP rate is the total number of DCPs divided by the current project week. Figure 10 plots the average DCP rate against the actual DCP count per week. Even with the 3 weeks of inactivity, another interpretation of this graph is that the number of DCPs per week is increasing. This might be a reasonable expectation as people on the project become more familiar with the project requirements, and the modeling skills might be improving.


DCP Rate Relationships

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13


DC

Ps Actual DCPs

Average DCP rate

Figure 10. DCP Rate Relationships

By plotting the DCPs shown in Figure 10, another related measure, the average DCP rate, becomes apparent. At the end of the project, the average DCP rate establishes the baseline for a product DCP rate, which is the average DCP rate for producing one or more project releases of a given product. There also could be a DCP rate for each business unit that is the average DCP rate for one or more products. Similarly, there is a project object mapping rate that represents the rate of production of object mappings, and this is discussed in Section 6.4.

6.2 Estimated Completion of Requirement Modeling Indicator The following example is simplified to focus on the completion of the requirements modeling without adding analysis for the object mapping data. If historical data is available, then the estimated completion of requirement modeling can be predicated using Formula 1.

For example, assume the total number of requirements is 50, and the DCP density is 16. This means that there will be an estimated 800 DCPs. If the DCP rate is 40 DCPs per week, then the estimated project will require approximately 20 weeks to complete. The accuracy of this prediction is only as good as the historical information.

Organizations attempting to predict the estimated completion date of the first TAF project will not have existing information; however TAF measures can be used if an actual or predicted number of requirements is known. If the DCP density and DCP rates are not known ahead of time, they can be derived as the project progresses. By using a real-time calculation of the DCP rate and DCP density during the project, a prediction can be made about the estimated completion date for modeling the requirements. A variant of Formula 1 is used to continuously predict the remaining number of weeks to complete the requirement modeling. The calculation is as follows (Formula 2).

Formula 1 _ _ | | Estimated Weeks | (Total Requirements*DCP Density) | to Complete = |__________________________________| Requirements | | | Average DCP rate | |_ _|


Formula 2 _ _ Real-time | | Estimated Weeks | (Total Requirements*DCP Density[Current week]) | to Complete = |________________________________________________| - Current week Requirements | | | Average DCP rate[Current week] | |_ _|

The hybrid graph in Figure 11 shows three different information components. The x-axis represents the current week, and the y-axis shows information for the following:

• Average DCP rate (circle data points)

• DCP density (square data points)

• Estimated weeks remaining before project completion (triangle data points)

Hybrid Graph

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11 12 13

Current Week

Average DCP rate DCP Density Estimated Weeksto Complete

Figure 11. Hybrid Graph Estimating Weeks Remaining in Project

Table 4 provides a measurement instrument for the estimated project completion date. A critical part of the indicator is the decision criteria, which should estimate the probability of accuracy of the indicator. Such formulas or expressions should be developed by each organization. An example decision criteria conditional expression is shown in Table 4. This interpretation is: when the change of the estimation from one week to the next (i.e., rate of change per week) begins to converge down to zero, the accuracy of the prediction increases.

Table 4. Real-Time Estimated Project Completion Date Indicator

Indicator Name Real-Time Estimated Completion of Requirement Modeling

Information Need Real-time estimate of the number of weeks it will take to complete the requirement modeling

Derived Measure(s) DCP density DCP rate


Indicator Name Real-Time Estimated Completion of Requirement Modeling

Base Measures Total DCPs Total requirements Project weeks

Function(s) See Formula 2.

Analysis Model This construct uses predicted value as a function of the current weekly values for DCP density and DCP rate.

Decision Criteria Rate of change per week >= 3 then 50% accurate Rate of change per week < 3 then 80% accurate Rate of change per week < 2 then 90% accurate Rate of change per week < 1 then 95% accurate

Graphical Representation Line graph. See Figure 11.

6.3 Indicators for Object Mappings An object mapping is required for model inputs and output variables. Object mappings define the relationship between modeled variables and the interfaces for setting the input values and retrieving the output values. From a development and measurement point-of-view, object mappings have two states: introduced and specified. Once a model variable is introduced into the model, an object mapping is introduced, but it is not specified; therefore object mappings' data can be tracked according to these states. The number of variables is known in advance of the object mapping specification. This is reflected by Figure 12, which shows the actual number of variables, actual object mappings (specified object mappings), and the average rate of completing the specification for object mappings.

Object Mapping Data

02468

101214161820

1 2 3 4 5 6 7 8 9 10 11 12 13

Current Week

Actual variables Actual object mappings Average object mapping rate

Figure 12. Object Mapping Data

Another view on the object mapping data is the relationship between the introduced variables and the completion of the object mapping specification. Often, many objects are introduced early when the models are being created, but the completion of the object mappings may lag behind for various reasons, such as the interface is not yet formalized in the design or implementation. The object mapping process is complete when each input and output variable has a specified object mapping that supports the proper execution of the test drivers. Typically, more input variables are introduced early in the modeling process, while the number of outputs grows more linearly in relationship to the number of DCPs. It is common for several of the same input variables to be related to one or more output variables defined in the model. As


the project nears completion, the introduction of new variables slows. This is reflected in Figure 13, which shows that the number of variables and object mappings begins to converge.

Variables vs. Object Mappings

0

20

40

60

80

100

120

140

1 2 3 4 5 6 7 8 9 10 11 12 13

Current Week

Total variables Total object mappings

Figure 13. Variables Versus Object Mappings

6.4 Estimating the Number of Object Mappings When a project begins, there may be an estimate of the total number of requirements. With this estimate, it is possible to predict the estimated completion date of the requirement modeling when there is a DCP density and a DCP rate. For object mappings, it is usually more difficult because the number of variables per requirement can be application-specific. However, once modeling begins, the number of variables per DCP can be calculated as reflected in Figure 14. This sample data indicates that there is approximately one variable for every 4 DCPs at week 13. This means that the total number of variables that will be introduced in the model can be estimated.

Variables per DCP

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

1 2 3 4 5 6 7 8 9 10 11 12 13

Current Week

Variables per DCP

Figure 14. Variables per DCP


6.5 Combined Estimate-to-Complete There are two possible ways to look at the estimate-to-complete:

• Using historical data to predict the project duration

• Using project data to predict the number of weeks until completion

6.5.1 Estimating Project Duration

One goal of project planning is to predict the duration of the project given some set of resources that have predictable modeling and object mapping development rates. Given a predictable number of requirements and historically derived measures for DCP density, object mapping rate, and variables per DCP, the duration of a project can be calculated (Formula 3). The accuracy of this prediction is only as good as the historical information.

6.5.2 Estimated Weeks to Project Completion

The estimated weeks to project completion uses information gathered during the execution of the project to predict the number of remaining weeks before the modeling and object mappings will be completed. This measure is useful when there is no historical information, but it is also useful as a way to track project progress. There also are some other benefits of this measure:

• DCP density is more accurate because it reflects the project-specific density.

• DCP and object mapping rates are more accurate because they are directly related to the current staff performing the work.

• Variables per DCP is more accurate because the number of variables per DCP can vary significantly from one application to another.

The formula to continuously predict the remaining number of weeks to complete the object mappings (see Formula 4) is similar to the formula for calculating the real-time estimate for completing requirement modeling (see Formula 2).

Formula 3 _ _ | | Estimated | (Total Requirements * DCP Density * Variables per DCP) | Project = |________________________________________________________| Duration | | | Object mapping rate | |_ _|

Formula 4 _ _ | Total Requirements | | * DCP Density[Current week] | Estimated Weeks | * Variables per DCP[Current week] | to Complete = |____________________________________________| - Current week Object Mappings | | | Average object mapping rate[Current week] | |_ _|


The accuracy of the formula is typically poor early in the project when no object mappings have been specified because the object mapping rate is zero; and this causes a divide-by-zero computation. Even if the number of object mappings is very small, the computation is still grossly inaccurate because the denominator is very small compared to the numerator. Figure 15 shows two data lines: estimated weeks to complete requirements (diamonds – Formula 2), and estimated weeks to complete object mappings (squares – Formula 4). The project is typically not complete until the object mappings are complete, but during the first part of the project, there are usually no object mappings specified; therefore, the best early estimate is provided by the estimated weeks to complete requirements, which is described in Section 6.2.

Combined Estimate to Complete

05

101520253035404550

1 2 3 4 5 6 7 8 9 10 11 12 13

Current Week

Wee

ks to

Com

plet

e

Estimated Weeksto CompleteRequirements

Estimated Weeksto Complete Object Mappings

Constraint on the object mapping weeks if greater than 2 x weeks for requirements.

Figure 15. Combined Estimate–to-Complete

Table 5 provides an indicator referred to as the TAF estimate-to-complete indicator, which is based on a conditional function that uses Formulas 2 and 4. Early during the project, Formula 2 is a better predictor, but at some point during the project, when there are a substantial number of object mappings completed, the estimate-to-complete based on the object mapping rate (Formula 4) becomes a better predictor than the estimated weeks to complete requirements. The decision criteria for estimated weeks to complete object mappings should estimate the probability of accuracy of the indicator, like the indicator shown in Table 4. Again, such formulas or expressions should be developed by each organization based on its own statistical data. An example decision criteria conditional expression is shown in Table 5.

Table 5. TAF Estimate-to-Complete Indicator

Indicator Name TAF Estimate-to-Complete

Information Need Real-time estimate of the number of weeks it will take to complete the object mappings, which will complete after the requirement modeling is complete

Derived Measure(s) DCP density DCP rate Variables per DCP Average object mapping rate


Indicator Name TAF Estimate-to-Complete

Base Measures Total DCPs Total requirements Total variables Total object mappings Project weeks

Function(s) If (Formula 4 < (2 * Formula 2)) then Formula 4 else Formula 2

Analysis Model Assume that if Formula 4 values are greater than 2 times Formula 2 values, then it is inaccurate because of the low number of specified object mappings, and use Formula 2 as an estimator.

Decision Criteria Rate of change per week >= 3 then 50% accurate Rate of change per week < 3 then 80% accurate Rate of change per week < 2 then 90% accurate Rate of change per week < 1 then 95% accurate

Graphical Representation Line graph. See Figure 15.

7 Summary This document provides introductory guidance for project measurement based on using measures derived through TAF. It aims to provide practical guidance for estimating and tracking software-intensive projects using well-established measurement techniques.

7.1 Process Summary Figure 16 provides a perspective on the key measurement information and how it relates to the typical TAF method of interface-driven requirements modeling. With this approach, there are four key base measures. Requirement engineers are responsible for producing requirements, which results in the base measure number of requirements. A test engineer or modeler works in parallel with developers to refine requirements and build models to support iterative testing and development. Modeling introduces model variables, and this results in the base measure number of variables. After model translation and processing, the model requirements are converted into DCPs, which is a base measure related to requirements. Finally, to support test driver generation and test execution and results analysis, the base measure number of object mappings is used. Object mappings relate model variables to the implementation interfaces.

Section 6 discussed how these various base measures can be plotted as graphs to provide raw data information related to project progress in producing the TAF artifacts. Section 6 also discussed ways to transform these base measures into derived measures. Finally, base measures and derived measures can produce indicators to provide key information related to project duration or estimated project completion.

This measurement-related information should help managers and project leads with predicting schedule duration and estimating project completion dates. Historical measurement information can be used prior to the start of a project, but it also is important to use data derived during the project. This document provided a suggested set of information needs for project managers and presented guidance for developing related indicators and their supportive measures.


Test Vector Generator

Test Driver Generator

TAF ModelTranslator

Model

Tester/Modeler

RequirementsAnalyst

Requirementsspecification

Number of Requirements

Number of DCPs

Number of Variables

Number of Object mappings

Base Measures

DCP Rate Relationships

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13


DC

Ps Actual DCPs

Average DCP rate

Variables per DCP

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

1 2 3 4 5 6 7 8 9 10 11 12 13

Variables per DCP

Object Mapping Data

0

5

10

15

20

1 2 3 4 5 6 7 8 9 10 11 12 13

Current Week

Actual variables Actual object mappings Average object mapping rate

Combined Estimate to Complete

05

101520253035404550

1 2 3 4 5 6 7 8 9 10 11 12 13

Current WeekW

eeks

to C

ompl

ete

Estimated Weeksto Complete Req

Estimated Weeksto Complete OM

Variables vs Object Mappings

0

20

40

60

80

100

120

140

1 2 3 4 5 6 7 8 9 10 11 12 13

Current Week

Total variables Total object mappings

Weekly Number of DCPs

0

10

20

30

40

50

60

70

80

1 2 3 4 5 6 7 8 9 10 11 12 13


DC

Ps

Raw Data Derived Measures Indicators Figure 16. Process View of TAF Measurement

7.2 Future Work This paper is not a comprehensive treatment of issues related to TAF measurement. Based on member requests, the Consortium may continue to extend this work to address the following:

• Project-related TAF measurement for the enterprise. This topic would focus on analyzing TAF measures, establishing baselines, and relating the production rates of TAF artifacts across various enterprise organizations and project types.

• Product-related TAF measurement guidelines. This focus of product measurement is on the need of project managers and engineers to ensure that a given product or service meets the customer’s quality requirements.

• Process-related TAF measurement guidelines. The focus of process measurement is on tracking and improving the effectiveness of the technical processes across the organization to increase predictability, productivity, and quality and to reduce cycle time and cost.

7.3 Acknowledgments The author wishes to thank the following people for their thoughtful review and comments:


• Phil Boettge of Lockheed Martin

• Joe Seppy of Software Productivity Consortium

• Sean Cassell of Software Productivity Consortium

• Mike Polen of Software Productivity Consortium

7.4 For More Information Members with general questions or comments on any of the topics in this paper or related topics, or members interested in applying TAF with Consortium assistance, should contact the author or their member account director (see http://www.software.org/pub/keycontacts.asp).

For more on TAF, see the Consortium’s TAF Website at http://www.software.org/pub/taf/testing.html or contact the author.

8 References

Blackburn, M.R., R.D. Busser, and A.M. Nauman 2002

Interface-driven Model-based Test Automation, STARWEST, November.

Blackburn, M.R., R.D. Busser, A.M. Nauman, R. Knickerbocker, and R. Kasuda 2001

Mars Polar Lander Fault Identification Using Model-based Testing, In Proceedings, IEEE/NASA 26th Software Engineering Workshop.

Busser, R. D., M. R. Blackburn, and A. M. Nauman 2001

Automated Model Analysis and Test Generation for Flight Guidance Mode Logic. In Proceedings, Digital Avionics System Conference.

Card, D., G. Jorgensen, R. MacIver, J. Marple, C. Miller, and B. Phillips 2003

Integrated Measurement Series, SPC-2002062-MC, version 01.00.00. Volume 2, Measurement for Project Management. Herndon, Virginia: Software Productivity Consortium.

Heitmeyer, C., R. Jeffords, and B. Labaw 1996

Automated Consistency Checking of Requirements Specifications. ACM TOSEM, 5(3):231-261.

Kelly, V., E. L. Safford, M. Siok, and M. R. Blackburn 2001

Requirements Testability and Test Automation. Lockheed Martin Joint Symposium.

MacIver, R., and D. Card 2001

Integrated Measurement Series, SPC-2001010-MC , version 01.00.00. Volume 1, Measurement and Analysis Guidebook. Herndon, Virginia: Software Productivity Consortium.

Safford, Ed L. 2000

Test Automation Framework, State-based and Signal Flow Examples. Twelfth Annual Software Technology Conference, April 30 - May 5, 2000.


Statezni, David 2000

Test Automation Framework, State-based and Signal Flow Examples, Twelfth Annual Software Technology Conference, April 30 - May 5, 2000.

2001 T-VEC’s Test Vector Generation System, Software Testing & Quality Engineering, May/June 2001.


Keywords: Test automation framework, requirement-based testing, measurement framework, project measurement

Trademarks: DynaComm is a registered trademark of Future Soft, Inc. MATRIXx is a registered trademark of Wind River Systems. Simulink is a registered trademark of The MathWorks. Stateflow is a registered trademark of The MathWorks. WinRunner is a registered trademark of Mercury Interactive Corporation. T-VEC is a registered trademark of T-VEC Technologies, Inc.

guidelines for using test automation framework measures · with measures and analysis techniques...

Documents