15 nov 2005 jones - iasted/sea 1 automated support for test-driven specification edward l. jones...

15 Nov 2005 Jones - IASTED/SEA 1

Automated Automated Support for Support for Test-Driven Test-Driven

SpecificationSpecificationEdward L. JonesEdward L. Jones

IASTED SEA ConferenceIASTED SEA Conference

Phoenix, ArizonaPhoenix, Arizona

October 14-16, 2005October 14-16, 2005

15 Nov 2005 2Jones - IASTED/SEA

OutlineOutline

Test-Driven Specification ParadigmTest-Driven Specification Paradigm Background Background An ExampleAn Example Automation SchemeAutomation Scheme Future WorkFuture Work


The Paradigm: Test Driven The Paradigm: Test Driven SpecificationSpecification

Test Cases

+

Assess Test Cases

SpecificationAssess

Specification

Software Development

Software Testing

Time


BackgroundBackground

Early Attention to Testing is GoodEarly Attention to Testing is Good V-Model of Software Testing (planning)V-Model of Software Testing (planning) W-Model of Software Testing (verification)W-Model of Software Testing (verification) Test-Driven Development (coding phase)Test-Driven Development (coding phase)

Requirements/Specification remain the hard part of software Requirements/Specification remain the hard part of software engineeringengineering

Verification options – prototyping, testing, model checking, Verification options – prototyping, testing, model checking, automated test case generationautomated test case generation


Specifics of Our WorkSpecifics of Our Work

Black-box: focus is software functionBlack-box: focus is software function Specification Language = Decision TableSpecification Language = Decision Table Decision table-based test adequacy Decision table-based test adequacy

criteriacriteria Automated conversion of decision table Automated conversion of decision table

to analyzer programto analyzer program Data-driven verification to reveal model Data-driven verification to reveal model

anomalies and test data inadequaciesanomalies and test data inadequacies


Decision TablesDecision Tables

Logic based, cause-effect specificationLogic based, cause-effect specification Within family of state-based specificationsWithin family of state-based specifications Old technology from 1950s-70sOld technology from 1950s-70s

Automated programming systemsAutomated programming systems Basis for software testingBasis for software testing

Specification language for rule based Specification language for rule based systems systems Vanthienen’ Prologa tool, other decision Vanthienen’ Prologa tool, other decision

management tools for decision sciencemanagement tools for decision science


An Example SpecificationAn Example Specification

Calculate employee pay, including overtime paid at 1.5 times the hourly rate of hourly employees for qualifying hours. The normal work week is 40 hours; the maximum number of paid hours is 80. Salaried employees earn over $30 per hour, and are paid overtime only for hours in excess of 60 hours. Salaried employees working less than 40 hours are paid for the normal work week. Hourly employees earn less than $30 per hour, and are paid for all hours worked up to the maximum; they are paid overtime for hours in excess of 40.


Stimulus-Response View of Stimulus-Response View of Specification Specification

Calculate Pay

hours

rate

regular_pay

overtime_pay


Decision Table for Decision Table for Specification (Incomplete)Specification (Incomplete)

---------------------------- ---------- hours > 40 | N N rate >= 30 | Y N ----------------------------- --------- overtime_pay = 0; | X X regular_pay = hours * rate; | X regular_pay = 40 * rate; | X ---------------------------- ----------

ConditionsDecision

Rules

ActionsAction Rules

Computation

Rule


Decision Table based Decision Table based Adequacy CriterionAdequacy Criterion

Possible adequacy criteria w.r.t. Possible adequacy criteria w.r.t. conditions, actions, rulesconditions, actions, rules

Functional CoverageFunctional Coverage Measures the thoroughness of testing Measures the thoroughness of testing

w.r.t. decision table rules w.r.t. decision table rules

FC = FC = # computation rules satisfied # computation rules satisfied

# computation rules in DT # computation rules in DT


Automated CheckingAutomated Checking

Specification anomaliesSpecification anomalies Ambiguous rulesAmbiguous rules - - test tuple satisfies test tuple satisfies

multiple rulesmultiple rules Missing ruleMissing rule -- -- test tuple satisfies no ruletest tuple satisfies no rule

Test data set adequacyTest data set adequacy Superfluous test data – rule satisfied by Superfluous test data – rule satisfied by

multiple test tuplesmultiple test tuples Incomplete test data – rule not satisfied Incomplete test data – rule not satisfied

by any test tupleby any test tuple Computation rule defines test oracle Computation rule defines test oracle


AutomationAutomation

Build Analyzer

Generate Analyzer

Decision TableRun

Analyzer Test Data Set

TestCoverage

Report

SpecCoverage

Report


DT Version #1DT Version #1

---------------------------- ---------- hours > 40 | N N rate >= 30 | Y N ----------------------------- --------- overtime_pay = 0; | X X regular_pay = hours * rate; | X regular_pay = 40 * rate; | X ---------------------------- ----------

Missing Rules


Analysis Reports - Analysis Reports - Version #1Version #1

TEST SET

hours rate 39 9 41 29 40 30 45 40

SPEC COVERAGE REPORT

VALID 35 9 315 0MISSING 41 29 MISSING MISSINGVALID 40 30 1200 0MISSING 45 40 MISSING MISSING

TEST COVERAGE REPORT

Rules -->Test -- --Case 1 2==== -- -- 1 X 2 3 X 4RULES NOT SATISFIED:FUNCTIONAL COVERAGE = 2/2


DT Version #2DT Version #2------------ ------------------------------------------N N Y Y - - | hours > 40- - - - Y - | hours > 60- - - - - Y | hours > 80N Y N Y Y Y | rate >= 30------------ ------------------------------------------X - - - - - | regular_pay = hours * rate;- X X X X X | regular_pay = 40 * rate;X X - - - - | overtime_pay = 0;- - X - - - | overtime_pay = 1.5 * rate * (hours - 40);- - - X X X | overtime_pay = 1.5 * rate * (hours - 60);------------ ------------------------------------------

TEST SET hours: 35 41 40 45 70 90 rate: 9 29 30 40 40 50




Rules -->Test -- -- -- -- -- --Case 1 2 3 4 5 6==== -- -- -- -- -- -- 1 X 2 X 3 X 4 X 5 X X 6 X X X

RULES NOT SATISFIED:FUNCTIONAL_COVERAGE = 6 / 6


VALID 35 9 315 0VALID 41 29 1160 43.5VALID 40 30 1200 0VALID 45 40 1600 -900AMBIGUOUS 70 40 1600 600AMBIGUOUS 70 40 1600 600AMBIGUOUS 90 50 2000 2250AMBIGUOUS 90 50 2000 2250AMBIGUOUS 90 50 2000 2250


DT Version #3DT Version #3------------ ------------------------------------------N N Y - - - | hours > 40- - - Y Y - | hours > 60- - - - N Y | hours > 80N Y N Y Y Y | rate >= 30------------ ------------------------------------------ X - - - - - | regular_pay = hours * rate;- X X X X X | regular_pay = 40 * rate;X X - - - - | overtime_pay = 0;- - X - - - | overtime_pay = 1.5 * rate * (hours - 40);- - - X X X | overtime_pay = 1.5 * rate * (hours - 60); ------------ ------------------------------------------

TEST SET hours: 39 41 40 45 70 90 rate: 9 29 30 40 40 50

No similar rule for salaried employee.




Rules -->Test -- -- -- -- -- --Case 1 2 3 4 5 6==== -- -- -- -- -- -- 1 X 2 X 3 X 4 X 5 X X 6 X X

RULES NOT SATISFIED:FUNCTIONAL_COVERAGE = 6 / 6


VALID 35 9 315 0VALID 41 29 1160 43.5VALID 40 30 1200 0MISSING 45 40 MISSING MISSINGAMBIGUOUS 70 40 1600 600AMBIGUOUS 70 40 1600 600AMBIGUOUS 90 50 2000 2250AMBIGUOUS 90 50 2000 2250


ConclusionConclusion Have presented a black-box approachHave presented a black-box approach

Test-driven specification as a paradigm Test-driven specification as a paradigm

for interaction between developer and for interaction between developer and

testertester

Demonstrated benefits of automationDemonstrated benefits of automation

Claim that approach is applicable to Claim that approach is applicable to

existing decision based specification existing decision based specification

methodsmethods


Future WorkFuture Work

Extend to State-Based SpecificationsExtend to State-Based Specifications Mill’s State-Box modelMill’s State-Box model SCR state tablesSCR state tables Pre-/Post-condition specificationsPre-/Post-condition specifications

Consider random generation of test Consider random generation of test data used to verify specificationdata used to verify specification

Use tool in software engineering Use tool in software engineering coursescourses


Questions?Questions?


Motivation / ReferencesMotivation / References

Hoffman & Strooper’s 2001 paperHoffman & Strooper’s 2001 paper

“Prose + Test Cases = Specification”

Pamela Zave’s IEEE-TSE 1991 paper, a post-mortem on Pamela Zave’s IEEE-TSE 1991 paper, a post-mortem on her 1980’s work on the Paisley operational specifications her 1980’s work on the Paisley operational specifications system system

“ “An Insider’s View of Paisley”An Insider’s View of Paisley”

15 nov 2005 jones - iasted/sea 1 automated support for test-driven specification edward l. jones...

Documents

test tuplecomputation

jones iastedseadt version

test coverage report

jones iastedseaoutlinetest

jones iastedseastimulus

qualifying hours

paid overtime

decision table rules