Testing the Polymorphic Interactions betweenClasses �

Robert McDaniel

John D. McGregor

Department of Computer Sci ence

Cl emson Uni versi ty

Cl emson, SC29634-1906

johnmc@cs. cl emson. edu

Abstract

To date, much of the e�ort spent in research wi th object -or i ented technology has been i n

the des i gn and i mpl ementat i on phase. An i mportant act i vi ty of the obj ect - or i ented sof tware

devel opment proces s whi ch has not been the f ocus of a great deal of research i s that of tes t i ng.

In part i cul ar , t es t i ng the i nteract i ons between two or more cl as ses has recei ved vi r tual l y no

f ocused attent i on. Thi s not i on, termed inter-class testing, when compared to i nter - procedural

tes t i ng, i nvol ves some of the same probl ems and concerns wi th f unct i onal i ty and data. How-

ever , the combi nat i on of such f eatures as obj ect s tate, pol ymorphi sm, and dynami c bi ndi ng

that make the obj ect - or i ented paradi gmattract i ve to devel opers , i nt roduce new obstacl es un-

seen i n any other tes t i ng envi ronment . Thi s work exami nes tes t i ng i nteract i ons , or message

passes , between cl as ses and pi npoi nts probl ems i ntroduced by the obj ect - or i ented paradi gm.

The combi nat i on of obj ect s tate and the poss i bi l i ty of pol ymorphi c subst i tut i ons can, i f con-

s i dered exhaust i vel y, compound the tes t i ng e�ort s i gni �cant l y when tes t i ng the i nteract i on of

two cl as ses vi a a message. To ai d i n reduci ng the tes t i ng e�ort i ntroduced by these combi ned

f eatures of the obj ect - or i ented paradi gm, a pol ymorphi c message pass i s de�ned i n terms of an

orthogonal array. The resul t i ng tes t domai n coverage generated, based on theory of a method

cal l ed Robust Testing, i s exami ned i n terms of three l evel s of de�ned adequacy. Addi t i onal l y,

some gui del i nes f or i nter - cl as s tes t i ng based on the method devel oped i n thi s paper are gi ven.

�The current version of thi s techni cal report i s heavi l y based on Robert McDani el 's masters thesi s enti tl edThe

E�ects of Polymorphism, Dynami c Bi ndi ng, and Object State on Testi ng the Interacti ons of Cl asses

Content s

1 Introduction 6

2 De�nitions andDescriptions 7

3 Interprocedural Testing 9

3.1 A Model for Integrat ion Test i ng :: : : : : : : : : : : : : : : : : : : : : : : : : : : : : 9

3. 2 Test Case Devel opment f or Error Detect i on :: : : : : : : : : : : : : : : : : : : : : : 10

3. 2. 1 I nter f ace Tests :: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 11

3. 2. 2 Funct i onal Tests :: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 11

3. 3 I nterprocedural Concl us i ons :: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 11

4 Inter-Class Testing 12

4. 1 Assumpti ons : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 13

4. 2 Cl ass I nteract i on :: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 13

4. 3 Pol ymorphi c Subst i tut i ons :: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 15

4. 4 Dynami c Bi ndi ng : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 16

4. 5 Object State :: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 17

5 Testing Based onObject State andDynamic Substi tutions of Objects 20

5. 1 Level s of Adequacy :: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 20

5. 2 An Overvi ew of obust Test i ng :: : : : : : : : : : : : : : : : : : : : : : : : : : : : : 24

5. 2. 1 Appl yi ng obust Test i ng Techni ques :: : : : : : : : : : : : : : : : : : : : : : 25

5. 2. 2 Steps i n Perf ormi ng obust Test i ng :: : : : : : : : : : : : : : : : : : : : : : 26

5. 3 Sel ect i ng Test Cases :: : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 28

5. 4 An Al ternat i ve Techni que : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 31

6 rror Detection nalysi s 34

2

7 uidel ines for erforming Inter-Class Testing 36

8 Conclusions and uture ork 39

Calculati ng the umber of Test Cases to xhaustively Test a iven essage

Interaction. 42

B T o actor Orthogonal rrayConstruction for a essage i th o arameters 44

C Orthogonal rrayConstruction for a essage i thOne arameter 48

D Orthogonal rrayConstruction for a essage i thOne arameter Considering

State as Independent 51

3

is t o i res

1 Sampl e cl ass hi erarchi es sendi ng and recei vi ng a message. :: : : : : : : : : : : : : : 14

2 Message to a cl ass hi erarchy wi th vi rtual methods. :: : : : : : : : : : : : : : : : : : 16

3 ueue cl ass hi erarchy wi th states . :: : : : : : : : : : : : : : : : : : : : : : : : : : : 19

4 Consi derat i ons f or Exhaust i ve Coverage. :: : : : : : : : : : : : : : : : : : : : : : : : 22

5 Consi derat i ons f or Mi ni mal Coverage. :: : : : : : : : : : : : : : : : : : : : : : : : : 23

6 Consi derat i ons f or epresentat i ve Coverage. :: : : : : : : : : : : : : : : : : : : : : : 24

7 Sampl e cl ass hi erarchi es sendi ng and recei vi ng a message. :: : : : : : : : : : : : : : 29

8 ueue cl ass hi erarchy wi th states . :: : : : : : : : : : : : : : : : : : : : : : : : : : : 32

9 Al gor i thm : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 37

10 Al gor i thm : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 38

11 Cal cul at i ng poss i bl e test cases cons i der i ng state and pol ymorphi c subst i tut i ons . :: : 42

12 Obj ect and Parameter i erarchi es f or a Pol ymorphi c Message Pass . :: : : : : : : : 48

4

i s t o les

1 Standard L9 orthogonal array. :: : : : : : : : : : : : : : : : : : : : : : : : : : : : : 26

2 Mi ni mal Test Case f or ueue Exampl e : : : : : : : : : : : : : : : : : : : : : : : : : 33

3 Mi ni mal Test Case f or ueue Exampl e : : : : : : : : : : : : : : : : : : : : : : : : : 34

4 Cal cul at i ng the degrees of f reedomi n EOA. : : : : : : : : : : : : : : : : : : : : 38

5 Poss i bl e representat i ons f or the parameter obj ect . :: : : : : : : : : : : : : : : : : : 42

6 Orthogonal Array f or Two Factors , 8-l evel s . :: : : : : : : : : : : : : : : : : : : : : 46

7 Tabl e 6 Conti nued. : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 47

8 Cal cul at i ng the mi ni mumnumber of test cases requi red. :: : : : : : : : : : : : : : 49

9 Cal cul at i ng the degrees of f reedom. :: : : : : : : : : : : : : : : : : : : : : : : : : : 52

10 Orthogonal Array L25. : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : : 53

11 Abbrevi at i ons f or Test Case Expans i on. :: : : : : : : : : : : : : : : : : : : : : : : : 55

12 Orthogonal Array L25 expanded to showactual test cases . :: : : : : : : : : : : : : 56

5

nt r o t i on

Obj ect or i ented technol ogy has grown over the past decade i nto a wel l de ned and i ncreas i ngl y

popul ar styl e of des i gn and programmi ng, and wi th good reason. The technol ogy o ers great

bene ts f or code mai ntenance, extens i bi l i ty and reuse. Powerf ul f eatures such as encapsul at i on and

i nher i tance, that support the bene ts of obj ect- or i ented technol ogy i n the sof tware devel opment and

mai ntenance process , i ntroduce newprobl ems i nto the area of sof tware ver i cat i on and val i dat i on.

To date, much of the e ort spent i n research wi th obj ect- or i ented technol ogy has been i n the des i gn

and i mpl ementat i on phases . An i mportant act i vi ty of the obj ect or i ented sof tware devel opment

process , whi ch has not been the f ocus of a great deal of research, i s that of test i ng. I n part i cul ar ,

test i ng the i nteract i ons among two or more cl asses has recei ved vi rtual l y no f ocused attent i on. I n

the obj ect or i ented paradi gm, i nteract i on between two cl asses occurs i n the f ormof a essa e passed

f roma run- t i me i nstance of one cl ass , commonl y ref erred to as an obj ect , to another obj ect . These

messages are sent and recei ved by obj ects i n part i cul ar states , andmay al so i ncl ude parameters that

are obj ects i n part i cul ar states . Some combi nat i ons of the obj ects and states i nvol vedwi th a message

send coul d potent i al l y cause unexpected react i ons that a devel oper may have over l oo ed dur i ng

i mpl ementat i on. I n a gi ven obj ect or i ented sof tware proj ect , there may be thousands of messages

passed between obj ects , and each message coul d potent i al l y cause a ser i ous error . Theref ore, the

success or f ai l ure of each i ndi vi dual message i s a concern i n the test i ng process .

The not i on of test i ng messages between cl asses , termed inte - ass testin , i s the f ocus of

thi s research. I n thi s wor we exami ne test i ng i nteract i ons , or message passes , between obj ects i n

a run- t i me envi ronment. i th no substant i al e ort on i nter- cl ass test i ng i n exi stence, wor done

on i nter- procedural test i ng i s exami ned as a start i ng poi nt . The purpose of exami ni ng the i nter-

procedural wor i s to i dent i f y the types of errors f ound i n the procedural paradi gmand chec f or

the exi stence of these or s i mi l ar errors i n the obj ect or i ented paradi gm. As i ni t i al l y expected, some

of the same probl ems wi th f unct i onal i ty and data that exi st i n the procedural paradi gmare f ound

to be val i d concerns i n the obj ect or i ented paradi gm. owever , f eatures such as pol ymorphi sm,

dynami c bi ndi ng, and obj ect state f ound col l ect i vel y i n the obj ect or i ented paradi gmi ntroduce

some newprobl ems and concerns . The i mpl i cat i on of each of these f eatures on test i ng i s exami ned.

Fromthese i ndi vi dual exami nat i ons we combi ne pol ymorphi smand obj ect state and show thei r

6

e ects on compoundi ng the test i ng e ort that i s requi red dur i ng i nter- cl ass test i ng. e exami ne a

sof tware test i ng method cal l ed st estin and exami ne i ts appl i cabi l i ty to i nter- cl ass test i ng

f or reduci ng the test i ng e ort i ntroduced by the combi nat i on of pol ymorphi smand obj ect- state.

An al ternat i ve f ormof obust Test i ng der i ved f or address i ng pol ymorphi smand obj ect state i s

used to generate a set of test cases . These test cases are then exami ned i n terms of three de ned

l evel s of adequacy to determi ne thei r abi l i ty to detect errors at the i nter- cl ass l evel .

The next sect i on of thi s paper gi ves de ni t i ons and descr i pt i ons of f eatures of the obj ect-

or i ented paradi gmas they are vi ewed i n thi s paper . Sect i on 3 exami nes i nterprocedural wor to

i dent i f y the types of errors that coul d occur and cl ass i es test cases that can l ead to detect i on of

these types of common errors . Sect i on 4 anal yzes i nter- cl ass test i ng, the commonal i t i es between

i nterprocedural and i nter- cl ass test i ng, and exami nes the e ects of pol ymorphi sm, dynami c bi ndi ng,

and obj ect state on test i ng. Sect i on 5 deal s wi th test i ng based on obj ect state and the poss i bi l i ty of

pol ymorphi c subst i tut i ons encountered wi th a message send. Three l evel s of test i ng adequacy f or

deal i ng wi th these two concerns are presented. Then, a tradi t i onal l y procedural sof tware test i ng

method cal l ed st estin , based on the use of orthogonal arrays and used f or sel ect i ng a subset

of test cases f roma l arger set , i s out l i ned. Fromthi s expl anat i on, the current probl emof the

combi nat i on of pol ymorphi smand obj ect state i s de ned i n terms of an orthogonal array and test

case combi nat i ons f or a s i mpl e exampl e are generated based on the theory of obust Test i ng us i ng

OATS Orthogonal Array Test i ng System. The set of combi nat i ons generated are exami ned i n

terms of the three l evel s of adequacy. Sect i on 6 descr i bes some types of errors expected f or an

exampl e presented i n the paper and the abi l i ty of thi s techni que to produce test cases to detect

such errors as wel l as the cl ass i cat i ons of errors establ i shed i n sect i on 4. Sect i on 7 out l i nes some

pseudo al gor i thms f or perf ormi ng i nter- cl ass test i ng, and sect i on 8 o ers concl us i ons and f uture

wor .

e ni t i ons n es r i t i ons

Because the term e t iente can be i nterpreted di erent l y, i t i s worth expl ai ni ng the vi ewpoi nt

ta en i n thi s research. Obj ect- or i ented ref ers to the use of cl asses , obj ects , and i nher i tance. A

ass i s a stat i c ent i ty that cons i sts of data and member f unct i ons , or methods, that represent

7

the operat i ons that can be perf ormed on the data. An e t i s a run t i me instan e of a cl ass

that has i ts own copy of the pr i vate data of the cl ass , and encapsul ates both state and behavi or .

A sub- cl ass can in e i t data and or methods f roma parent cl ass . ew cl asses may be de ned

i n terms of a pre- exi st i ng cl ass wi th some newmethods and or data added. I nher i tance has two

di st i nct perspect i ves i n sof tware devel opment, one termed \i nher i tance of i mpl ementat i on", and

the other \i nher i tance of speci cat i on. " The ai mof i nher i tance of i mpl ementat i on i s to bui l d the

new cl ass us i ng some i mpl ementat i on detai l s f roman exi st i ng cl ass . I nher i tance of speci cat i on,

on the other hand, bui l ds a newcl ass by i ncl udi ng the speci cat i on of an exi st i ng cl ass as a subset

of the speci cat i on of the newcl ass . Some obj ect- or i ented l anguages al l ow ti e in e i tan e, i n

whi ch a sub- cl ass i nher i ts data and or methods f rommul t i pl e parent cl asses .

I nstead of the procedural i dea of i nputs and outputs f or procedures , essa es are sent to an

obj ect . Amessage requests execut i on of a method of an obj ect , poss i bl y wi th arguments f or the

method. The requested method can modi f y the i nternal state of i ts obj ect and or send messages to

other obj ects . hen a method compl etes execut i on, i t returns the thread of control , and poss i bl y

a return val ue, to the obj ect that sent the message request i ng the operat i on.

The termpol ymorphi smmeans the abi l i ty to ta e more than one f orm. orson and Mc-

regor [7] de ne a pol ymorphi c ref erence i n an obj ect- or i ented l anguage, as one that can, over

t i me, ref er to i nstances of more than one cl ass . Because of thi s abi l i ty to ref er to more than one

cl ass of obj ect , a pol ymorphi c ref erence has both a stat i c and a dynami c type associ ated wi th i t

[ 7] . The stat i c type i s nown at compi l e t i me and determi nes the set of val i d types that can be

accepted by the obj ect at run t i me. The dynami c type i s not nown at compi l e t i me and can

change dur i ng execut i on of the program. orson and Mc regor [ 7] al so state that i n a strongl y

typed obj ect- or i ented l anguage the \i s a" nature of i nher i tance i s t i ght l y coupl ed wi th the i dea of

pol ymorphi sm. For exampl e, i f Bi nher i ts f romA, Bhas al l the f eatures of Apl us any newf eatures

added, theref ore Bi s an A, and anywhere an i nstance of Ai s expected an i nstance of Bi s al l owed.

A f eature cl osel y associ ated wi th i nher i tance and pol ymorphi sm i s na i in in . A

method i mpl emented i n a parent cl ass can be rede ned i n an i nher i ted cl ass . These dynami cal l y

bound methods are ref erred to, i n some obj ect or i ented l anguages , as i t a methods. I f a method

i s de ned i n cl ass Aand rede ned i n an i nher i ted cl ass B, when a message i s recei ved by cl ass

8

A request i ng method , the actual code that gets bound to method and executed depends

on the dynami c type of cl ass A. ence, the method i s dynami cal l y bound to the actual code

i mpl ementat i on. on- vi rtual methods are stat i cal l y bound and the code executed f or the part i cul ar

method i s the same f or any dynami c type of the cl ass .

nt er r o e r l es t i n

An i ni t i al concern wi th i nter- cl ass test i ng was to i dent i f y the types of errors that coul d occur when

test i ng the i nteract i on of two obj ects . i th no exi st i ng wor speci cal l y address i ng these concerns ,

wor done on procedural i ntegrat i on test i ng by Leung and hi te [ 4] was exami ned as a bas i s . Thi s

wor o ered cl ass i cat i ons of errors and cl ass i cat i ons of test cases to chec f or these common

types of procedural i ntegrat i on errors . Thi s sect i on summari zes the cl ass i cat i ons establ i shed by

Leung and hi te.

.

Based on an anal ys i s of the i nput domai ns of f unct i ons , Leung and hi te [ 4] devel oped a model

f or i ntegrat i on test i ng that stresses concentrat i on on three types of errors : 1 extra f unct i on, 2

mi ss i ng f unct i on, and 3 i nter f ace errors . The authors expl ai n that detect i on of these three types

of errors shoul d be the pr i mary obj ect i ve of i ntegrat i on test i ng, and even though these errors may

not be present i n a gi ven sof tware system, a poor l y des i gned systemmay contai n them. The three

types of errors as de ned by Leung and hi te [ 4] are descr i bed bel ow.

xtra unction rrors - i en t e s e i ati ns es an , t e e a e s e

n ti na i ties n t e i e , an t e e a e s e in ts t i a in e

t ese n ti ns an a se an e in . e e e e as n t nsi e e t e t e n ti na i ties

i a e n t e i e , t et in e in ts .

Si mpl y put, certai n i nputs to a f unct i on may i nvo e some addi t i onal f unct i onal i ty. Cons i der

an exampl e of a bounded queue and the inse t f unct i on associ ated wi th the bounded queue. I f

we i nvo e inse t wi th a f ul l bounded queue, we typi cal l y expect the f unct i on to return an error

code i ndi cat i ng that the bounded queue was f ul l . owever , a devel oper may have added i n the

9

f unct i onal i ty that when no i temcan be added to a f ul l bounded queue, the f unct i on stores the i tem

that was to be added and upon next i nvocat i on of the e ete f unct i on, adds the i temaf ter one has

been removed. I f the user of a f unct i on does not real i ze thi s addi t i onal f unct i onal i ty exi sts or has

not f ul l y cons i dered the i mpl i cat i ons of thi s addi t i onal f unct i onal i ty on f uture operat i ons wi th the

bounded queue, probl ems coul d resul t . The i nput of a f ul l bounded queue i nto the inse t f unct i on

has i nvo ed some addi t i onal f unct i onal i ty unseen wi th other i nputs of the inse t f unct i on.

i ssing unction rrors - e e a e s e a ia es se as in t t i

a e tsi e t e s e i ati ns t is an e ie e as ai in t s a t e n ti na i ties

e i e .

I n s i mpl er terms, the f unct i onal i ty of B f al l s short of what was expected of i t . I f we expect

an error code f roma queue when the e ete f unct i on i s i nvo ed on an empty queue and one i s not

returned, then the f unct i onal i ty of e ete i s not as expected. The e ete f unct i on f ai l s to suppl y al l

the f unct i onal i ty expected when i nvo ed wi th an empty queue.

Interface rrors - ninte a e e s ene e t e inte a e stan a et eent

es is i ate . e a e, t e a a ete s a n t e in e t e an t e a

n t e t e i t ata t es, ats, an in t t t es. n a i ti n, t e a a ete es

e. . a - - a e a - - e e en e a n t e e e . t s e t ese e s a e

ete te an a an e i e , e ann t a a s e n a i e .

I nter f ace errors are s i mpl y vi ol at i ng the i nter f ace to a modul e. For exampl e, i f a poi nter to

a queue i s expected as one argument to inse t and i nstead we try and pass the queue i tsel f , we have

vi ol ated the i nter f ace of the inse t f unct i on.

.

Based on the three cl ass i cat i ons of common errors that shoul d be detected dur i ng procedural

i ntegrat i on test i ng, Leung and hi te [ 4] propose us i ng two types of tests to ai d i n detect i ng these

cl ass i cat i ons of errors . These two types of tests are: 1 i nter f ace tests , and 2 f unct i onal tests .

10

3.2.1 Interface Tests

The i nter f ace test i s des i gned to chec the cal l i ng i nter f ace between the two i nteract i ng modul es .

There are two i nds of i nter f ace tests , the rst appl i cabl e to syntact i c cal l s and the second to

dynami c cal l s . The syntact i c i nter f ace tests shoul d chec the data type, f ormat and the parameter

pass i ng rul es of each parameter , and the order and number of parameters . Thi s type of test i ng can

typi cal l y be done us i ng stat i c anal ys i s . The second type of test , ref erred to by Leung and hi te [ 4]

as extremal tests , tests the extreme val ues of the i nput var i abl es to a f unct i on. Extreme val ues f or

the cal l i ng and the cal l ed modul e shoul d be cons i dered and can be retai ned f rommodul e test i ng to

use dur i ng i ntegrat i on test i ng. Compari son of extremal val ues f or each modul e can l ead the tester

to sel ect test cases that chec f or extra and mi ss i ng f unct i on errors dur i ng i ntegrat i on test i ng.

3.2.2 unctional Tests

The f unct i onal test case devel opment i s based on the wor done by owden [ 3] , who f ormal i zed the

f unct i onal test i ng method and provi ded gui del i nes f or sel ect i ng f unct i onal tests . s i ng owden's

methods, the tester i dent i es the f unct i ons that are supposed to be i mpl emented by the tested

program, and sel ects test data that can be used to chec the correctness of the i mpl emented

f unct i ons . Asubstant i al manual e ort i s requi red i n i dent i f yi ng f unct i ons and sel ect i ng test data

dur i ng modul e test i ng, but thi s e ort need not be repeated when perf ormi ng i ntegrat i on tests .

Compari son of the test data generated f or the modul e tests can l ead the tester to sel ect test data

that wi l l chec f or extra and mi ss i ng f unct i on errors dur i ng i ntegrat i on test i ng.

.

Leung and hi te f eel that detect i ng the three types of errors l i sted ear l i er i s the pr i mary goal of

i ntegrat i on test i ng. e f eel that the rst two types of errors , extra f unct i on and mi ss i ng f unct i on,

coul d be cl ass i ed as one type of error , a f unct i onal i ty error . ence, we have f unct i onal i ty errors

between modul es and i nter f ace errors , whi ch t the test case cl ass i cat i ons of Leung and hi te,

and to whi ch we can appl y f unct i onal and i nter f ace tests .

11

nt er -Cl s s es t i n

I nter- cl ass test i ng i s a subset of a l evel of obj ect or i ented test i ng ref erred to as i ntegrat i on test i ng

I ntegrat i on test i ng i s the l evel of obj ect or i ented test i ng i n whi ch we are concerned about the i nter-

act i ons of methods wi thi n a cl ass intra-class and the i nteract i ons between cl asses vi a messages

inter-class . There i s no di st i nct boundary f or when i ntegrat i on test i ng begi ns and test i ng i ndi -

vi dual cl asses ends. hen a s i ngl e cl ass i s tested, we test the i ndi vi dual methods wi thi n the cl ass ,

whi ch may i nvol ve methods request i ng other methods wi thi n the cl ass to f ul l l the speci cat i ons

of the tested method. Theref ore, we are perf ormi ng i ntra- cl ass test i ng as we progress through

i ndi vi dual cl ass test i ng, concerni ng oursel ves wi th the correctness of the f unct i onal i ty of the tested

method, correctness of the i nter f ace usage f or methods, and i ntegr i ty of any common data shared

between the methods. I nter- cl ass test i ng can al so occur as i ndi vi dual cl asses are tested as wel l as

af ter i ndi vi dual cl ass test i ng has been perf ormed. hen test i ng i ndi vi dual cl asses , methods may

request a method f romanother cl ass as part of the f unct i onal i ty of the method current l y bei ng

tested and theref ore two obj ects must i nteract . I nteract i ons between cl asses may al so ar i se af ter

i ndi vi dual cl ass test i ng i s compl ete because of devel oped appl i cat i on code that resul ts i n obj ects

i nteract i ng. Even though there i s no di st i nct boundary f or these phases of test i ng, us i ng \stubs"

f or requests to external cl asses al l ows a tester to del ay i nter- cl ass test i ng unt i l the i ntra- cl ass phase

of test i ng i s compl ete. i th i nter- cl ass test i ng, the tester must al so be concerned wi th the correct-

ness of the f unct i onal i ty of the method bound to the message, correctness of the method i nter f ace

bound to the method, and the e ects of the i nteract i on on the i ndi vi dual obj ects i nvol ved as wel l

as the sof tware systemas a whol e. I nter- cl ass test i ng i s onl y concerned wi th one message at a t i me,

and so wi th the potent i al l y thousands of messages poss i bl e i n a obj ect or i ented sof tware system,

i nter- cl ass test i ng can compose a maj or i ty of the i ntegrat i on test i ng f or obj ect or i ented sof tware.

Thi s sect i on exami nes i nter- cl ass test i ng wi th the i ntent of expos i ng concerns and poss i bl e

probl ems. The appl i cabi l i ty of Leung and hi te' s cl ass i cat i ons gi ven i n the previ ous sect i on are

cons i dered and when poss i bl e, i nter- cl ass i ssues are mapped i nto the i nter- procedural f ramewor .

As expected, there are concerns that do not map di rect l y and theref ore must be addressed.

12

.

The f ol l owi ng assumpti ons are made f or i nter- cl ass test i ng.

1. e assume a test i ng oracl e exi sts that can ver i f y the correctness of test case execut i on.

2. Cl asses are wel l - de ned. Awel l - de ned cl ass i s one that has the l i mi ts of al l i ts methods

de ned cl ear l y and appropr i atel y i mpl emented. Thi s i mpl i es that the correct speci cat i ons

f or the cl asses are avai l abl e.

3. Cl ass hi erarchi es are avai l abl e f or exami nat i on.

4. Al i st of the poss i bl e states of each obj ect i s avai l abl e so that we nowthe states to test .

5. al i d arguments exi st f or the constructor of an obj ect that al l ow an obj ect to be put i nto

each of i ts states .

6. I n the test i ng process f or each i ndi vi dual cl ass , the tester has progressed through test i ng

i ndi vi dual methods, i ncl udi ng i ntra- cl ass i ntegrat i on test i ng, and has used \stubs" to avoi d

i nter- cl ass test i ng. The tester i s nowat the poi nt where there are no \stubs" i n pl ace, al l

cl asses are f ul l y i mpl emented and avai l abl e f or test i ng.

7. Test i ng wi l l be done based on the i nteract i on of two obj ects at a t i me, addi t i onal obj ect

i nteract i on can be tested i ncremental l y.

8. l obal val ues wi l l be treated i n the same manner as parameters .

These assumpti ons are not i ntended to be restr i ct i ve, but are i nstead normal processes or pi eces of

the test i ng process that shoul d be avai l abl e i n an obj ect or i ented sof tware devel opment proj ect .

.

As previ ous l y stated, i nteract i on between i nstances of cl asses occurs i n the f ormof a message passed

f romone i nstance of a cl ass to an i nstance of another cl ass request i ng an operat i on on the l atter

i nstance. Thi s i nteract i on between two i nstances vi a a message can occur i n two di st i nct ways.

I n the s i mpl est case, a cl ass wi th no subcl asses can be the target of a message. I f the cl ass has

subcl asses then pol ymorphi c subst i tut i on f or the recei ver may be poss i bl e dependi ng upon the type

13

of ref erence made to the obj ect . I n thi s wor we di rect l y address the case i n whi ch the poss i bi l i ty

of pol ymorphi c subst i tut i ons f or obj ects exi sts .

A

B C

messageD

E F

Fi gure 1: Sampl e cl ass hi erarchi es sendi ng and recei vi ng a message.

Fi gure 1 shows two si mpl e cl ass hi erarchi es . Amessage i s bei ng passed between i nstances of

cl asses Aand D, and thus, these two obj ects are i nteract i ng. There are several goal s f or the test i ng

of thi s obj ect i nteract i on. I f a set of pre and post condi t i ons exi sts f or the method bound to the

message, we must exami ne these condi t i ons and ver i f y that they are not vi ol ated. Addi t i onal l y,

we want to i nsure that the f unct i onal i ty of the message i s correct and that the i nter f ace to cl ass

D, the recei vi ng cl ass , i s not vi ol ated. The l atter two concerns map di rect l y to cl ass i cat i ons of

errors gi ven by Leung and hi te. er i f yi ng that the f unct i onal i ty of a method i s correct may be a

t i me consumi ng process . I t requi res a f ul l understandi ng of the method and, of course, the abi l i ty

to i dent i f y and der i ve the correct resul ts of the method. i th regards to i nter f ace vi ol at i ons , we

woul d typi cal l y chec f or such thi ngs as 1 the number of parameters i nvol ved i n the message i f

any , 2 the correctness of the order of the parameters , 3 the correctness of the type s of the

parameters i nvol ved and 4 poss i bl e val ues f or the parameters . For these i nter f ace concerns , we

woul d expect a good compi l er to detect the rst three and, i f our obj ects are wel l de ned, the range

of bounds f or concern number 4 shoul d be de ned and i mpl emented f or the method bound to the

message. owever , as Leung and hi te state [ 4] , detect i ng i nter f ace errors remai ns a pr i mary goal

of i ntegrat i on test i ng, so exami nat i on f or i nter f ace errors i s a val i d concern i n i nter- cl ass test i ng.

I n addi t i on to f unct i onal and i nter f ace concerns , we want to chec f or \s i de e ects" f rom

message parameter val ues , both i mpl i ci t and expl i ci t , i n both obj ect Dand obj ect A. I n general , we

want to observe the e ect s of pass i ng the message f romobj ect Ato obj ect Dand determi ne i f the

obj ect i nteract i on was a success or f ai l ure. Success or f ai l ure i n thi s regard can be determi ned by

14

observi ng e ects on the data wi thi n each obj ect , on common data between both obj ects , and the

state of the sof tware systemas a whol e. Deal i ng wi th these \s i de e ect" i ssues may be part i cul ar l y

di cul t i n the obj ect or i ented paradi gm, even more so than i n the procedural wor l d. I n C , f or

exampl e, the l evel s of data vi s i bi l i ty are gl obal , publ i c, pr i vate, protected, and f r i end. Each of these

l evel s of data vi s i bi l i ty al l owvaryi ng degrees of access to methods and or data, dependi ng on i ts

decl arat i on and the scope of the cal l i ng component.

For two s i ngl e i nteract i ng cl asses , the goal s f or i nter- cl ass test i ng are synonymous wi th the

goal s of i nter- procedural test i ng. I n order to nd poss i bl e addi t i onal probl ems i n the obj ect- or i ented

paradi gmwe must l oo cl osel y at f eatures f ound col l ect i vel y wi th the i ntent of uncover i ng thei r

i mpact on test i ng. Three such f eatures are pol ymorphi c subst i tut i ons f or obj ects , dynami c bi ndi ng

of methods, and obj ect state.

.

Si nce the termpol ymorphi smwas expl ai ned ear l i er , we nowthat i nher i ted cl asses are al l owabl e

subst i tutes f or parent cl asses i n the run- t i me envi ronment. Loo i ng bac at Fi gure 1, we see that

cl asses Band Ci nher i t f romcl ass Aand theref ore are poss i bl e pol ymorphi c subst i tut i ons f or cl ass

Aat run- t i me. Thi s means that an i nstance of ei ther cl ass Bor cl ass C, i nstead of cl ass A, coul d

l egal l y send the message to an i nstance of the recei vi ng cl ass D. But cl ass Dal so has two poss i bl e

pol ymorphi c subst i tutes , E and F, i nstances of whi ch coul d l egal l y accept the message f romthe

obj ect sendi ng the message at run- t i me. i thout the poss i bl e pol ymorphi c subst i tutes , we woul d

onl y test cl ass Awi th cl ass D. owever , nowwe must al so cons i der the run- t i me poss i bi l i ty of cl ass

Aand al l i ts poss i bl e pol ymorphi c subst i tut i ons i n combi nat i on wi th cl ass Dand al l i ts poss i bl e

pol ymorphi c subst i tut i ons . I nstead of one set of cl asses to test , we nowmust cons i der ni ne sets of

i nteract i ng cl asses to cover a s i ngl e message between two obj ects .

I f the message sent f romone obj ect to another contai ns parameters that are obj ects , we

must cons i der the pol ymorphi c subst i tut i ons f or the parameters when the message i s tested. Pol y-

morphi smagai n compounds the number of poss i bl e test cases to cons i der . As an exampl e of the

magni tude of test cases that can resul t , suppose the message between the two obj ects A and D

i n Fi gure 1 has two parameters , and , both of whi ch have 3 sub- cl asses . I n order to test the

15

message exhaust i vel y j ust between Aand D, we woul d have to construct s i xteen di erent messages

to cover al l combi nat i ons of and and thei r poss i bl e pol ymorphi c subst i tutes . These s i xteen

di st i nct representat i ons of the same message onl y account f or parameters and do not account f or

poss i bl e pol ymorphi c subst i tut i ons f or cl ass Aor cl ass D, whi ch when cons i dered mean construct i ng

one hundred and f orty f our di erent test cases f or s i mpl i ci ty assumi ng one test case per combi -

nat i on to cover al l poss i bl e combi nat i ons of and and thei r poss i bl e pol ymorphi c subst i tutes ,

al ong wi th A, B, C, D, E, and F. emember, thi s i s the number requi red to cover a s i ngl e message

passed between two obj ects . For each of these combi nat i ons of obj ects i nvol ved wi th the message

we wi sh to test , we have to be concerned about the f unct i onal , i nter f ace, and data i ssues rai sed i n

the previ ous sect i on.

.

I n l i ght of the assumpti on made regardi ng the presence of pol ymorphi c subst i tut i ons f or obj ects

bei ng the s i tuat i on of concern i n thi s research, the probl emof dynami c bi ndi ng of methods ar i ses .

i th a dynami cal l y bound method, the code that gets bound to the method and executed at run

t i me depends on the dynami c type of the obj ect . I n a l anguage such as C , dynami c bi ndi ng i s

onl y a probl emwhen poss i bl e pol ymorphi c subst i tut i ons f or obj ects exi st . owever , i n Smal l tal ,

al l methods are dynami cal l y bound, whi ch means the probl emi s al ways present . i th a stat i cal l y

bound method, the speci c code that wi l l execute when the method i s requested i s nown at compi l e

t i me and bound to the message. Theref ore, when a speci c method i s requested, the same code i s

al ways executed regardl ess of the dynami c type of the obj ect .

draw D

E F

virtual draw ( );

virtual draw ( ); virtual draw ( );

Fi gure 2: Message to a cl ass hi erarchy wi th vi rtual methods.

Fi gure 2 shows the a message bei ng passed to a recei vi ng obj ect D. Because of the poss i bl e

16

pol ymorphi c subst i tut i ons f or cl ass D, the i ntended recei ver , we al so have to cons i der cl asses Eand

F. The a method i n cl ass Di s a dynami c, or vi rtual , method that i s rede ned i n cl asses Eand

F. I f the dynami c type of the requested obj ect causes obj ect Dto get the request f or a , then the

a method i mpl emented i n Dwi l l get executed. Li ewi se a dynami c type of cl ass Egets a f rom

cl ass Eand of type Fgets a f romcl ass F. Even though the part i cul ar method that gets i nvo ed

can change, the obj ect envi ronment surroundi ng the i nvo ed method remai ns constant. Because

we assume the cl asses have been tested to some extent i n i sol at i on, we are rel at i vel y con dent that

the a method wor s correct l y i n the cl ass i n whi ch i t i s de ned. e are concerned wi th how

the a method i nvol ved i n i nteract i on a ects the sendi ng obj ect and any poss i bl e parameters .

Thi s concern i s compounded because the method that gets i nvo ed f or a part i cul ar cal l wi l l depend

on the dynami c type of the obj ect , so we must be concerned wi th test i ng the sendi ng obj ect and

parameter obj ect wi th each de ni t i on of the vi rtual method.

I f the a method had been stat i cal l y de ned i n cl ass Dand i nher i ted i n cl asses E and

F, then regardl ess of the dynami c type of the obj ect recei vi ng the a message, the same a

method woul d have been i nvo ed. I n thi s case, we need to be concerned wi th the e ect the method

has on the cl ass i nteract i on wi th the sender and any parameters . Consequent l y, the type of a

method, ei ther stat i c or dynami c, must al so be cons i dered i n i nter- cl ass test i ng.

Dynami c bi ndi ng f orces us to cons i der the f unct i onal , i nter f ace, and data i ssues accordi ngl y

f or the type of the method that i s i nvol ved. owever , because the part i cul ar method that gets

i nvo ed can change, cons i der i ng al l these poss i bi l i t i es f or the message i nvol ves test i ng one obj ect

wi th al l poss i bl e methods that exi st , whi ch i s another compoundi ng f actor .

.

Obj ects are run- t i me ent i t i es that ta e up space i n a system' s memory. The arrangement of bi ts i n

an obj ect ' s al l ocated memory at any part i cul ar t i me determi ne the state of that obj ect . Theref ore,

i n addi t i on to the behavi or encapsul ated by an obj ect through i ts f unct i ons and data, obj ects

al so encapsul ate state. I n thi s research, the term\state" ref ers to des i gn- l evel state descr i bed by

Mc regor and Dyer [ 6] . Des i gn- l evel state i s not concerned wi th the i ndi vi dual data attr i butes of

an obj ect because of the poss i bl y i n ni te number of states that coul d ar i se. I nstead, des i gn- l evel

17

state i s based on sets of attr i bute val ues that are cl ear l y di erent i abl e and determi ne an observabl e

behavi or of the obj ect [ 6] . Thi s i dea can be i l l ustrated by a stac of n i tems. Fol l owi ng the

tradi t i onal vi ewof state, whi ch cons i ders a separate state f or each data attr i bute, the stac woul d

have n states . Thus, i n order to test the n poss i bl e states f or the stac , we woul d have to des i gn

a mi ni mumof n test cases , assumi ng that at l east one test case i s requi red f or each state. The

des i gn- l evel state representat i on i s onl y concerned wi th the stac bei ng f ul l , empty, or at some

poi nt i nbetween. Thi s reduces the mi ni mumnumber of states f romn to 3.

I n exami ni ng state- based test i ng of cl asses , Turner [ 12] cl ai ms that a method can a ect the

state of an obj ect i n one of f our poss i bl e ways:

1. I t can change the obj ect ' s state to an appropr i ate newstate.

2. I t can l eave the obj ect ' s state as i t i s .

3. I t can change the obj ect ' s state to an unde ned state, whi ch i s an error .

4. I t can change the obj ect ' s state to an i nappropr i ate state, whi ch al so i s an error .

Turner says that of these poss i bi l i t i es , 3 and 4 are de ni tel y errors , and 2 can be an error i f the

e ect shoul d have been 1. These concerns establ i shed by Turner are de ned i n terms of uni t test i ng,

whi ch concerns one cl ass and i ts methods. owever , these state error concerns are cl ear l y present

when i nter- cl ass test i ng i s per f ormed. The message sent by an obj ect may have arguments of a type

or val ues whi ch, when recei ved by another obj ect i n a part i cul ar state, cause the recei vi ng obj ect

to react as expected or to react unexpectedl y. Duri ng i nter- cl ass test i ng, we want to i nvest i gate

whether an obj ect reacts proper l y to a message f romanother obj ect i n al l poss i bl e states of the

recei vi ng obj ect or i ts subst i tut i ons . Addi t i onal l y, we want to observe howthe sendi ng obj ect reacts

when i t recei ves any return val ues .

To i l l ustrate why the states of an obj ect are i mportant i n test i ng, cons i der the s i mpl e exampl e

of a bounded queue as the recei vi ng obj ect . I n the state , when the en e e e t message

i s recei ved, we expect an error code returned i ndi cat i ng the bounded queue i s f ul l and no newi tem

can be added. I n the state a tia , however , we expect the i temto be added to the bounded

queue and a return code i ndi cat i ng the i temwas success f ul l y added. e certai nl y expect a bounded

queue to have a di erent behavi or to the en e e e t message i n state than i n state a tia

18

. As a resul t , we need to test each of the states of the bounded queue. Cons i der i ng al l poss i bl e

states by i tsel f coul d be a monumental tas . owever , there are current vi ews on i nher i tance and

obj ect state that may o er some i ns i ght i nto reduci ng the burden of test i ng obj ect state i n i nher i ted

cl asses and poss i bl y reduce the test i ng e ort f or i nter- cl ass test i ng.

Mc regor and Dyer' s [ 6] de ni t i on of \str i ct" i nher i tance assumes that a subcl ass contai ns

the publ i c i nter f ace of i ts parent s as a subset of i ts publ i c i nter f ace. Further , the i ntent of the

methods i nher i ted remai ns constant f romthe parent cl ass i nto the sub- cl ass es as j udged by the

pre and post condi t i ons f or the method. Assumi ng str i ct i nher i tance, the states of a cl ass wi l l exi st ,

i n some f orm, i n a subcl ass that i nher i ts f romthe cl ass . The subcl ass may contai n the state exact l y

as i n the parent cl ass , the state of the parent cl ass may be di vi ded i nto substates i n the subcl ass ,

the state of the parent cl ass may not be exerci sed i n the subcl ass , or the methods that compri se

the state may be modi ed, but the state st i l l exi sts whol l y i n the subcl ass .

Queue

Bounded Queue

Empty

Empty Empty

Priority Queue

Not Empty Not Empty

Full

NotEmpty

New TopElement

NewShape of

Queue

PartialFull

Fi gure 3: ueue cl ass hi erarchy wi th states .

Fi gure 3, modi ed f romMc regor and Dyer [ 6] , shows poss i bl e states of a queue, al ong wi th

a bounded queue and pri or i ty queue that i nher i t f romthe queue. Fol l owi ng the model of str i ct

i nher i tance, the two states of the queue, e t and n t e t , are present i n the two i nher i ted

cl asses . The state e t remai ns cons i stent i n the three cl asses , but n t e t , al though cons i stent

19

i n i ts i ntent , i s modi ed to model the behavi or of the two i nher i ted cl asses . hen a state i s

cons i stent throughout a cl ass hi erarchy, retest i ng every obj ect i n that state may not be necessary.

An i ncons i stent state, however , such as n t e t , or a newstate of an obj ect shoul d be re tested

i n i nher i ted cl asses . Further research i n thi s area i s a necessary step i n devel opi ng the overal l

test i ng process f or obj ect or i ented sof tware. hen f ul l y expl ored, i f the not i on cl ai med ear l i er of

not havi ng to re- test i nher i ted states that are unchanged proves correct , i t may a ect i nter- cl ass

test i ng. For our purposes i n thi s research and unti l f urther research proves concl us i ve, we assume

the i nher i ted states of an obj ect do need to be tested i n i nher i ted cl asses , and theref ore we must be

concerned about each state of al l obj ects and pol ymorphi c subst i tutes i nvol ved i n a message send.

es t i n s e on e t t t e n n i s t i t t i ons o

e t s

hen there are poss i bl e pol ymorphi c subst i tutes f or ei ther the obj ect sendi ng the message, the

obj ect recei vi ng the message, or the message parameters , and or mul t i pl e states of concern i n

any of these obj ects , these mul t i pl e obj ects and states must be cons i dered. owever , exhaust i vel y

cons i der i ng the e ects of obj ect state and pol ymorphi smon i nter- cl ass test i ng can cause probl ems

f or the tester i n terms of manpower, al l otted test i ng t i me, and choos i ng a set of test cases that

o er some l evel of con dence f romthe l arge number of poss i bi l i t i es . As i l l ustrated i n Appendi x

A, account i ng f or these poss i bi l i t i es can be a maj or tas . The exampl e gi ven i n Appendi x A i s a

s i mpl e scenar i o resul t i ng i n ve hundred and twel ve poss i bl e combi nat i ons of the sender , recei ver ,

and parameters i n thei r var i ous states , s i mpl y too many poss i bi l i t i es to test f or 1 message. As a

resul t , we need a method to ass i st i n ma i ng test case sel ect i ons f romthe l arge number that coul d

exi st .

.

I n Mandl ' s [ 5] wor wi th orthogonal l at i n squares appl i ed to compi l er test i ng, he i dent i es two

l evel s of adequacy f or test i ng an i dent i ed domai n spanned by a ni te number of var i abl es each

wi th a range of poss i bl e val ues . Mandl ci tes the approaches of 1 exhaust i ve test i ng and 2 ma i ng a

randomsel ect i on of test cases f romthe i dent i ed test i ng domai n and poi nts out that exhaust i vel y

test i ng the i dent i ed domai n i s the i deal choi ce, but wi th a l arge test i ng domai n i t i s al so the

20

i mpract i cal choi ce. The randomsel ect i on may uncover some de ci enci es , but wi th thi s approach i t

i s di cul t to assess the l evel of con dence der i ved. Mandl proposes a l evel of con dence based on

the premi se that al l combi nat i ons of test cases can be di vi ded i nto a reasonabl e number of di st i nct

groups. Combi nat i ons end up i n the same group i f an onl y i f the test i ng of one combi nat i on

provi des nothi ng newover what i s avai l abl e f romtest i ng another combi nat i on f romthe same group

each i n combi nat i on wi th a test case f romanother group. Mandl ' s wor resul ts i n a set of test

cases der i ved f or near \exhaust i ve" test i ng us i ng l at i n squares and i ncompl ete bl oc des i gn [ 5] that

cover the test i ng domai n by i ncl udi ng each val ue of each var i abl e i n some test case combi nat i on.

Lat i n squares contai n symbol s . .n such that these symbol s appear at l east once i n every rowand

every col umn. Mandl ' s l evel of adequacy achi eves a greater l evel of con dence than randomtest i ng

because i t i s a systemati c approach and does not resul t i n a bi as toward any one port i on of the

test i ng domai n, whi ch coul d resul t f romrandomsel ect i on. The method of sel ect i ng combi nat i ons

resul ts i n a set of combi nat i ons evenl y di str i buted over the test i ng domai n.

I n l i ght of the wor done by Mandl , we expl ai n three l evel s of adequacy to appl y to the prob-

l emof test i ng obj ect state and the presence of pol ymorphi sm. An addi t i onal l evel correspondi ng

to Mandl ' s randomsel ect i on i s omi tted because of i ts de ni t i on, whi ch i s randoml y sel ect i ng test

cases f romthe test i ng domai n. The three l evel s expl ai ned bel ow i n terms of the obj ect or i ented

paradi gmare exhaust i ve, mi ni mal , and representat i ve.

xhaustive - To achi eve exhaust i ve test i ng, the tester has to cons i der al l poss i bl e i nterac-

t i ons . I n the obj ect or i ented approach thi s means al l combi nat i ons of the sendi ng obj ect ,

recei vi ng obj ect , parameter obj ect s , and al l poss i bl e pol ymorphi c subst i tut i ons and states

f or each of these obj ects . enerat i ng these test case combi nat i ons al one may be more e ort

than a tester has the abi l i ty to admi ni ster . owever , when exhaust i ve test i ng i s appl i ed, the

tester achi eves a hi gh degree of con dence i n the tested domai n of the product . Fi gure 4

i l l ustrates the coverage gi ven by an exhaust i ve test set . Al l combi nat i ons of the i ntended

obj ects A, , and D, and al l pol ymorphi c poss i bi l i t i es f or each of these, al ong wi th al l states

of each obj ect and the pol ymorphi c poss i bi l i t i es , get tested i n combi nat i on. Thi s resul ts i n

ve hundred and twel ve combi nat i ons of obj ects and states , i l l ustrated i n Appendi x A, f or

thi s part i cul ar exampl e f or a s i ngl e message. Theref ore, exhaust i ve test i ng i s not a hi ghl y

21

3 states 3 states

Parameter X

2 states

3 states 3 states

D

E FB C

A

3 states 3 states

2 states 2 states

Hierarchy of Parameter X

Y Z

message( class X )

Fi gure 4: Cons i derat i ons f or Exhaust i ve Coverage.

f avored or exerci sed opt i on.

inimal - i th the mi ni mal l evel of test i ng, the tester i s onl y concerned wi th test i ng the

combi nat i on of the i ntended sendi ng, recei vi ng, and parameter obj ects , each i n a part i cul ar

state the tester expects the obj ect to be i n at the t i me of i nteract i on. The remai ni ng states of

the i ntended obj ects and the pol ymorphi c subst i tut i ons and thei r states are not tested. i th

thi s l evel of test i ng, we can achi eve a hi gh degree of con dence f or the i ntended sendi ng,

recei vi ng, and parameter obj ects and thei r i ntended states , but vi rtual l y no con dence f or

combi nat i ons of any of the pol ymorphi c poss i bi l i t i es i n any of thei r states or the remai ni ng

states of the i ntended obj ects . Fi gure 5 i l l ustrates the coverage gi ven by the mi ni mal set .

Onl y the i ntended sender , recei ver , and parameter obj ect , each i n an appropr i ate state, get

tested. For thi s exampl e, the resul t i s 1 test case combi nat i on i nvol vi ng obj ects A, D, and

and one state of each. ei ther addi t i onal states f or the i nvol ved obj ects nor the poss i bl e

pol ymorphi c subst i tutes nor thei r states get cons i dered f or test i ng.

The l evel of adequacy that we des i re i s one where the i ntended domai n i s tested, al ong wi th

some sel ected port i on of the remai ni ng poss i bl e test i ng domai n. A l evel of adequacy that al l ows

22

A Dmessage ( class X )

Fi gure 5: Cons i derat i ons f or Mi ni mal Coverage.

the tester to choose a sel ect i ve set of combi nat i ons of obj ect states and poss i bl e pol ymorphi c sub-

st i tut i ons f or the sender , recei ver , and parameters of the message passed to test i s needed to reduce

the i nter- cl ass test i ng e ort f romexhaust i ve whi l e provi di ng a hi gher l evel of product con dence

than that of mi ni mal test i ng. i ven our de ni t i on of a des i gn l evel state, the representat i ve l evel

of adequacy de ned bel owparal l el s Mandl ' s \exhaust i ve" l evel .

epresentative - i th representat i ve we want to cover the i ntended sender , recei ver , pa-

rameters , and states of each of these obj ects , as wel l as cover i ng a sel ected set of the poss i bl e

pol ymorphi c subst i tutes and states of the obj ects i nvol ved. The set of sel ected obj ects and

states f rombeyond the i ntended domai n shoul d be evenl y di str i buted over the remai ni ng

pol ymorphi c poss i bi l i t i es , and shoul d contai n at l east one occurrence of every poss i bl e val ue

obj ects and states . Choos i ng test case combi nat i ons i n thi s manner does not i gnore one

regi on ent i rel y, and i t tests every val ue, even though every poss i bl e combi nat i on does not get

tested. Theref ore, thi s set of test cases i s representat i ve of the test i ng domai n. Fi gure 6 i l l us-

trates the coverage provi ded by the epresentat i ve set . Each obj ect i n each state gets tested,

but not al l combi nat i ons of every obj ect i n each state wi th every other obj ect i n each of i ts

states . The coverage has expanded f romthat gi ven by mi ni mal coverage to cover poss i bl e

pol ymorphi c subst i tutes and more states of the obj ects .

eighted epresentative - Thi s l evel of coverage provi des the opportuni ty to more r i gor-

ous l y test certai n combi nat i ons than others . The combi nat i ons that are to be exami ned more

cl osel y are typi cal l y chosen vi a a r i s anal ys i s procedure. The procedure uses cr i ter i a that ,

f or thi s s i tuat i on, i ndi cate i nteract i ons that have a hi gher potent i al f or errors . These combi -

nat i ons wi l l be tested more of ten than others and wi l l , as a resul t , be i ncl uded i n combi nat i on

wi th other f actors as wel l .

23

3 states 3 states

Parameter X

2 states

3 states 3 states

D

E FB C

A

3 states 3 states

2 states 2 states

Hierarchy of Parameter X

Y Z

message( class X )

Fi gure 6: Cons i derat i ons f or epresentat i ve Coverage.

.

owthat we have establ i shed the type of coverage we want to generate, we wi l l exami ne a techni que

that wi l l ass i st i n thi s manner . An engi neer i ng methodol ogy des i gned by Taguchi [ 11] cal l ed st

esi n was devel oped to hel p engi neers des i gn hi gh- qual i ty products and processes at l ow cost .

Des i gn engi neers have to cons i der many var i abl es that a ect the qual i ty and cost of thei r end

resul t , whether i t be a newproduct or process . The obust Des i gn method uses a mathemati cal

tool cal l ed t na a a s [ 5, 8, 9, 11] to study a l arge number of deci s i ons var i abl es wi th a smal l

number of exper i ments , thereby eepi ng cost to a mi ni mumand qual i ty hi gh. Thi s resul t paral l el s

the des i re i n any test i ng envi ronment, a hi gh l evel of product con dence wi thout exhaust i ve test i ng.

An adapt i ve techni que, cal l ed st estin , i s a systemati c test method used to test sof tware

that uses the OATS Orthogonal Array Test i ng System tool to produce a conci se set of tests f rom

product requi rements . These test cases o er coverage of al l val ues of each var i abl e, but not al l

combi nat i ons of val ues . Case studi es on obust Test i ng i n Brownl i e et al . [ 1] have shown i t to

be twi ce as product i ve, by cutt i ng test i ng t i me and ndi ng more f aul ts , as convent i onal test i ng

pract i ces when obust Test i ng can be appl i ed.

24

5.2.1 pplying obust Testing Techni ues

I n order to appl y obust Test i ng techni ques to a sof tware product , as wi th most test i ng techni ques ,

a thorough understandi ng of the requi rements of the product must be acqui red. These requi rements

general l y i ncl ude such thi ngs as overal l f unct i onal i ty of the program, poss i bl e hardware and sof tware

con gurat i ons f or the product , and the poss i bl e types of data the end user wi l l expose to the product .

I n other words, the tester must be aware of al l the var i abl es that a ect the f unct i onal i ty of the

product . A conventi onal test i ng method, as descr i bed i n Brownl i e et al . [ 1] , requi red the tester

to test the product f unct i ons i n a representat i ve set of hardware sof tware con gurat i ons . The

bas i s of the i dea i s that a f unct i on wor i ng or not wor i ng proper l y i n one con gurat i on woul d

perf ormcomparabl y i n any other con gurat i on. Detect i ng and correct i ng errors tr i ggered by a

change i n one var i abl e that can cause ser i ous probl ems f or the end user i s i mperat i ve to assure

a qual i ty product . The type of conventi onal test i ng techni que descr i bed above may mi ss errors

caused by some of these changes , and exhaust i vel y test i ng f or al l the poss i bl e var i abl e changes may

be i mposs i bl e due to l ac of test i ng t i me and or su ci ent manpower. s i ng the same nowl edge

of product requi rements as requi red by the convent i onal test i ng method, obust Test i ng can be

used to generate a representat i ve set of test cases that provi des adequate coverage f or the var i abl es

i nvol ved.

To appl y obust Test i ng to a test i ng probl emi n general , the var i abl es of concern to the

tester , as wel l as the poss i bl e range of val ues f or the var i abl es , must be nown. Once the var i abl es are

establ i shed, an orthogonal array, whi ch transf orms to a matr i x whose rows represent poss i bl e test

cases and whose col umns represent the test parameters , i s generated. The resul t i ng orthogonal array

gi ves the tester a set of test cases whi ch, when used compl etel y, provi des \adequate" coverage of the

de ned var i abl es . To cl ar i f y the term\adequate", ref erenci ng Tabl e 1, we have test parameters A,

B, C, and D. I f we extract any two col umns, say Band Cf or exampl e, we can see that every val ue

of Bgets tested wi th every val ue of C. owever , when Band Care exami ned i n combi nat i on wi th

Aand D, we can see that not al l combi nat i ons of A, B, C, and Dget tested together . Each poss i bl e

val ue of any test parameter gets tested at l east once, but not al l combi nat i ons , and we termthi s

coverage \adequate". i th the orthogonal array i n Tabl e 1, the mai n e ects of al l parameters are

exami ned, and not the speci c e ects of the combi nat i on of any two or three f actors , whi ch can

25

Parameters

Test Case A B C D

1 1 1 1 1

2 1 2 2 2

3 1 3 3 3

4 2 1 2 3

5 2 2 3 1

6 2 3 1 2

7 3 1 3 2

8 3 2 1 3

9 3 3 2 1

Tabl e 1: Standard L9 orthogonal array.

be exami ned by hol di ng the var i abl es i n quest i on constant over thei r range of val ues and varyi ng

the remai ni ng var i abl e val ues . An orthogonal array l i e the one i n Tabl e 1 gi ves combi nat i ons of

var i abl e val ues that al l owal l val ues of the var i abl es i nvol ved to be tested at l east once. Thi s same

type i nf ormati on can be val uabl e i n i nter- cl ass test i ng. The tester needs to nowwhi ch var i abl es ,

i n thi s case states and pol ymorphi c subst i tutes , to test i n whi ch combi nat i ons to cover a port i on

of the test i ng domai n whi l e provi di ng the tester a l evel of con dence greater than that whi ch was

previ ous l y obtai ned.

5.2.2 Steps in erforming obust Testing

The maj or tas i n obust Test i ng i s de ni ng the probl emi n terms of an orthogonal array. Phad e

[ 8] l i sts the steps requi red to devel op an orthogonal array representat i on. Two bas i c steps , perhaps

the most i mportant, are i dent i f yi ng the a t s and e e s of the exper i ment. a t s can be vi ewed

as the parameters f or the test cases and e e s as the poss i bl e val ues f or these parameters . I n

Taguchi ' s or i gi nal appl i cat i on of orthogonal arrays i n obust Des i gn, the f actors and l evel s were

l argel y i ndependent because col umns of the generated orthogonal array coul d be removed to study

the e ects of i ndi vi dual f actors wi thout destroyi ng the orthogonal i ty of the ent i re matr i x.

Once f actors and l evel s are establ i shed, the mi ni mumnumber of test cases requi red to test

the f actors and l evel s are cal cul ated. Thi s number, nowas the e ees ee f or the test i ng

exper i ment, can be i n uenced by the i ntent of the test i ng process . Degrees of f reedomcan be

26

thought of i n terms of i ndependent compari sons of a f actor , and each f actor i nvol ved i n a probl em

de ni t i on has a degree of f reedomassoci ated wi th i t . A 6 l evel f actor has 5 degrees of f reedom

because once a val ue f or the f actor i s chosen, there are onl y 5 val ues remai ni ng f or the f actor , or

5 ways l ef t to change the f actor . The degrees of f reedomf or al l f actors i nvol ved i n the probl em

de ni t i on, pl us 1 degree of f reedomf or the mean of the exper i ment, are summed together to arr i ve

at the degrees of f reedomf or the exper i ment. The 1 degree of f reedomf or the overal l mean of the

exper i ment i s standard f or anal ys i s of var i ance cal cul at i ons f romwhi ch thi s cal cul at i on stems [ 10] .

I f the i ntent of the exper i ment i s to cons i der the mai n e ects of al l i ndependent f actors and l evel s on

the sof tware system, the degrees of f reedomcan be cal cul ated as descr i bed. I f we wi sh to cons i der

the e ects of the combi nat i on of two or more f actors poss i bl e by combi ni ng the two or more f actors

and devel opi ng a new f actor [ 8, 10] i n addi t i on to the mai n e ects of al l f actors , the degrees of

f reedomwi l l i ncrease because the number of f actors i ncrease. The cal cul ated degrees of f reedom

i s used i n conj unct i on wi th the number of f actors and l evel s to generate an orthogonal array. The

orthogonal array can resul t i n one of two ways. Based on the f actors , l evel s , and degrees of f reedom

a neworthogonal array can be generated, or the probl emcan be t to a standard orthogonal array

whi ch resul ts i n no newarray cal cul at i on. Standard orthogonal arrays are precal cul ated matr i ces

that correspond to patterns of f actors and l evel s . They are termed standard because the number of

f actors and number of l evel s appear most of ten i n exper i mentat i on, and thei r array representat i on

i s stored to avoi d recal cul at i on.

Orthogonal arrays possess a bal anci ng property whi ch al l ows themto gi ve excel l ent coverage

of the ent i re test domai n de ned by the parameters of the test cases . For each pai r of col umns i n an

orthogonal array, al l - parameter l evel combi nat i ons occur cons i stent l y. The standard L9 orthogonal

array used i n Brownl i e et al . [ 1] i s shown i n Tabl e 1. ar i abl es A, B, C, and Drepresent test

parameters , and the numbers 1, 2, and 3 represent the poss i bl e val ues of these parameters . The

parameter- l evel combi nat i ons 1, 1 , 1, 2 , 1, 3 , 2, 1 , 2, 2 , 2, 3 , 3, 1 , 3, 2 , and 3, 3 occur i n

al l pai rwi se col umns. Exami ni ng Tabl e 1, we can see that al l poss i bl e val ues of Aare tested wi th

al l poss i bl e val ues of B, and al l poss i bl e val ues of B are tested wi th al l poss i bl e val ues of C, but

not al l poss i bl e val ues of A, B, and Care tested together i n combi nat i on. Conduct i ng the 9 test

cases suggested by L9 woul d detect any cons i stent probl emwi th any l evel of any s i ngl e parameter

and any cons i stent probl emwi th pai rwi se compati bi l i ty of parameters . The L9 array de nes a set

27

of tests that systemati cal l y eval uate the mai n e ects of al l test parameters rel evant to the test i ng

requi rements .

.

I n the real mof i nter- cl ass test i ng, we have i dent i ed obj ect state and pol ymorphi smas f eatures that

can compound the var i abl es and test cases i nvol ved i n test i ng obj ect i nteract i on. Fromthese poss i -

bi l i t i es , we need to sel ect a representat i ve set of test combi nat i ons to test f unct i onal i ty, i nter f aces

and data i ntegr i ty. obust Test i ng, i dent i ed as a sel ect i ve process f or test cases , may o er some

i ns i ght i nto a sol ut i on f or the probl ems i ntroduced by obj ect state and pol ymorphi sm. I n order to

appl y obust Test i ng and standard orthogonal arrays to the probl emof pol ymorphi c subst i tut i ons

and mul t i pl e obj ect states , i t must be cl ear what the expected outcome i s . e want to use standard

orthogonal arrays to deci de whi ch combi nat i ons of obj ects and states to test i n i nteract i on. e

l i mi t the use of orthogonal arrays to onl y standard orthogonal arrays i n order to determi ne the

val i di ty of thi s techni que. Techni ques f or construct i ng orthogonal arrays are wel l establ i shed and

have been exami ned i n [ 9, 11] and exampl es of thei r use can be seen i n [ 1, 5] . Once we nowwhi ch

combi nat i ons of obj ects and states to test , we can devel op a part i cul ar test case to execute f or the

message. The test case shoul d exerci se the message i n the obj ect envi ronments generated by the

orthogonal array and al l owver i cat i on of f unct i onal i ty, i nter f ace, and data i ntegr i ty. Addi t i onal l y,

we are concerned wi th the success or f ai l ure of the test case as a whol e and i ts e ect on the state

of the sof tware system.

i th thi s goal i n mi nd, we can l i st some areas that have an a ect on test i ng the message

passed between two obj ects .

Sendi ng Obj ect

State of the Sendi ng Obj ect

ecei vi ng Obj ect

State of the ecei vi ng Obj ect

Parameter Obj ect s

28

State s of each Parameter Obj ect

Of these s i x, there i s one that coul d be el i mi nated. I f we cons i der the state of the sender to

be correct , otherwi se the obj ect sendi ng the message woul d not be sendi ng the part i cul ar message

we are i nterested i n test i ng but woul d i nstead be sendi ng a di erent message, there woul d be one

l ess f actor . owever , cons i der i ng the state of the sender i s necessary i n determi ni ng the success

or f ai l ure of the test case as a whol e. i thout nowi ng and test i ng the sendi ng obj ect ' s state, we

coul d not determi ne the state of the systemaf ter the message send i s compl ete. The message coul d

be cons i dered as two messages , one the message send and the other the return message f romthe

or i gi nal recei ver , but thi s woul d compound the test i ng e ort even f urther . As a resul t , we wi l l f ul l y

cons i der al l s i x f actors l i sted when perf ormi ng i nter- cl ass test i ng.

A

B C

messageD

E F

Fi gure 7: Sampl e cl ass hi erarchi es sendi ng and recei vi ng a message.

For a s i mpl e exampl e us i ng the de ned f actors and l evel s , cons i der the message i se t , wi th

no parameters , passed f roman i nstance of cl ass Ato an i nstance of cl ass Dshown i n Fi gure 7. For

s i mpl i ci ty, we wi l l assume cl asses Aand Dare both e e cl asses and thei r i nher i tance hi erarchy

i s that of the e e shown i n Fi gure 3. e have f our f actors to de ne, the sender , the state of

the sender , the recei ver , and the state of the recei ver . I f we wi sh to mai ntai n total i ndependence

of f actors , we cannot cons i der the obj ects and thei r states as two separate f actors because states

wi thout associ ated obj ects woul d be meani ngl ess . e theref ore reduce the number of f actors to 2

and cons i der sender state and recei ver state. The l evel s f or each of these f actors woul d be:

Sender State - The l evel s f or the sender state f actor i ncl ude the i ntended sender and al l i ts

poss i bl e pol ymorphi c subst i tutes i n each of thei r poss i bl e states .

ecei ver State - The l evel s f or the recei ver state f actor i ncl ude the i ntended recei ver and al l

i ts poss i bl e pol ymorphi c subst i tutes i n each of thei r poss i bl e states .

29

Appendi x B expl i ci t l y shows al l l evel s of these two f actors and the constructed orthogonal

array. As shown i n Appendi x B, wi th sender state and recei ver state as the i ndependent f actors ,

us i ng an orthogonal array to generate test cases f or a message that has no pol ymorphi c parameters

resul ts i n the representat i ve and exhaust i ve cr i ter i a bei ng equi val ent . Al though there i s no reduct i on

i n the number of test cases , al l the test case poss i bi l i t i es are generated, whi ch i n i tsel f can be a

bene t . I f the tester admi ni sters al l test cases suggested by the orthogonal array i n Appendi x B,

the resul t woul d be exhaust i ve coverage of the test i ng domai n. Theref ore, us i ng an orthogonal

array wi th two f actors exami ni ng onl y mai n e ects o ers exhaust i ve coverage of the f actors and

l evel s i nvol ved wi th no reduct i on i n the number of test cases .

ow cons i der a message that contai ns a parameter that i s an obj ect . Obj ect A e e

i s pass i ng the message en e e e t to obj ect D e e . For thi s exampl e, we wi l l assume

the obj ect bei ng passed as an argument f or en e e i s al so a e e and has a cl ass hi erarchy

s i mi l ar to the hi erarchy shown i n Fi gure 3. For each parameter we i ncl ude i n the orthogonal array

construct i on, we must add an addi t i onal f actor , whi ch i s descr i bed bel ow.

Parameter State - The l evel s f or the parameter state f actor i ncl ude the i ntended obj ect pa-

rameter and al l i ts poss i bl e pol ymorphi c subst i tutes i n each of thei r poss i bl e states .

Thi s new f actor i s f ormed i n the same manner as the f actors f or the sendi ng and recei vi ng

obj ect , except that i t i nvol ves the parameter obj ect . There are several opt i ons f or test i ng thi s

message. For s i mpl i ci ty, we wi l l assume that choos i ng two obj ects and a parameter to test i n

i nteract i on can be tested wi th one test case. The choi ces f or test i ng are 1 exhaust i ve test i ng,

whi ch as shown i n Appendi x Acons i sts of ve hundred and twel ve test cases , 2 mi ni mal test i ng -

whi ch i f our i ntended parameter and recei ver are e e cl asses , and assumi ng there i s one i ntended

state f or test i ng i n each obj ect , cons i sts of one test case, and 3 representat i ve - whi ch by cal cul at i ons

shown i n Appendi x C, Tabl e 8, cons i sts of a mi ni mumtwenty two test cases .

Aprobl emwi th the above exper i ment de ni t i on, shown i n Appendi x C, i s that the number

of l evel s per f actor exceeds any de ned f or the standard orthogonal arrays , whi ch eeps the de ned

probl emf rom tt i ng a standard orthogonal array. I n the tabl es l i sted i n Phad e [ 8] there are no

standard orthogonal arrays wi th any 8- l evel f actors , whi ch our current probl emde ni t i on requi res .

Thi s means that i f the de ned exper i ment has 5000 i nter- cl ass messages to test , and f or each

30

message an orthogonal array has to be generated, the test i ng t i me f or the proj ect coul d i ncrease

because of the generat i on t i me associ ated wi th each matr i x. As a resul t , we propose an al ternat i ve

method f or the probl emde ni t i on that wi l l reduce the number of l evel s per f actors and al l owthe

probl emto be t to a standard orthogonal array, whi ch resul ts i n no newarray cal cul at i on.

.

I f the tester deci des that the state of a part i cul ar obj ect or a part i cul ar obj ect i tsel f does not need

to be tested, or perhaps the tester deci des to exhaust i vel y test al l states or al l pol ymorphi c subst i -

tut i ons of a part i cul ar obj ect , us i ng the de ni t i on of f actors and l evel s gi ven ear l i er sender state,

etc. woul d cause probl ems. s i ng those de ni t i ons , a state coul d not s i mpl y be removed wi th-

out al so removi ng the obj ect associ ated wi th i t , nor coul d al l states be tested wi thout test i ng al l

pol ymorphi c subst i tut i ons of the obj ect . Si nce the test cases generated are i ntended to be used

whol l y, we rel ax one constrai nt pl aced on the f actors f or the orthogonal arrays i n obust Test i ng.

I nstead of cons i der i ng the obj ect and i ts di erent states as one f actor , we wi l l brea themup i nto

two f actors . The f actors and l evel s woul d nowbe as f ol l ows.

Sender - The l evel s f or the sender f actor are the sender i tsel f and the cl asses that serve as

pol ymorphi c subst i tut i ons f or the cl ass .

State of the Sender - I n Fi gure 8 the states e t and n t e t present i n cl ass e e

are al so present i n n e e e and i i t e e. The state e t remai ns cons i stent

throughout al l the cl asses , whi l e state n t e t gets di vi ded i nto two substates i n both

i nher i ted cl asses . At an abstract l evel , there are st i l l onl y two states i n each of these cl asses ,

but i n real i ty n e e e and i i t e e each have three poss i bl e states to test .

The number of l evel s f or thi s f actor i s the maxi mumnumber of phys i cal states of any one

poss i bl e obj ect , ei ther the i ntended obj ect or a poss i bl e pol ymorphi c subst i tute. I n thi s

exampl e, where the cl ass hi erarchy used f or the obj ect i s model ed af ter the e e hi erarchy,

the number of l evel s f or the state f actor woul d be a , , . Choos i ng the maxi mum

number of states guarantees that we do not el i mi nate any newstates i n the i nher i ted cl asses .

As f or the parent cl ass , whi ch i s the i ntended recei ver , i t may onl y mean overtest i ng the states

31

Queue

Bounded Queue

Empty

Empty Empty

Priority Queue

Not Empty Not Empty

Full

NotEmpty

New TopElement

NewShape of

Queue

PartialFull

Fi gure 8: ueue cl ass hi erarchy wi th states .

of that part i cul ar obj ect . Thi s i s because the i ntended cl ass may have f ewer states than the

i nher i ted cl asses and theref ore a e e wi l l be i ntroduced.

Adummy l evel i s a non exi stent l evel used s i mpl y to mai ntai n orthogonal i ty of the matr i x.

The dummy l evel i s cons i stent l y repl aced by a val ue the tester f eel s may need more test i ng,

thus more test i ng i n thi s case wi l l resul t f or states of the obj ect wi th f ewer uni que states .

hen us i ng orthogonal arrays the number of l evel s f or al l f actors shoul d be the same, so

dummy l evel s are al so i mposed to provi de bal ance. Because of thi s repet i t i on of one or more

l evel s f or test i ng, thi s suggests that the l evel s shoul d be ran ed i n order of i mportance so the

tester can subst i tute more i mportant l evel s of the f actors f or the dummy l evel s .

ecei ver - The l evel s f or the recei ver f actor are the recei ver i tsel f pl us the cl asses that serve

as pol ymorphi c subst i tut i ons f or the cl ass .

State of each ecei ver - The l evel s f or the recei ver state f actor are devi sed i n the same manner

as the states f or the sendi ng obj ect .

Parameter s - The l evel s f or the parameter f actor are the parameter i tsel f pl us the cl asses

that serve as pol ymorphi c subst i tut i ons f or the cl ass .

32

State of each Parameter - The l evel s f or the parameter state f actor are devi sed i n the same

manner as the states f or the sendi ng obj ect .

Each parameter establ i shes two addi t i onal f actors i n the cal cul at i on of the orthogonal array.

One f actor accounts f or the parameter and i ts pol ymorphi c subst i tut i ons and the second accounts

f or the states of the parameter .

Bef ore, the number of f actors and l evel s di d not t a standard orthogonal array, whi chmeant

each t i me an array was needed i t woul d have to be cal cul ated. s i ng the newvi ewpoi nt expl ai ned

above and i l l ustrated i n Appendi x D, f or a message wi th one parameter , we have the f actors and

l evel s shown i n Appendi x D, whi ch do t a standard orthogonal array and generate the test cases

shown i n Appendi x D, Tabl e 12.

Our opt i ons f or test i ng noware 1 exhaust i ve test i ng, whi ch agai n cons i sts of ve hundred

and twel ve test cases , 2 mi ni mal test i ng - whi ch i f our parameter and recei ver are e e cl asses

agai n i nvol ves one test case, and 3 representat i ve test i ng whi ch by cal cul at i ons i n Appendi x D,

Tabl e 9, f or the al ternat i ve method i nvol ves a mi ni mumof thi rteen test cases .

Lets exami ne the coverage gi ven by the representat i ve test case set generated i n Appendi x D

f or a s i ngl e parameter message pass . Agai n, our message i s en e e e t f romobj ect A e e

to obj ect D e e wi th a message parameter of cl ass type e e. Further , we wi l l assume the

tester i s concerned wi th each obj ect bei ng i n state t t . For thi s exampl e, the mi ni mal test

set woul d cons i st of the test case shown i n Tabl e 2.

Sender Sender State ecei ver ecei ver State Parameter Parameter State

ueue ot Empty ueue ot Empty ueue ot Empty

Tabl e 2: Mi ni mal Test Case f or ueue Exampl e

i thi n the test set shown i n Appendi x D, we can see that the mi ni mal test set i s generated

as the rst case i n the representat i ve test set . Fromthi s , we can gather that the representat i ve

test set wi l l at l east detect the same errors as the mi ni mal test set . owever , i f we l oo i n the

representat i ve set f or the test case f romthe exhaust i ve gi ven i n Tabl e 3, we see that i t i s not present .

I t i s evi dent that the representat i ve test set does not contai n the exhaust i ve test set because the

representat i ve set i s cons i derabl y smal l er .

33

Sender Sender State ecei ver ecei ver State Parameter Parameter State

Bounded ueue Ful l Bounded ueue Ful l Bounded ueue Ful l

Tabl e 3: Mi ni mal Test Case f or ueue Exampl e

One bene t of thi s al ternate method that i s not rel ated to the number of test cases or

thei r e ect i veness i s a better chance to t the de ned probl emto a standard orthogonal array,

whi ch el i mi nates the need to recal cul ate an orthogonal matr i x each t i me one i s needed. Al so, thi s

al ternat i ve techni que al l ows removal or addi t i on of any f actors whi ch may not need test i ng or whi ch

may need test i ng exhaust i vel y. For exampl e, i f we deci de that we do not want to test the state

of the recei ver , we can remove the col umn of the orthogonal array that represents thi s part i cul ar

f actor and st i l l have an orthogonal array that covers the remai ni ng f actors and thei r l evel s . I f we

deci de, however , that we want to exhaust i vel y test a f actor , such as the state of the recei ver , we

can remove the col umn that represents thi s f actor and i mpl ement, i n each state of the recei ver , the

remai ni ng test cases shown i n the array. Thi s vi ewo ers more exi bi l i ty and t i me savi ngs i f the

tester wi shes to i sol ate one of the i nvol ved f actors f or speci al cons i derat i on. o recal cul at i on of an

orthogonal array woul d be necessary.

r r or et e t i on n l s i s

I n order to determi ne the e ect i veness of thi s techni que i n produci ng test cases that detect errors ,

i t i s necessary to deci de what types of errors we want to detect . I nstead of tryi ng to generate a

l i st of poss i bl e i mpl ementat i on errors f or whi ch to search, we chose to i ntroduce usage errors i nto

the ear l i er e e exampl e that coul d uncover potent i al i mpl ementat i on errors . The usage errors

were col l ected f roma group of col l eagues who were gi ven the names of the cl asses i nvol ved and

the state i nf ormati on f or each cl ass . They were as ed to thi n of potent i al errors that coul d resul t

f romcombi nat i ons of these cl asses i n use. The errors that were returned were ones that coul d be

detected i n combi nat i ons of obj ects , but more s i mpl y they coul d be detected by test i ng a part i cul ar

obj ect i n one of i ts part i cul ar states . The errors are l i sted bel owand categor i zed accordi ng to the

cl ass they a ect .

ueue

34

1. emovi ng an i temf roman empty ueue

2. Cal l i ng the destructor on a ueue that i s not empty

Bounded ueue

1. emovi ng an i temf roman empty Bounded ueue

2. Addi ng an i temto a f ul l Bounded ueue

3. Cal l i ng the destructor on a Bounded ueue that i s not empty

Pri or i ty ueue

1. emovi ng an i temf roman empty Pri or i ty ueue

2. Cal l i ng the destructor on a Pri or i ty ueue that i s not empty

3. As i ng f or the top pr i or i ty f roman empty Pri or i ty ueue

I n essence, errors caused i n obj ect i nteract i on can resul t f romtwo thi ngs : 1 A proper

return code i s not returned to the obj ect sendi ng the message, and or 2 the return code recei ved

by the sendi ng obj ect i s not proper l y handl ed. Errors resul t i ng f romthese thi ngs are cl ear l y

i mpl ementat i on errors centered around the part i cul ar states of an obj ect , and shoul d be detected

dur i ng some phase of test i ng. Thi s suggests that ma i ng sure some test i ng occurs i n each state of

al l poss i bl e obj ects i nvol ved may be a vi tal poi nt i n ver i f yi ng the correctness of the sof tware. I f

we choose to test al l states of an obj ect dur i ng the i ntegrat i on phase, us i ng the representat i ve set

generated wi l l al l owus to do so. s i ng the mi ni mal test set , as a compari son, onl y o ers the tester

coverage of one state of each i ntended obj ect . The exhaust i ve test set o ers coverage of al l states ,

but al so carr i es a much l arger penal ty i n terms of requi red t i me and manpower, and i n real i ty may

not gi ve much more than the representat i ve test set .

I f errors resul t f romthe two reasons l i sted above, proper l y test i ng each obj ect i n each of i ts

states shoul d gi ve the same l evel of product con dence f or the states of an obj ect as test i ng every

combi nat i on of the i nvol ved obj ects i n every state. So, i f our representat i ve set of test cases tests

every i nvol ved obj ect i n each of i ts states , we shoul d have a hi gh l evel of con dence that the system

wi l l not \f ai l " f or the end user , and can concentrate on the f unct i onal i ty, i nter f ace adherence, and

data i ntegr i ty whi l e an obj ect i s i nvol ved i n a part i cul ar i nteract i on.

35

hen we exami ne the representat i ve set generated i n Appendi x D, we can see that every

obj ect gets tested i n every state, whi ch i s one of our concerns . s i ng a set of test cases onl y 4. 2

percent as l arge as the exhaust i ve set , we cover every obj ect i n every state. Theref ore, we have

reduced the test i ng e ort s i gni cant l y wi thout sacr i ci ng one part i cul ar obj ect or state.

I n terms of moni tor i ng the e ects obj ect i nteract i on has on the data i nvol ved i n every poss i bl e

i nteract i on, we cannot observe f romthe representat i ve test set nor the mi ni mal test set al l poss i bl e

data i nteract i ons . But, we can observe some i nvol vement of each obj ect , whi ch i f probl ems ar i se

can l ead us to exami ne more cl osel y one part i cul ar obj ect .

Fi nal l y, wi th regards to i nter f ace errors and f unct i onal i ty, ma i ng sure that every obj ect i s

i nvol ved i n some test combi nat i on shoul d uncover any potent i al errors . I f we are cons i stent wi thi n

each obj ect and wi thi n hi erarchi es f or dynami cal l y bound methods, us i ng a method once gi ves us

con dence that the method i nter f ace shoul d be cons i stent throughout the hi erarchy.

i el i nes or er or i n nt er - Cl s s es t i n

I n cons i der i ng i mpl ementat i on of the proposed techni que, we want to achi eve as much automati on

as poss i bl e wi thout i ntervent i on because of the magni tude of the i nter- cl ass test i ng e ort . I nter-

cl ass test i ng i nvol ves exami ni ng each i nter- cl ass message i n each cl ass , whi ch coul d potent i al l y be

thousands of messages and resul ts i n a concern f or the t i me i nvol ved. I n order to achi eve as much

automati on as poss i bl e, we wi l l use as i nput to our al gor i thma l i st of cl asses wi th thei r de ni t i ons .

The de ni t i on of each cl ass wi l l i ncl ude i nf ormati on s i mi l ar to that f ound i n a C header l e. From

thi s we shoul d be abl e to determi ne hi erarchi cal rel at i onshi ps and cl ass methods. Addi t i onal l y a

de ni t i on wi l l i ncl ude a representat i on of the states of a cl ass , f romwhi ch we can der i ve the number

of states per cl ass and the names of the states . The al gor i thm E E ATECASES shown i n Fi gure

9 out l i nes the construct i on of the test cases f or i nter- cl ass test i ng.

The step whi ch returns a ref erence to an orthogonal array cal l i ng EOAi s a compl i cated

step whi ch i n i tsel f has several di st i nct phases . A necessary step i n us i ng an orthogonal array

i s tryi ng to t the probl emde ni t i on to a standard orthogonal array, whi ch requi res that we

nowthe number of f actors , l evel s , and the degrees of f reedomf or the exper i ment. At the poi nt

i n E E ATE CASES where E OAi s cal l ed, we now the sender , recei ver , and parameters

36

algori thm E E ATE CASES Cl assLi st : l i st of cl asses wi th de ni t i onsea ass in ass ist

SenderCl ass cl asscal cul ate the set of pol ymorphi c subst i tutes f or SenderCl ass

ea et in en e ass

ea inte - ass essa e in et

determi ne the recei vi ng cl assdetermi ne the parameter cl ass or cl asses

cal cul ate sets of pol ymorphi c subst i tutes f or recei ver and parameter cl assesi n i s sti t tes e ist sen e , e ei e , an a a ete s t en

construct test message

e se

Oa ef E OA sender , recei ver , l i st of parameters

TestCaseArray T A SLATE A A l i st of cl asses , Oa efea test ase in est ase a

construct test messageen ea inte - ass essa e

en ea et

en ea ass

Fi gure 9: Al gor i thm

i nvol ved wi th the message. i th thi s i nf ormati on, we can cal cul ate the number of f actors . There

wi l l al ways be at l east f our f actors i nvol ved i n thi s cal cul at i on, whi ch represents the sender , the

sender ' s state, the recei ver , and the recei ver ' s state.

The number of l evel s f or each obj ect f actor i s equal to the number of cl asses that coul d

represent that cl ass at run- t i me. I f , f or exampl e, we have a sendi ng cl ass wi th 4 cl asses that i nher i t

f romi t , the number of l evel s f or sender f actor woul d be 5. The number of l evel s f or each state

f actor i s the maxi mumnumber of states of the cl ass i tsel f or any pol ymorphi c subst i tute f or the

cl ass whose state i s bei ng tested. Agai n, i f we have a sender wi th 2 states , and 4 i nher i ted cl asses

each wi th 3, 4, 3, and 4 states respect i vel y, the number of l evel s f or the sender state f actor woul d be

maxi mum2, 3, 4, 3, 4 4.

The degrees of f reedomf or the exper i ment can be cal cul ated as shown i n Tabl e 4 by summi ng

up number of l evel s - 1 f or each f actor and addi ng an overal l mean of 1, whi ch i s constant f or

every probl em. As an exampl e, i f we return to our above exampl e, and f urther assume the recei ver

has 4 l evel s , recei ver state has 3, parameter has 6 l evel s , and parameter state has 7, the degrees of

f reedomf or the exper i ment woul d as cal cul ated i n Tabl e 4.

37

Factor Degrees of Freedom

Sendi ng Obj ect 5 - 1 4

of States of the Sendi ng Obj ect 4 - 1 3

ecei vi ng Obj ect 4 - 1 3

of States of the ecei vi ng Obj ect 3 - 1 2

Parameter Obj ect 6 - 1 5

of States of the Parameter Obj ect 7 - 1 6

Overal l Mean 1

Total 24

Tabl e 4: Cal cul at i ng the degrees of f reedomi n EOA.

algori thm E OAin t: sender , recei ver , and parameter s f or current message

cal cul ate number of f actorscal cul ate number of l evel scal cul ate degrees of f reedomf or exper i ment

ea stan a it n e s e ees ee

i number of f actors max f actors f or OA and

number of l evel s max l evel s f or OAretr i eve standard OA

i no standard OAmatchesgenerate an orthogonal array

return ref erence to orthogonal array

Fi gure 10: Al gor i thm

Once the degrees of f reedom, number of f actors , and number of l evel s are cal cul ated, we can

try and t the probl emto a standard orthogonal array. I n order to do so, we must compare these

three val ues to the s i mi l ar val ues f or the standard orthogonal array. Each standard orthogonal

array can be represented i n memory by the number of rows i n the array, the maxi mumnumber of

f actors i n the array, and the maxi mumnumber of l evel s per f actor . I f the degrees of f reedomf or

the exper i ment i s at l east equal to the number of rows, and the numbers of f actors and l evel s are

l ess than the max val ues f or the standard array, the array i s a vi abl e choi ce and a ref erence to the

chosen array can be returned. The E OAal gor i thmshown i n Fi gure 10 out l i nes thi s process of

sel ect i ng an orthogonal array.

Af ter the orthogonal array i s returned, the remai ni ng port i on i nvol ves mappi ng the numeri cal

38

representat i on f romthe orthogonal array to the state and cl ass names, whi ch may i ncl ude the use

of dummy l evel s to l l the di mens i ons of the standard orthogonal array. Thi s process can be done

us i ng trans l at i on tabl es and i n E E ATECASES i s the cal l to T A SLATE A A . Af ter

the compl et i on of E E ATE CASES, the remai ni ng steps f or compl et i ng the i nter- cl ass test i ng

process woul d be the execut i on of the test cases generated us i ng an orthogonal array.

At thi s poi nt , thi s method poses several di cul t i es i n terms of compl ete automati on. er-

i f yi ng the correctness of the execut i on of test cases i s a probl emthat res i des i n not onl y thi s but

most al l test i ng processes and i s the typi cal l y the j ob of the \test i ng oracl e. " Actual generat i on of

an orthogonal array when no standard orthogonal arraymatches the probl emde ni t i on i s a process

whi ch warrants f urther i nvest i gat i on bef ore i mpl ementat i on.

Con l s i ons n t r e or

Thi s paper has exami ned a rel at i vel y unexpl ored area of obj ect- or i ented research, that of i nter- cl ass

test i ng. I n essence, the same probl ems wi th f unct i onal i ty, i nter f aces , and data i ntegr i ty exi st i n the

obj ect or i ented paradi gmthat exi st i n the procedural paradi gm, onl y to a greater extent . Areas

of concern, such as poss i bl e pol ymorphi c subst i tut i ons , dynami c bi ndi ng, and obj ect state have

been exami ned and some poss i bl e pi t f al l s regardi ng these areas have been exposed. Two of these

concerns , pol ymorphi c subst i tutes and obj ect state, were exposed as poss i bl e compoundi ng f actors

f or test i ng i nter- cl ass i nteract i ons . I n regards to these two areas , an exi st i ng test i ng methodol ogy

cal l ed st estin was exami ned and al tered to produce a set of test cases to reduce some of

the test i ng e ort that these two areas can cause. The probl ems associ ated wi th dynami c bi ndi ng

do not exi st i ndependent of pol ymorphi sm, and so by address i ng the probl ems i ntroduced by

pol ymorphi sm, we al so address the probl ems of dynami c bi ndi ng.

The test case combi nat i ons generated us i ng orthogonal arrays provi de better error detect i ng

capabi l i t i es over that provi ded by appl yi ng mi ni mal coverage, s i mpl y because the set cons i ders

the poss i bl e pol ymorphi c subst i tutes and thei r states . The capabi l i t i es of the representat i ve set

do not guarantee the error detect i on capabi l i t i es of exhaust i ve test i ng i n terms of data errors

and f unct i onal i ty, but nei ther i s i t as l arge as the exhaust i ve set . For our test i ng exampl e, the

representat i ve set was 4. 2 percent of the exhaust i ve set , and possessed the same coverage f or obj ect

39

state as that of the exhaust i ve set . The tester must deci de, based on the obj ect i ves of the test pl an

and the constrai nts , e. g. , t i me and money, whi ch l evel of adequacy i s best sui ted f or thei r part i cul ar

proj ect . The exami nat i on of a s i ngl e message i n thi s wor onl y represents a smal l port i on of the

wor i nvol ved f or an actual sof tware systemwi th potent i al l y thousands and thousands of messages .

I n some obj ect or i ented l anguages , such as Smal l tal , where al l methods are dynami c and al l

communi cat i on i s done vi a messages , the e ort requi red f or i nter- cl ass test i ng may be greater than

other obj ect or i ented l anguages .

i th the l arge number of messages poss i bl e i n a gi ven sof tware system, f uture wor shoul d

begi n wi th the automati on of thi s techni que, to the greatest extent poss i bl e. Af ter automati on, an

extended study of the e ect i veness of thi s techni que i n detect i ng errors i n a sof tware devel opment

envi ronment dur i ng i nter- cl ass test i ng i s needed. Future wor i n thi s area al so i nvol ves expl or i ng the

poss i bi l i ty of expandi ng the not i on of st estin to other areas of obj ect- or i ented test i ng where

these same f actors may be present . Addi t i onal l y, the potent i al exi sts f or the combi nat i on of thi s

techni que wi th i ncremental test i ng techni ques , such as the i erarchi cal I ncremental Test i ng IT

Al gor i thm[ 2] , to try and l essen the test i ng e ort evenmore based on i nher i tance rel at i onshi ps . The

presented method f or obust Obj ect Ori ented Test i ng separates an obj ect and i ts states , whi ch

al l ows f or removal of these i ndi vi dual f actors f or test i ng cons i derat i on. I f f uture wor i n the area

of test i ng cons i stent states i n i nher i ted cl asses concl udes that retest i ng i s unnecessary, the deci s i on

to separate obj ect and state i n thi s test i ng techni que wi l l be f urther j ust i ed. Future wor al so

i ncl udes wor wi th the not i on of a wel l de ned cl ass and the i mpact of wel l de ned cl asses on the

wor presented i n thi s paper .

40

e er en es

[ 1] obert Brownl i e, ames Prowse, and Madhav S. Phad e, \ obust Test i ng of AT T

PM StarMAIL si ng OATS, " e ni a na , May- une 1992, pp. 41- 47.

[ 2] Mary ean arrol d, evi n . Fi tzpatr i c , and ohn D. Mc regor , \I ncremental Test i ng of

Obj ect- Ori ented Cl ass Structures , " ee in s t e nte nati na n e en e n t a e

n inee in , May 1992, pp. 68- 80.

[ 3] . E. owden, i n \Funct i onal programtest i ng and anal ys i s , " Mc raw- i l l , 1987.

[ 4] areton . . Leung and Lee hi te, \AStudy of I ntegrat i on Test i ng and Sof tware egress i on

at the I ntegrat i on Level , " ansa ti ns n t a e n inee in ,

[ 5] obert Mandl , \Orthogonal Lat i n Squares : An Appl i cat i on of Exper i ment Des i gn to Compi l er

Test i ng, " ni ati ns t e , October 1985, ol . 28, o. 10, pp. 1054- 1058.

[ 6] ohn D. Mc regor and Dougl as M. Dyer, \A ote on I nher i tance and State Machi nes , " t-

a e n inee in tes , vol . 18, no. 4, pp. 61 - 69, October 1993.

[ 7] ohn D. Mc regor and Ti m orson, \ nderstandi ng Obj ect- Ori ented: A ni f yi ng Paradi gm. "

ni ati ns t e , vol . 33, no. 9, pp. 40- 60, September 1990.

[ 8] Madhav S. Phad e, \ ual i ty Engi neer i ng s i ng obust Des i gn, " Prent i ce al l , Engl ewood

Cl i s , , 1989.

[ 9] Damaraj u aghavarao, \Construct i ons and Combi nator i al Probl ems i n Des i gn of Exper i -

ments , " ohn i l ey Sons, I nc. , ew or , 1971.

[ 10] Phi l l i p . oss , \Taguchi Techni ques f or ual i ty Engi neer i ng - Loss Funct i on, Orthogonal

Exper i ments , Parameter and Tol erance Des i gn, " Mc raw- i l l , ew or , 1988.

[ 11] eni chi Taguchi , \Systemof Exper i mental Des i gn" Engl i sh Trans l at i on, raus I nternat i onal

Publ i cat i ons , hi te Pl ai ns , ew or , 1987.

[ 12] C. D. Turner and D. . obson, \The Test i ng of Obj ect- Ori ented Programs, " Techni cal eport :

T - 13 92, ni vers i ty of Durham, Engl and, ovember 1992.

41

C l l t i n t e er o es t C s es t o s t i el es t

i en es s e nt er t i on.

3 states 3 states

Parameter X

2 states

3 states 3 states

D

E FB C

A

3 states 3 states

2 states 2 states

Hierarchy of Parameter X

Y Z

message( class X )

Fi gure 11: Cal cul at i ng poss i bl e test cases cons i der i ng state and pol ymorphi c subst i tut i ons .

Based on the representat i on gi ven i n Fi gure 11, we have one parameter , , whi ch can be represented

at run- t i me by one of 3 di erent cl asses , , , or . i ven that each of the hi erarchi es shown above

are model ed af ter the ueue hi erarchy, the resul t i s 8 di st i nct ways to represent the same message

wi th regards to the parameter obj ect and the state of the parameter obj ect i nvol ved i n the message

send. These are shown i n Tabl e 5.

Obj ect State

ueue Empty

ueue ot Empty

Bounded ueue Empty

Bounded ueue Part i al Ful l

Bounded ueue Ful l

Pr i or i ty ueue Empty

Pri or i ty ueue ewTop El ement

Pri or i ty ueue ewShape of ueue

Tabl e 5: Poss i bl e representat i ons f or the parameter obj ect .

42

The sender of the message can be one of 3 di erent obj ects . Cl ass A ueue has 2 states ,

and each of i ts pol ymorphi c poss i bi l i t i es Band C Bounded ueue and Pri or i ty ueue respect i vel y

has 3 states , f or a total of 8 di erent ways to represent the sender , s i mi l ar to the 8 representat i ons

l i sted i n Tabl e 5 f or the parameter obj ect . Si mi l ar l y, the recei ver can al so be represented i n one of

8 di erent ways.

Together , there are 64 combi nat i ons of the sendi ng and recei vi ng obj ects . Each of these 64

combi nat i ons can i nvol ve 8 di erent f orms of the same message. Thi s means we have 64 8 512

di erent combi nat i ons of sender , recei ver , parameter , and states i f we cons i der the pol ymorphi c

subst i tut i ons and states of each obj ect exhaust i vel y.

43

o t or r t o on l r r Cons t r t i on or es s e

i t o r et er s

Thi s appendi x expands us i ng the sender state and the recei ver state as f actors f or orthogonal array

construct i on. The cl ass hi erarchi es i nvol ved are those shown i n Fi gure 7. The sender and recei ver

hi erarchi es are patterned af ter the ueue cl ass hi erarchy shown i n Fi gure 3 so that we may have

state names to use i nstead of numeri c val ues when the test cases generated are expanded. The

l evel s of these two f actors are shown bel ow.

Factor - Sender StateLevel s - 1 ueue - Empty

2 ueue - ot Empty3 Bounded ueue - Empty

4 Bounded ueue - Part i al Ful l5 Bounded ueue - Ful l

6 Pri or i ty ueue - Empty7 Pri or i ty ueue - ewTop El ement

8 Pri or i ty ueue - ewShape of ueue

Factor - ecei ver State

Level s - 1 ueue - Empty2 ueue - ot Empty

3 Bounded ueue - Empty4 Bounded ueue - Part i al Ful l

5 Bounded ueue - Ful l6 Pri or i ty ueue - Empty

7 Pri or i ty ueue - ewTop El ement8 Pri or i ty ueue - ewShape of ueue

The message passed has no parameters , thus there are onl y two f actors . The orthogonal

array constructed i s shown Tabl e 6. There i s not a standard orthogonal array f or two f actors wi th

a maxi mumof 8 l evel s , but the cal cul at i on i s rather s i mpl e. Each combi nat i on of l evel s of the

sender state and l evel s of recei ver state must appear once i n the pai rwi se col umns. Si nce there are

onl y two col umns, every combi nat i on of the two f actors appears once. Assumi ng i t ta es one test

44

case to test each combi nat i on of sender state and recei ver state, there i s no reduct i on us i ng an

orthogonal array as compared to test i ng al l poss i bl e combi nat i ons . s i ng thi s method f or test i ng a

message wi th no parameters wi l l not be bene ci al .

45

Test Parameters

Test Case Sender- State ecei ver- State

1 1 ueue - Empty 1 ueue - Empty

2 1 ueue - Empty 2 ueue - ot Empty

3 1 ueue - Empty 3 Bounded ueue - Empty

4 1 ueue - Empty 4 Bounded ueue - Part i al Ful l

5 1 ueue - Empty 5 Bounded ueue - Ful l

6 1 ueue - Empty 6 Pri or i ty ueue - Empty

7 1 ueue - Empty 7 Pri or i ty ueue - ewTop El ement

8 1 ueue - Empty 8 Pri or i ty ueue - ewShape of ueue

9 2 ueue - ot Empty 1 ueue - Empty

10 2 ueue - ot Empty 2 ueue - ot Empty

11 2 ueue - ot Empty 3 Bounded ueue - Empty

12 2 ueue - ot Empty 4 Bounded ueue - Part i al Ful l

13 2 ueue - ot Empty 5 Bounded ueue - Ful l

14 2 ueue - ot Empty 6 Pri or i ty ueue - Empty

15 2 ueue - ot Empty 7 Pri or i ty ueue - ewTop El ement

16 2 ueue - ot Empty 8 Pri or i ty ueue - ewShape of ueue

17 3 Bounded ueue - Empty 1 ueue - Empty

18 3 Bounded ueue - Empty 2 ueue - ot Empty

19 3 Bounded ueue - Empty 3 Bounded ueue - Empty

20 3 Bounded ueue - Empty 4 Bounded ueue - Part i al Ful l

21 3 Bounded ueue - Empty 5 Bounded ueue - Ful l

22 3 Bounded ueue - Empty 6 Pri or i ty ueue - Empty

23 3 Bounded ueue - Empty 7 Pri or i ty ueue - ewTop El ement

24 3 Bounded ueue - Empty 8 Pri or i ty ueue - ewShape of ueue

25 3 Bounded ueue - Part i al Ful l 1 ueue - Empty

26 3 Bounded ueue - Part i al Ful l 2 ueue - ot Empty

27 3 Bounded ueue - Part i al Ful l 3 Bounded ueue - Empty

28 3 Bounded ueue - Part i al Ful l 4 Bounded ueue - Part i al Ful l

29 3 Bounded ueue - Part i al Ful l 5 Bounded ueue - Ful l

30 3 Bounded ueue - Part i al Ful l 6 Pri or i ty ueue - Empty

31 3 Bounded ueue - Part i al Ful l 7 Pri or i ty ueue - ewTop El ement

32 3 Bounded ueue - Part i al Ful l 8 Pri or i ty ueue - ewShape of ueue

33 3 Bounded ueue - Ful l 1 ueue - Empty

34 3 Bounded ueue - Ful l 2 ueue - ot Empty

35 3 Bounded ueue - Ful l 3 Bounded ueue - Empty

36 3 Bounded ueue - Ful l 4 Bounded ueue - Part i al Ful l

37 3 Bounded ueue - Ful l 5 Bounded ueue - Ful l

38 3 Bounded ueue - Ful l 6 Pri or i ty ueue - Empty

39 3 Bounded ueue - Ful l 7 Pri or i ty ueue - ewTop El ement

40 3 Bounded ueue - Ful l 8 Pri or i ty ueue - ewShape of ueue

Tabl e 6: Orthogonal Array f or Two Factors , 8- l evel s .

46

Test Parameters

Test Case Sender- State ecei ver- State

41 3 Pri or i ty ueue - Empty 1 ueue - Empty

42 3 Pri or i ty ueue - Empty 2 ueue - ot Empty

43 3 Pri or i ty ueue - Empty 3 Bounded ueue - Empty

44 3 Pri or i ty ueue - Empty 4 Bounded ueue - Part i al Ful l

45 3 Pri or i ty ueue - Empty 5 Bounded ueue - Ful l

46 3 Pri or i ty ueue - Empty 6 Pri or i ty ueue - Empty

47 3 Pri or i ty ueue - Empty 7 Pri or i ty ueue - ewTop El ement

48 3 Pri or i ty ueue - Empty 8 Pri or i ty ueue - ewShape of ueue

49 3 Pri or i ty ueue - TE 1 ueue - Empty

50 3 Pri or i ty ueue - TE 2 ueue - ot Empty

51 3 Pri or i ty ueue - TE 3 Bounded ueue - Empty

52 3 Pri or i ty ueue - TE 4 Bounded ueue - Part i al Ful l

53 3 Pri or i ty ueue - TE 5 Bounded ueue - Ful l

54 3 Pri or i ty ueue - TE 6 Pri or i ty ueue - Empty

55 3 Pri or i ty ueue - TE 7 Pri or i ty ueue - ewTop El ement

56 3 Pri or i ty ueue - TE 8 Pri or i ty ueue - ewShape of ueue

57 3 Pri or i ty ueue - S 1 ueue - Empty

58 3 Pri or i ty ueue - S 2 ueue - ot Empty

59 3 Pri or i ty ueue - S 3 Bounded ueue - Empty

60 3 Pri or i ty ueue - S 4 Bounded ueue - Part i al Ful l

61 3 Pri or i ty ueue - S 5 Bounded ueue - Ful l

62 3 Pri or i ty ueue - S 6 Pri or i ty ueue - Empty

63 3 Pri or i ty ueue - S 7 Pri or i ty ueue - ewTop El ement

64 3 Pri or i ty ueue - S 8 Pri or i ty ueue - ewShape of ueue

Tabl e 7: Tabl e 6 Conti nued.

47

C r t o on l r r Cons t r t i on or es s e i t ne -

r et er

3 states 3 states

Parameter X

2 states

3 states 3 states

D

E FB C

A

3 states 3 states

2 states 2 states

Hierarchy of Parameter X

Y Z

message( class X )

Fi gure 12: Obj ect and Parameter i erarchi es f or a Pol ymorphi c Message Pass .

Thi s appendi x expands us i ng the sender state, recei ver state, and parameter state combi nat i ons

as f actors f or orthogonal array construct i on. The sendi ng hi erarchy, recei vi ng hi erarchy, and pa-

rameter hi erarchy shown i n Fi gure 12 are patterned af ter the ueue cl ass hi erarchy shown i n Fi gure

3 so that we may have state names to use i nstead of numeri c val ues when the test cases generated

are expanded. The l evel s of these three f actors are shown bel ow.

Factor - Sender StateLevel s - 1 ueue - Empty

2 ueue - ot Empty3 Bounded ueue - Empty

4 Bounded ueue - Part i al Ful l5 Bounded ueue - Ful l6 Pri or i ty ueue - Empty

7 Pri or i ty ueue - ewTop El ement8 Pri or i ty ueue - ewShape of ueue

48

Factor - ecei ver StateLevel s - 1 ueue - Empty

2 ueue - ot Empty3 Bounded ueue - Empty

4 Bounded ueue - Part i al Ful l5 Bounded ueue - Ful l

6 Pri or i ty ueue - Empty7 Pri or i ty ueue - ewTop El ement

8 Pri or i ty ueue - ewShape of ueue

Factor - Parameter State

Level s - 1 ueue - Empty2 ueue - ot Empty

3 Bounded ueue - Empty4 Bounded ueue - Part i al Ful l

5 Bounded ueue - Ful l6 Pri or i ty ueue - Empty

7 Pri or i ty ueue - ewTop El ement8 Pri or i ty ueue - ewShape of ueue

The message passed has a sender , recei ver , and a parameter wi th whi ch we are concerned.

Each f actor i nvol ved has 8 l evel s . I n order f or a standard orthogonal array to be a vi abl e choi ce f or

an exper i ment, the number of rows i n the array must at l east be equal to the degrees of f reedom

requi red f or the exper i ment. The cal cul at i on of the degrees of f reedomi s shown i n Tabl e 8.

Factor Degrees of Freedom

Sender 8 - 1 7

ecei ver 8 - 1 7

Parameter 8 - 1 7

Overal l Mean 1

Total 22

Tabl e 8: Cal cul at i ng the mi ni mumnumber of test cases requi red.

Tabl e 8 shows a mi ni mumof 22 test cases to systemati cal l y eval uate the e ects of al l f actors

49

rel evant to the test i ng requi rements .

The cl osest standard orthogonal array wi th at l east 22 rows i s L25. owever , L25 i s de ned a

maxi mumof 6 f actors , wi th at most 5- l evel s . One probl emencountered wi th us i ng f actors and l evel s

as de ned i s that a l arger number of l evel s resul t per f actor , whi ch contr i butes to the probl emnot

tt i ng a standard orthogonal array. Theref ore, each t i me an array i s needed i s must be cal cul ated.

As a resul t , we wi l l cons i der an al ternat i ve method that wi l l reduces the number of l evel s per f actor

and al l ows a better chance of tt i ng the probl emto a standard orthogonal array, so we can exami ne

the val i di ty of thi s approach to test i ng.

50

r t o on l r r Cons t r t i on or es s e i t ne -

r et er Cons i er i n t t e s n e en ent

Factor - Sender

Level s - 1, 4 Cl ass A ueue2, 5 Cl ass BBounded ueue3 Cl ass CPri or i ty ueue

Factor - ecei verLevel s - 1, 4 Cl ass D ueue

2, 5 Cl ass EBounded ueue3 Cl ass FPri or i ty ueue

Factor - Parameter 1

Level s - 1, 4 Cl ass ueue2, 5 Cl ass Bounded ueue

3 Cl ass Pri or i ty ueue

States f or Cl ass A

1, 3, 5 ot Empty2, 4 Empty

States f or Cl ass B1, 4 Part i al Ful l

2, 5 Ful l3 Empty

States f or Cl ass C1, 4 ewTop El ement

2, 5 ewShape of ueue3 Empty

States f or Cl ass D

1, 3, 5 ot Empty2, 4 Empty

States f or Cl ass E1, 4 Part i al Ful l

2, 5 Ful l3 Empty

States f or Cl ass F1, 4 ewTop El ement

2, 5 ewShape of ueue3 Empty

States f or Cl ass

1, 3, 5 ot Empty2, 4 Empty

States f or Cl ass1, 4 Part i al Ful l

2, 5 Ful l3 Empty

States f or Cl ass1, 4 ewTop El ement

2, 5 ewShape of ueue3 Empty

Al l f actors wi l l be represented gener i cal l y by the numbers 1, 2, etc i n the orthogonal array

cal cul at i on. Trans l at i ng these numbers to actual obj ects and states occurs i n the test case com-

bi nat i on trans l at i on and can be done by l oo i ng above at the numeri c representat i on gi ven bef ore

each l evel of each f actor . The appropr i ate dummy l evel s have been added to each f actor above,

51

whi ch wi l l be necessary i n tt i ng the de ned probl emto a standard orthogonal array. ot i ce, some

states and obj ects wi l l be represented by mul t i pl e entr i es i n the array. Thi s means these l evel s wi l l

get tested more of ten. As a resul t , we have l i sted the l evel s of each i n an order of i mportance.

For the obj ect f actors , the i ntended obj ect i s l i sted as the rst f actor . For the state f actors , any

states newl y added i n a cl ass wi l l be l i sted rst , f ol l owed by an order i ng of l east cons i stent state

throughout the hi erarchy to the most cons i stent . ot i ce, f or the state f actors , the state t i s

l i sted l ast because thi s state remai ns unchanged throughout the hi erarchy.

I n order f or a standard orthogonal array to be a vi abl e choi ce f or an exper i ment, the number

of rows i n the array must at l east be equal to the degrees of f reedomrequi red f or the exper i ment.

I n Tabl e 9, the degrees of f reedomf or the exper i ment are cal cul ated.

Factor Degrees of Freedom

Sendi ng Obj ect 3 - 1 2

State of the Sendi ng Obj ect 3 - 1 2

ecei vi ng Obj ect 3 - 1 2

State of the ecei vi ng Obj ect 3 - 1 2

Parameter Obj ect 3 - 1 2

State of the Parameter Obj ect 3 - 1 2

Overal l Mean 1

Total 13

Tabl e 9: Cal cul at i ng the degrees of f reedom.

For thi s exper i ment, we have 6 f actors wi th a maxi mumof 3 l evel s f or any f actor . I n terms

of a standard orthogonal array, thi s model doesn' t t any standard array exact l y. The cl osest array

wi th at l east 13 rows i s L16, whi ch onl y al l ows f or two- l evel f actors . L16' has 16 rows but onl y

al l ows f or a maxi mumof 5 f our- l evel f actors . L18 al l ows a maxi mumof 8 f actors , 7 of whi ch may

be 3- l evel f actors .

Perhaps the most vi abl e choi ce i s L25. L25 cons i sts of 25 test cases , wi th a maxi mum

of 6 f actors , al l of whi ch may be 5- l evel f actors . e can use dummy l evel s to map our exper i -

ment to L25. e wi l l use two and three dummy l evel s as needed f or each f actor , whereby ma i ng

the number of l evel s per f actor equal to 5. Tabl e 10 di spl ays the gener i c representat i on of thi s array.

52

Col umn

Expt. o. Snd. Snd. St . cv. cv. St . P1 P1St.

1 1 1 1 1 1 1

2 1 2 2 2 2 2

3 1 3 3 3 3 3

4 1 4 4 4 4 4

5 1 5 5 5 5 5

6 2 1 2 3 4 5

7 2 2 3 4 5 1

8 2 3 4 5 1 2

9 2 4 5 1 2 3

10 2 5 1 2 3 4

11 3 1 3 5 2 4

12 3 2 4 1 3 5

13 3 3 5 2 4 1

14 3 4 1 3 5 2

15 3 5 2 4 1 3

16 4 1 4 2 5 3

17 4 2 5 3 1 4

18 4 3 1 4 2 5

19 4 4 2 5 3 1

20 4 5 3 1 4 2

21 5 1 5 4 3 2

22 5 2 1 5 4 3

23 5 3 2 1 5 4

24 5 4 3 2 1 5

25 5 5 4 3 2 1

Tabl e 10: Orthogonal Array L25.

Asi mpl e test case expans i on i nvol vi ng no dummy l evel s woul d be test case 2.

Test case 2 represents :

Sender - 1 cl ass A- ueue

State of the Sender - state 2 Empty

ecei ver - 2 cl ass E - Bounded ueue

State of the ecei ver - state 2 Ful l

Parameter 1 - 2 cl ass - Bounded ueue

State of Parameter 1 - state 2 Ful l

53

Sl i ght l y more compl i cated expans i ons i nvol ve dummy l evel s . owever , trans l at i on i s a matter

of mappi ng numeri c val ues to correspondi ng obj ects or states .

Test case 7 represents :

Sender - 2 cl ass B - Bounded ueue

State of the Sender - state 2 Ful l

ecei ver - 3 cl ass F - Pri or i ty ueue

State of the ecei ver - state 4 ewTop El ement

Parameter 1 - 5 cl ass - ueue

State of Parameter 1 - state 1 ot Empty

Test case 13 represents :

Sender - 3 cl ass C- Pri or i ty ueue

State of the Sender - state 3 Empty

ecei ver - 5 cl ass E - Bounded ueue

State of the ecei ver - state 2 Ful l

Parameter 1 - 4 cl ass - ueue

State of Parameter 1 - state 1 ot Empty

Test case 21 represents :

Sender - 5 cl ass B - Bounded ueue

State of the Sender - state 1 Part i al Ful l

ecei ver - 5 cl ass E - Bounded ueue

State of the ecei ver - state 4 Parti al Ful l

Parameter 1 - 3 cl ass - Pr i or i ty ueue

State of Parameter 1 - state 2 ewShape of ueue

i th the number of dummy l evel s used, an i ni t i al thought was that dupl i cate test cases

woul d exi st wi thi n the set generated. owever , af ter expans i on and cl oser exami nat i on, we f ound

54

ueue

B Bounded ueue

P Pri or i ty ueue

E ot Empty

E Empty

PF Part i al Ful l

F Ful l

TE ewTop El ement

S ewShape of ueue

Tabl e 11: Abbrevi at i ons f or Test Case Expans i on.

thi s to be f al se. o dupl i cate test cases were f ound wi thi n the generated set .

s i ng the set of abbrevi at i ons i n Tabl e 11 and the numeri c mappi ngs i ndi cated when sender ,

recei ver , parameter , and states were descr i bed i ni t i al l y i n Appendi x D, the gener i c array L25 can

be trans l ated i nto the matr i x shown i n Tabl e 12. The goal was to produce an orthogonal array to

represent the test cases us i ng a standard orthogonal array. The resul t i ng orthogonal array wi l l be

used to exami ne the usef ul ness of thi s proposed techni que.

55

Test Parameters

Expt. o. Snd. Snd. St . cv. cv. St . P1 P1St.

1 E E E

2 E B F B F

3 E P E P E

4 E E E

5 E B F B F

6 B PF B E E

7 B F P TE B PF

8 B E E E

9 B PF B PF B E

10 B F P S P TE

11 P TE P S B PF

12 P S E P S

13 P E B F E

14 P TE E B F

15 P S B PF E

16 E E B E

17 E B E E

18 E E B F

19 E B F P TE

20 E P TE E

21 B PF B PF P S

22 B F E E

23 B E B PF B PF

24 B PF P S E

25 B F E B PF

Tabl e 12: Orthogonal Array L25 expanded to showactual test cases .

56