"a framework for assessing the adequacy and effectiveness of software development methodologies

8/10/2019 "A framework for assessing the adequacy and effectiveness of software development methodologies

1/47

A Framework for Assessing the Adequacy and Effectiveness of

Software Development Methodologies

James

D

Arthur and Richard

E

Nance

1

Virginia Tech d :

ndex

Terms

Software Development Methodologies, Proc edural Evaluat ion, Evaluat ing M ethod-

ologies, S oftware Engineering Object ives, Software Engineering Principles , In duced

Attributes , Indicators .

1

Introduction

Over t he past two decades the software development process has changed dramatical ly. Early

software developm ent practices were guided by seat-of-the-pants prog ram ming styles. Recog-

nizing maintenance difficulties associated with such styles, the software development cornmunit?

began to inve stigate an d identify software engineering principles th a t could significantly enha nc e

the maintainabi l i ty an d qual i ty of a result ing product. Consequent ly, development techriques t ha t

exploi ted software en giv er ing principles l ike abstract ion [LISB75], information hiding [Pr\RD7 ]

and stepwise refinement [WIRN 71] were formulated and integrated into m any software

development

processes.

Th e subsequent de mand for increasingly complex software systems. however, mandated the

coordinated use of complementary principles, guided by an encompassing software development

philosophy th at recognized project level goals an d objectives, i .e. a methodological approach to

software developm ent. Today, myriad of tools, techniques, an d developm ent methodo logies

scl

dress th e chal lenging task of prod ucing high qual i ty software. For example. SC R

[HENKiY]

alitl

.

-..

-

Work supported by the

U S

Navy throu gh the Systems Research Cen ter under Basic Ordering


2/47

DARTS [GOMH8 ] are development methodologies that emphasize specific software engineering

goals (reduc ing software development costs and facilitati ng the design of real-tim e system s, respec-

tively). SREM [ALFM85] and SADT [ROSD77] are methodology based en v im nm cn k Both focus

on particular phases of the software life cyd e an d are suppo rted by unif ied se ts of complementary

tools.

The steady proliferation of design methodologies, however, has nct been without

a

price. In

particula r, users find increasing difficulty in choosing n appropr ia te methodolop jca l ap p i .~ a nd

recognizing reasonable expecta tions of a design or development methodology. Addressing th is

concern, the research described in this paper outlines a well-defined procedure for

eva luat ing the adequacy of a software development methodology relative

t

project goals.

a n d

assessing th e effectiveness of a methodology relative to th e quality of the produ ct produced.

The evaluation procedure is based on a substantiated set of l inkages among accepted software

engineering objectives, principles, and attributes. These linkages reflect an assessment perspective

structured by the needs, process, and product sequence for system development. and enable

comparative scale for determining the adequacy and effectiveness of the supporting development

methodology. Th e identification of code and docu men tation p roperties and t he definition of metrics

for these properties enables an accumulative determination of software engineering attributes.

principles and objectives.

To provide a uniform basis for discussion, Section

2

outlines the role of a me tho do log in

the software development process. Section discusses th e relationship of software engineering

objectives, pr inciples and attr i bute s to the software development effort . Section identif ies th e

commonly accepted objectives, pr inciples and attr ibutes , defines th e relationships am on g t ien i .

an d then discusses how one evalua tes a methodology based on the those relationships. Finally.

Section describes an application of th e assessment procedure to two Sav y software development

methodologies.


3/47

2. W h a t C o n s t i t u t e a M e t h o d o l o g y

Fundamental to the research presented in this paper is a common understanding of what

consti tutes a method dogyn. Simply state d, a methodology is a collection of complementary

methods , and a se t of rul s for applying them FREP771. More specifically,

a

methodology

1) organizes and structures the tasks comprising the effort to achieve a global objective,

establishing the relationships among tasks,

(2) defines methods for accomplishing individual tasks (within the framework of the global

objective), and

3 )

prescribes

n

order in which certain classes of decisions are made. and ways of making

those decisions th at lead to th e overall desired objective.

In

general, software development m ethodologies should be guided

by

accepted softw are engineerin s

prina ples th at , when applied to the defined process, achieve a desired goal. ased on this coninloll

understanding of what constitutes

a

methodology, the follo~.ving ections pres ent

a

procedural

approach to evaluating the dequ cy and eflectiveness of software development methodologies.

3 T h e R o le of O b j e ct i ve s , P r i n c ip l e s, a n d A t t r i b u t e s i n S o f t w a r e D e v e l o p m e n t

The development of large, complex software systems is considered a project ac t iv i ty , i~ ivo l v i~l g

several analysts an d programmers and a t least one manager. W hat then is the role of a methodology

in this setti ng and how does it relate to objectives, principles and attrib utes ? Figure assists ill

providing a n answ er to this question.

In general terms, an objective is 'something aimed at or striven for. More specific t o the

software development context, an objective pertains to a project desirable a characteristic tllat

can be definit ively judged only at the co m pl c. .~ ~,f th e project.

3


4/47

DOCUMP CTATlON

+)

PROGRAMS

Rchcd onplrriy

Wdl Dbmsd ~ L O C S

~rr b i l i~y ATlRIBUTES

Visibility ebtrmr

h l yBm

Figure

Illustration of the Relationships Among Objectives, Principles, Attributes

in the Software Development Process

A

software engineering

principle

describes an aspect of

how

the process of software developm ent

should b e done. T he process of software development, if it is to achieve the st ipul ated obje ctives,

mu st be governed by these 'rules of righ t conduct.

t tr ibutes ar e the intangible characteris tics of the product: the software produced by project

personnel following t he principles set fo rth by th e methodology. Unlike objectives, which per tain

o n ly t o t h e tot l project activity, attributes may be observed in one unit of the product a n d

absent in another . T h e dai m of presence o r absence of an a ttr ibute is based on th e recognit ion

o f

properties which contri bute evidence suppo rting th e claim. Properties are observable. a nd can b e

subjective as well as objective in nature.


5/47

Influenced by Fritz Ba uer s original defi nition of softwa re engineering

[BAUF72]

and reflecting

the abo ve description cf software engineering objectives, principles an d att ribu tes , t he rationale for

the evaluation procedure described in this paper is founded on the philosophical argument that:

T h e

mison

d etrr

of any software development methodology is the achievement of one

or more objec ivu th rough a

proiress

governed by defined

principles.

In turn. adherence

t o a process governed y those principles should result in a product ( p r o g r a m s a n d

docum entation) that possesses

at t r ibutes

considered desirable and beneficial.

This philosophy, exemplified by Figure 1, is tempered by practical concerns:

(1) While

a

se t of softw are engineering objectives can be identified, this se t might no t be

complete, and additions or modification should be permitted.

2) Objectives can be given different emphasis within a methodology or in applications of a

methodology.

(3) Att ribu tes of a large software product might be evident in one compone nt yet missing in

another.

4 A F r a m e w o r k

f ~ r

v a l u a t i n g S o f t w a r e a e v e l o p m e n t M e t h o d o l og i e s

A broad review

of

software engineering l i tera ture (BERG81,

CHMLSO

G A F J 7 8 , JXChIi5.

PARD79, PA-9D72, SCOL78 WARJIB] leads to the identification of seven objectives com~nonly

recognized in th e num erous methodologies:

1) Maintainability th e ease with which corrections can be made to respond to recogr.ized

inadequacies.

2) Corre ctness stric t adherence to specified requirements,

(3) Reusability he use of developed software in oth er applications,

4) Testab ility he ability to evaluat e conformance with requirements,

5) Relia bility the error-free perform ance of softw are over time.

6) Portab ility the ease in transferring software from one host system to anothe r. and


6/47

7) AdaptabiLity th e ease with which softw are

c n

accommodate to change.

The authors note that these definit ions,

as

well

s

others presented in this section, are abridged:

they are primarily intended to reflect

a working

understa nding based on general lite ratur e usage.

Achievement of these objectives comes through th e application of principles supported en-

couraged, enforced) by a methoddogy. Th e principles enumerated bdo w are extracted from the

references cited above and others)

as

mandatory in the creative process producing high qualie

programs and documentation.

1) Abstraction defining each program segment at a given level of refinement.

a ) Hierarchical Decomposit ion components defined in a t o p d w n manner.

b)

Functional Decomposition components partitioned along functional boundaries.

2 ) Information Hiding insulating the internal details of component behavior.

3)

Stepwise Refinement utilizing a convergent design.

4) Stru cture d Program ming using a restricted set of control construc ts.

5)

Concurrent Documentation management of supportin g docum ents system sp ec ifia -

tions, user manu al, etc.) thro ugh out the life cycle.

6) Life Cyc le Verification verification of requiremen ts thro ugh out the design. developmenr.

and ma inte nan ce phases of t he Life cycle.

Employment of well-recognized principles should result in software products possessing

z

tribu tes considered to be desirable an d beneficial. short definition of each att rib ute is

ivm

below.

1) Cohesion the binding of stat em ents within a software component.


7/47

2) Coup ling th e interdependence among sof tware components.

3) Complexity an abstract measure of work assodated with a software component relative

to human understanding and/or machine execution.

4) Well-defined Interfaces th e definitional da rit y and comp leteness of a shared bou ndary

between a pai r of comp onents hardw are or software).

5) Readabil ity th e diff iculty in understanding a software comp onent related to complex ity).

6) Ease of Ch ang e the ease with which software accomm odates enha ncem ents or extensions.

7) Traceability th e ease in retracing the complete history of a software comp onent from its

cur rent s t a tu s t o i t s des ign incept ion.

8) Visibility of. Beh avior th e provision of a review process for e n o r checking.

9) Early E rror Detection indication of faults in requirements specification and design prior

to implementa tion.

T h e software developm ent process, illustrated in Figure 1, depicts

a

natuml relationship that

l inks objectives to p r indp les and pr inciples to at tributes. T h a t is. one achieves the objectives of

a software developm ent m ethodology by applying fundamental principles which. in turn, induce

part icular code and documentation

attr ibutes

From a more detailed perspective. Figure 2 defines

the

specific

se t oflinka ges relating objectives to principles and principles to att ribu tes. -4s described

below, these linkages provide a framework for assessing both th e

adequacy

of a methodology

well as its effectiveness

4.1 ssessing h e dequacy of

a

M e t h o d o l o g y

T h e enunciat ion of objectives should be the f irst step in the definition of aso ftw are development

methodology. Closely

following is the sta teme nt of pr indples t ha t , employed properly, lead to the

attainment of those objectives.

In tur- . the applicat ion of those pr inc iples wi thin a j t r u c t u ~ d

software development process will yield a product th at exhibits desirable at tr ibute s. Th e importan t


8/47

Early Error Detection

Visibility of Behavior

Well-Defined Inurfacc

Figure

Linkages Among t h e Object ives, Principles and A ttribute s

correspondence between th e object ives an d principles and between th e principles an d at t r ibu tes is

shown in Fjgnre ; a literature confirmation of these relationships is discussed in [. \RTJSi].

T he ade quacy of asof tw are development methodology ca n be defined s its a bility t o achieve tlic

software engin eer kg object ives corresponding to those dictated by system needs and cequirem el~ts .

Intnitiveiy, the adequacy of

a

methodology is assessed through a top-down evaluation sc11e111e

s tar t ing wi th an examinat ion of s tated methoddogical object ives relat ive to system needs and

requirements. This step is then followed by a comparison of stated methodological principles

and at t r ibutes wi th those deemed most appropriate. An exa min ation of linkages defined by th e

e d n a t i o n p r o ce d ur e re ve als t h e most appropriate" set. Relative tr the framework depicted

by

F i v e and he s et s of l inkages defined in Figure 2,

an

application of th e evaluat ion proc edr~ re

o

the assessmezlt of method ological deqwcy is outlined below.


9/47

Objectives cf the Methodology: The identification of objectives and the relationships tying objec-

tives to needs an d requirements is usually accomplished by reading th e descriptions of software

development methodology. Evaluation a t this level is quite subjective; however, the absen ce of

de r

statement of objectives for a methodology should trigger

an

alarm: Is the 'methodology

only tool or an incomplete set of tools without coherent stru ctur e?

A

methodology should not

be faulte d, however, for emphasizing certain objectives at th e expense of others; such prioritization

can be highly depen dent on t h e application domain.

Principles Defining the Process Based on the objectives emphasized by the methodology and the

predefined se t of linkages am on g objectives and principles, th e next ste p in assessing the adequacy

of a methodology is an nvestigation of the software development process Tha t is, gven a stated set

of methodological objectives, on e asks: Are the principles supported by the methoddogy consistent

with th ose deemed necessary (through linkage exam ination ) to achieve the s tate d set of objecti*;es?

Th e presence of principles without corresponding objective(s) or vice versa should evoke concerns.

.iUthough this level of evaluation is inherently subjective, some analytical qualit:/ is introduced

through t h e established objective/principle correspondence.

Attributes of the Product T h e third st ep in the assessment process, formulating the set of erpected

product attributes, is based on the fact that principles govern the process by which a software

product i s produced. Th at is, given se t of principles should induce a consequent set of product

attribute s. Obviously, the expec ted set of product attr ibut es should correspond to those desired by

the software engineer , and t o some extent, be implied or stated in t he description cif the software

development methodology. Mo re objectivity is introduced a t this level because, although intangible.

evidence of the at tr ibu tes should be discernible in th e product.

4 2

Assessing the ffectiveness

o f

a

M e t h o d o l o g y

While a top-down evaluation process reveals deficiencies of a softw are deve lop ne ~i tmetliod-

ology, the

effectiveness

of a methodology is assessed through a bot tom -up evaluation process. is


10/47

the term implies, the effectiveness of a methodology is defined as the degree to which a method-

ology produces

a

desired result. In part icul ar, th e etfrctiveness of a metho dology is reflected by

a product s conformance t o the software development process de in ed by th at methodology. We

note, however, that elements independent of the methodology can influence its effectiveness, e.g.

an inadequa te un derstand ing and/or use of the methodology.

The E tence

of

Product Attributes:

Assessing methodological effectiveness starts with an exwni-

nation of the software produc t (cod e and docum entation) for the presence or absence of a t tr ibutes.

Because attr ib utes are intangible, subjective quali ties, the current evaluation is based o a defined

pr c~ er t ie s ha t p rovide evidence s to the presence or absence of attribu tes. More specifically.

the com puta tion of metric values reflect th e exte nt to which particula r properties are observed.

In

tur n, this information is used to synthesize the extant set of product at tr ibutes. Clearly , the

set of at tr ibutes determined from a product evaluation should agree with those induced hy tt.e

corresponding development methodology. Se t mismatch can imply an inapprop riate software de-

velopment methodology, an inadequate understanding of the methodology. or perhaps, the failure

of users to ad here to th e prinap les advocated by t he methodology.

Implied PTinriples and Object ives: Knowing which attributes are present in the product. and

the exte nt to which the y are assessed presen t, provides a basis for imply ing the use of software

engineering principles in t he software development process. Th e rationale for such a state me nt is

based o n the observation th at a p rinciple-to-a ttribute Linkage conversely indicates

an

a t t r ibu te- to -

principle relationship Implying principle usage must be tempered, however, because of the many-

te o n e relationships evist inq between at tr ibu tes and principles. Similarly . using the established

Linkages amo ng objectives and principles, on e can sp eculatr on the achievement of stated software

engineering objectives.

In

summ ary, the thr ee levels of examin ation defined by top-down eva luation process establishes

a procedure for determining how well a methodology can support perceived needs, requirements.

and t he software development process. Conversely, the bot tom -up evd ua tio n process reveals how

Yance

V P

age

1 ol


11/47

F i g u r e

Lllustration of the Evaluation Process

well the methodology

is

applied in the software deueloprr ent process through the

ns

o ~~uaniitolice

measures to support

an

objective,

qt~al i ta t ive

ssessment.

4 3 An Illustration

of

t h e

Evaluation Process

o illus trate how th e eva1ua:ion scheme can be applied. we r er th e reader t o F i g r e

3

whiie

considering th e single objective of

maintainnbility

Formally, maintainability can be deiined a s :he

ease with which m aintenance can be performed to

a

functional unit in accordance

with

prescribed

requirements. Accep ting maintainability as

n

object ive mandates the indusioa o f x p r i n d p e s

(hierarchical decomposit ion, functional decomposit ion, information hiding, st q w is e ref in em at,

structured programming and concurrent documentation) contr ibuting to the realization of tbat


12/47

object ive. T h a t is, if methodology em ph siz es maintainabili ty

s

an object ive, then i t should

also stress th e use of t he six principles :hat ar e related to maintainabil ity.

Exp and ing on one of those prinap les, information hiding, we note the five at t r ibu tes reduce d

coupling, enhanced cohesion, well-defined interfaces, ease of change, and low complexity) that

should be evident in software developed using a process governed by the principle of information

hiding. This set of expected a t t rib ute s is the n com pared to the desired set for correspondence.

n general , a methodology sho uld emphasize th e sam e set of software engineering object ives

derived from project level requirements. T h e methodology should correspondingly stress the set of

principles l inked to those object ives. Additionally, th e expected set of product at t r ibu tes defined

by th e linkages among prin aple s an d at t r ibu tes) should agree with the set deemed m ost desirable

by th e project manager. If the above condit ions are met . then the can dida te methodology is

assum ed t o b e a deq uate relat ive to project level , software engineering object ives.

On assessing the effectiveness of

an

methodology, le t us first observe th e relationship between

par t icu lar a t t r ibute

nd

specific pr o d ~ c t haracteris t ics . Reierring again to

Fiqurr? 3 nd

~ ~ a r r o w i i ~ q

ou r atte ntio n t o one of th e attrib ute s, well-defined interfaces, we identify one set of chara cteristics

related t o the well-defined interfaces at t ribu te. The se characteris t ics form the set of ob sert nblr

propert ies which contribute to t he claim th at a piece of software exhibi ts a w ell-defined interface.

Alth oug h th e prop erties shown a r e only subset of those previously identified [.ARTJSG]. they

represent k t confirmin g and co ntras ting eleme nts. For example, the use of global variables ha s

negat ive imp ac t on well-defined interfaces. T h e use of s truc tured d at a in para me ter cal ls . however.

h s

positive impact.

Hence, t o determine th e effect iveness of methodology one assesses the extent t o which p roduct

at t r ibu tes are present or absent ) , and then propagate the resu lt s o i tha t assessment through the

se ts of linkages defined by th e evaluation procedu re. As discussed by Kearney [ K E x J ~ ~ ]owever.

th at assessment process must be predicated o n val idated metrics .


13/47

5.

A p p l i c a t io n o f t h e E v a l u a t io n P r o c e d u r e

Based on the defined set of linkages among objectives, principles, and attributes, the opera-

tional specification of the evalu ation procedure is guided by two fundamen tal axioms:

1)

the methodology description and project requirements provide standards, conventions,

and gui elines t h a t descri e hour t o

produce

a product, and

(2) the project documentation, code, a nd code documentation reflect ow we l l the develop-

ment process prescribed by the methodology is followed.

A s

described below,

n

application of the evaluation procedure, guided by the above two

oms, illustrates t he utility and intrinsic prower of th e evaluation procedure in assessing the

adequa cy an d effectiveness of a methodology. Provide d in this illustration is a characteriza tion

of the c ompon ents used. in t he evaluation process, an individual assessment of two me th od ol og

descriptions, an analysis of associated products, and a summary of the results. Th e authors

note that a substantial part of the characterization and assessment process is embodied in the

operational aspects of applying the evaluation procedure.

Length restrictions, however, prevent

their discussion. For information on the operationa l aspects the autho rs refer the intereste d reader

t o

[ARTJ86 ARTJ87j.

5.1

ata S o u r c e s

A joint investigatio n of two comparable Navy software development methodologies and respec-

tive produc ts is detailed in [NANR SJ]. Th e investigation effort utilizes:

a

Four so ftware deve lopment m e t h o d o l o ~ ocuments for

1) identifying th e pronounced software engineering objectives. principles. an d a ttribu tej.

and

VPI

age

3of46


14/47

(2) assessing the adequacy of each methodology through the objective/prinaple/attribate

linkages defined by the evaluation procedure, and

Eight software system documents an d 118 routines, comprising 53 sou rce lines of code,

for

(1) determining th e evident set of product at tr ibute s, and

(2) via the

attribute/principle/objective

linkages, empirically assessing th e principles and

objectives emphasized during product development.

The following section provides

a

summary of the results and i l lustrates the uti l i ty

anc

2rsatiiity

of the procedural approa ch t o evaluating softwa re development methodologies. For simplicity. we

refer to the software systems

as

system A and system B

(and

methoddogy A. methodology B

respectively).

5 2 A n a l y z i n g t h X i e t h o d o l o g i c a l D e s c r i p t i o n a n d A s s o ci a te d P r o d u c t

Th e initial step in the evaluation process is to perform a ' top-down" analysis of rnethodolo gies

and B, t o reveal th e set of claimed software engineering objectives. principles, arid attri bute s.

Because bo th m ethodologies have experienced evolutionary development. a clear s ta tement o f t ie i r

respective methodologicai objectives is lacking. Nonetheless, as detailed in Figure

4

the docu-

mentation for methodology 4 does appear to stress the objectives of rrliability and correctness

supported by the principles of structured progmm ming hiemrchical decom position

d

uncticiml

decomposi t ion Following th e objective/principle relationships defined y the eva lua t ion pnce-

dure. for each objective stressed in methodology

A

only three of t he necessary six principles are

emp hasiz ed. The impli cation is, tha t unless th e principles of Life-cycle i-erification. inform ation

hiding an d stepwise ref inement are implicit ly assumed and utilized, correctness and reliability.

are

compromised.


15/47

Figure

Pronounced Objectives. Principles,

and

Xttributes

Using metric values and propert ies , a corresponding bot tom-u pn e~ am in at io n f product x

provides some interest ing resul ts . Th e Kiviat graph displayed in Figure ja i l lustrates t h e extent t

which each at t r ibute is

nssessed as

present in the product . (Att ribu te ratings are restricted to

~ Z

arbitrarily chosen 1-10 scale.) Note th at (r educe d) complexity atta ins the highest rating 8 0 out

o

10.0, closely followed by rea dability (7.4) and cohesion (6.8 ). Based o n th e thr ee principles stressed

i n m e t h o d o l o g

A,

th e evaluation procedure predicts th at (red uce d) complexity, readability. and

cohesion should, in fac t , be am ong the product a t t r ibutes .

n

concert with the stated objectives and principles for methodology A, f ig ure 5b reveal s tha t

s t ructured programming (7.7) is the prominent pnnaple used in developing system

A

fdlowed

by stepwise refinement (6.i), hierarchical decomposition 6.41,

and

functional decomposition 6.4).

Figure 5c depic ts th e results of emphasizing these principles in ter ms of methodology objectives.

In


16/47

Visibility Complexity

of Bchrvior

Interface

ase

of

Readability

Be

Hiemchid

Dccompositim

lmct id

Decomposition

Hid

Progmmhg R e f i e n t

Mainlainability

Figure

Detected Presence of Objectives, Principles, a nd Attri hu tes

particular, reliability is rated as the major software development objective 6.7). . i l though correct-

ness

s

also

stressed by methodology A, ascertaining correctness necessitates life-cycle verification.

This prin dp le is neithe r emphasized by methodology A, nor evident in th e software product . Is

i l lustrated by Figures Sa, 5b a nd 5c, other objectives an d pr inciples are given some em phas is du rin g

th e software developm ent process for system A. It is the auth ors opinions. however. t h a t because

they a re n ot explicit ly stressed in methodology A, the a ssod ated product suffers.

For methodology B, the objectives enun da ted in th e docu mentation are maintai~nabilif

udapl

ability, and reliability. Structured prcgmmming and concumnt documenta t ion

re

t h e e m p k i z e d

principles. Like methodology A however, a complete set of supporting principles are not -tared.


17/47

Hiemrchicnl decomposit ion, junctional decomposition, and to som e extent injon naiio n

hi ing

are

implici t ly assumed

as

underlying principles of methodology B. According to the l inkages among

objectives and pr inciples, l of the above pr inciples (both stated and assumed) are required

to

achieve the objectives exp liat l y state d in methodology B.

Subsequent analysis of product B and a bottom-up n propagation of the results through the

Linkages defined by th e evaluation procedure reveals stru ctu red prog ramm ing

as

the m ost prominent

principle

8.3) ,

closely followed by concurrent documentation

7.0).

hforeover, the evaluation also

indicates tha t the im plia t ly assumed pr inciples of methodology

B

are utilized ste pu ise refinement.

hierarchical decomposition, functional decomposition, and information hiding rate 6.9, 6.7.6.7 and

6.3

espectively. Finally, th e results imply th at d uring th e development of produ ct

B

the objectives

of maintainabil i ty, adaptabil i ty, and rel iabil i ty are most emphasized. The above assessments are

i ll u s tr a te d i n F igure s 5 4 5b, and 5 ~ .

To summarize, the evaluation procedure reveals th at both m ethodologies lack

a

d e a r s t at e m en t

of goals

and

objectives, as well

as

sufficient principles for achieving the ohjectives tha t a re em -

phtsized. hioreover. glaring deficieacies are apparent in both software development methodologies.

T h a t is, both fail to actively sup po rt the principle of information hiding an d also hav e difficulties

in incorporating the desirable at tr ibutes of traceabil i ty and well-defined interfaces in respective

system products. In general , the evaluation procedure does accurately assesses the sof tware engi-

neering

cbjectives, principles, and attri bu tes espoused by methodologies and B. Of particular

significance, however, is tha t the objectives an d principles determined to be 'emphasized du rin g

th e product development process, yet not state d in the methodology docum entation. are precisely

those tha t ar e implici t ly assumed imp ortant by the sof tware engineers developing products Iand

B.

A more detailed account of the evaluation can be found in [NXNR85].

0 onclusion

T d . techniques, environments, and methodologies dominate the sof tware enginer iny: l i t -

erature, but relat ively l i t t le research in the evaluation of nethodologies is evident . T 5 s work


18/47

reports

an

init ial at tempt to develop a procedural approaLh io evaluating software development

methodologies. Prom inent i n this approach are:

1)

a n explication of role of a methodo logy in th e software development process,

(2)

the development of a procedure based on linkages among objectives. principles, and

at t r ibu tes , and

3) t,he establis hme nt of basis for reduction of the subjec tive natu re of th e evalua tion thro ugh

th e introdu ction of properties.

in applica tion of the evaluation pro cedure to two Navy methodologies has provided consistent

results t h a t demo nstrate the u ti l i ty and versatil i ty of the evaluation procedure NANRSS]. Curren t

research efforts focus on the continued refinement of th e evaluation procedure thr oug h

(a ) th e the identif ication and integration of produc t quali ty indicators reflective of attribute

presence, and

( b ) t h e

ccllid tion

of metri cs sup por ting the measure of those indicators.

T he consequ ent refinement of the evaluation procedure offers promise of

a flexible approach that

admits t o change as the field of knowledge ma ture s. In conclusion, the procedural a ppr oac h

presented in this pap er represents a promising path toward th e end goal of objectively evalu ating

softvare engineering methoddogies.

ism

VPI

age

8

d 6


19/47

References

[ALFMS] ALford, M., SREM t the age of

Eight;

T h e Distributed Computing Design System."

*

IEEE Computer

V d .

18, No. 1 Ap ril 1985, pp. 36-54.

(ARTJ861 Art hur , J.D., Nance, R E . and Henry, S.M., "A P rocedural -1pproach to Ev alu atin g So ft-

w re

Development Methodologies: T h e Foundation," T ech nial Report T R - S 2 4 , Virginia

Tech, 1986.

[ARTJ87 Ar thu r, J.D. and Nance, R.E., "Developing an Automated Procedure for E~alua t ing

S o ft xa e Development Methoddogies and Assodated Products. ' T e ch n id Rep ort SRC-87-

007, Systems

Research

Center,

Virginia

Tech, 1987.

[BAUF72] Bauer, F.L. "Software Engineering," Information Pmcesing 71 North Holland Pub-

lishing Company, 1972.

(BERG811 Bergiand, G.D. A Guided Tour of Program Design Me & od ol og id Computer

v01.

14, Xo. 10, Oc tob er 1981, pp . 13-36.

[CHMLSO] Chmu ra, L.J. Norcio, A.f., and Wi ans ki, T.J., 'Evdu atiag Software Design Processes

by Analyzing Change Data Over Time." IEEE T m n s ~c t i o n s n S o f t w a y

Engineennc.

Val

16

No. 7,

J u l y

1990, pp. 729-740.

[FR EP 77] Freeman, P., "The Nat ur e of Design," -4 Tutorial on S o f tw m D esign Techniques.Second

edition, IEEE Computer Society Press, 1977,

pp.

29-36.

[ G A F J a l ] G d e n e y ,

J.

E.,

"Metrics in S oftw are Quality Assurance,'

Proceedings of

t h e

.Yational

AC U Conference November 1981. pp. 126-130.

[GOMHM]Gona a , H. "A Software Design Method for Real-Time S -stems," Communications of

the ACM ol. 27, No. 9, Sep tem be r 1984, pp. 938-9-19.

[HE NK 78] Heninger, K. L., J. W. Kallander, J. E. Shor e, and D. L. Parna s. %o ftw are Require-

ments for the A-7E Aircraft," NRL Me mora ndu m Report 3876. Yaval Research Labocztory,

Washington, D. C., November, 1978.

[JACM75] Jackson, M., Principles of rogram Design London: Academic Press. 1975.

[KEAJ86 Kearney,J., el .

al. 'Software Com plexit y Measurement.' C o m m n n i m t i o m of

fhe

A.C..U. Vo1.29, No. 11, Nov ember 1986, pp. 1044-1050. Yo ntere y,

CX

March 19.37.

pp.

238-252.

R

S-

VPI

1 9 d Y


20/47

[LISB75]

Liskov,

B.

Zlles, S., 5pecification Techniques For Data Abstraction,"

IEEE

Tmnsm-

t i o m o n S of tw ar e E n g i n ec h g

Vol. S E -I , NO.

1

March 1975, pp. 7-19.

[ N A N W ] N ance,

RE.

Arth ur, J.D . and Dande kar, A.V. "E valuation of Software Development

Methoddogies,"

X

Final R eport of t he Immediate Software Development Project, T h e

De-

partment of Computer Saences Virginia Tech, December 1985.

.

[PAR 376] P arn as,

D.

"On the Design and Development of Program

Families, IEEE Tmnsactiorrci

on Software Engineering V d SE-2,

NO. 1

March 1976, pp. 1-9.

(PARD721 Pa rna s, D.. "On the Criteria to be Used in Decomposing Systems into Modules.'

Communimtions

of

the A m ,

ol. 15,

NO.

5, May 1972, pp. 330-336.

[ROSD77] Ross D., "S truc ture d Analysis: A Language for Com mu nicati ng Ideas," I E E E T m m -

actions on Software Enginem'ng, Vol. SE-3, No. 1 January, 1977, pp. 16-34.

[SCOL78] Scott ,

L. .h

Engineering Methodology for Presenting Softw are Functional

.-\rchitec-

ture, Proceedings ofthe hird Internotiona Conference on Software Engineeriny YY

973.

pp.222-229.

[WARJ76] Warnier, J. Logical

C m l r u c t i o n o f

Pmgmms 3rd edition, trans . B. Flanagan, YY:Van

Nostrand Reinhold, 1976.

[W IR N 71 ] W ~ r t h ,

.

*Program Development by Stepwise Refinement,"

Comm unications of

the

ACM

Vol. 14, No.4, Xpril, 1971, pp. 221-227.

Y a u

VPI

Rge dU


21/47

VIEWGR PH

M TERI LS

FOR

THE

R

N NCE

PRESENT TION


22/47

FR MEWORK

FQR

SSESSING THE

DEQU CY ND EFFECTIVENESS OF

SOFTW RE DEVELOPMENT METHODOLOGIES

Presentation

to

the

Fifteenth nnual Software Engineering Workshop

Richurd

E

Nance

James D rthur

Systems Research C enter

and

Department of Computer Science

Virginia Polytechnic

Institute

and State University

Blacksburg Virginia

28

November 199

PRECEDING

P GE BL NK OT FILMED


23/47

TH ORI IN

Immediate Software Development Issues

for

Embedded Systems Applications

in

Surface Combatants

(25 March 5September 1985)

Issue: Multiple Software Development Methodologies

(a)

Review

two software development methodologies A and B)

b)Compare

and

evaluate

A and

c )

Assess

the costs and benefits of:

Continuing witn multiple sofnvare development

methodologizs

Using only one software development methodology

Transitioning to an alternate s o f ~ a r eevelopment

me hodology


24/47

OUTLINE

Evaluation Approach

Objectives Principles Attributes Framework

Developm ent of

an

Evaluation Procedure

Software Engineering and Software Development

Structured Evaluation Procedure

Data Sources

Application of th Evaluation Procedure

Summary of Sam ple Data

Illustration of Procedure Application

Summary of Results

Future Work


25/47

EV LU TION PPRO CH

1

Develop

n


hat is a methodology?

How

can they be

compared?

2 pply

the


In consonance with our Navy sponsor and with

Contributions from software development sites

and

oversight agencies.


26/47

O METHODOLOGES

W hat is a methodology?

methodology is

a

collection of complementary methods,

an d a set of rules for a.pplying them. More specifically, a

methodology

1) organizes and structures tasks comprising the effort to

achieve a global objective, establishing the

relationships am ong tasks,

2) defmes methods for accomplishing individual tasks

within the context of the global objective),

and

3) prescribes an order in

which

certain classes of

decisions are made,

and

ways of making those

decisions that lead to the desired objective.


27/47

RATIONALE FOR

TH EVALUATION

PROCEDURE

A set of ob ie ct iv es can be identified that include those

postulated by any software engineering methodology.

A

methodology defines those principles that characterize a

proper and appropriate development process. Adherence to a

process governed by these principles should result in a

prod uct program s and docum entation) that possesses

attributes considered desirable and beneficial.

Philosophy tempered by practical concerns:

1) Sets of objectives, principles, attributes areopen.

2) Prioritization of objectives recognized.

3)

Com ponents of large software system vary attribute

sampiing.

4)

Flexible application

of

evaluation procedure

consonant with project objectives.


28/47

FRAMEWORK FOR SOFTWARE DEVELOPMENT

O

ECTNES

Maintainability

Correctness

Reusability

Testability

Reliability

Portability

Adaptability

E Imaus

Hierarchical Decomposition

Functional Decomposition

Information

Hiding

Stepwise Refinement

Structured

rog raming

Life Cycle Verification

Concurrent Documentation

llln m

Reduced

Coupling

Enhanced Cohesion

Reduced Complexity

Well Defmed Interfaces

Readability

Ease of Change

Traceability

Visibility o Behavior

Early E m r Detection

PROCESS

PRODUCT

DOCUMENTAnON +) PROGRAMS

Properties

r n u T E S

Y a m

VP l

p ge l


29/47

PROCEDURE DEVELOPMENT

1 Identlfy Objectives

What qualities are desirable

2

Define Principles

How

are desirable qualities obtained

3 ink Principles to Objectives

Which principles contribute to each objective

4 Defme Resulting Attributes

Use of a principle induces what desirable attributes

5

Defme Properties Associated with Attributes

What properties give evidence of attribute presence or

absence

How to

measure properties


30/47

PRIMARY SOFTWARE

ENGINEERING

OB.TECTM3S

1)

Adaptability -

the ease with which software can

I

accomm odate to changing requirements

2) Correctness -

strict adherence to specifications

I

3)

Maintainability

-

the ease with which corrections

can e made to respond to recognized

inadequacies

I

4) Portability -

the ease

in

transferring software

to

another host environment

5)

Reliability

-

the error free behavior of software over

time

6)

Reusability -

the use of developed software

in

other

.

applications

.

4

7) Testability -

the ability to evaluate conformance

.

with specifications


31/47

PRIMARY SOFTWARE

ENGINEERING

PRINCTPLES

1)

Abstraction

defming each program segment at a

given level of refmement

a)

Hierarchical ecomposition

- components

defmed in a top-down manner

@ Functional ecomposition

components

partitioned along functional boundaries

2) Concurrent Documentation -management of

supporting documents system specifications, user

manuals, etc) throughout the life cycle

3)

Information Hiding

insulating the internal details

of com ponent behavior

4) Life Cycle Verification -

verification of

requirements throughout the design, development,

and maintenance phases of th life cycle

5) Stepwise Refinement utilizing convergent design

6) Structured Programming .:sing a restricted set of

program control constructs


32/47

OBJECTIVES PRINCIPLES LINK GES

daptability

Correctness

Maintainability

Portability

Reliability

Reusability

Testability

Concurrent Documentation

Functional Decomposition

Hierarchical Decomposition

Information Hiding

Life Cycle Verification

epwise Refinement

Structured

Programming


33/47

PRIM RY SOITWARE

ENGINEERING

ATTRIBUTES

1)

Cohesion The binding of statements within a

software component

2)

Complexity an abstract measure of work

associated with a software component

3)

Coupling the interdependence among software

components

4)


indication of faults

in

requirements, specification and design prior to

implementation

5 )

Ease of Change software that accommodates

enhancements or extensions

6)

Readability

the difficulty in ~nderstanding

software component

7) Traceability

the ease

in

retracing the complete

history of a software component from its current

status to its design

8) Visibility

of

Behavior

the

provision of a review

process for error checking

9)

Well-Defined Interfaces

the definitional clarity

nd

completeness of a shared boundary between

software and/or hardware software/software,

softw arehardware)


34/47

PRINCIPLES

TTRIBUTES

LINK GES

Documentation

Early

Enor Detection

Information Hiding

Well Defined Interfaces


35/47

ILLUSTR TION O

TH

EV LU TION PROCEDURE

Use

of lobal

1

Variables

xcessive

of arameters

Use

ofData


36/47

SETS

OF

DEFINED

LINK GES

a m

VPl

PalcUd 6


37/47

THE OPA FRAMEWORK FOR EVALUATION

SUMMARY

Fundm en ta l to the evaluation procedure are several sets of

linkages:

Linkages

Defined

Substantiated

Objectives

/

Principles

33).

33)

Principles

/

Attributes 24)

24)

Attributes

/

Properties

(125)

1 14)

66 Automatable

Assessing the adequacy of a methodology is achieved through a

op-down evaluatior? process.

Assessing the

effe tiveness

of a methodology is achieved through

a bottom-up evaluation process.

R S a m

VPI

Page

6 4 6


38/47

APPLICATION O F

THE

PROCEDURE

SUMMARY OF SAMPLE DATA

Documents Pr imary

A

The Combat System Developmext Plan

The Computer Programming M anual

The

rogram

Development

anual

SixNumbered Documents PDS, IDS)

Functional Description Document

Two

Numbered Documents PDS, IDS)

Source

Code:

A: Routines 17

SysProcs

2

Routines

99

SLOCS 117

SLOCS

137

SLOCS 5729


39/47

D T

SOURCES

ND

IMPLIC TIONS

Methodology

Description Standards Objectives

onventions do it Principles

Project Guidelines

Requirements

roject

Documentation

PPS

I S

How well {Principles

P S is it done

ttributes

ode and

Programs

ode Documentation


40/47

N ACCUMULATION OF

EVIDENCE

Demonstrating that software possesses a desired attribute

(or does not) is not a proof exercise; rather, it resembles

an exercise

in

civil litigation in that evidence

is

gathered

to

support oth contentions (the presence or absence) and

weighed on the scales of comparative judgement.

Offsettin

Measurement Scale


41/47

ELEMENTS METRICS

AND PROPERTIES

Relationship

Elements

Subjective

-

Code Example

Documentation

Cil)

of Distinct

Parameterless

Calls

Example

Awareness

of

V V

Y2)

of Distinct

Calls


42/47

ASSESSING METHODOLOGICAL EFFE TIVENESS

(ATTRIBUTES)

Coupling

Cohesion

Complexity

Well Defmed Interfaces

Readability

Ease of Change

Visibility of Behavior


Traceability

th

have difficulty with Traceability

and

Well Defied

Interfaces

u t

VP

4 d 6


43/47

Coupling

Ease of Readability

Change

K M T GR PH FOR

Methodology

Mc~lrodology


44/47

Hierarchical

Decomposition

oncumnt

/

I

Information

Lifccyclc Hiding

Verification


45/47

Maintainability

KIVI T

GR PII

FOR O D J E m

Methodology

Methodology


46/47

R SULTS

O PROCEDURE PPLIC TION

Assessing Methodological Adeauacv

A:

Stresses Objectives of Reliability and Correctness

Emphasizes Principle of Structured Program ming

Methodology A was (and is) an evolving

m e hodology

Stresses Objectives of Mainrainability,

Adaptability, Reliability,

and

Correcmess

Emphasizes Principles of Modular Decomposition,

Structured Programing and Concurrent

Documentation

At the Objectives level, both methodologies support stated

project objectives.

t th

Principles level, both methodologies lack the

enunciation of proper Principle usage.

No reference to desired Attributes is found


47/47

FUTUR

RESE RCH

pplying the Evaluation Procedure to

Sofbvare Quality ssurance

Predicting and/or assessing software quality necessitates

Systematic

approach to

s s ~ s s ing

product or process) confo rm a~ cewith

cceptance criteria

standards and guidelines)

The Evaluation Procedure

Currentiy

supports

a well-defmed, svstematic

approach for evaluating software products, and

Provides a rigorous framework for

Relating acceptance criteria based on attributes

to software engineering principles and

objectives, and

Defining acceDtance levels based on m easures

reflecting the achievement of objectives,

adherence to principles

nd

realization

of

attributes