Menzies/Hihn - 1

STARSeeking New Frontiers in Cost Modeling

Tim Menzies (WVU)

Jairus Hihn (JPL)

Oussama Elrawas (WVU)

Karen Lum (JPL)

Dan Baker (WVU)

22nd International Forum on COCOMO and Systems/Software Cost Modeling (2007)

Menzies/Hihn - 2

STAR

STAR has three key advancements over traditional methods and even 2cee

– Provides an integrated set of COCOMO models• COCOMO II

• COQUALMO

• COCOMO II Risk (threats) Assessment Model

– Can be used to systematically analyze strategic and tactical policy decisions• Searches for optimal combination of inputs that jointly reduce effort, defect rates

and threats

• Uses constraints to restrict search

– Free, Floating, Fixed

– Can be tuned/calibrated with constraint sets instead of traditional historical data records

• Seek stable conclusions in space of all tunings

• Abduction: View it as an alternative to Bayesian methods based

STAR is an abductive inference engine that applies simulated annealing to a treatment learner (TAR)

Menzies/Hihn - 3

Note• This talk is an extension of

material presented in “The Business Case for Automated Software Engineering”

– IEEE ASE 2007

– Menzies, Elwaras, Hihn Feather, Madachy, Boehm

http://menzies.us/pdf/ 07casease.pdf

Menzies/Hihn - 4

Method

• Stagger across the space of known tunings and inputs (Monte Carlo)

• For N staggers, score N runs by an index we call energy:

– Ef = (effort - minEffort ) / (maxEffort - minEffort)

– De = (defects - minDefects ) / (maxDefects - minDefects)

– Th = (threats - minThreats) / (maxThreats - minThreats)

• Save the one with lowest energy index

normalization0 <= x <= 1

Menzies/Hihn - 5

How to Stagger• Simulated annealing (Von Neuman)

– Pick input ranges and internal values at random

– Do many runs • starting from “boiling hot”

(when you stagger around like a

drunk)

• to “cooler” (No staggering walk straight to your destination)

• Keep track of multiple solutions– Current– New– Best

Sample runs from STAR

(after 500 runs, little improvement)

Bad

Good Best 10%

Menzies/Hihn - 6

Staggering the Tunings Range of effort multipliers (COCOMO)

• COCOMO effort estimation

– Effort multipliers are straight (ish) lines

– when EM = 3 = nominal…• multiple effort by one (I.e. nothing)

– i.e. they pass through the point {3,1};

cplx, data, docupvol, rely, ruse,stor, time

Increase effort

acap, apex, ltex, pcap,pcon, plex,sced,

site,toool

decrease effort

Menzies/Hihn - 7

After staggering,select best things

• Sort all ranges by their “goodness”

– Try the first ranked range,

– Then the first and second,

– Then the first and second and third

– And so on

• Seek the “policy”

– The fewest ranges

– that most reduce • threats,

• effort,

• defects

Bad

Good

22 good ideas

38 not-so- good ideas

Menzies/Hihn - 8

Staggering the inputs : 5 different ways

1. COCOMO II: stagger over entire model input space

“Values” = fixed

“Ranges”= Loose (select within these ranges)

1

2 4

53

Menzies/Hihn - 9

All Flight Groundpmat = 5 pmat=3 ltex=4 acap = 5 ltex=4 plex=4 pcap = 5 plex=4 site=6 apex = 5 automatedanalysis=6 peer reviews=6 ltex = 5 site=6 automatedanalysis=6 prec = 4 apex=5 pcon=5 plex = 5 peer reviews=6 prec=5

site = 6 execution testing and tools = 6

execution testing and tools = 6

peer reviews = 6 pcon=5 apex=5 execution testing and tools = 6 prec=5 acap=5 automatedanalysis=6 acap=5 pcap=5 pcon=5 pcap=5 pmat=2 pcap=4 prec=4

Strategic

Making Strategic Decisions

Full range of modelConstrained by Jairus’ guess

at JPL environment

Menzies/Hihn - 10

Results : OSP

• One advantage of this output display

– If you can’t accept the full policy…

– … you can see what trade-offs arise with some partial policy

• But partial polices cannot include many choices. For example note the missing values:

– Peer reviews < 6

– Execution testing & tools < 6

– Automated analysis < 5

Menzies/Hihn - 11

Results: OSP2• OSP2 was a more constrained

environment as it was a follow-on from OSP and ‘inherited’ the – Team– Development Environment– Design– Etc.

• Again note the missing values:– Peer reviews < 6– Execution testing & tools < 6– Automated analysis < 6

Menzies/Hihn - 12

Results: all experiments• No point in half-hearted defect removal

– Never found in any policy• Peer reviews in 1,2,3,4

• Execution testing & tools in 1,2,3,4

• Automated analysis in 1,2,3,4

• Beware spurious generalities

– X= one of {cocomo or osp or osp2 or flight or ground}

– Y= one of {cocomo or osp or osp2 or flight or ground}

– Not(X = Y)

– X’s best policy is not Y’s best policy

– Exception …

• … Use more automated analysis (model checking, etc)

– Automated analysis = 5 or 6 always in best policy

Menzies/Hihn - 13

Calibrating/Tuning Models

• Traditional Approach Current cost models are tuned to local contexts– LC (Boehm, 1981)

• Tuned to local data using LC– Hard to tell when old data no longer locally relevant

• Suffers from the “large outlier problem”• Row pruning done heuristically

• Traditional approach Next Step– 2CEE (Menzies, Jalali, Baker, Hihn, Lum, 2007)

• Tuned to local data using LC • Tunes and validates every time it runs• Tames outliers primarily with column pruning• Uses Nearest Neighbor for row pruning

– Not all flight software is equal– Culls old data that is no longer relevant

– Both of these approaches require you get more data which may be hard to obtain

• STAR– Current research results suggest that we may be able to estimate almost as well without local

data and LC.• Use est vs actual instead of energy as evaluation metric

– Constrain parameter ranges based on • project being estimated• knowledge of what typically varies in your environment

– Assumes basic COCOMO tunings are ‘representative’• Seems reasonable

Menzies/Hihn - 14

Comparisons

Mre = abs(predicted - actual) /actual

Diff = ∑ mre(lc)/ ∑ mre(star)

“” same at 95% confidence (MWU)

“” same at 99% confidence (MWU)

Very little difference– Half the time: insignificantly different– Otherwise, median diffs = +/- 25%

Why so little difference?– Most influential inputs tightly constrained– Most of the variance comes from uncertainty

in the SLOC,• Not from noise of internal staggering

diff same

diff diff

same same

diff diff

same same

∑ mre(lc) / ∑ mre(star) strategic tactical

ground 66% 63%

all 91% 75%

OSP2 99% 125%

OSP 112% 111%

flight 101% 121%