frestimate software quick start guide ann marie neufelder softrel, llc [email protected]
TRANSCRIPT
Helpful information
Press F1 key at any time to see relevant help Mouse over fields to see tooltips Electronic copies of the user’s manuals can be found
at http://www.softrel.com/support.htm
Tips for Installing FREstimate Install on a recent Windows Operating System If you are installing onto Vista
a. Save the installation file to your hard drive instead of installing from the internet b. Right click on the download file and select "Run as adminstrator“ c. After installation you will need to download this application to support the
Frestimate help files http://www.microsoft.com/downloads/details.aspx?FamilyID=6ebcfad9-d3f5-4365-8070-
334cd175d4bb&DisplayLang=en Shut down all other programs prior to installing Log in to Windows as a user with system admin privileges as the install process
requires write access to the windows/system32 folder Do not install on a network drive or any drive that you do not have write
privileges for It is recommended that you install onto a “C:” drive
If you notice any error messages during installation, write them down and continue with the install. You may notice error messages if you are installing over a previous version of Frestimate.
After the software is successfully installed, you can launch it from Windows Start->All programs or by launching the FREstimate icon from the folder that you installed to. Default install folder is c:/Frestimate
Definitions
All definitions and formulas are defined in the technical manuals and help files Some help files are not provided with the
evaluation edition Press F1 to see the help file containing all
formulas The formulas and inputs are summarized in
the next few pages There are also wizards to help you
understand the reliability prediction inputs
Definitions
Software Reliability is a function of Inherent defects
Introduced during requirements translation, design, code, corrective action, integration, and interface definition with other software and hardware
Operational profile Duty cycle Spectrum of end users Number of install sites/end users Product maturity
Definitions
Prediction models versus reliability growth models Prediction models used before code is even
written Uses empirical defect density data Useful for planning and resource management
Reliability growth models used during a system level test
Extrapolates observed defect data Used too late in process for most risk mitigation Useful for planning warranty/field support
Definitions
Defect density Normalized measure of software defects Usually measured at these 2 milestones
Delivery/operation also called escaped or latent defect density
System level testing Useful for
Predicting reliability Benchmarking Improving efficiency and reducing defects
KSLOC – 1000 executable non-comment, non-blank lines of code
EKSLOC – Effective size adjusting for reuse and modification
Basic Formulas Normalized size – Size normalized to EKSLOC of assembler via use of
standard conversion tables Delivered Defects (Ndel)= predicted normalized size * predicted delivered
defect density Critical defects = delivered defects * ratio of defects predicted to be
critical in severity Testing defects (N0) = predicted normalized size * predicted testing
defect density Interruptions = (Ratio of restorable events to all others) * Total predicted
defects Restorable event - Usually the definition of an interruption is based
on time in minutes (i.e. if the system can be restored in 6 minutes than it’s an interruption)
Critical interruptions = interruptions * ratio of defects predicted to be critical in severity
Basic Formulas MTTF (i) – Mean Time To Failure at some period in time i=
T/ (N (exp (-Q/TF)*(i-1))-exp((-Q/TF)*(i) ) N = total predicted defects Q = growth rate TF = growth period (approximate number of months it takes for all residual defects to
be discovered) T = duty cycle for period i (this can be > 24/7 if multiple sites)
MTTCF (i) – Mean Time To Critical Failure Same formula as MTTF except that Critical defects is substituted for N
MTBI (i)– Mean Time Between Interruptions = Same formula as MTTF(i) except that N is substituted by predicted Interruptions
MTBCI (i)– Same formulas as MTTF(i) except that N is substituted by predicted critical interruptions
Failure Rate (i) = 1/MTTF(i) Critical Failure Rate(i) = 1/MTTCF(i) Interruption rate (i) = 1/MTBI(i) Critical interruption rate (i) – 1/MTBCI(i)
Basic Formulas End of Test MTTF = T/N End of Test failure rate = N/T Reliability(i) = Exp(-mission time * critical failure rate(i))
Mission time -duration for which software must continually operate to complete the mission
Availability(i) = MTTCF(i) / (MTTCF(i) + MTSWR) MTSWR = Weighted average of workaround time, restore time and repair
time by predicted defects in each category Average MTTF – Average of each point in time MTTF(i) over this release
Similarly for the average MTTCF, Availability, Reliability, failure rate, critical failure rate, MTBI, MTBCI
MTTF at next release – Point in time MTTF for the milestone which coincides with the next major release. Similarly for the MTTCF, Availability, Reliability, failure rate, critical failure
rate, MTBI, MTBCI at next release
Example of growth over a release
Industry Model C:\PROGRAM FILES\FRESTIMATEMANAGER'S EDITION\SAVEFILE2222.mdb
MTTFin hours
Months after delivery
0
200
400
600
800
Release milestone
Next scheduled
major release
Average MTTF is
average of all of these
MTTFs
Overview of Software Reliability Prediction and Management Process
12
Step 1Completedetailed survey
Step 1Completedetailed survey
Score
World class
Distressed
Very goodGood
AverageFair
Imparied
Where you’d like your project to be
Step 3. Identify gaps between your survey responses and average responses for next percentile group
Where your project is predicted to be now
Step 4 - Assess for each gap•Existence of all prerequisites•Relative startup cost•Relative startup time
Step 5 – Mitigate gaps with most efficiency
Step 6. Compare cost investment of implementing selected gaps vs. tangible and intangible cost savings of shipping about half as many defects and being late about 25% less often
.011
2.069
.060
.112
.205
.6081.111
Percentile
10%
100%
20%25%36%85%
100%
NormalizedFieldedDefectDensity
Probabilitylate
deliveryWhen improving to
next percentile•Average defect reduction = 55%•Average p(late) reduction = 25%
Step 2. Predict current •defect density percentile•defect density•probability of late delivery
Starting up Frestimate
After you launch FREstimate you will see the license agreement.
Once you accept the agreement you will see the Frestimate Main Menu.
The File Menu is enabled so that you will open an existing FREstimate file or create a new one.
The very first thing you do whenever you launch Frestimate is open or create a project file.
Step 1. Open a file
This is the first thing that you will see after accepting the license agreement.
The evaluation edition does not permit creation of new files. Select File and then open the demoprog.mdb file
Step 1. Open a File
When you open an existing file the results page will be populated as shown here.
Step 1. Main results page with new file
If you are using either the Standard or Manager’s edition this page will be displayed after you create a new project. The results are not populated until a prediction of the effective size is input using the General inputs button.If you are using the evaluation edition, you will not see this view.
Step 2. Enter General inputs and size
When starting a new prediction, you will need to enter a size prediction to see any results. The other inputs have default values which should be reviewed and modified.
There are wizards to help you enter these inputs.
If you are using the evaluation edition, the size has already been filled in for a real example.
Surveys
Select a prediction model and then select the “Survey Inputs for this Model”.
You will then be directed to the survey for the selected model.
Types of surveysModel Number of
inputsKey strengths/weaknesses
Historical model Very few If data is recent and similar, usually most accurate. Most organizations do not have any.
SEI CMMi lookup chart 1 Lookup table of average defect density by SEI CMMi level. Quick but not most accurate.
Industry lookup chart 1 Lookup table of average defect density by application type. Quick but not most accurate.
Shortcut model 22 Relatively easy to answer but more accurate than lookup tables. Can be useful for tradeoffs.
Full-scale model Between 96 and 300
Requires work to complete. Best model for doing tradeoffs/improvements.
Closest DB match 96 Find the project in our DB which is most like yours. The database contains many different types of projects, but if none are similar another model should be used.
SEI CMMi level lookup table
Select the SEI CMMi model from the main pull down menu and press the “Survey Inputs for this Model” button. Then select which of the SEI CMMi levels pertains to this organization. The results are then updated according to your selection.
Industry type lookup chart
Select the industry model from the main pull down menu and press the “Survey Inputs for this Model” button. You be shown the general inputs page. Go to the application type field and select the industry or application type that best fits this application. The results are then updated according to your selection.
About the Shortcut and Full-scale Surveys ALL prediction surveys were developed by a research organization that
collected and organized lots of defect data from many real projects SoftRel, LLC has been collecting this since 1993 on more than 100
real software projects More than 600 software related characteristics Actual fielded and testing defects observed Actual normalized size Actual capability for on time releases Relative cost and time to implement certain practices
All surveys were developed using traditional statistics and modeling Predictive models are not novel The only thing that is relatively novel is applying them to software
defects
The Shortcut model surveyThis is the first of 2 pages for the Shortcut Survey model. The questions are in 2 categories – opportunities and obstacles.
The defect density is predicted by how many of each you check yes.
The prediction formula can be viewed by pressing the Help button.
One page of the Full-scale survey
This is one page of the Full-scale model survey.
Some surveys have one question, some have a few questions and some have many questions.
Step 3. View results, profiles, trends
The results will be populated once you have entered a size prediction. They will stay populated from that point onwards.
The tables shown here map to the data flow diagram that we saw previously.
The results are filtered by criticality.
View profiles
All of the profiles that we saw on the data flow diagram can be viewed by pressing the appropriate button.
A profile is a metric with respect to some particular point in time.
View trends
Press the Trends button. Select any one of the trends from the list.
The trends are graphical representations of the profiles and results.
You can save them as a bitmap or copy to clipboard or print.
Step 4. Tailor the results page
If you are only interested in a few of the resulting metrics, you can pick and choose which ones to hide/show by selecting the “Filter Report” button
Step 5. Generate a formatted report or print the results page
You can generate a formatted report (.txt, spreadsheet, word processing) by selecting the “Reports” button. You can print an exact image of this page with the “Print” button.
This feature is disabled in the evaluation edition.
Step 6. Compare the results to others in our DB
Once your prediction is complete you may want to compare it that of projects that are most similar to yours.
This feature is disabled in the evaluation edition.
Compare your prediction to actual defect density from projects similar to yours
This is your prediction
These are actual defect
densities from other
organizations like your
Step 7 – Review cost scenarios
If you have completed the shortcut and full-scale surveys, you can see the quantitative impact of certain improvements
Cost scenariosThis feature displays the answers that you entered for the surveys. You can sort the survey questions based on relative cost, schedule time, impact and correlation to defects. You can then create a scenario to move to the next percentile prediction using the most optimized set of changes.
This is the Manager’s edition view. The standard edition has a basic view only interface. This feature is disabled in the evaluation edition.
Cheat sheet for fastest way to improve by 1 percentile group
% Suggested techniques for fastest /cheapest way to transition to next percentile
Predicted Average Defect Reduction
99 Remove obstacles so that there is no more than 3 obstacles and at least 1 opportunity. Increase survey B score by about 6 points.
46%
90 Remove one additional obstacle. Increase survey B score by about 6 points.
44%
75 Make sure that the number of obstacles and opportunities are about even. Increase survey B score by about 4 points.
60%
50 Increase survey A score by at least 10 points. Make sure that opportunities outnumber obstacles by at least 2.
55%
25 Remove all obstacles. Make sure that there are 3 or 4 opportunities. Increase survey A score by about 3 points.
47%
10 Remove all obstacles. Make sure that there are at least 5 opportunities.
81%
Key Practices to embrace by percentile group
35
•Get control of changes and versions•Get a comprehensive test plan (versus ad hoc testing)•Independently test every change•Track and record all defects and changes
Key practices are cumulative
None of the world class organizations skipped the practices at the bottom or
middle •Get all parts of lifecycle in place from requirements to support•Review and prioritize changes
•Maintain domain expertise
•Formalize unit testing with non-peer review•Define “shall nots”•Measure line or branch coverage
•Write test plans before code written•Testers involved in requirements definition•Require developer unit testing•Plan ahead (predict size, defects, resources)•Collect field data for predicting next project
Based on actual benchmarking
results vs. opinion
Group
World Class
Very Good
Average/Good
Fair
Ugly/Bad
Key gaps to avoid by percentile group
36
•Wrong coding standards•Reinventing wheel•Using short term contractors for line of business code•Testers come on project at 11th hour•Using automated tools before you know how to perform the task manually
Group
World Class
Very Good
Average/Good
Fair
Ugly/Bad
•Too much focus on coding, not enough focus on everything else•Old code not protected/managed well•Unsupported Operating Systems/Compilers
•“Big blobs” – large executables, versions, projects•Incorrect application of life cycle models•Failing to define “shall nots”
Eliminate obstacles from the
bottom first
Based on actual benchmarking
results vs. opinion
Step 8. Enter testing/growth data (Manager’s edition)
When you press this button you will see the main menu for the reliability growth models which are used exclusively during a system level test or later.
Step 9. Enter fielded data (when available)
Once fielded data becomes available, you may want to enter it here. This is the ultimate verification of the predictions that you did earlier in the life cycle.