Download - Metrics Selection Guidelines
Metrics Selection Guidelines
Metrics
S.No Metrics Type Metrics Name Explanation
1 Project Metrics Schedule variance(%)
2 Project Metrics Effort variance(%)
A metric is a mathematical number that shows the relationship between two variables. It is a quantitative measure of the degree of which a system, component or process, possesses a given attribute.
Actual calendar time is compared with the planned calendar time and the Schedule variance is calculated.
To Note :a. Planned Calendar days is as per the Plan and Actual Calender days should be the actuals spent. b. This is done for both completed and ongoing states. When ongoing, an approximation is made by estimated days to completion and % completed.
Actual effort is compared against the planned effort and the effort variance is calculated.
To Note :a. Planned effort is the effort estimated upfront. b. Actual effort is the effort as per the actualsc. This is done for both completed and ongoing states. When ongoing, an approximation is made by estimated days to completion and % completed.
3 Product Metrics Size variance
4 Project Metrics
Actual size is compared with estimated size and Size variance is calculated. Project size is the total size of the project measured in terms of LOC or Function points or Feature Points.
To Note :a. Business model governs when the size measure is done - during proposal or after requriements b. Use the same measure for planned and actuals.
Requirement Stability Index
Requirement changes occur as the project progresses. Requirement changes are logged and traced to design, code and test cases. For every change in requirement, impact analysis is done for schedule and effort. Requirement stability index is calculated to indicate the stability of the requirements
To Note :a. Always Requirement changes have to be updated. B. All defects raised in reviews or testing with type 'RC' have to be incorporated separately in Change Track sheet and closed to completionc. Changes need to be grouped and counted. Similar changes need to be grouped and this decision is done by the PM/PL and person responsible for Requirements change management in the project. d. Higher the stability index value, the higher is the stability of requirements. As it nears zero (or) negative values, it shows requirements are unstable
5 Product Metrics
6 Product Metrics Productivity in coding
7 Product Metrics
8 Product Metrics
Productivity in project(Size measure FP or Feature / Day)
Productivty of the project is the rate at which the project is made. Total size of the project is factored against the total effort and this is calculated. This is done at the end of the project generally. Midway through the project, there can be an estimation done on size completed and factored against the effort put in.
To Note:Producitivty is done as per the size measure. Comparison between projects should be made in similar measures.
Coding Productivity is measured by the size of the code (LOC) and the effort spent on coding.
To Note :a. Group it as per Development environment if applicable
Productivity in Test Case Preparation
Test Case Preparation Productivity is measured by the size of the test case (number of test cases) and the effort spent on preparing the test cases.
To Note :a. If there are test cases, test data or test programs, have different measures for them.
Productivity in Test Case Execution
Test Case Execution Productivity is measured by the number of test cases executed and the effort spent on executing the test cases.
To Note :a. If there are test cases, test data or test programs, have different measures for them.
9 Product Metrics
10 Project Metrics
11 Project Metrics
12 Project Metrics
Productivity in Defect Detection
Defects occur in projects as a result of reviews and testing.Productivity in Defect Detection is measured by the total number of defects detected as a result of reviews and testing and the effort spent on reviews and testing.
Productivity in Defect fixation
Defects occur in projects as a result of reviews and testing.Productivity in Defect Fixation is measured by the total number of defects fixed as a result of reviews and testing and the effort spent on fixing the defects found during reviews and testing.
Schedule Variance across phases
Actual calendar time is compared with the planned calendar time and the Schedule variance is calculated for each phase.
To Note :a. Planned Calendar days is as per the Plan and Actual Calender days should be the actuals spent for phase. b. This is done for both completed and ongoing states. When ongoing, an approximation is made by estimated days to completion and % completed.
Effort Variance across phases
Actual effort for a phase is compared against the planned effort for the phase and the effort variance is calculated for the particular phase.
To Note :a. Planned effort is the effort estimated upfront for phase. b. Actual effort is the effort as per the actuals for phase.c. This is done for both completed and ongoing states. When ongoing, an approximation is made by estimated days to completion and % completed.
13 Project Metrics
14 Project Metrics
15 Project Metrics Cost of Quality
16 Project Metrics Cost of Poor Quality
Effort distribution across phases
Project is planned as per the Phase – Task – Activity hierarchy. Planned effort and actual effort are recorded for each activity, task and phase. Phases for a project are decided during commencement of the project. Effort distribution across phases is calculated at the end of the project and acts as an input to future projects, further planning and resource allocation.
To Note : Give the distribution as per the Project phases. There could be a few projects where the phases are not applicable. Let them not be shown with a distribution of 0.
Effort Distribution across Activity Types
Project is planned as per the Phase – Task – Activity hierarchy. Planned effort and actual effort are recorded for each activity, task and Activity Type. Activities are grouped by Activity Types as per the organization standard.
To Note:This metric could be very useful when you have to measure the effort spent on a few heads like Measurement, Miscellaneous, Quality assurance etc which do not exactly fall under a phase.
Cost of Quality is the total effort spent for Testing, Reviews, QA, Measurement, Training and Configuration Management
Cost of Poor Quality is the total effort spent for rework done for reviews, testing and verification.
17 Product Metrics Defect density
18
19 Product Metrics Defect age
20 Project Metrics Review Efficiency(%)
21 Project Metrics Testing Efficiency(%)
Defects occur in projects as a result of reviews and testing. Defect density for the project is defined as the total number of defects per project size (KLOC/FP).
To Note :Could be at project level or at work product level. It is just the same formula with different applicability.
Residual Defect Density
RDD is defined as the no of defects caught in Acceptance per project size (KLOC/FP).
Defect age is the difference between the Phase detected and Phase introduced. Defect age is calculated for a defect and the average defect age is calculated for all the defects.
To Note :Phase detected has to be the phase sequence of the project. Phase introduced is also the sequence of the phase and should identfiy clearly which phase would have introduced this defect
Defects are detected via two detection mechanism, Reviews and Testing. Review efficiency is the number of defects detected in Reviews compared to the total defects. This metric along with the Phase occurred could determine the strength of the review process.
Defects are detected internally in reviews and testing. Testing efficiency indicates how many defects are detected in Testing before being detected at customer site.
22 Process Metrics
23 Process Metrics
24
25 Process Metrics
26 Process Metrics
Overall Defect Removal Efficiency(%)
Defects are detected internally in reviews and testing. Defect Removal efficiency indicates how many defects are detected before being detected at customer site. This inclusive of Review and Testing.
Phasewise Detection Effieincy
Defects detected, occurred and escaped in each phase are calculated and phasewise detection effiency is arrived at.
Defect Distribution across phases
Defects are analyzed based on various phases. Number of defects in each phase is proportioned with the total number of defects and the ratio is given. This metric is applied on the total defects in the project in reviews and testing.
Defect Distribution across Origin
Defects are analyzed based on Origin. Number of defects for each item in the classification is proportioned with the total number of defects and the ratio is given. This metric is applied on the total defects in the project in reviews and testing.
Defect Distribution across types
Defects are analyzed based on various types. Number of defects for each item in the classification is proportioned with the total number of defects and the ratio is given. This metric is applied on the total defects in the project in reviews and testing.
27 Process Metrics
28 Process Metrics
29 Process Metrics
30 Process Metrics
31 Process Metrics
32 Process Metrics
33 Process Metrics
Defect Distribution across causes
Defects are analyzed based on various causes. Number of defects for each item in the classification is proportioned with the total number of defects and the ratio is given. This metric is applied on the total defects in the project in reviews and testing.
Audit Execution Effectiveness
Audit Execution Effectiveness indicates the no of audits conducted versus planed
Process Complaince Index
process complaince indicates the adherence of the projects to the project process established
SEPG Execution Effectiveness
SEPG Execution Effectiveness indicates the no of QMS Release Planed versus the Actual no of releases
% of Improvement suggestions addressed
Indiactes the no of improvement information addressed
% of deviations for a specific process
No of deviations requested by projects are collected to idnetify
Average Time for deployment per QMS release
Mean Time period taken by projects to incoporate the changes given in the QMS release
Metrics Selection Guidelines
Formula Calculation Tips
A metric is a mathematical number that shows the relationship between two variables. It is a quantitative measure of the degree of which a system, component
Schedule variance = ((Actual Calendar days - Planned calendar days) + Start Variance) / Planned calendar days * 100
1. Original planned calendar days for the project was 200 calendar days2. Actual calendar days for the project was 225 calendar days3. Start variance is the difference between Planned and Actual Start4. Start Variance was 5 calendar days5. Schedule variance = 15.00%The above measure is for the project as a whole.
Effort variance = (Actual Effort - Planned Effort)/Planned Effort * 100
1. Planned Effort for the project was 1000 person days2. Actual Effort for the project was 1125 person days3. Effort variance = 12.50 %
Size variance = (Actual size - Estimated Size) / Estimated Size * 100
Estimated Size is 800 FPActual Size(FP/KLOC) is 1000 FPSize variance = 25.00 %
RSI = 1- ((No of changed + No of deleted + No of added) / Total no of Initial requirements) *100
No. of Changed is 6No. of Deleted is 2No. of Added is 4Total No. of Requirements is 120RSI = 90.00%
Productivity in project = Actual Project Size / Actual Effort spent for the project
Actual Project Size = 1000 FPActual Effort Spent = 1125 Person daysProductivity for the project = 0.89 FP/Day
Productivity in Coding = Actual LOC or FP / Actual Effort spent for Coding
Actual Size = 100 KLOCEffort Spent for Coding = 400 Person DaysProductivity in Coding = 250 LOC/Day
Productivity in Test Case Preparation = Actual No of test cases / Actual Effort spent on Test Case preparation
Acutal No. of Test Cases prepared = 3000Actual Effort Spent = 50 Person DaysProductivity in Test Case Preparation = 60 TC/Day
Productivity in Test Case Execution = Actual No of Test cases (Planned + Adhoc) / Actual Effort spent on Testing
Actual No. of Test Cases = 3000Adhoc Test Cases = 150Actual Effort Spent on Testing = 30 PDProductivity in Test Case Execution = 105 TC/Day
Productivity in Defect Detection = Actual no of Defects (Review + Testing) / Actual effort spent on (Review + Testing)
Actual No. of Review Defects = 650Actual No. of Testing Defects = 450Actual Effort Spent for Review = 25 PDActual Effort Spent for Testing = 30 PDProductivity in Defect Detection = 20
Productivity in Defect Fixation = Actual No of defects fixed / Actual Effort spent on Defect fixation
Actual No. of Defects Fixed = 1100Actual Effort Spent on Defect Fixation = 50 PDProductivity in Defect Fixation = 22
Schedule variance for a Phase = (Actual Calendar days for a phase – Planned Calendar days for a phase + Start Variance for a phase) / (Planned Calendar days for a phase) * 100
Schedule varaince for the various phases arePresales is 10.00%Requirements is 20.00%Design is 8.00%Development is 21.43%Testing is 5.00%Acceptance 0.00 %Customer implementation 0.00 %
Effort variance for a phase = (Actual Effort for a phase – Planned Effort for a phase) / (Planned Effort for a phase) * 100
Effort varaince for the various phases arePresales is 7.14%Requirements is 17.65%Design is 15.38%Development is 7.14%Testing is 25.00%Acceptance 11.11 %Customer implementation 11.11 %
Distrbution of effort in each phase / Total effort * 100
Planning is 5%Requirements is 10%Design is 25%Development is 35%Testing is 15%Acceptance is 10%
Distrbution of effort in each ActivityGroup / Total effort * 100
SDLC Regular is 10%Review is 10%Test is 15%Rework-Review is 10%Rework-Test is 10%Verification-Review is 5%Verification-Test is 5%Quality Assurance is 5%Configuration mgt is 5%Measurement is 10%Marketing is 5%Training is 10%
Cost of Quality = (Review + Testing + Verification-Review + Verification Testing + QA + Config Mgmt + Measurement + Training + Re-Work Review + Re-Work Testing) / Total Effort * 100
Testing Effort = 80 PDReview Effort = 100 PDQA Effort = 30 PDMeasurement Effort = 20 PDTraining Effort = 20 PDConfig Mgt Effort = 20 PDCost of Quality = 24.00 %
Cost of Poor Quality = Rework Effort / Total Effort * 100
Actual Effort Spent for Rework on Testing = 40 PDRework on Reviews = 40 PDRework on verification = 50 persondaysCost of Poor Quality = 11.5 %
Defect density for the project = Total number of defects / Project size in KLOC or FP
Defect density for each work product = Total number of defects / Work product size measure
Total No of Defects = 1100Project Size = 200 KLOC or 1000 FPDefect Density = 5.5 Defects/KLOC or 1.1 Defects/FP
Residual Defect Density = (No. of Defects caught in Acceptance/Project Size)
No of Defects in Acceptance = 50Project Size = 200 KLOC or 1000 FPRDD = 0.25 Defects/KLOC or 0.05 Defects /FP
Defect age = sigma (Phase detected – Phase introduced) ^2 / Total no of defects
Defect Introduced: During Requirements - Phase 2Defect Detected: During Testing - Phase - 5Defect Age is calculated by using this phase difference.
Review Efficiency = (Number of defects caught in reviews) / (Total number of defects caught ) * 100
Total no. of defects in Reviews = 650Total No of Defects in (Reviews + Testing) = 1100Review Efficiency = 59.09 %
Testing Efficiency = 1-((Defects found in Acceptance) / Total No of Testing Defects) * 100
Total no. of defects in Testing = 450No of Defects in Acceptance Testing = 50Testing Efficiency = 88.88 %
Defect Removal Efficiency = (1- (Total defects Caught by customer / Total No of Defects)) * 100
No of Defects caught by customer during Review, Acceptance and Implementation = 80
Total no. of defects in Review and Testing = 1100
Defect Removal Efficiency is 92.72 %
Depending on the Defects detected in each phase compared to the incoming escapes and overall escapes
Phasewise detection effieincy Requirements - 50%Design - 45%
% of defects across various phases
Defect Distribution across various phasesPlanning is 5%Requirements is 10%Design is 25%Development is 35%Testing is 15%Acceptance is 10%
% of defects across various origin
Defect Distribution across various severitiesRequirements<10%Design<15%Coding<40%Integration Testing<20System Testing< 15%
% of defects across various types
Defect Distribution across various typesRun Time Error < 5Missing Implementation < 10 Validation < 10%Data Invariant < 5% Logical Error< 5 % Functional Specification < 5% Build Error < 10% GUI Error < 20%Performance Errors< 20 %Design Errors < 10%
% of defects across various causes
Defect Distribution across various causes Lack of Standards 20%Process Deficiency 10%Lack of Training 10%Schedule Pressure 15%New Technology 10%Clerical Error 5%Others 10%Lack of Domain 20%
(No. of Audits Counducted *100)/ No. of Audits Planned
(No of check points complied with during audit) /(Total of applicable check points)
Note : 1) No of check points complied with during audit = Total No. of Check Points - No of Check Points not applicable-No of check points not complied with during audit
2) Total of applicable check points = Total No. of Check Points - No of Check Points not applicable
(No. of QMS Release *100)/ No. of QMS Release Planned
(Nof improvement addressed* 100)/No of improvement suggested
No of deviations requested for a process* 100/total no of projects
Sum of time taken by projects / nof of projects
Date Version Section Author
3-Mar-09 1.0 Initial Draft Raja Sekher
Description of the Change