traps in test estimation
TRANSCRIPT
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 1/18
Traps in Test EstimationTraps in Test Estimation
Shrini KulkarniShrini KulkarniPrincipal ConsultantPrincipal Consultant -- TestingTesting
iGATE Global SolutionsiGATE Global SolutionsBangalore, IndiaBangalore, India
[email protected]@igate.com
http://shrinik.blogspot.comhttp://shrinik.blogspot.com
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 2/18
What is a Trap and what is anEstimation?
Trap: noun - Something (often something deceptively attractive) that catches you unaware or place of confining or embarrassing
situation
Estimation: An approximate calculation of quantity or degree orworth of some activity or an object
of value
Test Estimation: An Act or processof estimation applied to a testing task
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 3/18
Consider this « A typical situation
Test Size
Test Effort
Schedule
Key QuestionKey Question ±± How many testers for how long? How many testers for how long?
Inputs
R equirements Test cases Use cases
User guide Expert Opinions
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 4/18
I s there any Scientific Method for Test estimation? I s there any Scientific Method for Test estimation?
A List of Commonly Used Techniques «A List of Commonly Used Techniques «
Guestimation ± Wild Ass Guess ± rule of thumb
A percentage of development effort
R isk Based methods ± Determine what to test and how much
Use of historical Data + Industry Standards
Work-break down Structure ± WBS method
Use of Dev Size estimates ± Use case, function points, COCOMO
Test point Analysis method
Delphi ± Wideband Delphi Method with WBSConsensus opinion in a team of expert estimators
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 5/18
But what all is included in ³Product´ andBut what all is included in ³Product´ andin the phrase ³test´ ?in the phrase ³test´ ?
How long it takes to youHow long it takes to youtest this Product ?test this Product ?
It all starts with this Motherhood Question «It all starts with this Motherhood Question «
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 6/18
Getting Trapped?Getting Trapped?
How will you know you will be able to complete Testing as per the
estimate ? We d on't. Why pret end we d o? We d on't. Why pret end we d o? - Matthew Heusser
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 7/18
Trap # 1Trap # 1 -- Model of TestingModel of Testing
There's a slippery slope between asking for good faith estimates ("Knowing what
you know now, when do you think you can deliver?") and predicting the future.
- Mathew Heusser
A simplified model of Testing is assumed.
Testing is an open ended search for problems ± decision to
stop testing is often decided by the stakeholders depending upon the parameters that are out side the purview of testing
Actual Testing model is complex hence estimates tend to gowrong
Testing is assumed to be complete when planned test casesare executed
Dependencies on environment, availability of Stable build totest and test data are typically ignored or down played
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 9/18
Trap # 3Trap # 3 -- Unknown FutureUnknown Future
W e¶re too optimistic, with short memories that mask the painful overruns
from previous projects ± Karl E. W iegers
Things change during the course of the project in unimaginable ways
Each test cycle presents its own challenges and issues
There might be be too many or too less bugs discovered in any cycle
There might be developer delays that eats up test time
There is might be pressure to skip tests and constantly change test sets
Development might lag behind bug find rate
There might be requirements changes hence development delays
There might be f ailures in Build verification tests
There might be significant number of ³non reproducible defects´
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 10/18
Trap # 4Trap # 4 -- Too many VariablesToo many Variables
G iving an Estimate would mean a commitment ± which you are not
sure about fulfilling ± that is a trap
Test scope ± that keeps changing depending upon available timefor testing
Set of features available in a given build Time available for testing for any given cycle Test environment availability
Stability of build for testing Unknown number of bugs discovered any given cycle Variable bug find and fix rate Additional cycles of Testing for every new bug or regression bug
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 11/18
Trap # 5Trap # 5 ±± Match that MAGIC numberMatch that MAGIC number
Role Play «Role Play «
U nlike other forms of estimation, I can make the estimate totally
predictable: I'll stop when the allotted time is up ± Michael Bolton
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 12/18
Michael Bolton¶s Recipe for Estimation «Michael Bolton¶s Recipe for Estimation «
Testing is an open ended search for problems ± we would never say ³we are
Done´ ± instead we temporarily stop and pass on the information to the
Stakeholder and wait for next opportunity to start «.
When do we stop then ? Several ways «.
Please see Next slide «
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 13/18
Heuristics for ³Stopping Testing´Heuristics for ³Stopping Testing´
� If we've modeled the product in lots of different ways, and then tested in accordance with those models, we can say that we've done enough testing and stop.
� If we're satisfied that we have addressed the list of compelling questions that we set outto answer (along with the other questions that we realized are important along the way), then we can say that we've done enough testing and stop.
� If the product is so horribly broken that it has to be send in for major rework, then we can
say that we've done enough testing and stop.
� If management decides that it has sufficient confidence to ship, and ships, then we cansay that we've done enough testing and stop.
� If management decides that it must ship the product, even though confidence in its quality is less than what we'd like it to be, we can say that we've done enough testing and stop.
� If we're testing on behalf of some one who is trying to decide whether the software isacceptable, and they say it is, we can say that we've done enough testing and stop.
� If we're testing on behalf of someone who is trying to decide whether the software is acceptable, and they say it isn't, until we get another version, we can say that we've
done enough testing and stop.
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 14/18
So « what is the solution?
I hear you, I hear you«´ So Mr. Big Man, what¶ syour solution?´ «.. the truth is
I don¶t have one.
I don¶t think that¶s a failure, but a realization ±
I¶m not the lone ranger, and the estimation
problem may be a bugbear, but it¶ s not awerewolf .Source : htt p://bl og.geeksmi t hol ogy .c om/2006/11/14/est imat i ng-is-bunk
But I do have some suggestions «
Ask what is in your sponsor's mind ± what date?
Take overall testing as cycle by cycle
Propose an Adaptive and iterative approach R esist the temptation to commit for ONE date
Say ± you will test until ship date and until told not to test«
H ell, t here are no r ules here± we¶re t ryi ng to acc omplish somet hi ng. -
T homas A. Edison
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 15/18
And «And «
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 16/18
Questions?Questions?
8/7/2019 Traps in Test Estimation
http://slidepdf.com/reader/full/traps-in-test-estimation 17/18
Thank youThank you
http://www.igate.com
http://shrinik.blogspot.com
F or f ur t her disc ussi ons:
Shrini Kulkarni
Principal Consultant - Testing
iGATE Global Solutions Ltd
Bangalore