Download - Simple measurements
Agile Cambridge 2011© Schalk W. Cronjé
Simple MeasurementsSimple MeasurementsSchalk W. CronjéSchalk W. Cronjé
ysb33r @ gmail.comysb33r @ gmail.com@ysb33r@ysb33r
Agile Cambridge 2011© Schalk W. Cronjé
What is Agile?What is Agile?
Why are you not measuring Why are you not measuring your process today?your process today?
How do you know when your How do you know when your delivery was successful?delivery was successful?
Agile Cambridge 2011© Schalk W. Cronjé
““What is Agile?” - ResponsesWhat is Agile?” - Responses
● “Continuous development done in small iterations”
● “Adapting to rapid change, predictable periodic delivery of usable software”
● “Developers and testers work together from beginning to end”
● “You'd only get a cynical answer from me”
Agile Cambridge 2011© Schalk W. Cronjé
What is Agile Fundamentally?What is Agile Fundamentally?
Delivering value to stakeholders● Customers● Users● Developers & Testers● Governments & Legislators● Societal Institutions
● Increased revenue● Reduction in costs● Faster response
times● Less rework
Agile Cambridge 2011© Schalk W. Cronjé
Opposing Forces?Opposing Forces?
Software Engineering
Accuracy
Craftsmanship
Precision
Agile Cambridge 2011© Schalk W. Cronjé
Software EngineeringSoftware Engineering
● Implies measurement● Implies Kolb / Shewhart / Deming Cycles ● Is not code craftmanship● SE receives bad press– Perceived difficulty in measurement– Lack of measurement in S/W dev
Agile Cambridge 2011© Schalk W. Cronjé
AccuracyAccuracy
Accuracy is to precision as
engineering is to mathematics
Agile Cambridge 2011© Schalk W. Cronjé
AccuracyAccuracy
We need to be accurate enough to get there, but not more accurate
Agile Cambridge 2011© Schalk W. Cronjé
Properties of a Kanban SystemProperties of a Kanban System
Agile Cambridge 2011© Schalk W. Cronjé
Taking an Economic ViewTaking an Economic View
● It helps to quantify the effects of multiple interacting variables
● It helps us to understand that the customer is not the only judge of value
● By using an economic framework it can allow us to maximise value, including
– cycle time– product cost– development expense
● It helps to communicate with non-technical decision makers
Agile Cambridge 2011© Schalk W. Cronjé
BackgroundBackground
3 Teams – 3 Locations – 18 months
Agile Cambridge 2011© Schalk W. Cronjé
David Anderson's Six StepsDavid Anderson's Six Steps
● Focus on quality● Reduce Work-in-Progress● Deliver often● Balance demand against throughput● Prioritise● Attack sources of variability to improve
predictability
Agile Cambridge 2011© Schalk W. Cronjé
VisualisationVisualisation
Agile Cambridge 2011© Schalk W. Cronjé
VisualisationVisualisation
● Honesty!● Makes it transparent to everyone that is
going on● Visual cue for bottlenecks● Manual board and easy-to-use online system
Agile Cambridge 2011© Schalk W. Cronjé
Business Value IncrementsBusiness Value Increments
● Delivery is in business value increments (BVIs)
● Only “done” when the increment is delivered end-to-end
● No burning down of tasks ! (false economy)● Each increment must have at least one
metric of the value it will deliver
Agile Cambridge 2011© Schalk W. Cronjé
deliveredbuild
Batch DevelopmentBatch Development
Feature 3 DevFeature 2 Dev
Feature 4 Dev
Feature 1 Dev Feature 1 QAFeature 2 QAFeature 3 QAFeature 4 QA
Agile Cambridge 2011© Schalk W. Cronjé
deliveredbuild
Batch DevelopmentBatch Development
Feature 3 DevFeature 2 Dev
Feature 4 Dev
Feature 1 Dev
BugDB
Feature 1 QAFeature 2 QAFeature 3 QAFeature 4 QA
defect trickle feed
Total Time = devt1 .. devt
4 + qat
1 …qat
4 + fixdelay + fixtime + retest time
Feature 5+ Dev
Feature 5+ QA
deliveredbuild
Retest fixes QA
Agile Cambridge 2011© Schalk W. Cronjé
deliveredbuild
Time FactorsTime Factors
Feature 3 DevFeature 2 Dev
Feature 4 Dev
Feature 1 Dev
BugDB
Feature 1 QAFeature 2 QAFeature 3 QAFeature 4 QA
defect trickle feed
Feature 5+ Dev
Feature 5+ QA
deliveredbuild
Retest fixes QA
understandingspecs
time from raising defect until it is available for testing
Building testinfrastructure
time tore-test
basic buildverification
Agile Cambridge 2011© Schalk W. Cronjé
""If you only quantify one thing, If you only quantify one thing,
quantify the cost of delayquantify the cost of delay""Don Reinertsen
Agile Cambridge 2011© Schalk W. Cronjé
Cost of Delay – Quality IssueCost of Delay – Quality Issue
● One small feature, not properly tested● Found post-release● Time wasting by users having to manual
work quantified => £35,000
TotalManualTimeSpent x AvgCostToCompanyPerUser + TimeToRework x AvgCostToCompanyPerDev
Agile Cambridge 2011© Schalk W. Cronjé
Limit Work-in-ProgressLimit Work-in-Progress
● Use a pull system● Managing queues are easier than managing timelines● Allows for better throughput utilisation● Identifies bottlenecks in the system● Measuring cycle time allows better prediction than
traditional estimation
Ready QA QAComplete
ReleasedSpec Dev DevComplete
SpecComplete
Agile Cambridge 2011© Schalk W. Cronjé
PoliciesPolicies
● Each stage has a policy● Describes generic activities required● “Definition of Done” for each stage
Ready QA QAComplete
ReleasedSpec Dev DevComplete
SpecComplete
Agile Cambridge 2011© Schalk W. Cronjé
FeedbackFeedback
● Faster feedback makes learning faster and more efficient
● Faster feedback provides a sense of control
● Large batches leads to slower feedback
Agile Cambridge 2011© Schalk W. Cronjé
Measuring Cycle TimeMeasuring Cycle Time
● Measure average time in system of each BVI● Less time is spent on estimating future delivery
times● Historic data automatically takes into account any
disruptions such as– Operational issues– Days off sick
AvgTimeInSystem x NoOfBVIs Time = --------------------------------- Concurrency
Agile Cambridge 2011© Schalk W. Cronjé
CadencesCadences
● Delivery is done at regular cadences.● What is ready is delivered, what is not, is
left out● No artificial time-boxing or break points as
in SCRUM-like approaches● Cycle-time leads to better predictability of
what can actually be delivered
Agile Cambridge 2011© Schalk W. Cronjé
Historic Data vs EstimationHistoric Data vs Estimation
Agile Cambridge 2011© Schalk W. Cronjé
Simple Flow ModelSimple Flow Model
Specify Develop QA
Ready QA QAComplete
ReleasedSpec Dev DevComplete
SpecComplete
Kanban Board
Agile Cambridge 2011© Schalk W. Cronjé
Reducing Learning TimeReducing Learning Time● Writing test specifications after the
development is costly in time– Knowledge decay– Leads to incomplete designs
● Move the test specification up-front before any development starts
– This is part of requirements discovery– It broadens the perspective of the programmer– Allows us to distinguish between automated
checks and human testing
Agile Cambridge 2011© Schalk W. Cronjé
Reducing Learning TimeReducing Learning Time
Specify Develop QA
+Time to specify
+Time to develop
-Time to re-work
-Time to re-test
Test specification
Agile Cambridge 2011© Schalk W. Cronjé
Change the Flow ModelChange the Flow Model
Specify Develop & Check Verify
● Ensure the all automated checks are included in the development stage
● Free up a Verification stage for exploratory testing and validating that process has been fulfilled.
Agile Cambridge 2011© Schalk W. Cronjé
Change the Flow ModelChange the Flow Model
Specify Develop & Check Verify
+Time to develop
+More testing people
-Time to re-work
-Time to re-test
Agile Cambridge 2011© Schalk W. Cronjé
Real-world measurementsReal-world measurements
S1 S2 S3 S4
READY 4 6 5 1h
SPEC 1 0.5 6 0.5
SPEC/Completed 2 0.5 1 0.5
DEV + Check 2 1 13 4
DEV + Check / Completed
3 6 4 18
VERIFY 3 7 8 1
VERIFY/Completed 2 5 6 18
Blockages 0.5 1h 2 1h
Total 17 26 43 42
Specify Develop & Check Verify
Average cycle days per feature
Agile Cambridge 2011© Schalk W. Cronjé
What is wrong?What is wrong?
S1 S2 S3 S4
READY 4 6 5 1h
SPEC 1 0.5 6 0.5
SPEC/Completed 2 0.5 1 0.5
DEV + Check 2 1 13 4
DEV + Check / Completed
3 6 4 18
VERIFY 3 7 8 1
VERIFY/Completed 2 5 6 18
Blockages 0.5 1h 2 1h
Total 17 26 43 42
Average cycle days per feature
Agile Cambridge 2011© Schalk W. Cronjé
What is wrong?What is wrong?
S1 S2 S3 S4
READY 4 6 5 1h
SPEC 1 0.5 6 0.5
SPEC/Completed 2 0.5 1 0.5
DEV + Check 2 1 13 4
DEV + Check / Completed
3 6 4 18
VERIFY 3 7 8 1
VERIFY/Completed 2 5 6 18
Blockages 0.5 1h 2 1h
Total 17 26 43 42
Average cycle days per feature
Excessive long time between feature availability and testing. Is there enough testing bandwidth? Or is this team just cheating?
Agile Cambridge 2011© Schalk W. Cronjé
Quantifying BVIsQuantifying BVIs
Agile Cambridge 2011© Schalk W. Cronjé
Quantifying BVIsQuantifying BVIs
● Know what you want to achieve● Establish a scale for measurement● Decide how to measure● Make constraints explicit● Communicate
Agile Cambridge 2011© Schalk W. Cronjé
Quantify BVIs - ExampleQuantify BVIs - Example
For all customer submissions that meets the following criteria, 80% should be
handled by an automation system instead of a human and resolution achieved within
30 minutes.
Criteria ACriteria B
etc.
Agile Cambridge 2011© Schalk W. Cronjé
PlanguagePlanguage
● Created by Tom Gilb● Provides a structured way of quantifying
requirements● Allows for reuse of specifications
Agile Cambridge 2011© Schalk W. Cronjé
Quantify BVI - PlanguageQuantify BVI - Planguage
Name: Customer submissions resolved by automation systems
Scale: Percentage of submissions resolved by automation in relation to all submissions
Meter: Query on database using the following statement: SELECT … FROM … WHERE ...
Goal: 80% [Criteria A, Criteria B]
Agile Cambridge 2011© Schalk W. Cronjé
Applying the Economic ModelApplying the Economic Model
● Compare to average human time to respond and volume covered
● Extrapolate to time savings or cost savings● This allows for two measures of success– Technical – Financial
Agile Cambridge 2011© Schalk W. Cronjé
Kanban fails when people don’t want to face the truth.
- Hillel Glazer
Agile Cambridge 2011© Schalk W. Cronjé
It all fell apart ...It all fell apart ...
● Restructure – 3 teams + 3 managers● 2 managers rejected Kanban– 1 manager wanted ScrumWorks– 1 manager used Microsoft Project
● 1 manager was forced not to use Kanban– Switched to iteration-based deliveries
● All teams forced back to time-based estimation
Agile Cambridge 2011© Schalk W. Cronjé
It fell apart even more ...It fell apart even more ...
● 1 team decided on 4 wks dev + 4 wks testing
– Quality was worse than before– Cycle time was at least 8 calendar weeks per BVI
● 1 team did not deliver for over 5 months– Finally delivered something the user did not want
● None of teams have any measurements on cycle time; cannot predict how long delivery will take
Agile Cambridge 2011© Schalk W. Cronjé
What did the team members say?What did the team members say?
● “I have lost visualisation. I have no idea what is going on”
● “We are not using Kanban, because the company forced us to SCRUM”
● “If I was given a choice, my the team and I would definitely go with Kanban”
● “It is not the company 'standard' as such you need to 'sell' the process frequently. “
Agile Cambridge 2011© Schalk W. Cronjé
Lessons LearntLessons Learnt
Agile Cambridge 2011© Schalk W. Cronjé
Apply Lean Principles Apply Lean Principles to your Team to your Team
Specify Develop & Check Verify
● Make the system pull-based● Reduce the batch size● Limit the work-in-progress● One feature end-to-end● If the flow breaks, fix it immediately● Measure cycle time● Quantify requirements● Use Cost of Delay where applicable
Agile Cambridge 2011© Schalk W. Cronjé
Paired TeamsPaired Teams
● Pair up people with different skills to work on same feature
– Feature teams are not a new concept● Could be perceived to have higher person
cost per feature– Instead of distributing the person cost over
multiple queues, the cost is combined into a single queue
– Can actually lead to reduced cycle time.
Agile Cambridge 2011© Schalk W. Cronjé
Simple measurementsSimple measurementsare a foundationare a foundation
of software engineeringof software engineeringand not for measuring humansand not for measuring humans