simple measurements

Agile Cambridge 2011© Schalk W. Cronjé

Simple MeasurementsSimple MeasurementsSchalk W. CronjéSchalk W. Cronjé

ysb33r @ gmail.comysb33r @ gmail.com@ysb33r@ysb33r

mailto:[email protected]


What is Agile?What is Agile?

Why are you not measuring Why are you not measuring your process today?your process today?

How do you know when your How do you know when your delivery was successful?delivery was successful?


““What is Agile?” - ResponsesWhat is Agile?” - Responses

● “Continuous development done in small iterations”

● “Adapting to rapid change, predictable periodic delivery of usable software”

● “Developers and testers work together from beginning to end”

● “You'd only get a cynical answer from me”


What is Agile Fundamentally?What is Agile Fundamentally?

Delivering value to stakeholders● Customers● Users● Developers & Testers● Governments & Legislators● Societal Institutions

● Increased revenue● Reduction in costs● Faster response

times● Less rework


Opposing Forces?Opposing Forces?

Software Engineering

Accuracy

Craftsmanship

Precision


Software EngineeringSoftware Engineering

● Implies measurement● Implies Kolb / Shewhart / Deming Cycles ● Is not code craftmanship● SE receives bad press– Perceived difficulty in measurement– Lack of measurement in S/W dev


AccuracyAccuracy

Accuracy is to precision as

engineering is to mathematics


AccuracyAccuracy

We need to be accurate enough to get there, but not more accurate


Properties of a Kanban SystemProperties of a Kanban System


Taking an Economic ViewTaking an Economic View

● It helps to quantify the effects of multiple interacting variables

● It helps us to understand that the customer is not the only judge of value

● By using an economic framework it can allow us to maximise value, including

– cycle time– product cost– development expense

● It helps to communicate with non-technical decision makers


BackgroundBackground

3 Teams – 3 Locations – 18 months


David Anderson's Six StepsDavid Anderson's Six Steps

● Focus on quality● Reduce Work-in-Progress● Deliver often● Balance demand against throughput● Prioritise● Attack sources of variability to improve

predictability


VisualisationVisualisation


VisualisationVisualisation

● Honesty!● Makes it transparent to everyone that is

going on● Visual cue for bottlenecks● Manual board and easy-to-use online system


Business Value IncrementsBusiness Value Increments

● Delivery is in business value increments (BVIs)

● Only “done” when the increment is delivered end-to-end

● No burning down of tasks ! (false economy)● Each increment must have at least one

metric of the value it will deliver


deliveredbuild

Batch DevelopmentBatch Development

Feature 3 DevFeature 2 Dev

Feature 4 Dev

Feature 1 Dev Feature 1 QAFeature 2 QAFeature 3 QAFeature 4 QA


deliveredbuild

Batch DevelopmentBatch Development


Feature 4 Dev

Feature 1 Dev

BugDB

Feature 1 QAFeature 2 QAFeature 3 QAFeature 4 QA

defect trickle feed

Total Time = devt1 .. devt

4 + qat

1 …qat

4 + fixdelay + fixtime + retest time

Feature 5+ Dev

Feature 5+ QA

deliveredbuild

Retest fixes QA


deliveredbuild

Time FactorsTime Factors


Feature 4 Dev

Feature 1 Dev

BugDB

Feature 1 QAFeature 2 QAFeature 3 QAFeature 4 QA

defect trickle feed

Feature 5+ Dev

Feature 5+ QA

deliveredbuild

Retest fixes QA

understandingspecs

time from raising defect until it is available for testing

Building testinfrastructure

time tore-test

basic buildverification


""If you only quantify one thing, If you only quantify one thing,

quantify the cost of delayquantify the cost of delay""Don Reinertsen


Cost of Delay – Quality IssueCost of Delay – Quality Issue

● One small feature, not properly tested● Found post-release● Time wasting by users having to manual

work quantified => £35,000

TotalManualTimeSpent x AvgCostToCompanyPerUser + TimeToRework x AvgCostToCompanyPerDev


Limit Work-in-ProgressLimit Work-in-Progress

● Use a pull system● Managing queues are easier than managing timelines● Allows for better throughput utilisation● Identifies bottlenecks in the system● Measuring cycle time allows better prediction than

traditional estimation

Ready QA QAComplete

ReleasedSpec Dev DevComplete

SpecComplete


PoliciesPolicies

● Each stage has a policy● Describes generic activities required● “Definition of Done” for each stage

Ready QA QAComplete


SpecComplete


FeedbackFeedback

● Faster feedback makes learning faster and more efficient

● Faster feedback provides a sense of control

● Large batches leads to slower feedback


Measuring Cycle TimeMeasuring Cycle Time

● Measure average time in system of each BVI● Less time is spent on estimating future delivery

times● Historic data automatically takes into account any

disruptions such as– Operational issues– Days off sick

AvgTimeInSystem x NoOfBVIs Time = --------------------------------- Concurrency


CadencesCadences

● Delivery is done at regular cadences.● What is ready is delivered, what is not, is

left out● No artificial time-boxing or break points as

in SCRUM-like approaches● Cycle-time leads to better predictability of

what can actually be delivered


Historic Data vs EstimationHistoric Data vs Estimation


Simple Flow ModelSimple Flow Model

Specify Develop QA

Ready QA QAComplete


SpecComplete

Kanban Board


Reducing Learning TimeReducing Learning Time● Writing test specifications after the

development is costly in time– Knowledge decay– Leads to incomplete designs

● Move the test specification up-front before any development starts

– This is part of requirements discovery– It broadens the perspective of the programmer– Allows us to distinguish between automated

checks and human testing


Reducing Learning TimeReducing Learning Time

Specify Develop QA

+Time to specify

+Time to develop

-Time to re-work

-Time to re-test

Test specification


Change the Flow ModelChange the Flow Model

Specify Develop & Check Verify

● Ensure the all automated checks are included in the development stage

● Free up a Verification stage for exploratory testing and validating that process has been fulfilled.


Change the Flow ModelChange the Flow Model


+Time to develop

+More testing people

-Time to re-work

-Time to re-test


Real-world measurementsReal-world measurements

S1 S2 S3 S4

READY 4 6 5 1h

SPEC 1 0.5 6 0.5

SPEC/Completed 2 0.5 1 0.5

DEV + Check 2 1 13 4

DEV + Check / Completed

3 6 4 18

VERIFY 3 7 8 1

VERIFY/Completed 2 5 6 18

Blockages 0.5 1h 2 1h

Total 17 26 43 42


Average cycle days per feature


What is wrong?What is wrong?

S1 S2 S3 S4

READY 4 6 5 1h

SPEC 1 0.5 6 0.5




3 6 4 18

VERIFY 3 7 8 1



Total 17 26 43 42



What is wrong?What is wrong?

S1 S2 S3 S4

READY 4 6 5 1h

SPEC 1 0.5 6 0.5




3 6 4 18

VERIFY 3 7 8 1



Total 17 26 43 42


Excessive long time between feature availability and testing. Is there enough testing bandwidth? Or is this team just cheating?


Quantifying BVIsQuantifying BVIs


Quantifying BVIsQuantifying BVIs

● Know what you want to achieve● Establish a scale for measurement● Decide how to measure● Make constraints explicit● Communicate


Quantify BVIs - ExampleQuantify BVIs - Example

For all customer submissions that meets the following criteria, 80% should be

handled by an automation system instead of a human and resolution achieved within

30 minutes.

Criteria ACriteria B

etc.


PlanguagePlanguage

● Created by Tom Gilb● Provides a structured way of quantifying

requirements● Allows for reuse of specifications


Quantify BVI - PlanguageQuantify BVI - Planguage

Name: Customer submissions resolved by automation systems

Scale: Percentage of submissions resolved by automation in relation to all submissions

Meter: Query on database using the following statement: SELECT … FROM … WHERE ...

Goal: 80% [Criteria A, Criteria B]


Applying the Economic ModelApplying the Economic Model

● Compare to average human time to respond and volume covered

● Extrapolate to time savings or cost savings● This allows for two measures of success– Technical – Financial


Kanban fails when people don’t want to face the truth.

- Hillel Glazer


It all fell apart ...It all fell apart ...

● Restructure – 3 teams + 3 managers● 2 managers rejected Kanban– 1 manager wanted ScrumWorks– 1 manager used Microsoft Project

● 1 manager was forced not to use Kanban– Switched to iteration-based deliveries

● All teams forced back to time-based estimation


It fell apart even more ...It fell apart even more ...

● 1 team decided on 4 wks dev + 4 wks testing

– Quality was worse than before– Cycle time was at least 8 calendar weeks per BVI

● 1 team did not deliver for over 5 months– Finally delivered something the user did not want

● None of teams have any measurements on cycle time; cannot predict how long delivery will take


What did the team members say?What did the team members say?

● “I have lost visualisation. I have no idea what is going on”

● “We are not using Kanban, because the company forced us to SCRUM”

● “If I was given a choice, my the team and I would definitely go with Kanban”

● “It is not the company 'standard' as such you need to 'sell' the process frequently. “


Lessons LearntLessons Learnt


Apply Lean Principles Apply Lean Principles to your Team to your Team


● Make the system pull-based● Reduce the batch size● Limit the work-in-progress● One feature end-to-end● If the flow breaks, fix it immediately● Measure cycle time● Quantify requirements● Use Cost of Delay where applicable


Paired TeamsPaired Teams

● Pair up people with different skills to work on same feature

– Feature teams are not a new concept● Could be perceived to have higher person

cost per feature– Instead of distributing the person cost over

multiple queues, the cost is combined into a single queue

– Can actually lead to reduced cycle time.


Simple measurementsSimple measurementsare a foundationare a foundation

of software engineeringof software engineeringand not for measuring humansand not for measuring humans

simple measurements

Technology