productcamp vancouver 2013

agile42 | We advise, train and coach companies building software www.agile42.com | All rights reserved. Copyright ©2007-2013

Lean Product ManagementThe Value of Experiments

Photo: ESAhttp://www.nbcnews.com/id/50677062/ns/technology_and_science-space

@DaveSharrockProductCampVancouver 2013


http://www.archives.gov.on.ca/english/on-line-exhibits/d-day/big/big_03_airline_assembly.aspxArchives of Ontario, Reference Code: C 190-5-0-0-21

Assembly Line Manufacturing has a high cost of change


• In software development high cost of change leads to inclusive thinking

• Any and every idea has to be captured in the first version of a requirements specification

• Creates waste - bloated documents, unwanted features and entitlement thinking

Inclusive thinkingFeature usage in

enterprise software

45%

19%

16%

13%

7%

NeverRarelySometimesOftenAlways

Standish Group - CHAOS Report, 1995, 2002


Cost of Change

Detail of Requirements

Hypothesized Requirements

Capture all possible needs


Software Development has commoditized cost of change

Continuous Delivery

Ruby on Rails

Coffeescript

jquery

Object-oriented languages


Cost of Change


Evolving Requirements


Emerging needs as development

progresses



• If change is cheap, requirements can change continuously

• We can evolve our thinking as we learn more about the product we are building

• Maximizing Return-on-Investment means validating new features early, before the total investment cost has been spent

Changing paradigms


Cost of Change


Validated Requirements

Evolving Requirements


Lean Startup experiments

Emerging needs as development

progresses



Shifting Paradigms

“I know that half of my advertising

budget is wasted, but I’m not sure

which half”Lord Leverhulme

B.G.|A.G.

http://www.marketingprofs.com/charts/2009/3161/more-smbs-use-online-media-than-traditionalhttp://marketing.blogs.ie.edu/archives/2011/01/trends-in-online-advertising-in-2011-in-the-united-states.php


Validated Learning

• Understand the problem a feature will solve

• Decide how to ask if your customer cares

• Validate before investing complete cost


Theory

Prediction

Experiment

Observe

Use the theory to make a prediction

Design an experiment to

test the predictionRun the

experiment

Modify or change your theory

The Scientific Method


The Theory of General Relativity

Einstein’s Theory of General Relativity

Light bends in a gravitational field

Solar Eclipse of 1919

Observations validated General Relativity

Einstein’s prediction

(1907)

Wait for 1919 solar eclipse to observe

whether or not light bends around the sun

Arthur Eddington observed that Light did bend around the sun

Gradual acceptance of General Relativity over Newtonian Mechanics

http://thethoughtstash.wordpress.com/2011/01/03/how-eddington-demonstrated-that-einstein-was-right/


User Model

Prediction

Experiment

Validate

Use the model to make a prediction

Design an experiment to

test the predictionRun the

experiment

Modify or change your user model

Applied to an Online Experience


Designed by KISSmetrics


User Story (Journey) Mapping

Story Mapping © Jeff Patton, all rights reserved, www.AgileProductDesign.com

Elements of Customer Journey

Diff

eren

t Cus

tom

er

Exp

erie

nces

Your model for customer interaction covers each step of the customer

journeyEach row matches a different

customer experience


Vanity vs. Actionable Metrics

Vanity metrics (like web hits or number of downloads) which only serve to document the current state of the product but offer no insight into how we got here or what to do next.

An actionable metric is one that ties specific and repeatable actions to observed results.


http://500hats.typepad.com/500blogs/2007/09/startup-metrics.html


Depersonalize Decision Making

• “HiPPO’s” rule the business world • It is never about you, it can’t be

about you • Benchmarking is awesome • Leverage competitive analysis • Experimentation / testing rock

• Execution strategy: • Transparency, standardization,

looking outside in • Be a slave to customer

centricity (its about your customers (internal & external))


“I can’t say it any better, HiPPO’s rule the world, they over rule your data, they impose their opinions on you and your company customers, they think they know best (sometimes they do), their mere presence in a meeting prevents ideas from coming up. The solution to this problem is to depersonalize decision making, simply don’t make it about you or what you think. Go outside, get context from other places. Include external or internal benchmarks in your analysis. Get competitive data (we are at x% of zz metric and our competition is at x+9% of zz metric).Be incessantly focussed on your company customers and dragging their voice to the table (for example via experimentation and testing or via open ended survey questions). Very few people, HiPPO’s included, can argue with a customer’s voice, the customer afterall is the queen / king! : )”

Avinesh Kaushik, kaushik.net

Interaction Model Backlog ValidationDoing Doing Deployed

For <target group>Who <has a specific need>The <product name>Is A <product category>That <key benefit>Unlike <primary competitors>Our Product <additional advantages>

Pathway#1

Pathway#2

•Visualize your User Interaction Model and adjust it as you learn•Come up with ideas for each pathway•Prioritize experimental testing•Spend time on measurement infrastructure (a test without ability to measure = an unvalidated feature)

Level of Detail / Close to done

•Always revisit validation tests for size •The execution board is equivalent to the team’s task board, but at the Idea Card level

Idea#1.1

ExecutionValidated?

Idea

#1.2

Idea

#1.3

Idea

#2.1

Idea

#2.2

To Do Observation

Idea

#2.3

Idea

#2.4

Idea

#1.4

Idea

#1.6

Idea

#1.5


• In 2009, Google ran approximately 12,000 experiments, of which about 10% led to business changes

• Linus Pauling famously said, "The best way to have a good idea is to have a lot of ideas."

• Focus on minimizing the incremental cost of testing ideas with your customers/users

How Much Effort on ValidationFeature usage in

enterprise software

45%

19%

16%

13%

7%

NeverRarelySometimesOftenAlways

Standish Group - CHAOS Report, 1995, 2002


Feedback loops make all the difference

In November 2007, economists in the Survey of Professional Forecasters — examining some 45,000 economic-data series — foresaw less than a 1-in-500 chance of an economic meltdown as severe as the one that would begin one month later.

http://www.nytimes.com/2012/09/09/magazine/the-weatherman-is-not-a-

moron.html

In 1940, the chance of an American being killed by lightning was about 1 in 400,000.

Today it’s 1 in 11 million.


Lean Product ManagementThe Value of Experiments

• Validate requirements trump everything• Incremental cost of validation is falling• Feedback means nothing without a model

Photo: ESAhttp://www.nbcnews.com/id/50677062/ns/technology_and_science-space

The race doesn’t go to the one with the best idea, but to the one with the most tested ideas

agile42 | We advise, train and coach companies building software www.agile42.com | All rights reserved. Copyright © 2007 - 2009.

thank you

[email protected]: dave.sharrock

twitter: @davesharrockslides: slideshare.net/davesharrock

“Coming together is a beginning. Keeping together is progress. Working together is success.”

Henry Ford

productcamp vancouver 2013

Technology

coach companies

software development

high cost of change

totalinvestment cost

change leads

changeyour user model

requirements specication

customer cares