iterate. calculate. communicate
TRANSCRIPT
ITERATE. CALCULATE. COMMUNICATE.
CLOSING THE GAP BETWEEN MEASURE + LEARN
LinkedIn, Medium & Twitter: @laurex
LAURE PARSONS SENIOR PRODUCT MANAGER
“THE GOAL OF BUILD-MEASURE-LEARN IS NOT TO BUILD A FINAL PRODUCT TO SHIP OR EVEN TO BUILD A PROTOTYPE OF A PRODUCT, BUT TO MAXIMIZE LEARNING THROUGH INCREMENTAL AND ITERATIVE ENGINEERING”
Steve Blank, author of The Four Steps to the Epiphany
BE ACTIONABLE!
1. WHAT SHOULD YOU TRACK? 2. WHAT MAKES A METRIC “GOOD?”
3. HOW CAN YOU GET ALIGNMENT WITH YOUR TEAM AND LEADERSHIP?
4. WHAT SHOULD YOU DO TO ENSURE YOU LEARN?
‣ CASE STUDY: BUILDING A NEW FEATURE!
ACQUISITION ACTIVATION REFERRAL RETENTION REVENUE
AARRR: PIRATE METRICS
ACQUISITION ACTIVATION REFERRAL RETENTION REVENUE TEAM
AARRRT: TEAM METRICS
THE ONE METRIC
From Lean Analytics by Benjamin Yoskovitz & Alistair Croll
UX TESTS MVP A/B TESTSCLASSIC LEAN MEASUREMENT
WHO, WHAT, WHEN, WHERE, WHY & HOW
▸WHO CUSTOMERS ARE
▸WHAT CUSTOMERS DO AND/OR TRY TO DO
▸WHEN DO USERS DISENGAGE
▸WHERE CUSTOMERS COME FROM
▸WHY CUSTOMERS CONVERT
▸HOW MUCH YOU CAN BUILD
GOOD METRICS
GOOD METRICS
COMPARABLE
RATIOS & RATES
GOOD METRICS
UNDERSTANDABLE
GOOD METRICS
ACTIONABLE
BE ACTIONABLE!
TWO TYPES OF ACTIONABLE DATA▸“Accounting” metrics can make your
predictions more accurate
▸“Experimental” metrics help optimize product, pricing, or market. Agree on what will change before you collect the data.
“IF YOU MEASURE SOMETHING AND IT’S NOT ATTACHED TO A GOAL, IN TURN CHANGING YOUR BEHAVIOUR, YOU’RE WASTING YOUR TIME.”
LEAN ANALYTICS by Ben Horowitz and Alistair Kroll
BE ACTIONABLE!
ALIGNMENT & COMMUNICATION
OKRS
WHO OWNS THE DATA?
WHAT ACTIONS WILL YOU TAKE?
WHO NEEDS TO KNOW?
HOW WILL THEY KNOW?
THE DATA TOOL YOUR TEAM WILL ACTUALLY USE
‣A single source of truth
‣Company-wide common understanding and definitions
‣Appropriate permissions
‣Transparency
‣Collaboration via shared data and commenting
‣Automated integration from your tools
‣Sharing in Slack, where your team is already talking
‣Dashboards to track the important things for each team
‣One-click reports to share with management and stakeholders
CASE STUDY
CASE STUDY QUESTIONS
DISCOVERY & LEAN CUSTOMER DEVELOPMENT▸ Based on our problem
hypothesis, what percentage of the 6-10 interviews we conduct confirm what we think is the problem?
CASE STUDY QUESTIONS
MVP
▸ Solves a problem in the simplest way possible
▸ Could actually be better than the intended product, with “concierge onboarding"
▸ Asks a question, which should be clear and measurable.
CASE STUDY QUESTIONS
EFFORT TO BUILD▸ How much dev effort is
anticipated in points or cost per developer hour?
▸ Reduce scope to the most essential parts of the feature. Build “nice to haves” when you have signal
▸ Track velocity, throughput, cycle times to anticipate & understand building costs
EFFORT TO BUILD
CASE STUDY QUESTIONS
WHO CARES?▸ Beta test the feature with customers
who’ve requested it and/or with freemium or opt-in customers
▸ If you have a large enough customer base, survey your customers with a tool like Intercom or Qualaroo when they use the feature
▸ Measure feature adoption in Mixpanel or other event-based tools. See how often customers try the feature and how many times they re-use it.
CASE STUDY QUESTIONS
COSTS BEYOND DEVELOPMENT
▸ Track support requests tagged with the feature name
▸ Track marketing spend on launching the feature
▸ Track escaped bugs
CASE STUDY QUESTIONS
THE BOTTOM LINE
▸ Do a cohort analysis of customers before and after the feature was introduced.
▸ Track number of signups, churn, and referrals.
▸ If the feature is plan-gated, you can also track by pricing segments.
WHAT ACTION FOLLOWS?
COMMUNICATION CHECKLIST
ARE YOU COMMUNICATING?▸ Does your whole team and company know why
you are tracking these metrics?
▸ Do you have agreed-upon actions based on the results?
▸ Do you have to track down people to get the data?
▸ Does someone own the metric?
DON’T STOP NOW!
LEARN MORE
▸Lean Analytics by Benjamin Yoskovitz & Alistair Croll
▸Getting Started with OKRs & Metrics Guide
▸The School of Little Data
▸The AARRRT of Product Metrics
THANK YOU!