product management for ai/ml

Product Management for AI/ML

The Product Mentor, Season 7

Resources available:https://goo.gl/TfUxac

Chris ButlerDirector of Prod Strat @ Philosophie NYC

The Best Product Person 2016

17 years of product and BD

Microsoft, Waze, Horizon Ventures, KAYAK, and started my own company (failed)

chrisbutler@philosophie.is

@chrizbot

Product management for AI/ML

● What do I need to know about these things?

● How do they impact product’s role○ Purpose and strategy

○ Learning

○ Building

○ Prioritizing

○ Measuring

○ Technical

What is...artificial intelligence?...artificial general intelligence?...narrow artificial intelligence?

...a neural network?...machine learning?

...deep learning?…?

Where to start?

Learns from from (good) data

Attempts to reduce an error against desired outcomes

Why are AI programs different?

● Content: models, not programs

● Process: training, not debugging

● Release: retraining, not patching

● Uncertainty: of objective

● Uncertainty: of action and recommendation

● Uncertainty: propagates through model

Training and inference

Types of problems it can solve (possibly)

● Ranking - Google search results

● Recommendation - Netflix movie recommendations

● Regression (or prediction) - Zillow predicting house prices

● Classification - Image is a cat or dog

● Clustering - Tumblr social network analysis to find groups

of topics

● Supervised

● Unsupervised

● Supervised

● Unsupervised

● Reinforcement

● Semi-supervised

● One shot

● Few shot

Types of learning

● Transfer

● Active

● Imitation

● Q

● Transduction

● ...

Resources to start learningBooks

● Programming Collective Intelligence by Toby Segaran● The Master Algorithm by Pedro Domingos● Introduction to Machine Learning by Nils Nilsson● Data Mining by Ian Whitten● Data Science for Business by Foster Provost● Neural Networks and Deep Learning by Michael

Nielsen● Make Your Own Neural Network by Tariq Rashid

Courses

● Introduction to Machine Learning by Andrew Ng (highly recommended)

● Machine Learning Engineer by Udacity● Machine learning is Fun! by Adam Geitgey● How to use Tensorflow for Classification by Siraj

Raval● Learning AI if you suck at Maths by Daniel Jeffries● Machine Learning Mastery by Jason Brownlee● Machine Learning by Georgia Tech (Udacity)

Must-Reads:

● WTF is Artificial Intelligence by Sam DeBrule● Machine learning for Product Managers by Ken

Norton● AI, Deep Learning, and Machine Learning: A Primer

by Frank Chen● Artificial Intelligence is the new electricity by Andrew

Ng● The current state of Machine Learning by Shivon Zilis● How Google is remaking itself a ‘Machine Learning

First’ company by Steven Levy● An executives guide to machine learning by Dorian

Pyle (Mckinsey)● Experience Design in the Machine Learning Era by

Fabien Girardin● A human’s guide to Machine learning by Sam DeBrule

(subscribe to his newsletter)● What every manager should know about Machine

Learning by Mike Yeomans● An introduction to Machine Learning theory and its

application by Nick Mccrea● Machine Learning pitfalls by Ben Hamner

https://hackernoon.com/machine-learning-and-product-managers-930b691b1b37

Fast.ai MOOC is very hands on

Unless you are in research, the real focus should be on what differentiates your product and gives it meaning. Not finding a better way to detect the difference between cats and dogs in ImageNet images.https://uxdesign.cc/robots-need-love-too-empathy-mapping-for-ai-59585ad3548d

Purpose and strategy

Without human purpose, a computer is just a rock that we tricked into thinking.

https://uxdesign.cc/robots-need-love-too-empathy-mapping-for-ai-59585ad3548d

Design Thinking, Lean, and Agile?

Design Thinking(AKA Human-Centered Design)

Human at the center being assisted and augmented by AI/ML

Lean

Start small with simple models that build confidence in what you are doing

Agile

Continuously improve the data, the model, training, etc.

Design Thinking, Lean, Agile, and AI are about emergent practice

Design Thinking, Lean, Agile, and AI are about learning and adapting

How to learn about purpose

When using AI, you want to know:

● Are we helping solve a problem?

● Do they trust the information?

● Do they feel comfortable giving feedback to the system?

Rich Picture

Empathy Mapping (for the machine)

Confusion matrix

Decision boundaries

Patterns for building with purpose

Self driving cars classifications

Does everything Learning Watching

Approving Confident Recommending

Veto’ing Proven Taking action

Human System State Machine Action

Intelligent CTA

“Calculating” and explainability

Be up front about possible errors

and sometimes wrong

Feedback

Wizard of Oz to learn

Bootstrapping

How to prioritize with purpose

Prioritization

Outcome Mapping

How to measure with purpose

Research questions

● Think back to the last time you did this, how did you come to

that decision?

● Do you trust these suggestions for what to do next?

● How do you think the system decided [action]?

● Was there enough information for you to [take action]?

● How much do you trust the system to make the right decision in

the future? It is more or less than before?

Gathering feedback from people

Gathering feedback from people

Back to the confusion matrix!

Technical concerns

Simulation and QA

Scaling

Technical debt

***

In closing

In closing

● We give machines their purpose - focus on problems you

are solving, not new toys

● Building these systems are a journey - iterate and learn

● We deal with nondeterministic systems all day in our

teams, industries, and markets - AI is no different

Don’t get stuck with a rock that doesn’t help you meet your purpose

Thank you

For more information:https://goo.gl/TfUxac

Appendix: removed slides

Background for internal review● Audience: product people (new and experienced)

● When: 9/17/17

● Alternate use: planning on using parts for Design Thinking for AI workshops

● Feedback needed:○ Good enough overview of AI? Design Thinking/Lean?

○ Does it feel like a good journey/order?

○ Anything unnecessary? Missing?

○ Did you learn something?

Definition

“A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.”

-Tom Mitchell, 1997

Design Thinking and Lean, not either/or

Diverge and converge

The “spiral”

Stanford d.school Design Thinking process

Learn

Build

Measure

Iterate

Lean

Empathy Mapping (for the machine)

Iterate

Perfect is the enemy of good… ...and not possible with AI