product management for ai/ml
TRANSCRIPT
Product Management for AI/ML
The Product Mentor, Season 7
Resources available:https://goo.gl/TfUxac
Chris ButlerDirector of Prod Strat @ Philosophie NYC
The Best Product Person 2016
17 years of product and BD
Microsoft, Waze, Horizon Ventures, KAYAK, and started my own company (failed)
@chrizbot
Product management for AI/ML
● What do I need to know about these things?
● How do they impact product’s role○ Purpose and strategy
○ Learning
○ Building
○ Prioritizing
○ Measuring
○ Technical
What is...artificial intelligence?...artificial general intelligence?...narrow artificial intelligence?
...a neural network?...machine learning?
...deep learning?…?
Where to start?
Learns from from (good) data
Attempts to reduce an error against desired outcomes
Why are AI programs different?
● Content: models, not programs
● Process: training, not debugging
● Release: retraining, not patching
● Uncertainty: of objective
● Uncertainty: of action and recommendation
● Uncertainty: propagates through model
Training and inference
Types of problems it can solve (possibly)
● Ranking - Google search results
● Recommendation - Netflix movie recommendations
● Regression (or prediction) - Zillow predicting house prices
● Classification - Image is a cat or dog
● Clustering - Tumblr social network analysis to find groups
of topics
● Supervised
● Unsupervised
● Supervised
● Unsupervised
● Reinforcement
● Semi-supervised
● One shot
● Few shot
Types of learning
● Transfer
● Active
● Imitation
● Q
● Transduction
● ...
Resources to start learningBooks
● Programming Collective Intelligence by Toby Segaran● The Master Algorithm by Pedro Domingos● Introduction to Machine Learning by Nils Nilsson● Data Mining by Ian Whitten● Data Science for Business by Foster Provost● Neural Networks and Deep Learning by Michael
Nielsen● Make Your Own Neural Network by Tariq Rashid
Courses
● Introduction to Machine Learning by Andrew Ng (highly recommended)
● Machine Learning Engineer by Udacity● Machine learning is Fun! by Adam Geitgey● How to use Tensorflow for Classification by Siraj
Raval● Learning AI if you suck at Maths by Daniel Jeffries● Machine Learning Mastery by Jason Brownlee● Machine Learning by Georgia Tech (Udacity)
Must-Reads:
● WTF is Artificial Intelligence by Sam DeBrule● Machine learning for Product Managers by Ken
Norton● AI, Deep Learning, and Machine Learning: A Primer
by Frank Chen● Artificial Intelligence is the new electricity by Andrew
Ng● The current state of Machine Learning by Shivon Zilis● How Google is remaking itself a ‘Machine Learning
First’ company by Steven Levy● An executives guide to machine learning by Dorian
Pyle (Mckinsey)● Experience Design in the Machine Learning Era by
Fabien Girardin● A human’s guide to Machine learning by Sam DeBrule
(subscribe to his newsletter)● What every manager should know about Machine
Learning by Mike Yeomans● An introduction to Machine Learning theory and its
application by Nick Mccrea● Machine Learning pitfalls by Ben Hamner
https://hackernoon.com/machine-learning-and-product-managers-930b691b1b37
Fast.ai MOOC is very hands on
Unless you are in research, the real focus should be on what differentiates your product and gives it meaning. Not finding a better way to detect the difference between cats and dogs in ImageNet images.https://uxdesign.cc/robots-need-love-too-empathy-mapping-for-ai-59585ad3548d
Purpose and strategy
Without human purpose, a computer is just a rock that we tricked into thinking.
https://uxdesign.cc/robots-need-love-too-empathy-mapping-for-ai-59585ad3548d
Design Thinking, Lean, and Agile?
Design Thinking(AKA Human-Centered Design)
Human at the center being assisted and augmented by AI/ML
Lean
Start small with simple models that build confidence in what you are doing
Agile
Continuously improve the data, the model, training, etc.
Design Thinking, Lean, Agile, and AI are about emergent practice
Design Thinking, Lean, Agile, and AI are about learning and adapting
How to learn about purpose
When using AI, you want to know:
● Are we helping solve a problem?
● Do they trust the information?
● Do they feel comfortable giving feedback to the system?
Rich Picture
Empathy Mapping (for the machine)
Confusion matrix
Decision boundaries
Patterns for building with purpose
Self driving cars classifications
Does everything Learning Watching
Approving Confident Recommending
Veto’ing Proven Taking action
Human System State Machine Action
Intelligent CTA
“Calculating” and explainability
Be up front about possible errors
and sometimes wrong
Feedback
Wizard of Oz to learn
Bootstrapping
How to prioritize with purpose
Prioritization
Outcome Mapping
How to measure with purpose
Research questions
● Think back to the last time you did this, how did you come to
that decision?
● Do you trust these suggestions for what to do next?
● How do you think the system decided [action]?
● Was there enough information for you to [take action]?
● How much do you trust the system to make the right decision in
the future? It is more or less than before?
Gathering feedback from people
Gathering feedback from people
Back to the confusion matrix!
Technical concerns
Simulation and QA
Scaling
Technical debt
***
In closing
In closing
● We give machines their purpose - focus on problems you
are solving, not new toys
● Building these systems are a journey - iterate and learn
● We deal with nondeterministic systems all day in our
teams, industries, and markets - AI is no different
Don’t get stuck with a rock that doesn’t help you meet your purpose
Appendix: removed slides
Background for internal review● Audience: product people (new and experienced)
● When: 9/17/17
● Alternate use: planning on using parts for Design Thinking for AI workshops
● Feedback needed:○ Good enough overview of AI? Design Thinking/Lean?
○ Does it feel like a good journey/order?
○ Anything unnecessary? Missing?
○ Did you learn something?
Definition
“A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P if its performance at tasks in T, as measured by P, improves with experience E.”
-Tom Mitchell, 1997
Design Thinking and Lean, not either/or
Diverge and converge
The “spiral”
Stanford d.school Design Thinking process
Learn
Build
Measure
Iterate
Lean
Empathy Mapping (for the machine)
Iterate
Perfect is the enemy of good… ...and not possible with AI