human-centered machine learning · 2016. 3. 4. · human-centered machine learning saleema amershi...
TRANSCRIPT
![Page 1: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/1.jpg)
Human-Centered Machine Learning
Saleema AmershiMachine Teaching Group, Microsoft Research
UW CSE 510 Lecture, March 1, 2016
![Page 2: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/2.jpg)
What is Machine Teaching?
![Page 3: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/3.jpg)
Can improve learning with better learning strategies:
•Note taking
• Self-explanation
•Practice
•Mnemonic devices
•….
![Page 4: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/4.jpg)
![Page 5: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/5.jpg)
![Page 6: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/6.jpg)
Machine Learning Machine Teaching
Images from http://thetomatos.com
![Page 7: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/7.jpg)
What is Machine Learning?
![Page 8: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/8.jpg)
What is Machine Learning?
“Process by which a system improves performance from experience.” – Herbert Simon
“Study of algorithms that improve their performance P at some task T with experience E” – Tom Mitchell
“Field of study that gives computers the ability to learn without being explicitly programmed” – Arthur Samuel
![Page 9: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/9.jpg)
Programming
6
5
3
1
8
7
2
1
2
3
5
6
7
8
![Page 10: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/10.jpg)
Programming
6
5
3
1
8
7
2
1
2
3
5
6
7
8
![Page 11: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/11.jpg)
Programming
![Page 12: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/12.jpg)
Programming
![Page 13: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/13.jpg)
f(x)≈y
![Page 14: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/14.jpg)
f( )≈2
![Page 15: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/15.jpg)
How do Machines Learn?
![Page 16: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/16.jpg)
How do Machines Learn?
Clean
DataModel
Apply Machine
Learning Algorithms
![Page 17: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/17.jpg)
How do Machines Learn?
Apply Machine
Learning Algorithms
Images from: https://www.toptal.com/machine-learning/machine-learning-theory-an-introductory-primer
![Page 18: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/18.jpg)
How do Machines Learn?
Clean
DataModel
Apply Machine
Learning Algorithms
Where do you get this data?
How should it be represented?
Which algorithm
should you use?
How do you know if its working?
![Page 19: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/19.jpg)
Investigating Statistical Machine Learning as a Tool for Software DevelopersPatel, K., Fogarty, J., Landay, J., and Harrison, B. CHI 2008.
![Page 20: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/20.jpg)
Methodology
Semi-structured interviews with 11 researchers.
5 hour think-aloud study with 10 participants. Digit recognition task.
![Page 21: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/21.jpg)
Applied Machine Learning is a Process
Collect Data Create Features Select Model Evaluate
Slide content from Kayur Patel
![Page 22: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/22.jpg)
![Page 23: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/23.jpg)
Applied Machine Learning is a Process
Collect Data Create Features Select Model Evaluate
![Page 24: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/24.jpg)
Applied Machine Learning is a Process
Collect Data Create Features Select Model Evaluate
![Page 25: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/25.jpg)
What About Music Recommendation?
Collect Data Create Features Select Model Evaluate
Genre: Rock
Tempo: Fast
Drums: Yes
Time of day: Afternoon
Recently heard: No
….
![Page 26: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/26.jpg)
Problems with current tools
Don’t support machine learning as an iterative and exploratory process.
Image from: http://www.crowdflower.com/blog/the-data-science-ecosystem
![Page 27: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/27.jpg)
Problems with current tools
Don’t support machine learning as an iterative and exploratory process.
Don’t support relating data to behaviors of the algorithm.
LogitBoostWith8To18EvenWindow-Iter=10.model
LogitBoostWith8To18EvenWindow-Iter=20.model
SVMWith8To18EvenWindow-Iter=10.model
….
![Page 28: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/28.jpg)
Problems with current tools
Don’t support machine learning as an iterative and exploratory process.
Don’t support relating data to behaviors of the algorithm.
Don’t support evaluation in context of use.
![Page 29: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/29.jpg)
Model Performance is Important
Clean
DataModel
Apply Machine
Learning Algorithms
![Page 30: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/30.jpg)
What are other considerations?
Collect Data Create Features Select Model Evaluate
![Page 31: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/31.jpg)
Computational Efficiency?
Collect Data Create Features Select Model Evaluate
![Page 32: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/32.jpg)
Data processing efficiency?
Collect Data Create Features Select Model Evaluate
“Data scientists, according to interviews and
expert estimates, spend 50 percent to 80 percent
of their time mired in this more mundane labor
of collecting and preparing unruly digital data.”
- New York Times, 2014
![Page 33: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/33.jpg)
Understandability?
Collect Data Create Features Select Model Evaluate
“TAP9 initially used a decision tree algorithm
because it allowed TAP9 to easily see what
features were being used…Later in the
study…they transitioned to using more complex
models in search of increased performance.”
![Page 34: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/34.jpg)
Considerations for Machine Learning
Model performance
Computational efficiency
Iteration efficiency
Ease of experimentation
Understandability
….
New opportunities for HCI research!
Need to make tradeoffs!
![Page 35: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/35.jpg)
Interactive Machine LearningFails, J.A. and Olsen, D.R. IUI 2003.
![Page 36: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/36.jpg)
Crayons: IML for Pixel Classifiers
![Page 37: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/37.jpg)
What tradeoffs did Crayons make?
Collect Data Create Features Select Model Evaluate
![Page 38: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/38.jpg)
What tradeoffs did Crayons make?
Collect Data Create Features Select Model Evaluate
![Page 39: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/39.jpg)
“Classical” ML Interactive ML
![Page 40: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/40.jpg)
What tradeoffs did Crayons make?
Rapid iteration
• Fast training
• Integrated environment
Simplicity
Model performance
Flexibility
• Automatic featuring
• No model selection
![Page 41: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/41.jpg)
When are these tradeoffs appropriate?
Rapid iteration
Simplicity
Novices
Large set of available features
Data can be efficiently viewed and labeled
Model performance
Flexibility
Experts
Custom features needed
Data types that can’t be viewed at a glance
Labels obtained from external sources
![Page 42: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/42.jpg)
Flock: Hybrid Crowd-Machine Learning ClassifiersCheng, J. and Bernstein, S. CSCW 2015.
![Page 43: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/43.jpg)
Collect Data Create Features Select Model Evaluate
“At the end of the day, some machine learning
projects succeed and some fail. What makes the
differences? Easily the most important factor is
the features used.”
[Domingos, CACM 2012]
![Page 44: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/44.jpg)
How do people come up with features?
Look for features used in related domains.
Use intuition or domain knowledge.
Apply automated techniques.
Feature ideation – think of and experiment with custom features.
![Page 45: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/45.jpg)
A Brainstorming Exercise
![Page 46: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/46.jpg)
How do people come up with features?
Look for features used in related domains.
Use intuition or domain knowledge.
Apply automated techniques.
Feature ideation – think of and experiment with custom features.
“The novelty of generated ideas increases as participants
ideate, reaching a peak after their 18th instance.”
[Krynicki, F. R., 2014]
![Page 47: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/47.jpg)
Workflow
User specifies a concept and uploads some unlabeled data.
Crowd views data and suggests features.
![Page 48: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/48.jpg)
What makes a cat a cat?
![Page 49: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/49.jpg)
What makes a cat a cat?
![Page 50: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/50.jpg)
Workflow
User specifies a concept and uploads some unlabeled data.
Crowd compares and contrasts positive and negative examples and suggests “why” they are different. Reasons become features.
Reasons are clustered.
User vets, edits, and adds to features.
Crowd implements feature by labeling data.
Features used to build classifiers.
![Page 51: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/51.jpg)
Collect Data Create Features Select Model Evaluate
Collect Data Create Features Select Model Evaluate
Cra
yons
Flo
ck
![Page 52: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/52.jpg)
Collect Data Create Features Select Model Evaluate
Positives
Negatives
Standard Ranked List Split Technique (Best/Worst Matches)
[Fogarty et al., CHI 2007]
![Page 53: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/53.jpg)
Collect Data Create Features Select Model Evaluate
Traditional Labeling
Grouping and
tagging surfaces
decision making.
Moving, merging
and splitting
groups helps
with revising
decisions.
[Kulesza et al., CHI 2014]
Structured Labeling
![Page 54: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/54.jpg)
Collect Data Create Features Select Model Evaluate
[Amershi et al., CHI 2015]
Sum
mary
Sta
tsM
od
elT
rack
er
![Page 55: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/55.jpg)
Collect Data Create Features Select Model Evaluate
[Patel et al., IJCAI 2011]
![Page 56: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/56.jpg)
Collect Data Create Features Select Model Evaluate
Rule
-base
d e
xpla
natio
n
Keyw
ord
-base
d e
xpla
natio
n
Sim
ilarity
-base
d e
xpla
natio
n
[Stumpf et al, IUI 2007]
![Page 57: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/57.jpg)
Collect Data Create Features Select Model Evaluate
![Page 58: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/58.jpg)
Experts Everyday PeoplePractitioners
![Page 59: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/59.jpg)
Experts Everyday PeoplePractitioners
How are these scenarios different?
User experience impacts what you can expose.
Interaction focus impacts attention and feedback.
Accuracy requirements impacts expected time and effort.
…..
![Page 60: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/60.jpg)
Principles for human-centered ML?
Traditional User Interfaces
Visibility and feedback
Consistency and standards
Predictability
Actionability
Error prevention and recovery
….
Intelligent/ML-Based Interfaces
Safety
Trust
Manage expectations
Degrade gracefully under uncertainty
….
![Page 61: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/61.jpg)
Traditional Machine Learning
Clean
DataModel
Apply Machine
Learning Algorithms
![Page 62: Human-Centered Machine Learning · 2016. 3. 4. · Human-Centered Machine Learning Saleema Amershi Machine Teaching Group, Microsoft Research UW CSE 510 Lecture, March 1, 2016](https://reader034.vdocuments.us/reader034/viewer/2022051902/5ff245970b81ab0d4372ef69/html5/thumbnails/62.jpg)
Human-Centered Machine Learning
Collect Data Create Features Select Model Evaluate