![Page 1: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/1.jpg)
The “Assembly Line” for the Information Age
Human-Computer Cooperation for Large-Scale Product Classification
Jianfu ChenComputer Science Department,
Stony Brook University
![Page 2: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/2.jpg)
Machines Transform Human History
![Page 3: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/3.jpg)
People have always been seeking the optimal way of integrating
machine and human labor.
![Page 4: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/4.jpg)
20th Century
Ford Assembly LineIntegrates Machine and Human Labor Efficiently
![Page 5: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/5.jpg)
21st Century – Information Age
“Mass Production” of Information
![Page 6: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/6.jpg)
We want to find the optimal ways to integrate machine and human intelligence.• NOT all products could be produced fully
automatically by machines– assembly line integrated machine & human labor
• NOT all information can be produced fully automatically by computers– We want to find optimal ways to integrate machine
and human intelligence• What’s the “Assembly Line” for the Information
Age?
![Page 7: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/7.jpg)
A Case Study: Large scale product classification
Kindle Fire HD 8.9" 4G LTE Wireless8.9" HD Display, Dolby Audio, Dual-Band Dual-Antenna Wi-Fi, 4G LTE, 32GB or 64GB
Goal: • optimally integrate computer and human effort• Achieve a lower unit cost for product classification• More precisely, optimize the accuracy-cost tradeoff
![Page 8: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/8.jpg)
An “Assembly Line” for Human Computer Cooperation
3Com V.35 cable V.35 cable ( DTE ) - DB-50 (M) - M/34 (V.35) (M) - 10 ft
26121609
A list of K candidate classes
System Accuracy
Machine Accuracy
Human AccuracyX
Cost is Human labor cost, i.e., the salary paid to workers, which is proportional to the working time spent.
=
![Page 9: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/9.jpg)
A quick glance at Accuracy-Cost Relation
• Assume K determines the Accuracy and Cost.• System Accuracy • Machine Accuracy increases as K increases
• Human Accuracy decreases as K increases
• Cost increases as K increases
![Page 10: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/10.jpg)
A quick glance at Accuracy-Cost Relation
![Page 11: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/11.jpg)
There is an optimal cost that gives the highest accuracy.
![Page 12: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/12.jpg)
Towards a more realistic analysis of accuracy-cost relationship
• With the above “assembly line” model, human accuracy and working time are influenced by a set of factors– K– Task difficulty– Expertise
• I am familiar with office supplies, but not familiar with nuts and bolts.
– Cognitive characteristics• Careful, smart, quick• Independent of the task
![Page 13: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/13.jpg)
Use a probabilistic graphical model to capture the cognitive process of human
classification• A probabilistic graphical model shows how the
above different factors interact with each other, and influence the accuracy and cost.
• Specifically, we use Bayesian Network, which characterizes the causal relationships of different factors.
![Page 14: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/14.jpg)
Use a Bayesian Network to predict accuracy and cost
𝑨𝒉 𝑨𝒎
𝑨
𝑻𝒊𝒎𝒆
𝑪𝒐𝒔𝒕
𝑪𝒐𝒈𝑴𝑲
𝑻𝒄 𝑬𝑫
![Page 15: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/15.jpg)
Not only visually intuitive, but also formal
𝑨𝒉 𝑨𝒎
𝑨
𝑻𝒊𝒎𝒆
𝑪𝒐𝒔𝒕
𝑪𝒐𝒈𝑴𝑲
𝑻𝒄 𝑬𝑫
𝑃 (𝐴h=1|𝐷 ,𝐾 ,𝑀 ,𝐶𝑜𝑔 )= 11+exp (− (𝑤1
𝑇𝐷+𝑤 2𝐾 +𝑤3𝑀 +𝑤4𝐶𝑜𝑔 )−𝛼)𝑇𝑖𝑚𝑒=𝑤 1
𝑇𝐷+𝑤2𝐾 +𝑤3𝑀+𝑤4𝐶𝑜𝑔+𝛽
![Page 16: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/16.jpg)
Inference and learning
• with this Bayesian Network, we predict the accuracy and cost by
• Training data– A set of examples with known class labels– We let each human worker work on multiple examples,
record the correctness and the working time• EM algorithm learns the parameters and the
hidden variables
![Page 17: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/17.jpg)
usage of the model
• Predict the accuracy-cost tradeoff– Given certain budget, what’s the highest accuracy
we can achieve?– To achieve certain accuracy, what’s the lowest
expected cost?• How to charge customers?
• Optimally assign the workers to the tasks
![Page 18: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/18.jpg)
Related Works
• time and motion study– Scientific management (Taylorism)
• Crowdsourcing– Amazon Mechanical Turk– learning worker expertise and accuracy
• Item Response Theory– Psychometrics• IQ test, GRE, GMAT
![Page 19: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/19.jpg)
Conclusion
• In information age, we need a new “assembly line” to integrate human and machine intelligence.
• We try to model human accuracy and working time by considering the interactions of a set of relevant factors, using a probabilistic graphical model.
• We use the model to predict the accuracy-cost tradeoff, decide how to charge customers, and optimally assign tasks to human workers.
![Page 20: The “ Assembly Line ” for the Information Age](https://reader035.vdocuments.us/reader035/viewer/2022081604/56816133550346895dd08bbf/html5/thumbnails/20.jpg)
Thank you!