incentivize crowd labeling under budget constraint qi zhang, yutian wen, xiaohua tian, xiaoying gan,...
TRANSCRIPT
![Page 1: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/1.jpg)
Incentivize Crowd Labeling under
Budget Constraint
Qi Zhang, Yutian Wen, Xiaohua Tian,
Xiaoying Gan, Xinbing Wang
Shanghai Jiao Tong University, China
![Page 2: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/2.jpg)
2
Outline Introduction to Crowdsourcing Mechanism
Problem Formulation and Mechanism Setting
Mechanism Analysis
Performance Evaluation
2
![Page 3: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/3.jpg)
Background
Crowdsourcing systems leverage human wisdom to
perform tasks, such as:
Image classification
Character recognition
Data collection
3
![Page 4: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/4.jpg)
Types of Tasks
Tasks can be divided into two categories:
Structured response format
Binary choice
Multiple choice
Real Value
Unstructured response format
Logo design
4
![Page 5: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/5.jpg)
5
Motivating Example
Example: Image classification
Workers
Allocation
Crowdsourcing
Platform
Task Dog
Dog
Cat
Cat
Dog
Inference
AlgorithmDog
![Page 6: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/6.jpg)
6
Framework: Reverse Auction
(1)Tasks
(2)Bids
(3)Winning bids determination
(4)Winning bids
(6)Payments
(5)Answers
![Page 7: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/7.jpg)
7
Major Challenges(1)
To design a successful crowdsourcing system
Task Allocation (winning bids)
• Tasks should be allocated evenly
Payment Determination:
• Must provide proper incentives (monetary rewards)
Inference Algorithm:
• Should improve overall accuracy
• Should address the diversity of the crowd
![Page 8: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/8.jpg)
8
Major Challenges(2)
We need to model on
Diverse task difficulty
• Dog or Cat
• Older than 30 or Not
Diverse worker quality
Cat
![Page 9: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/9.jpg)
9
Model on Tasks(1)
We focus on binary choice tasks
Each task is a 0 – 1 question
(Assumption) Each worker is uniformly reliable
Task Soft Label
• Probability that the task is labeled as 1( by a reliable worker)
Crowd Label 0 or 1
![Page 10: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/10.jpg)
10
Model on Tasks(2)
The soft label is viewed as a random variable drawn from Beta distribution
Update parameters (a,b) by Bayes rule
Inference
The task is inferred as 1
Prior Parameters
PriorPosterior
Likelihood
More than half agree
![Page 11: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/11.jpg)
11
Framework: Reverse Auction
The platform publicizes a set of binary tasks
Workers reply with a set of bids
• Each bid is a task-price pair
(Allocation) The platform sequentially decide winning bids
(Payment) Winning workers provide labels and get payment
![Page 12: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/12.jpg)
12
Crowdsourcing Platform Utility
After observing all crowd labels , the distribution is updated as
Platform Utility: KL Divergence between
the initial and the final distribution
![Page 13: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/13.jpg)
13
Problem Formulation
We want:
Platform utility maximization under budget constraint
Individual rationality
Truthful about the cost
Truthful bid Untruthful bid
Computation Efficiency
![Page 14: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/14.jpg)
14
Allocation Scheme (1)
The task allocation(winning bid determination) is sequential :
Candidate selection
• one candidate a round
Proportional rule check
Answer collection & Soft label update
The allocation scheme repeats the 3 steps until
Remaining bids Candidate
Discard
Winning bid
All bids
Discard Winning bids
![Page 15: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/15.jpg)
15
Allocation Scheme (2)
The candidate selection is greedy
• The largest platform utility gain per unit price
• Platform utility gain:
PU Gain
Price
Candidate
Updated distributionCurrent distribution
![Page 16: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/16.jpg)
16
Allocation Scheme (3)
Proportional rule check
Soft label update
• Collect the answer from the winning bid
• Update the soft label according to Bayes rule
price
budget
fraction ratio
![Page 17: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/17.jpg)
17
Allocation Scheme (5)
Candidate selection
Proportional rule check
Soft label update
Computationally efficient !
![Page 18: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/18.jpg)
18
Payment Scheme(1)
p(C) = max {b1,b2, b3, b4}
Winning bids
{A, B, C}
Discard
{D, E, F}
Kick out C
{ A,B,D,E,F }
Winning bids
{A, B, D, E}
Discard
{F}
C
b1b2 b3 b4
b1 is the minimum price
that bid C can replace bid A
![Page 19: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/19.jpg)
19
Payment Scheme(2)
(Proposition)The winning bid C is paid threshold payment.
p(C) C’s payment, b(C) C’s bid
if b(C) < p(C), C is a winning bid
if b(C) > p(C), C is discarded
p(C)=max { b1, b2, b3, b4}
Winning bids
{A, B, D, E}
C
b1b2 b3 b4
![Page 20: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/20.jpg)
20
Payment Scheme(3)
(Proposition)The incentive mechanism is truthful
Each bid has a cost
Workers will truthfully reveal the cost as asked price
Why?
Proof: Threshold payment + Greedy candidate Selection
![Page 21: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/21.jpg)
21
Individual Rationality
(Proposition)The incentive mechanism is individual rational
The utility of a winning bid is nonnegative
Proof : Let us consider the winning bid C
1. C is the 3rd
winning bid.
2. The first 2 bids are the same
3. b3 is the minimum price
that bid C can replace the new 3rd
bid (D)
It is true that b3 > b(c) !
p(C) = max {b1, b2, b3, b4}, p(C) > b3
p(C) > b(C)
New Winning bids
{A, B, D, E}
b1b2 b3 b4
{ A, B, C}
Original wining bids
![Page 22: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/22.jpg)
22
Budget Feasibility
(Proposition, Payment Bound) Payment to each winning bid
is upper bounded by
• Proportional rule:
• Set
![Page 23: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/23.jpg)
23
Performance Evaluation(1)
Benchmark
1. Untruthful Allocation: Workers’ cost is public information
2. Random Allocation: Candidate selection is random
Truthful Running Time
Platform Utility
Benchmark 1 High
Benchmark 2 Low Low
My Mechanism High
![Page 24: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/24.jpg)
24
Performance Evaluation(2)
Metric 1 : Platform Utility
• Platform utility vs. Budget
Price of Truthfulness
Gain over random allocation
![Page 25: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/25.jpg)
25
Performance Evaluation(3)
Metric 2 : Budget Utilization
• Payment / Budget
Budget utilization gain
Over random allocation
![Page 26: Incentivize Crowd Labeling under Budget Constraint Qi Zhang, Yutian Wen, Xiaohua Tian, Xiaoying Gan, Xinbing Wang Shanghai Jiao Tong University, China](https://reader038.vdocuments.us/reader038/viewer/2022110319/56649c755503460f94928f21/html5/thumbnails/26.jpg)
Thank you !
Presented by : Qi Zhang