a case study of bias in bug-fix datasets
TRANSCRIPT
![Page 1: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/1.jpg)
SAIL, School of Computing, Queen’s University, Kingston, Canada
A Case Study of Bias in Bug-Fix Datasets
Thanh H. D. Nguyen, Bram Adams, Ahmed E. Hassan
![Page 2: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/2.jpg)
2
We need bug prediction• Problem:
• Quality improvement resource is limited.• Solution:
• Bug prediction identifies defect-prone modules.
Our focus is data quality
![Page 3: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/3.jpg)
3
What if there is sample
bias?
![Page 4: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/4.jpg)
We should consider bias in our studies
Stanford graduate student housing survey
![Page 5: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/5.jpg)
![Page 6: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/6.jpg)
6
1
2
#1
#2
#2
Unlinked bugs have:Higher severityLess experience[Bird al et. 2009] Linkage Bias
![Page 7: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/7.jpg)
7
1
2
#1
#2
#2
![Page 8: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/8.jpg)
8
1
2
#1
#2
#2
Tagging BiasAbout 2/3 of all bugs
reports are not defects[Antoniol al et. 2008].
![Page 9: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/9.jpg)
9
Biases are threats to validity of software quality studies
• Because of linkage bias, our models:• neglect higher severity bugs.• neglect less experienced developers.
• Because of tagging bias, our models:• inaccurately consider more bugs that existed.
Do biases really exist? How do biases
affect our research?
![Page 10: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/10.jpg)
10
![Page 11: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/11.jpg)
11
![Page 12: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/12.jpg)
12
Near ideal data:Linkage is enforced.Tagging is provided.
![Page 13: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/13.jpg)
13
Severity
Experience
Maturity
Release pressure
Collaboration
✔✔−−−
−✔✔−−
Conjecture: Biases are properties of the
software process, not of missing links.
Do linkage biases exist in Jazz?
![Page 14: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/14.jpg)
14
Severity
Experience
Maturity
Release pressure
Collaboration
✔✔−−
✔
Question:How does
tagging biases affect our research?
Do tagging biases exist in Jazz?
![Page 15: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/15.jpg)
15
How tagging biases affect our research?
Files Defects + Tasks
A 5B 4C 6D 1
Defects only
3441
Not biasWhich we should use
BiasWhich we
normally use
Spearman: .94Pearson: .97
Conjecture: It might be ok to
use biased data.
![Page 16: A Case Study of Bias in Bug-Fix Datasets](https://reader035.vdocuments.us/reader035/viewer/2022062503/587484d71a28abc62f8b4d4f/html5/thumbnails/16.jpg)
16