improving family search indexing efficiency and quality
DESCRIPTION
RootsTech workshop presentationTRANSCRIPT
D E R E K H A N S E N , J A K E G E H R I N G , PAT R I C K S C H O N E , A N D M AT T H E W R E I D
FAMILY HISTORY TECHNOLOGY WORKSHOPFEBRUARY 3, 2012
IMPROVING INDEXING EFFICIENCY & QUALITY:COMPARING A-B-ARBITRATE AND PEER REVIEW
FAMILYSEARCH
FAMILYSEARCH INDEXING
A-B-ARBITRATE PROCESS (A-B-ARB)
A
B
ARB
THE PROBLEM
Time
Am
ou
nt Scanned
Documents
OUR APPROACH
•Historical Data Analysis•Field Experiment comparing quality control models
HISTORICAL DATA ANALYSIS
• Quality (estimated based on A-B agreement)• Measures difficulty more than actual quality• Underestimates quality, since an experienced Arbitrator
reviews all A-B disagreements• Good at capturing differences across people, fields, and
projects
• Time (calculated using keystroke-logging data)• Idle time is tracked separately, making actual time
measurements more accurate• Outliers removed
A-B AGREEMENT BY FIELD
A-B AGREEMENT BY LANGUAGE
English Language
• Given Name: 79.8• Surname: 66.4
French Language
• Given Name: 62.7%• Surname: 48.8%
1871 Canadian Census
A-B AGREEMENT BY EXPERIENCE
Birth Place: All U.S. Censuses
B (
novic
e ↔
exp
ert
)
A (novice ↔ expert)
A-B AGREEMENT BY EXPERIENCE
Given Name: All U.S. Censuses
B (
novic
e ↔
exp
ert
)
A (novice ↔ expert)
A-B AGREEMENT BY EXPERIENCE
Surname: All U.S. Censuses
B (
novic
e ↔
exp
ert
)
A (novice ↔ expert)
A-B AGREEMENT BY EXPERIENCE
Gender: All U.S. Censuses
B (
novic
e ↔
exp
ert
)
A (novice ↔ expert)
A-B AGREEMENT BY EXPERIENCE
U.S. - English Canada - English
Canada - FrenchMexico - Spanish
TIME & KEYSTROKE BY EXPERIENCE
TIME & KEYSTROKE OF ARB
A NEW APPROACH? (A-R-ARB)
• Peer review model• Efficiency ++•Quality ?
PEER REVIEW PROCESS (A-R-ARB)
A R ARB
Already Filled In Optional?
FIELD EXPERIMENT
• Develop Truth Set of 2,000 1930 Census images• Use historical A-B-ARB data• Create new A-R-ARB dataset by having
new indexers review and arbitrate• Compare quality & efficiency• Qualitatively identify types of errors
DISCUSSION
IMPLICATIONS• Transition users from novice to expert• Recruit foreign language indexers• Intelligent matching based on expertise
(in A-B-ARB &/or A-R-ARB)
FUTURE POSSIBILITIES• Peer review by algorithms?• Initial indexing by algorithms?
QUESTIONS
• Derek Hansen ([email protected])• Jake Gehring ([email protected])• Patrick Schone ([email protected])• Matthew Reid ([email protected])