Reconstructing historical populations from genealogical dataAn overview of methods used for aggregating data from GEDCOM files
Corry Gellatly
Department of Historyand Art HistoryUtrecht University
Workshop on Population ReconstructionIISH Amsterdam, 19-21 February 2014
1. Overview
Why build a large genealogical database by aggregating hundreds of genealogical data (GEDCOM) files?
Research increasingly requires big data, to:
Understand large-scale population dynamics
between regions
over time
social, biological, cultural and economic aspects
Detect rare or ‘small-effects’
epidemiology (disease and intervention)
inheritance (genetics)
comparative life histories
2. GEDCOM files
Why use GEDCOM files for population reconstruction?
Pros a standard file structure for representing information about
familial relationships and life events
most popular format for storage and exchange of genealogical data
used internationally and widely available online
Cons it is a highly flexible format that allows users to enter wildly
incorrect information (if they wish to)
3. Data aggregation
Single GEDCOM files typically contain only a few hundred individuals, so we import hundreds of files into a single genealogical database
There are broadly 3 steps between import of files and the output, which is usable research datasets
1.Screening (to reject poor quality files)
2.Data cleaning
3.Linkage / de-duplication
4. Screening
Screening is carried out for various errors, e.g.
low mean number of offspring per family individuals younger than 0 or very old (>110) impossible relationships (due to age difference between individuals) individuals occurring as different sexes missing individuals
If errors are detected, then the file is either:
removed (in the case of obvious errors)
retained for further checking (in the case of ambiguous errors):
e.g. where individuals have more than two parents – this can be due to adoption or incorrect family links between individuals
5. Cleaning
Example: date errors
If DOB is 1857
Born to 10 year old mother?
Wife 17 years older?
First of 5 children born atthe age of 39?
If DOB is actually 1875
Born to 28 year old mother?
Wife 1 year younger?
First of 5 children born atthe age of 21?
6. Dataset extraction
Definition of datasets is driven by research questions:
which timespan? which region? do we need complete families? do we need dates of birth, death, marriage?
The identification of links between genealogies (or removal of duplicate individuals) is done during the process of dataset extraction
7. Linkage, de-duplication
Linkage fields
Day of birth, marriage or death (DOB, DOM, DOD) Year of birth, marriage or death (YOB, YOM, YOD) Surname Given names Sex
Problems
YOB, YOM, YOD more common than DOB, DOM, DOD (particularly in older data) but less unique to each individual
High inconsistency in recording of given names Middle names included or excluded Middle names used instead of first names Abbreviated names Nicknames (sometimes in brackets)
8. Linkage, de-duplication
Trade-off between data coverage and quality
Surname, given name, DOB Low risk of false linkages, but high risk of missing linkages (due
to problems with given names) and low data coverage
Surname, DOB Low risk of false linkages, but low data coverage
Surname, YOB High risk of false linkages, but high data coverage
9. Group-linking method
Individuals are identifiable by those they are related to
This principle is being applied to the problem of genealogical data, in which many records have YOB, but not DOB and given names are somewhat unreliable for linking
Group-linking string
10. Group-link test
Test with single GEDCOM file containing no duplicates
2,082 individuals; 971 marriages; 681 conceptive relationships; 1,913 conceptions
11. Group-link test
Percentage data coverage x Percentage of unique records within that data (÷ 100) gives an estimation of linkage power
12. Missing data
What about missing information?
The information on the siblings of these individuals is probably missing. Why? Because they appeared at marriage
This data is left censored, because these individuals appeared in the data after the event we are measuring (i.e. number and sex of siblings).
13. Missing data
Depending on what type of links we are trying to find, we may want to break up the string
String to link individuals based on their siblings String to link individuals based on their marriages and children
14. Record de-duplication (1600-1699)
De-duplication of 17th century records from the genealogical database
Febrl program (Freely Extensible Biomedical Record Linkage)
17,488 records with Surname and YOB
Indexes Surname > YOB Surname > Group-link string 2 (sex + siblings) Surname > Group-link string 3 (sex + marriages + offspring)
Comparison function Winkler
Classifier KMeans
16. Record de-duplication (1600-1699)
Results
Examples of matches in highest weight category (1,914 matches)
17. Record de-duplication (1600-1699)
Results
Examples of matches in lower weight category (10,434 matches)
17. Further work
Record linkage
Refine a method of probabilistic data matching that can identify linkages
where typo errors or name variations occur possible date typos exist there are missing persons in the family structure
Group-linking algorithm
Using the group-linking string as a start point to then check for existence of birth, marriage and death dates of relatives (where these exist) and performing matches on these variables
Inherently based on probabilistic matching
18. Acknowledgements
Netherlands Organisation for Scientific Research (NWO)
Project number 276-53-008: “Nature or nurture? A search for the institutional and biological determinants of life expectancy in Europe during the early modern period”
Colleagues at Utrecht University
Tine De Moor Institutions for Collective Action team: