a grammar-based entity representation framework for data cleaning authors: arvind arasu raghav...
TRANSCRIPT
A Grammar-based Entity Representation Framework forData Cleaning
Authors: Arvind Arasu Raghav Kaushik
Presented by Rashmi Havaldar
Problems Poor data quality is due
to lack to unique representations for real world entities
Eg: California can be represented as California, Calif, CA, etc
Although textually different, these 5 records correspond to just 2 authors
Problem Definition Main problem in data cleaning is to determine whether
or not two representations are duplicate i.e. correspond to same real world entity.
Cosine similarity and Edit distance use textual similarity. But it can be misleading.
Two representations of same entity can be highly dissimilar
Conversely, two representations that are textually very similar can correspond to different entities
Solution: Programmable Framework
Basic Definitions The Program is a collection of triples of the form <R,P,A> where R is
the grammar rule, P is predicate and A is action The grammar rule has a head and body. Head is single non terminal
and body is sequence of non terminals, terminals and variables Terminals are words and punctuation Non terminals are represented by angular brackets
terminals using single quoted strings (eg:’Jeff’) and variables using uppercase letters
Example: Framework program
Expanded program G’ for program G Expanded program G’, like G is a collection of augmented rules To construct G’, we consider each augmented rule R=<R,P,A> and
enumerate all possible assignments of constant values to variables in R so that predicate P evaluates to true i.e. <R’, true, A’>
Parse Tree:
Handles variations in the order in which the first name and last name appear
Program handles variations resulting from the use of nick name
Weights: Non negative real numbers are assigned to each augmented
rule in G’ The weight of an output record is the sum of weights of
augmented rules involved in the parsing of output record Lower weights indicate high confidence Programmer can use “loose” rules, rules that the programmer
is not very confident about. Higher weights assigned to “loose” rules If R’ is augmented rule in expanded program G’, the weight of
R’ is the log of number of rules in G’
Implementation Given a program G, we can construct expanded program G’.
Given an input record r, we can use traditional parsing technique to parse r
But the main problem with this approach is that the scale of the expanded program G’ can be very large
Instead, construct Gr’, a partially expanded program at query time.
To construct Gr’, consider R=<R,P,A> and enumerates all possible assignment of constants to variables in R such that P evaluates to true
Enforce an additional constraint, if variable X occurs in R, then the constant c assigned to variable X should be a substring of the record r.Dictionary (X): P(X,.…)
Eg: Smith Andy, J: Dictionary (N): Nicknames (I,N,F,G)
Case studies1. UCD people data
Quality of record matching and Record matching
2. Author Affiliation Dataset
Program:
Discussion
Record matching: Previous works on record matching focused on similarity design
function This framework indicates that, with right pre processing the
need for approximate equality when performing record matching is minimized and often eliminated
How ever string similarity joins are needed to capture variations such as typos and misspellings
This framework does not intend to replace this body of work
Pay as you go: The goal of this framework is not to clean the entire
dataset, because doing so is difficult This framework rather approaches “pay as we go” where
they use example reference tables that cover only part of data to clean a subset of data
Lineage: Parse trees constitute a natural notion of lineage that can be
used to program on top of the module For eg. Data cleaning developer using this framework can
choose not to use rule weighting options and use if- then- else logic to capture parse tree preferences
Uncertainty: Framework provides a tool to manage uncertainty in the data Framework incorporates “possible worlds”. Thus it allows
multiple possible variations of same entity. Framework also returns multiple parse trees for same input
record with accompanying score.
Questions???
Thank you!