AnHai DoanUniversity of Wisconsin-Madison

Data Quality Challenges in Data Quality Challenges in Community SystemsCommunity Systems

Joint work with Pedro DeRose, Warren Shen, Xiaoyong Chai, Byron Gao, Fei Chen, Yoonkyong Lee, Raghu Ramakrishnan, Jeff Naughton

Numerous Web CommunitiesNumerous Web Communities

Academic domains– database researchers, bioinformatists

Infotainments– movie fans, mountain climbers, fantasy football

Scientific data management– biomagnetic databank, E. Coli community

Business– enterprise intranets, tech support groups, lawyers

CIA / homeland security– Intellipedia

Much Efforts to Build Community PortalsMuch Efforts to Build Community Portals Initially taxonomy based (e.g., Yahoo style) But now many structured data portals

– capture key entities and relationships of community

No general solution yet on how to build such portals

Cimple Project @ Wisconsin / Yahoo! ResearchCimple Project @ Wisconsin / Yahoo! Research

Researcher

Homepages

Conference

Pages

Group Pages

DBworld

mailing list

DBLP

Web pages

Text documents

* **

** * ***

SIGMOD-04

**

** give-talk

Jim Gray

Keyword search

SQL querying

Question answering

Browse

Mining

Alert/Monitor

News summary

Jim Gray

SIGMOD-04

**

Maintain and add more sources

Develops such a general solution using extraction + integration + mass collaboration

Mass collaboration

Prototype System: DBLifePrototype System: DBLife

Integrate data of the DB research community 1164 data sources

Crawled daily, 11000+ pages = 160+ MB / day

Data ExtractionData Extraction

Data IntegrationData Integration

Raghu Ramakrishnan

co-authors = A. Doan, Divesh Srivastava, ...

Resulting ER GraphResulting ER Graph

“Proactive Re-optimization

Jennifer Widom

Shivnath Babu

SIGMOD 2005

David DeWitt

Pedro Bizarrocoauthor

coauthor

coauthor

advise advise

write

write

write

PC-Chair

PC-member

Provide ServicesProvide Services DBLife system

Mass Collaboration: VotingMass Collaboration: Voting

Picture is removed if enough users vote “no”.

Mass Collaboration via WikiMass Collaboration via Wiki

Summary: Community SystemsSummary: Community Systems Data integration systems + extraction + Web 2.0

– manage both data and users in a synergistic fashion

In sync with current trends– manage unstructured data (e.g., text, Web pages)– get more structure (IE, Semantic Web)– engage more people (Web 2.0)– best-effort data integration, data spaces, pay-as-you-go

Numerous potential applications

But raises many difficult data quality challenges

Rest of the TalkRest of the Talk

Data quality challenges in 1. Source selection2. Extraction and integration3. Detecting problems and providing feedback4. Mass collaboration

Conclusions & ways forward

1. Source Selection1. Source Selection

Researcher

Homepages

Conference

Pages

Group Pages

DBworld

mailing list

DBLP

Web pages

Text documents

* **

** * ***

SIGMOD-04

**

** give-talk

Jim Gray

Keyword search

SQL querying

Question answering

Browse

Mining

Alert/Monitor

News summary

Jim Gray

SIGMOD-04

**

Maintain and add more sources

Mass collaboration

Current Solutions vs. Cimple Current Solutions vs. Cimple

Current solutions– find all relevant data sources

(e.g., using focused crawling, search engines)– maximize coverage – have lot of noisy sources

Cimple – starts with a small set of high-quality “core” sources– incrementally adds more sources

– only from “high-quality” places– or as suggested by users (mass collaboration)

Start with a Small Set of “Core” SourcesStart with a Small Set of “Core” Sources

Key observation: communities often follow 80-20 rules– 20% of sources cover 80% of interesting activities

Initial portal over these 20% often is already quite useful

How to select these 20%– select as many sources as possible– evaluate and select most relevant ones

Evaluate the Relevancy of SourcesEvaluate the Relevancy of Sources Use PageRank + virtual links across entities + TF/IDF

... Gerhard Weikum

G. Weikum

See [VLDB-07a]

Add More Sources over TimeAdd More Sources over Time Key observation: most important sources will

eventually be mentioned within the community– so monitor certain “community channels” to find themMessage type: conf. ann.Subject: Call for Participation: VLDB Workshop on Management of Uncertain Data

Call for Participation Workshop on "Management of Uncertain Data" in conjunction with VLDB 2007

http://mud.cs.utwente.nl ...

Also allow users to suggest new sources– e.g., the Silicon Valley Database Society

Summary: Source SelectionSummary: Source Selection

Sharp contrast to current work– start with highly relevant sources– expand carefully – minimize “garbage in, garbage out”

Need a notion of source relevance Need a way to compute this

2. Extraction and Integration2. Extraction and Integration

Researcher

Homepages

Conference

Pages

Group Pages

DBworld

mailing list

DBLP

Web pages

Text documents

* **

** * ***

SIGMOD-04

**

** give-talk

Jim Gray

Keyword search

SQL querying

Question answering

Browse

Mining

Alert/Monitor

News summary

Jim Gray

SIGMOD-04

**

Maintain and add more sources

Mass collaboration

Extracting Entity MentionsExtracting Entity Mentions Key idea: reasonable plan, then patch Reasonable plan:

– collect person names, e.g., David Smith– generate variations, e.g., D. Smith, Dr. Smith, etc.– find occurrences of these variations

ExtractMbyName

Union

s1 … sn

Works well, but can’t handle certain difficult spots

Handling Difficult SpotsHandling Difficult Spots Example

– R. Miller, D. Smith, B. Jones– if “David Miller” is in the dictionary

will flag “Miller, D.” as a person name

Solution: patch such spots with stricter plans

ExtractMbyName

Union

s1 … sn

FindPotentialNameLists

ExtractMStrict

Matching Entity MentionsMatching Entity Mentions Key idea: reasonable plan, then patch Reasonable plan

– mention names are the same (modulo some variation) match

– e.g., David Smith and D. Smith

Union

Extract Plan

MatchMbyName

s1 sn…Works well, but can’t handle certain difficult spots

Handling Difficult SpotsHandling Difficult Spots

Estimate the semantic ambiguity of data sources– use social networking techniques [see ICDE-07a]

Apply stricter matchers to more ambiguous sources

MatchMStrict

Extract Plan

MatchMbyName

Union

{s1 … sn} DBLP\

Extract Plan

DBLP

DBLP: Chen Li

· · ·41. Chen Li, Bin Wang, Xiaochun Yang.VGRAM. VLDB 2007.· · ·38. Ping-Qi Pan, Jian-Feng Hu, Chen Li.Feasible region contraction.Applied Mathematics and Computation.· · ·

Going Beyond Sources: Going Beyond Sources: Difficult Data Spots Can Cover Any Difficult Data Spots Can Cover Any

Portion of DataPortion of Data

MatchMStrict

Extract Plan

MatchMbyName

Union

{s1 … sn} DBLP\

Extract Plan

DBLP

Mentions that Match “J. Han”

MatchMStrict2

Summary: Extraction and IntegrationSummary: Extraction and Integration Most current solutions

– try to find a single good plan, applied to all of data

Cimple solution: reasonable plan, then patch So the focus shifts to:

– how to find a reasonable plan? – how to detect problematic data spots? – how to patch those?

Need a notion of semantic ambiguity Different from the notion of source relevance

3. Detecting Problems 3. Detecting Problems and Providing Feedbackand Providing Feedback

Researcher

Homepages

Conference

Pages

Group Pages

DBworld

mailing list

DBLP

Web pages

Text documents

* **

** * ***

SIGMOD-04

**

** give-talk

Jim Gray

Keyword search

SQL querying

Question answering

Browse

Mining

Alert/Monitor

News summary

Jim Gray

SIGMOD-04

**

Maintain and add more sources

Mass collaboration

How to Detect Problems?How to Detect Problems? After extraction and matching, build services

– e.g., superhomepages Many such homepages contain minor problems

– e.g., X graduated in 19998 X chairs SIGMOD-05 and VLDB-05 X published 5 SIGMOD-03 papers

Intuitively, something is semantically incorrect

To fix this, lets build a Semantic Debugger– learns what is a normal profile for researcher, paper, etc. – alerts the builder to potentially buggy superhomepages– so feedback can be provided

What Types of Feedback?What Types of Feedback? Say that a certain data item Y is wrong Provide correct value for Y, e.g., Y = SIGMOD-06 Add domain knowledge

– e.g., no researcher has ever published 5 SIGMOD papers in a year

Add more data– e.g., X was advised by Z– e.g., here is the URL of another data source

Modify the underlying algorithm– e.g., pull out all data involving X

match using names and co-authors, not just names

How to Make How to Make Providing Feedback Very Easy?Providing Feedback Very Easy?

“Providing feedback” for the masses– in sync with current trends of empowering the masses

Extremely crucial in DBLife context If feedback can be provided easily

– can get more feedback– can leverage the mass of users

But this turned out to be very difficult

Critical in our experience, but

unsolved

Provide a Wiki interface

How to Make How to Make Providing Feedback Very Easy?Providing Feedback Very Easy?

Say that a certain data item Y is wrong Provide correct value for Y, e.g., Y = SIGMOD-06 Add domain knowledge Add more data Modify the underlying algorithm

Provide form interfaces

Unsolved, some recent interest on

how to mass customize software

See our IEEE Data Engineering Bulletin paperon user-centric challenges, 2007

What Feedback What Feedback Would Make the Most Impact?Would Make the Most Impact?

I have one hour spare time, would like to “teach” DBLife– what problems should I work on?– what feedback should I provide?

Need a Feedback Advisor– define a notion of system quality Q(s)– define questions q1, ..., qn that DBLife can ask users– for each qi, evaluate its expected improvement in Q(s)– pick question with highest expected quality improvement

Observations– a precise notion of system quality is now crucial– this notion should model the expected usage

Summary: Detection and FeedbackSummary: Detection and Feedback

How to detect problems? – Semantic Debugger

What types of feedback & how to easily provide them?– critical, largely unsolved

What feedback would make most impact?– crucial in large-scale systems – need a Feedback Advisor– need a precise notion of system quality

4. Mass Collaboration4. Mass Collaboration

Researcher

Homepages

Conference

Pages

Group Pages

DBworld

mailing list

DBLP

Web pages

Text documents

* **

** * ***

SIGMOD-04

**

** give-talk

Jim Gray

Keyword search

SQL querying

Question answering

Browse

Mining

Alert/Monitor

News summary

Jim Gray

SIGMOD-04

**

Maintenance and expansion

Mass collaboration

Mass Collaboration: VotingMass Collaboration: Voting

Can be applied to numerous problems

Example: MatchingExample: Matching

Hard for machine, but easy for human

Mouse for Dell laptop 200 series ...

Dell X200; mouse at reduced price ...

Dell laptop X200 with mouse ...

ChallengesChallenges How to detect and remove noisy users?

– evaluate them using questions with known answers

How to combine user feedback?– # of yes votes vs. # of no votes

See [ICDE-05a, ICDE-08a]

Mass Collaboration: WikiMass Collaboration: Wiki

Community wikipedia– built by machine + human– backed up by a structured database

DataSources G

T

V1

V2

V3

W1

W2

W3

u1

V3’ W3’

T3’

M

Machine MachineHuman

Mass Collaboration: WikiMass Collaboration: Wiki

<# person(id=1){name}=David J. DeWitt #>

<# person(id=1){title}=Professor #>

<strong>Interests:</strong><# person(id=1).interests(id=3).topic(id=4){name}=Parallel Database #>

David J. DeWitt

Professor

Interests: Parallel Database

<# person(id=1){name}=David J. DeWitt #>

<# person(id=1){title}=John P. Morgridge Professor #>

<# person(id=1) {organization}=UW #> since 1976

<strong>Interests:</strong><# person(id=1).interests(id=3).topic(id=4){name}=Parallel Database #>

<# person(id=1){name}=David J. DeWitt #>

<# person(id=1){title}= John P. Morgridge Professor #>

<# person(id=1){organization}=UW-Madison#>since 1976

<strong>Interests:</strong><# person(id=1).interests(id=3).topic(id=4){name}=Parallel Database #>

<# person(id=1).interests(id=5).topic(id=6){name}=Privacy #>

David J. DeWitt

John P. Morgridge ProfessorUW-Madison since 1976

Interests: Parallel Database Privacy

Machine

Human

Sample Data Quality ChallengesSample Data Quality Challenges How to detect noisy users?

– no clear solution yet– for now, limit editing to trusted editors– modify notion of system quality to account for this

How to combine feedback, handle inconsistent data?– user vs. user– user vs. machine

How to verify claimed ownership of data portions?– e.g., this superhomepage is about me– only I can edit it

See [ICDE-08b]

Summary: Mass CollaborationSummary: Mass Collaboration

What can users contribute? How to evaluate user quality? How to reconcile inconsistent data?

Additional ChallengesAdditional Challenges

Dealing with evolving data (e.g., matching) Iterative code development Lifelong quality improvement Querying over inconsistent data Managing provenance and uncertainty Generating explanations Undo

ConclusionsConclusions Community systems:

– data integration + IE + Web 2.0– potentially very useful in numerous domains

Such systems raise myriad data quality challenges– subsume many current challenges– suggest new ones

Can provide a unifying context for us to make progress– building systems has been a key strength of our field– we need a community effort, as always

See “cimple wisc” for more detail Let us know if you want code/data