Learning to Rank from Distant Supervision: Exploiting Noisy Redundancy for Relational Entity Search

Mianwei Zhou, Hongning Wang, Kevin Chen-Chuan ChangUniversity of Illinois Urbana ChampaignLearning to Rank from Distant Supervision: Exploiting Noisy Redundancy for Relational Entity Search1Limitation of Traditional Entity Search2Entity Search: in most cases, what we want are not pages, but entities (Cheng 2007)

Thalmic Lab founded by ?LimitationFail to model the relation implied by the queryDifficult for user to enumerate different representations of a relation

Thalmic Lab founded by #personTraditional searchDue to its popularity, could not be found in Wikipedia yet.Limitation.2Our Task: Relational Entity Search3Relational Entity Search: relation-specific ranking functions Relational Entity SearcherFounder-Of RankerGrad-FromRankerAdvantagesTrain rankers from relational data to capture the relation semanticsRelieve users from the burden of specifying keywords.OutputStephen LakeAaron Grant Columbia Univ.Harvard Law SchoolInputFounderOf (Thalmic Lab)GraduateFrom (Barack Obama)Advantage 1: We develop a framework to train such rankers from relational data to capture the relation semantics, which is expected to achieve higher accuracy than the general ranking function in traditional entity search.

Advantage 2: Ranker 3Proposal: Relational Entity Search Framework4In order to realize such a relational entity searcher, we propose a relational entity search framework.

4Relational Entity Search Framework5Relational Entity SearcherFounder-Of Ranker

Entity-aware Searcher Keyword Indexes Entity Index Entity Indexes (#person, #location, #color, #medicine, ) #person: (bill gates, d1, 5)(Steven Ballmer, d2, 1)Snippets1: Microsoft was founded by Bill Gatess2: Steven Ballmer is CEO of MicrosoftAn entity-aware searcher, which is a backend system that supports the demo I shown at the beginning.

5Relational Entity Search Framework6Stephen LakeStephen Lake, co-founder of Thalmic LabDaniel DebowDaniel Debow investigates Thalmic LabEntity-SnippetRelational Entity SearcherFounder-Of Ranker

Entity-aware Searcher Keyword Indexes Entity Index Relational QueryFounderOf (Thalmic Lab)ResultStephen LakeAaron GrantTranslated QueryThalmic Lab,Founded byFounder,Started,,#PersonAn entity-aware searcher, which is a backend system that supports the demo I shown at the beginning.

6Training Relational Entity Ranker Offline7CompanyFounderMicrosoftBill Gates, Paul AllenIBMThomas WatsonFounderOf RelationBill GatesMicrosoft was founded by Bill Gates Paul AllenSteven BallmerSteven Ballmer is CEO of Microsoft

Entity-SnippetEntity-awareSearcher

Founder-Of Ranker

Such a requirement is practical, as nowadays, public knowledge bases such as actually provide a huge amount of relational data covering many different types, as training examples.

We will collect all person entities that ever co-occur with the company name, and retrieve their co-occurring snippets.Labeling.7Challenges on Accuracy and Efficiency8Note: the ranker should first have high accuracy to rank the correct entities at the top.

8Challenges on Accuracy and Efficiency: Distantly Supervised Ranking9Challenge 1 (Accuracy): How to avoid the Negative Effect brought by Noisy Snippets of Positive Entities?EntitySnippets e1: Bill GatesCompanyFounderMicrosoftBill Gates s11: Microsoft was founded by Bill Gates and s13: Bill Gates dropped out of college and started Microsoft. s12: Bill Gates met Microsoft CEO at his home NoiseNote: Since we simply collect snippets where the query and the entity co-occur, such snippets might be totally irrelevant with the target relation.

Tempt the ranker to learn an unexpected relation, and harm the accuracy.9Challenges on Accuracy and Efficiency: Distantly Supervised Ranking10Challenge 2 (Efficiency): How to limit the number of keyword features without sacrificing too much accuracy?started, #personEntity-aware Searcherfounded by, #personfounder, #personSnippetsRequire Expensive Index CheckingChallenge 1 (Accuracy): How to avoid the Negative Effect brought by Noisy Snippets of Positive Entities?In order to learn an entity ranker, we will need some keywords to help identify the target relation.

Remember in the framework, we are using entity-aware searcher, which refers to inverted indexes to check how keywords co-occur with entities. Since index checking operations take time, the number of 10Challenges on Accuracy and Efficiency: Distantly Supervised Ranking11Distantly Supervised RankingDistantly Supervised: Only Entity Labels, No Snippet LabelsRanking: Efficiency is RequiredChallenge 2 (Efficiency): How to limit the number of keyword features without sacrificing too much accuracy?Challenge 1 (Accuracy): How to avoid the Negative Effect brought by Noisy Snippets of Positive Entities?12Insight: Redundancy Ranking PrincipleLearn indicative patterns based on redundancy (Challenge 1: Accuracy)13Microsoft => Bill Gates Microsoft was founded by Bill Gates.IBM => Thomas Watson Founded by Thomas Waston, IBM is Facebook => Mark Zuckerberg Facebook was founded by Mark ZuckerbergBeforeEntity [founded by]Indicative Pattern: Some important patterns that are indicative of the relation. E.g., founded by, started, created 13Filter Noisy Snippets by Indicative Patterns (Challenge 1: Accuracy)14e: Bill GatesIndicative Patterns PEvidence Snippet: Snippets that contain at least one indicative patternIn terms of accuracy, in order to eliminate the negative effect brought by noisy snippets, we first need to determine the criteria of discriminating noisy snippets from evidence snippets.14A small number of indicative patterns are sufficient (Challenge 2: Efficiency)15Indicative PatternsBeforeEntity [started]BeforeEntity [founded by]Around [founder]BeforeEntity [created]Microsoft => Bill Gates Microsoft was founded by Bill Gates. Bill Gates created Microsoft IBM => Thomas Watson The founder of IBM is Thomas Facebook => Mark Zuckerberg Facebook was founded by Mark Zuckerberg Mark Zuckerberg created Facebook in 2006 Snippets15Redundancy Ranking Principle16e: Bill GatesWeb Redundancy0Redundancy Ranking Principle1617Redundancy Ranking PrincipleQuery-Entity DistanceQuery FrequencyBeforeEntity[founded by]BeforeEntity[started]Snippet Feature f(s)e: Bill Gates0Solution: Pattern-based Filter Network18Objective Function19Subject toFor efficiency concern, the number of indicative patterns should be smallMuch like other machine learning framework, we will maximize Subject to constraint for efficiency concernWhat parameters to learn19Model Redundancy Ranking Principle: Pattern-based Filter Network (PFNet)20Noise Filtering LayerEvidence Aggregation Layer1. Filter Noisy Snippets by Indicative Patterns.2. Aggregate Contribution from Evidence SnippetsNoise Filtering Layer in PFNet21Evidence Aggregation Layer in PFNet2222Likelihood for PFNet23Subject toFactor Design24Factor Design25Aggregate Contribution from Evidence Snippets. Optimization: Maximizing Likelihood by Greedily Adding Indicative Patterns 26Indicative Patterns P

Candidate Snippet FeaturesLog LikelihoodBeforeEntity [founded by]-100.20Given current P, calculate the maximized likelihood by gradient ascent on wOptimization: Maximizing Likelihood by Greedily Adding Indicative Patterns 27Indicative Patterns P

Candidate Snippet FeaturesLog LikelihoodAround[founder]BeforeEntity [founded by]Around [microsoft]BeforeEntity [started]-100.20-5600.21-76.13-200.43Around[founder]Optimization: Maximizing Likelihood by Greedily Adding Indicative Patterns 28Indicative Patterns P

Candidate Snippet FeaturesLog LikelihoodAround[founder]BeforeEntity [founded by]Around [microsoft]BeforeEntity [started]-60.20-3450.21-103.43Around[founder]BeforeEntity [founded by]Experiment Setting296 sets of different relationsDatasetBase TypeQuery NumPositive / Total Entity NumSnippet NumFounderOf#person371473 / 200611033507PublisherOf#organization323329 / 191661488347WriterOf#person669993 / 466832111565PlaceOfBirth#location350350 / 243481376995PlaceOfDeath#location350350 / 232461105738GraduateFrom#organization228228 / 855997916Experiment Setting30BaselinesEntityRank (Cheng 2007)Multi Instance Learning (MIL, Riedel 2010)SVMRank (Joachims2003)

BaselineFilter NoiseRedundancyMILYesNoSVMRankNoYes3031Ranking Performance on 6 Different Relations.First, we would like to see31321. Relation-specific ranking function performs better.2. It is important to leverage redundancy.3. It is necessary to filter noisy snippets.First, we would like to see32Larger Improvement on More Noisy Relations33RelationRunner-up Baseline (NDCG@5)PFNet(NDCG@5)

ImprovementPercentage of NoiseFounderOf0.6906480.7535579.11%62.7%PublisherOf0.54620880.67543523.7%72%WriterOf0.5664430.69115422.0%76.7%PlaceOfBirth0.5862410.67038414.3%76.7%PlaceOfDeath0.5187370.59331114.4%90.3%GraduateFrom0.7521520.7993036.3%49.4%Around 10 indicative patterns are sufficient341. Higher redundancy can achieve better results352. Filtering noise is helpful for queries of different redundancy1. Performance increases with more training examples.2. Around 90 training examples are sufficient for most relations.36We are also interested in checking how36Thanks. Q & A37