search quality at linkedin
DESCRIPTION
Presented to the Bay Area Search Meetup on February 26, 2014 http://www.meetup.com/Bay-Area-Search/events/136150622/ At LinkedIn, we face a number of challenges in delivering high quality search results to 277M+ members. Our results are highly personalized, requiring us to build machine-learned relevance models that combine document, query, and user features. And our emphasis on entities (names, companies, job titles, etc.) affects how we process and understand queries. In this talk, we'll talk about these challenges in detail, and we'll describe some of the solutions we are building to address them. Speakers: Satya Kanduri has worked on LinkedIn search relevance since 2011. Most recently he led the development of LinkedIn's machine-learned ranking platform. He previously worked at Microsoft, improving relevance for Bing Product Search. He has an MS in Computer Science from the University of Nebraska - Lincoln, and a BE in Computer Science from the Osmania University College of Engineering. Abhimanyu Lad has worked at LinkedIn as a software engineer and data scientist since 2011. He has worked on a variety of relevance and query understanding problems, including query intent prediction, query suggestion, and spelling correction. He has a PhD in Computer Science from CMU, where he worked on developing machine learning techniques for diversifying search results.TRANSCRIPT
Recruiting SolutionsRecruiting SolutionsRecruiting Solutions
Abhimanyu LadSatya KanduriSenior Software Engineer Senior Software Engineer 1
Abhi Satya
Search Quality at LinkedIn
2
tag: skill OR titlerelated skills: search, ranking, …
tag: companyid: 1337industry: internet
verticals:people, jobs
intent: exploratory
3
SEARCH USE CASES
How do people use LinkedIn’s search?
4
PEOPLE SEARCH
Search for people by name
5
PEOPLE SEARCH
Search for people by other attributes
6
EXPLORATORY PEOPLE SEARCH
7
JOB SEARCH
8
COMPANY SEARCH
9
AND MUCH MORE…
10
OUR GOAL
Universal Search– Single search box
High Recall– Spelling correction, synonym expansion, …
High Precision– Entity-oriented search: match things, not strings
11
QUERY UNDERSTANDINGPIPELINE
12
QUERY UNDERSTANDING PIPELINE
Spellcheck
Query Tagging
Vertical Intent Prediction
Query Expansion
Raw query
Structured query+
Annotations
13
QUERY UNDERSTANDING PIPELINE
Spellcheck
Query Tagging
Vertical Intent Prediction
Query Expansion
Raw query
Structured query+
Annotations
14
SPELLING CORRECTION
Fix obvious typos
Help users spell names
15
SPELLING OUT THE DETAILS
PEOPLE NAMESCOMPANIES
TITLES
PAST QUERIES
N-gramsmarissa => ma ar ri is ss sa
Metaphonemark/marc => MRK
Co-occurrence countsmarissa:mayer = 1000
marisa meyer yahoo
marissa
marisa
meyer
mayer
yahoo
16
SPELLING OUT THE DETAILS
PROBLEM: Corpus as well as query logs contain many spelling errors
Certain spelling errors are quite frequent
While genuine words (especially names) might be infrequent
17
SPELLING OUT THE DETAILS
PROBLEM: Corpus as well as query logs contain many spelling errors
SOLUTION: Use query chains to infer correct spelling
[product manger] [product manager] CLICK
[marissa mayer] CLICK
18
QUERY UNDERSTANDING PIPELINE
Spellcheck
Query Tagging
Vertical Intent Prediction
Query Expansion
Raw query
Structured query+
Annotations
19
QUERY TAGGING IDENTIFYING ENTITIES IN THE QUERY
TITLE CO GEO
TITLE-237software engineersoftware developer
programmer…
CO-1441Google Inc.
Industry: Internet
GEO-7583Country: US
Lat: 42.3482 NLong: 75.1890 W
(RECOGNIZED TAGS: NAME, TITLE, COMPANY, SCHOOL, GEO, SKILL )
20
QUERY TAGGING IDENTIFYING ENTITIES IN THE QUERY
TITLE CO GEO
MORE PRECISE MATCHING WITH DOCUMENTS
21
ENTITY-BASED FILTERING
BEFORE
22
AFTER
ENTITY-BASED FILTERING
BEFORE
23
BEFORE
ENTITY-BASED FILTERING
24
AFTER
ENTITY-BASED FILTERING
BEFORE
25
ENTITY-BASED SUGGESTIONS
26
ENTITY-BASED SUGGESTIONS
27
QUERY TAGGING : SEQUENTIAL MODEL
EMISSION PROBABILITIES(Learned from user profiles)
TRANSITION PROBABILITIES(Learned from query logs)
TRAINING
28
QUERY TAGGING : SEQUENTIAL MODEL
INFERENCE
Given a query, find the most likely sequence of tags
29
QUERY UNDERSTANDING PIPELINE
Spellcheck
Query Tagging
Vertical Intent Prediction
Query Expansion
Raw query
Structured query+
Annotations
30
VERTICAL INTENT PREDICTION
JOBS
PEOPLE
COMPANIES
(Probability distribution over verticals)
31
VERTICAL INTENT PREDICTION : SIGNALS
[Company]
1. Past query counts in each vertical + Query tags
2. Personalization: User’s search history
[Employees]
[Jobs]
[Name Search]
(TAG:COMPANY) (TAG:NAME)
32
QUERY UNDERSTANDING PIPELINE
Spellcheck
Query Tagging
Vertical Intent Prediction
Query Expansion
Raw query
Structured query+
Annotations
33
QUERY EXPANSION
GOAL: Improve recall through synonym expansion
34
QUERY EXPANSION : NAME SYNONYMS
35
QUERY EXPANSION : JOB TITLE SYNONYMS
36
QUERY EXPANSION : SIGNALS
[jon] [jonathan] CLICK
Trained using query chains:
[programmer] [developer] CLICK
Symmetric but not transitive!
[francis] ⇔ [frank][franklin] ⇔ [frank]
[francis] ≠ [franklin]
[software engineer] [software developer] CLICK
Context based!
[software engineer] => [software developer]
[civil engineer] ≠ [civil developer]
37
QUERY UNDERSTANDING PIPELINE
Spellcheck
Query Tagging
Vertical Intent Prediction
Query Expansion
Raw query
Structured query+
Annotations
38
QUERY UNDERSTANDING: SUMMARY
High degree of structure in queries as well as corpus(user profiles, job postings, companies, …)
Query understanding allows us to optimally balance recall and precision by supporting entity-oriented search
Query tagging and query log analysis play a big role in query understanding
39
ranking
WHAT’S IN A NAME QUERY?
kevin scott
≠
BUT NAMES CAN BE AMBIGUOUS
SEARCHING FOR A COMPANY’S EMPLOYEES
SEARCHING FOR PEOPLE WITH A SKILL
RANKING IS COMPLICATED
Seemingly similar queries require dissimilar scoring functions
Personalization matters– Multiple dimensions to personalize on– Dimensions vary with query class
Model
TRAINING
Documents for training
Features
Human evaluation
Labels
Machine learning model
TRAINING
Documents for training
Features
Human evaluation
Labels
Machine learning model
ASSESSING RELEVANCE
RELEVANCE DEPENDS ON WHO’S SEARCHING
What if the searcher is a job seeker?
Or a recruiter?
Or…
THE QUERY IS NOT ENOUGH
WE NEED USER FEATURES
Non-personalized relevance model:score = f(Document | Query)
Personalized relevance model:score = f(Document | Query, User)
COLLECTING RELEVANCE JUDGMENTS WON’T SCALE
TRAINING
Documents for training
Features
Human evaluation
Search logs
Labels
Machine learning model
CLICKS AS TRAINING DATAApproach: Clicked = Relevant, Not-Clicked = Not Relevant
CLICKS AS TRAINING DATAApproach: Clicked = Relevant, Not-Clicked = Not Relevant
CLICKS AS TRAINING DATAApproach: Clicked = Relevant, Not-Clicked = Not Relevant
CLICKS AS TRAINING DATA
Unfairly penalized?
Good results not seen are marked Not Relevant.
Approach: Clicked = Relevant, Not-Clicked = Not Relevant
User eye scan direction
CLICKS AS TRAINING DATAApproach: Clicked = Relevant, Skipped = Not Relevant
• Only penalize results that the user has seen but ignored
CLICKS AS TRAINING DATAApproach: Clicked = Relevant, Skipped = Not Relevant
• Only penalize results that the user has seen but ignored• Risks inverting model by overweighing low-ranked results
FAIR PAIRS
[Radlinski and Joachims, AAAI’06]
• Fair Pairs: • Randomize, Clicked= R,
Skipped= NR
FAIR PAIRS
Flipped
[Radlinski and Joachims, AAAI’06]
• Fair Pairs: • Randomize, Clicked= R,
Skipped= NR
FAIR PAIRS
Flipped
[Radlinski and Joachims, AAAI’06]
• Fair Pairs: • Randomize, Clicked= R,
Skipped= NR• Great at dealing with position bias• Does not invert models
EASY NEGATIVES
Page 1
Page 99
• Assumption: A decent current model would push out bad results to the very end.
• Easy Negatives: Some of the results at the end are picked up as negative examples
EASY NEGATIVES
• Use strategies that sample across the feature space• Searches with less results preferred• Always sample from a given page, say page 10
2 pages90+ pages
PUTTING IT ALL TOGETHER
Human evaluation is not practical for personalized searches
Learn from user behavior– Multiple heuristics depending on the need– Different pros and cons
66
EFFICIENCY VS EXPRESSIVENESS Build tree with logistic regression leaves. By restricting decision nodes to (Query, User) segments,
only one regression model can be evaluated for each document.
X 2=0
X2=?
X2=1
X4?
X 4=0
X4=1
SCORING
New document
Features
Machine
learning model
scoreNew
document
Features
Machine
learning model
scoreNew
document
Features
Machine learning model
score
Ordered listOrdered
listOrdered list
68
A SIMPLIFIED EXAMPLE
Yes
Name Query?
No
Skill Query?
Yes
No
69
TEST, TEST, TEST
a
b
c
d
g
h
b
e
a
f
g
h
Model 1 Model 2
a
b
c
e
d
f
Interleaved
[Radlinski et al., CIKM 2008]
Interleaving
SUMMARY
Query understanding leverages the rich structure of LinkedIn’s content and information needs.
Query tagging and rewriting allows us to deliver precision and recall.
For ranking, personalization is both the biggest challenge and the core of our solution.
Segmenting relevance models by query type helps us efficiently address the diversity of search needs.
71
Abhimanyu Lad Satya [email protected] [email protected]://linkedin.com/in/abhilad https://linkedin.com/in/skanduri