anatomy of relevance: from data to action
DESCRIPTION
Relevance denotes how well a search result satisfies the user information need. In addition to the search engine components (i.e., indexer and query parser), there are many other components that impact relevance. e.g., user understanding , data optimization, domain knowledge, etc. Improving relevance remains the main and most challenging goal of each search engine. Indeed, relevance can be subjective, therefore hard to measure and to improve. In this talk, Saïd will demystify the concept of relevance by defining its main components. For each component, he will present the technology enablers, the data, and processes that are required in order to measure and improve relevance. In this talk, attendees will learn how to provide a relevant user experience and track it over time.TRANSCRIPT
![Page 1: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/1.jpg)
Anatomy Of Relevance: From Data To Action
Saïd Radhouani, Ph.D. radhouani
Yellow Pages, Canada
![Page 2: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/2.jpg)
Agenda
• Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
![Page 3: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/3.jpg)
Who we are
• Leading media and marketing solutions company in Canada
• Champion the neighbourhood economy by connecting businesses and consumers
7.3 million Unduplicated unique visitors to our network of websites
6.5 million Downloads of our mobile apps
26% Internet users reached in Canada each month
300 000 Local and national advertisers
![Page 4: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/4.jpg)
Who I am
• Entrepreneur
• Ph.D. in Search and Knowledge Management • IT Director – Content, Search and Relevance
• Noble cause: leverage data to help build smarter organizations
![Page 5: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/5.jpg)
Agenda
• Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
![Page 6: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/6.jpg)
Local Search
![Page 7: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/7.jpg)
The Importance Of Relevance
Businesses Consumers
Source: unknown
![Page 8: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/8.jpg)
Agenda
• Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
![Page 9: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/9.jpg)
User Objective Understand the user intent to maximize their satisfaction Data • Who the user is (individual or group) • Where they come from (direct, SEs, …) • Their context (Location, TOD/DOW/Season, Device, …) • Their interactions history
KPIs • Coverage: %tracked user, %tracked user interactions • Freshness: Daily updates of User profiles & interactions
User
![Page 10: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/10.jpg)
Content Objective Have accurate and “fresh” content to satisfy your audience needs Data • Enriched Directory Content (Merchants, Products, Deals, etc.) • Rich Editorial Content (Neighborhoods, Smart Tips, etc.) • Third-party content (social networks, etc. ) • User-Generated Content
KPIs • Coverage: %Canadian businesses, %Neighborhoods • Completeness: e.g., for each merchant, have all required info • Freshness: Frequency of validation & updates Content User
![Page 11: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/11.jpg)
Knowledge Objective Help to bridge the gap between user queries and content data Data • Linguistic (synonyms, acronyms, etc.) • Rich and “fresh” geo data (location names, polygons, etc.) • Multilingual
KPIs • Coverage: %searched entities (keywords, location names, etc.) • Completeness: e.g., for each location: centroid, polygon,
synonyms, relationships, etc. • Freshness: Frequency of validation & updates
Content Knowledge User
![Page 12: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/12.jpg)
Search Objective Action-driven, personalized and contextualized relevant search experience KPIs Offline • Recall & Precision Online • #returned results • Mean Reciprocal Rank (MRR) – the multiplicative inverse of
the rank of the first correct answer. Perfect SE should have a MRR of 1
Content Knowledge
Search
User
![Page 13: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/13.jpg)
Presentation Objective Allow users to access quickly to actionable content through optimal & dynamic search results presentation e.g., “Movie Theater” vs. “Restaurant” KPIs [that really matter] • Page Views • Unique Visitors • Visit Duration • Bounce Rate • Return Visits Frequency • (Useful) Interactions per Visits Content
Knowledge
Presenta4on Search
User
![Page 14: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/14.jpg)
Optimization
UserSatisfaction improveRelevance() { While (alive) { Act; Measure; Learn; Grow; } }
Content Knowledge
Op4miza4on
Presenta4on Search
User
![Page 15: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/15.jpg)
Agenda
• Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
![Page 16: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/16.jpg)
A problem well-‐stated is half-‐solved.
Where To Start?
Charles Ke2ering
![Page 17: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/17.jpg)
Data-Driven Actions
• Implicit User Feedback (analytics & logs)
• Stable Test Collection
• Side-By-Side Comparison
• A/B Testing
• Explicit User Feedback (Survey)
• Find gaps in data and improve them
• Improve processes
• Improve presentation
• Improve algorithms
• Confirm hypothesis
Measure Act
KPIs
![Page 18: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/18.jpg)
Know Your Data
• Web & Mobile Analytics
• Search logs Analysis
• Build KPIs
![Page 19: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/19.jpg)
From Data To Action
Examples: • Missing geo-location • Missing businesses • Seasonal trends (adapt search algo)
Manual or automa8c ac8ons
![Page 20: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/20.jpg)
Test Collection KPIs: Recall & Precision • Predefined test collection built manually
• Variety of use cases that cover most users needs
• Evaluate new updates before going to production
![Page 21: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/21.jpg)
Side-By-Side Comparison
• Comparison of two (versions of the) Search Engines
• Pre-defined set of real searches selected from logs
• Pre-classified (keywords, business names, language, geo location, etc.)
• “Blind” comparison
• Significant number of testers per search
![Page 22: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/22.jpg)
A/B Testing
Principle • Experiment one or many ingredients at a
time with real users • Some non-trivial amount of traffic is
always being tested
KPIs • Page Views • Unique Visitors • Visit Duration • Bounce Rate • Return Visits Frequency • Useful Interactions per Visits
Simulation • Run new versions on a significant
search logs sample
• Estimate the impact prior deployment to prod and avoid bad surprises
![Page 23: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/23.jpg)
Relevance In Action
e.g., Bounce Rate – Why the user left quickly? • Who is the user? • Where they come from (direct vs. SE) • Their devices • Which content was provided? • Which page? • etc.
Valuable signals, NOT only for marketing!
![Page 24: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/24.jpg)
Agenda
• Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks
• Measuring and Improving Relevance
• Conclusion
![Page 25: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/25.jpg)
Conclusion
Stop coding and make your data talk!
![Page 26: Anatomy of relevance: From Data To Action](https://reader033.vdocuments.us/reader033/viewer/2022052909/5597a2211a28abd3218b4607/html5/thumbnails/26.jpg)
Anatomy Of Relevance: From Data To Action
Saïd Radhouani, Ph.D. radhouani
Yellow Pages, Canada