future of search | yury lifshits, yahoo! research

38
Yury Lifshits Yahoo! Research http://yury.name Future of Search St. Petersburg | Helsinki December 2008

Upload: yury-lifshits

Post on 08-May-2015

2.669 views

Category:

Technology


0 download

DESCRIPTION

http://yury.name

TRANSCRIPT

Page 1: Future of Search | Yury Lifshits, Yahoo! Research

Yury LifshitsYahoo! Researchhttp://yury.name

Future of Search

St. Petersburg | Helsinki

December 2008

Page 2: Future of Search | Yury Lifshits, Yahoo! Research

Outline

Structured Search Yahoo! Work in Search

SearchMonkey BOSS

Research Agenda

Page 3: Future of Search | Yury Lifshits, Yahoo! Research

Structured Search:work in progress

Page 4: Future of Search | Yury Lifshits, Yahoo! Research

Structured Search =Bring structured data to search users

M.K. Bergman. The Deep Web: Surfacing Hidden Value. 2001.

Page 5: Future of Search | Yury Lifshits, Yahoo! Research

Value Proposition

Coverage Real-time data Semi-private data

Structured queries Ordering and filtering results Straight-to-answers

Page 6: Future of Search | Yury Lifshits, Yahoo! Research

User Interface: Query Search assist: Yahoo! Selector: LinkedIn, VKontakte.ru Multiple search buttons: Gmail Search tabs: Yahoo / Google

Page 7: Future of Search | Yury Lifshits, Yahoo! Research

User Interface: Results

Federated page Facets Search transfer / search form

K.P. Yee, K. Swearingen, K. Li, M. Hearst. Faceted metadata for image search and browsing. CHI 2003.

Fernando Diaz. Aggregation of News Content Into Web Results. WSDM 2009.

http://glue.yahoo.com http://au.alpha.yahoo.com

Page 8: Future of Search | Yury Lifshits, Yahoo! Research
Page 9: Future of Search | Yury Lifshits, Yahoo! Research
Page 10: Future of Search | Yury Lifshits, Yahoo! Research

Data Supply Chain

Atomic fact Flight, Event, Patent

Data aggregatorUS Patents, Amadeus/Sabre flights, Upcoming.com

Domain searchExpedia, Spock

General purpose searchYahoo!, Google, Yandex, Baidu

Page 11: Future of Search | Yury Lifshits, Yahoo! Research

Getting structured data

Entity extraction Markup Feeds Search API (OpenSearch)

OR

Do a search transfer

Page 12: Future of Search | Yury Lifshits, Yahoo! Research

Give Us Your Data For …

Traffic via search transferFirefox search box

Better presentation in search SearchMonkey

Hosted searchBOSS Custom

Showing your adsYahoo Local + AT&T

Page 13: Future of Search | Yury Lifshits, Yahoo! Research

Yahoo! Work in Search

Page 14: Future of Search | Yury Lifshits, Yahoo! Research

Slides by:Paul Tarjan, Chief Technical Monkey

([email protected])

Full version http://www.slideshare.net/ptarjan/searchmonkey-presentation

Page 15: Future of Search | Yury Lifshits, Yahoo! Research

an open platform for using structured data to build more useful and relevant search results

Before After

What is SearchMonkey?

Page 16: Future of Search | Yury Lifshits, Yahoo! Research

Enhanced Result: Zagat

Key/Value Pairsor Abstract

LinksImage

Page 17: Future of Search | Yury Lifshits, Yahoo! Research

Infobar: Wikipedia Preview

Summary Blob

Page 18: Future of Search | Yury Lifshits, Yahoo! Research

Creating an Infobar

Infobar advantages Annotate someone else’s site Use links and images from other domains

• Mash up info from multiple sites• Affiliate / coupon links? Hmmm…

Can act on *, all websites• But these apps can be annoying if poorly

designed

Key design principles Put something useful in the summary Be creative with the HTML

Page 19: Future of Search | Yury Lifshits, Yahoo! Research

How to get data to SearchMonkey?

Humans see:• name• picture of a person• current job• industry, …

Computers see:an undifferentiatedblob of HTML

Can we make computers smarter?

Page 20: Future of Search | Yury Lifshits, Yahoo! Research

How does it work?

Acme.com’sdatabase

Index

RDF/Microformat Markup

site owners/publishers share structured data with Yahoo!. 1

consumers customize their search experience with Enhanced Results or Infobars

3

site owners & third-party developers build SearchMonkey apps.2

DataRSS feed

Web Services

Page Extraction

Acme.com’s Web Pages

Page 21: Future of Search | Yury Lifshits, Yahoo! Research

SearchMonkey Resources

Main: http://developer.yahoo.com/

searchmonkey

Lists and forums: searchmonkey-

[email protected] http://suggestions.yahoo.com/

searchmonkey

Page 22: Future of Search | Yury Lifshits, Yahoo! Research

Vik Singh (Architect)Graham Mudd (Senior PMM)

Page 23: Future of Search | Yury Lifshits, Yahoo! Research

BOSS = Build your Own Search Service

Open Yahoo’s core search features via web services to let 3rd parties revolutionize Search

Unrestricted

What

Page 24: Future of Search | Yury Lifshits, Yahoo! Research

Unrestricted:

• Unlimited queries• Blend, re-order, discard• Full presentation control• Non-search apps OK

Monetization: Free or CPM or Ads

What

Page 25: Future of Search | Yury Lifshits, Yahoo! Research

Barriers to entry are massive• $300M, top talent, a prayer to get to basic parity

No monopoly over great ideas

Search anywhere• Improve Vertical Quality w/ Web comprehensiveness• Fragment the market, foster more players, choice, competition

Yahoo extends advertising reach, 3rd parties revenue share

Why

Page 26: Future of Search | Yury Lifshits, Yahoo! Research

Why

Traditional Search Distribution

+ BOSS Distribution

Page 27: Future of Search | Yury Lifshits, Yahoo! Research

Tracks

API

A self-service, web services model for developers and start-ups to quickly build and deploy new search experiences.

• UIUC• CMU• Stanford• Purdue• IIT Bombay• MIT• UMass

CUSTOM

Working with 3rd parties to build a more relevant, brand/site specific web search experience.

This option is jointly built by Yahoo! and select partners.

ACADEMIC

Working with the following universities to allow for wide-scale research in the search field:

Interested in Custom? Email us [email protected]

Page 28: Future of Search | Yury Lifshits, Yahoo! Research

http://boss.yahooapis.com/ysearch/{vert}/v1/{q}

{vert} := {web, news, images, spelling}

@ requiredappid

@ optional (Y!OS compliant)start, count, lang, region, format, callback, sites

BOSS API v1

Page 29: Future of Search | Yury Lifshits, Yahoo! Research

Python (v2.5+) library

BOSS Search SDK plus …

SQL for remixing arbitrary XML/JSON sources

Loosely Functional programming paradigm

BOSS Mashup Framework

Page 30: Future of Search | Yury Lifshits, Yahoo! Research

Ported enhanced version of BMF to GAE platform

http://zooie.wordpress.com/2008/08/04/yahoo-boss-google-app-engine-integrated/

Easiest way to deploy a BOSS application online

BMF + Google App Engine

Page 31: Future of Search | Yury Lifshits, Yahoo! Research

http://www.4hoursearch.com

http://123people.com

Mashable! Contest for BOSS search engineshttp://mashable.com/boss/

Examples

Page 32: Future of Search | Yury Lifshits, Yahoo! Research

BOSS Custom for TechCrunch

Page 33: Future of Search | Yury Lifshits, Yahoo! Research

TechCrunch Network Search

CrunchBase + Posts + Web Sort by time / relevance Enhanced results Domain-specific facets Yahoo! sponsored search Real-time indexing Special results

Page 34: Future of Search | Yury Lifshits, Yahoo! Research

Research Agenda

Page 35: Future of Search | Yury Lifshits, Yahoo! Research

Structured Search

Analysis of search demand Intent classification General search vs. vertical

Incentives in data supply Push & real-time indexing Search user interface

One box vs. multi-box General vs. vertical

Deciding search transfer When? To whom?

Page 36: Future of Search | Yury Lifshits, Yahoo! Research

Key Scientific ChallengesDraft: http://research.yahoo.com/ksc

1. Search intent2. Quality metrics3. Web mining4. Multilingual IR5. Nextgen search

Synthesized result pages

6. World knowledge

A.Z. Broder. Taxonomy of web search. SIGIR 2002.

Page 37: Future of Search | Yury Lifshits, Yahoo! Research

More Problems

Discovery search

Web search vs. asking people

Event search

Page 38: Future of Search | Yury Lifshits, Yahoo! Research

Thanks for your attention!

Yury Lifshits http://yury.name [email protected]