enterprise search – where do we go from here? aya soffer, phd dgm, information and interaction...

Enterprise Search – Where do we go from here?

Aya Soffer, PhD

DGM, Information and Interaction Technologies

IBM Haifa Research Lab

25/2/08 2IBM Research Lab in Haifa © Aya Soffer

The World of SearchThe World of Search

Content management, groupware, “Intranet search”

E-commerce,

News, public documentation, government,…

Proprietary (owned & stored by

the SE) “local”

Market intelligence, news tracking, job data mining

Private

“inward facing”

Web searchPublic

“outward facing”

Public

(owned and stored by others) “global”Access

Content


Enterprise Search


User Expectations Shape Products

Global / Web search (public content)Users expect relevant answers to very low-content queries: elections, olympics

Search engines deliver!

Hence reinforcement of user expectations.

The global context is extremely popular Shapes user expectations at the workplace

Shapes enterprise search products

Enterprise search (proprietary content)Users expect a similar interaction style

Yet, users are more sophisticated


Enterprise Search — Harder than Web Search

Far fewer resources — but high expectations

Data is not “search friendly”

Must index everything - find everything I can access

Security - but show me only what I am allowed to see

Link-based methods not as effective – not enough linkage

Search is not cheap! - about 5-10 cents/ document/ year – yet ROI is hard to calculate


Smaller scaleCorpus is much smaller

Query load is much lower

Less anarchic Central authority

Data formats can be controlled

More potential structure Data is better organized and richer

Can tap into organizational knowledge

No spamAt least not intentional

Enterprise Search — Easier than Web Search


Major Players


Trends


Enterprise Search Trends

Information finding goes beyond simple keyword search in search bar

Text analytics and semantic search allows searching by concepts instead of by keywords Understand content, understand user intent

Collaboration technology is fundamentally changing how people organize and access content

Democratic tagging of data as a bridge across multiple taxonomiesDiscover relationships, find experts, utilize wisdom of the crowds and social networks to find best matches

Information finding isn’t enough – need to provide means to explore search results

New visualizations for search resultsCombined search / browse paradigm will be pervasiveCombination of search and BI BI for the Masses

Search goes mobileAccess information in context from mobile devices


Social Search


Social Search

With the advent and popularity of Web 2.0, the ability to share and find information is being expanded beyond keyword search mainly by use of tags

While Google's page rank can be viewed as one of the first applications of the Web 2.0 concept of wisdom of crowds, Search has yet to fully harness the power of Web 2.0

Community influenced search is beginning to appear in niche search engines and as Beta’s in major search engines

Wikia Search

Yahoo! MyWeb

Google Co-Op

Eurekster


1 billion global users80,000,000 web sitesContent ConsumersContent Providers

User-Generated ContentUser-Generated Content

Published ContentPublished Content

CollaboratorsFacilitators

Collective IntelligenceCollective Intelligence

Web 2.0: The wildly read-write Web 2.0


Search Today – Web 2.0 phenomena indirectly influencing search results

Indirect InfluenceIndirect InfluenceCollective IntelligenceCollective Intelligence

User-Generated ContentUser-Generated Content

Published ContentPublished Content

User-Generated MetadataUser-Generated MetadataSocial NetworksSocial Networks


Evolution of Search Technologies

Scope

Technology

Business Impact

User

World Wide Web (Html)

• Hyperlink analysis for relevance ranking (good for Web, not as good in enterprise)

• Categorization/ summarization

eCommerce and consumer market

Everyone

Web-based Search (2nd Generation)

Small, closed collections

• String matching• Boolean search• Basic relevance ranking

Vertical business domains (medical, legal)

Trained specialists

InformationRetrieval

(1st Generation)

Structured, semi-structured and unstructured information

• Text analytics with novel linguistic and semantic processing

• Ranking on structure• Faceted navigation

Pervasive use in business processes to realize value from unstructured information

Everyone & applications

Information Discovery

(3rd Generation)

Structured, semi-structured, unstructured information + networks

• Community input • Incorporation of wisdom of crowds

• Tags, tag clouds, social network data

Pervasive use in business processes to realize value from unstructured information

Everyone & applications

Social Information Discovery

(4th Generation)


Trend - Mutually Reinforcing Relationship

Data

Metadata

•Author

•Mentioned

•Bookmark

•Reader

•Associated

•TaggedByPeople

tagsfolder label

subject of emailtitle of document

anchor textquery

authordocument owner

email addressusername

codeuser profile

email messagesreceived documentsauthored documents

calendar entrieschat history


Web 2.0 Unified Search

The result set includes relevant documents, people, and tags. Ranking is affected by the volume of tagsand comments that are associated with each document.

Tags related to the result set – presented as a tag cloud

People related to the result set –sorted by relevance


Guided Navigation Meets Business

Intelligence (BI for the Masses)


Multifaceted search or guided navigation

Allows the simultaneous exploration of many aspects of a topic, and the gradual “zooming in” on the information target

Reduces frustration: ensures that only valid choices are presented, so zooming in never yields an empty result set

Solution to the problem of

“few terms too many results, more terms no results”

Supports browsing when the user doesn’t really know what to ask for in a multi-dimensional information space

Multifaceted search is very popular in e-Commerce solutions (Amazon, eBay, buy.com, …), but is also relevant to more traditional text search applications

New applications are emerging every day


Multifaceted Search – Example

Categories- {

Featured Dimensions

Categories

Category Counts

QueryCurrent Context

Other Dimensions

Search Results


Guided Navigation Meets Business Intelligence

Faceted Navigation LimitationsCurrent faceted navigation interfaces enable drilling down one facet at a time.Counts and context are similarly presented for each facet separately

Business Intelligence today is mainly for structured informationAssumes structure is known in advanceOffline processing Complex report generations

New direction: faceted navigation as a front end for business intelligence applications.

Facets can be defined on a combination of fields including aggregations and simple expressions.


MultiFaceted Search with BI - Example

Query

For each author: counts (as before) but also calculated values on result set

Average Rating per Sales Rank


Correlating Facets – BI on search results

Top-Ranked books do not

have the highest ranking

Older books have lower sales rank

Clicking on the value will bring

you to the books with high rating and top

sales rank


The World of SearchThe World of Search


E-commerce,



the SE) “local”


Private

“inward facing”

Web searchPublic


Public


Content


The World of SearchThe World of Search The boundaries are blurring


E-commerce,



the SE) “local”


Private

“inward facing”

Web searchPublic


Public


Content

enterprise search – where do we go from here? aya soffer, phd dgm, information and interaction...

Documents

enterprise search slide

web search slide

semantic search

masses search

linkage search

social search slide

enterprise search harder

ibm research lab