FAST Search & TransferVisiting CERN

Taking Search further

2

Todays agenda

– Information discovery and metadata extraction for scientific documents

– Data Cleansing

– Ad hoc analysis and visualization. Use cases: CERN administration, CDS, EDMS

– Processing and search solutions for complex structured data

– Extreme scalability

3

Search

The Emerging “OSI Layer Model”

ECM

Data Security

Data Management

Storage

Network / Protocols

Web Services

ContentManagement

ConsumptionManagement

Service Layer

VirtualizedInfrastructure

MCE RetailInvestment

BankingSecurities

ConsumerFinancialServices

GovermentLife

SciencesHealth Care

SearchSolutions

4

The Shift that follows

Users

Content

5 What are enabling content & services?

5 How to hard-wire services in portal?

3 Key user and business drivers?

3 What type of relevant analysis?

2 What “jobs” do I serve?

2 How do I store and manage content?

1 Who are my users?

1 What content do I have?

4 How to build uniqueness?

4 How to expose services?

6 Content refinement advantages?

6 “I hope someone like it...”

5

ContentRefinement

QueryAnalysis

ALERT

CONTENTPIPELINE

QUERYPIPELINE

RESULTPIPELINE

ResultAnalysis

SEARCHC

OLL

EC

TIO

NS

VIE

WS

…

…. . . . .

D

Q

CO

NTE

NT

API

QU

ERY

API

MANAGEMENT & APPLICATION SERVICES

Deployment Studio Business Studio

News

Intranet

Xyz…

FAST ESP Architecture

DecisionsInsightInformationData & Content

6

The Data processing Pipeline;Real-Time Content Refinement

FormatConversion

LanguageDetection

Lemmas(tenses, forms)

Synonyms

OntologiesCustomPLUG-IN

SpellChecking

indexContexts

TaxonomyClassification Speech2text

GeographyCompaniesPeopleNews

7

Information DiscoveryExample: scirus.com from Reed Elsevier

einstein big bang

8

9

” diving equipment for one legged woman ”

10

LOCATION NAVIGATORSPRODUCED FROM FULL DOCUMENT

Search for COUNTERTERRORISMNavigate on locations

CanadaWestern EuropeMassachusettsPittsburghGermanyGuantanamo BayMalibu

LOCATION NAVIGATORSPRODUCED FROM LOCAL CONTEXT

Search for COUNTERTERRORISMNavigate on locations

Guantanamo BayGermany

RELEVANTRELEVANT

IRRELEVANTIRRELEVANT

IRRELEVANT

IRRELEVANT

Contextual Insight – ‘The new Search’

IRRELEVANT

RELEVANTRELEVANT

11

Search 2.0 - Precision

12

Taking it further

1313

Leverage Content: Contextual Search

FAST ESPUsed for Data Quality improvement

15

’Search’ is mastering the challenge of relaxed matching

488941 britney spears40134 brittany spears36315 brittney spears24342 britany spears7331 britny spears6633 briteny spears2696 britteny spears1807 briney spears1635 brittny spears1479 brintey spears1479 britanny spears1338 britiny spears1211 britnet spears1096 britiney spears991 britaney spears991 britnay spears811 brithney spears811 brtiney spears664 birtney spears664 brintney spears664 briteney spears601 bitney spears601 brinty spears544 brittaney spears544 brittnay spears364 britey spears364 brittiny spears329 brtney spears269 bretney spears269 britneys spears244 britne spears244 brytney spears220 breatney spears220 britiany spears199 britnney spears163 britnry spears147 breatny spears147 brittiney spears147 britty spears147 brotney spears147 brutney spears133 britteney spears133 briyney spears121 bittany spears

121 bridney spears121 britainy spears121 britmey spears109 brietney spears109 brithny spears109 britni spears109 brittant spears98 bittney spears98 brithey spears98 brittiany spears98 btitney spears89 brietny spears89 brinety spears89 brintny spears89 britnie spears89 brittey spears89 brittnet spears89 brity spears89 ritney spears80 bretny spears80 britnany spears73 brinteny spears73 brittainy spears73 pritney spears66 brintany spears66 britnery spears59 briitney spears59 britinay spears54 britneay spears54 britner spears54 britney's spears54 britnye spears54 britt spears54 brttany spears48 bitany spears48 briny spears48 brirney spears48 britant spears48 britnety spears48 brittanny spears48 brttney spears44 birttany spears44 brittani spears44 brityney spears

44 brtitney spears39 brienty spears39 brritney spears36 bbritney spears36 briitany spears36 britanney spears36 briterny spears36 britneey spears36 britnei spears36 britniy spears32 britbey spears32 britneu spears29 britent spears29 brittnany spears29 britttany spears29 btiney spears26 birttney spears26 breitney spears26 brinity spears26 britenay spears26 britneyt spears26 brittan spears26 brittne spears26 btittany spears24 beitney spears24 birteny spears24 brightney spears24 brintiny spears24 britanty spears24 britenny spears24 britini spears24 britnwy spears24 brittni spears24 brittnie spears21 biritney spears21 birtany spears21 biteny spears21 bratney spears21 britani spears21 britanie spears21 briteany spears21 brittay spears21 brittinay spears21 brtany spears

21 brtiany spears19 birney spears19 brirtney spears19 britnaey spears19 britnee spears19 britony spears19 brittanty spears19 britttney spears17 birtny spears17 brieny spears17 brintty spears17 brithy spears17 brittanie spears15 brinney spears15 briten spears15 briterney spears15 britheny spears15 britneny spears15 brittamy spears15 brittmey spears15 brytnei spears15 btirney spears15 rittney spears14 brinet spears14 britneyy spears14 britten spears12 beritney spears12 bretiny spears12 briatny spears12 brieney spears12 brinany spears12 britaany spears12 britan spears12 britine spears12 britnea spears12 britnes spears12 britnez spears12 britnny spears12 brittenay spears12 brittneys spears12 brittony spears12 brteny spears12 btitany spears12 btittney spears

12 btriney spears12 btritney spears12 rittany spears11 braitney spears11 brettney spears11 britamy spears11 britery spears11 britnary spears11 brittent spears11 bruttney spears11 pritny spears10 bitaney spears10 brenty spears10 bristney spears10 britay spears10 britinny spears10 brittaany spears10 brittanys spears10 brittini spears10 brittniy spears10 brtieny spears10 brutany spears9 bitteny spears9 brinttany spears9 britanay spears9 britinany spears9 britn spears9 britnew spears9 britneyn spears9 britrney spears9 brtiny spears9 brtittney spears9 brtny spears9 brytny spears9 rbitney spears8 birtiny spears8 bithney spears8 brattany spears8 breitny spears8 breteny spears8 brightny spears8 brintay spears8 brinttey spears8 briotney spears

8 britanys spears8 britley spears8 britneyb spears8 britnrey spears8 britnty spears8 brittner spears8 brottany spears7 baritney spears7 birntey spears7 biteney spears7 bitiny spears7 breateny spears7 brianty spears7 brintye spears7 britianny spears7 britly spears7 britnej spears7 britneyu spears7 britniey spears7 britnnay spears7 brittian spears7 briyny spears7 brrittany spears7 brttiney spears7 btiteny spears7 btrittany spears6 beritny spears6 bhritney spears6 birthney spears6 breathney spears6 breaty spears6 bretany spears6 briatany spears6 brint spears6 britenney spears6 britian spears6 britinty spears6 brititney spears6 britnsy spears6 britrey spears6 britsny spears6 brittine spears6 brittnry spears6 brittsny spears

6 brtanny spears6 brtittany spears6 bruttany spears6 brytany spears6 btitny spears6 pretny spears6 ritany spears5 bbritany spears5 bbrittney spears5 biitney spears5 bitny spears5 breathny spears5 breney spears5 brethney spears5 brettny spears5 bridny spears5 brigney spears5 briittney spears5 brinaty spears5 brinay spears5 brinetty spears5 brinitty spears5 brinny spears5 brintery spears5 britary spears5 britbney spears5 brithany spears5 britieny spears5 britiey spears5 britnt spears5 britnys spears5 britrny spears5 brittenny spears5 brittnee spears5 brittnt spears5 brney spears5 broitney spears5 brotny spears5 bruteny spears5 btiyney spears5 btrittney spears5 gritney spears5 spritney spears4 bittny spears

4 bnritney spears4 brandy spears4 brbritney spears4 breatiny spears4 breetney spears4 bretiney spears4 brfitney spears4 briattany spears4 brieteny spears4 briety spears4 briitny spears4 briittany spears4 brinie spears4 brinteney spears4 brintne spears4 britaby spears4 britaey spears4 britainey spears4 britinie spears4 britinney spears4 britmney spears4 britnear spears4 britnel spears4 britneuy spears4 britnewy spears4 britnmey spears4 brittaby spears4 brittery spears4 britthey spears4 brittnaey spears4 brittnat spears4 brittneny spears4 brittnye spears4 brittteny spears4 briutney spears4 briyeny spears4 brnity spears4 brtteny spears4 brttiany spears4 bryney spears4 brythney spears4 brytne spears4 brytni spears4 brytnie spears

16

17

18

19

20

Cleansing Process :: Birdsview

XML

TELCO listings

GOVT. registers

Master database /persistant storage

Lookup to FAST for Matching

Logic for cleansing

clean data

manual inspection, correction, storage/learning

Ambigousdata

XMLXML

WEB Crawling

21

ABU BUK UKA KAB ABL BLA LAN AN, N, , K KH KHA HAL ALI LID ID D H HU HUS USS SSE SEI EIN IN N O OC OCE CEA EAN AN N B BL BLV LVD

3-Gram

Score

131 776

ABUKABLAN, KHALID HUSSEIN OCEAN BLVD 33062 POMPANO BEACH

Sample de-duping assessment

KABLAN, KHALID HUSEIN OCEAN BULEVARD POMPANO BEACH

3-Gram

KAB ABL BLA LAN AN, N, , K KH KHA HAL ALI LID ID D H HU HUS USE SEI EIN IN N O OC OCE CEA EAN AN N B BUL ULE LEV EVA VAR ARD

22

- N-Gram Score to compare strings on similarity- Lai Nancy, 501 N. Clinton St. ≈ Nancy J. Lai, 501 North Clinton

- Levenshtein edit distance algorithms- Calculates the number of edits that separates two strings - John Meyers ≈ John P. Meiers; Britney Spears ≈ Britney Spaers

- Phonetic match using Soundex, Kölner fonetic, Chapman and Jarowinkler- Calculates an integer from the strings, and matches on ’sounds like’- Christin ≈ Kristin; Kierkegaard ≈ Kierkegård

- Weighted Multi-field Compare / Context hits- Scores hits differently based on field importance

- Dictionaries to support ’proper spelling’ (names and addresses)

- Jaro and Jarowinkler- Looks for similarity around a string position, scoring starting of strings higher

- ’Charset’ normalization- For multinational chars and name combinations- André = Andre, Jürgen = Jurgen etc.

Features in FAST Data CleansingAccuracy Approaching 99.5%

23

Zero-term Search

Content Capture & Refinement

Including MARS preview

25

ContentRefinement

QueryAnalysis

ALERT

CONTENTPIPELINE

QUERYPIPELINE

RESULTPIPELINE

ResultAnalysis

SEARCHC

OLL

EC

TIO

NS

VIE

WS

…

…. . . . .

D

Q

CO

NTE

NT

API

QU

ERY

API

MANAGEMENT & APPLICATION SERVICES

Deployment Studio Business Studio

News

Intranet

Xyz…

Current FAST Architecture

Better ConnectorsPush & Pull

Operational ControlPerformance

A GUI for orchestration

CCR MARS

26

CCR content universe

REFINEMENTCONNECTORSCOLLECTIONS

MANAGEMENT

CONTENTREFINEMENT

STR

UC

TUR

EDD

ATA

RIC

H M

EDIA

UN

STR

UC

TUR

EDD

ATA

John Smith, George Johnsen13th of January, 2006

<sentence>1 GEOFFREY BIBLE, recently retired CEO of Philip Morris International. Born in Sydney in 1937, he now lives in New York.</sentence>

<person>GEOFFREY BIBLE</person>

2727

28

A more Complex CCR Graph

29

MARS: Vision and objectives

– Schema flexibility; no a priori schema definitions– Contextual search driving extreme precision and analytics– Advanced query operators including join, groupby, etc.– End-to-end XML support including XPath and XQuery– Improved performance, availability, and security– Flexibility in development and deployment – Complete disk – cache – pure memory flexibility– Support new HW and technology developments (multi-core,

64-bit, SAN/NAS)

«Project Mars designs the next generation search core that will strengthen FAST’s position as the most compelling vendor of advanced, mission critical search solutions for the next 10 years.»

Key Design Objectives:

30

Mars Node Module Decomposition

JRE 1.5/1.6

OSGi R4

IndexingESP Services

Core Services

Plug-In Engine

Java Runtime

Search ● ● ●

Platform

Node

Log Storage Config ● ● ●

Architecture Views

31

Content Processing and Querying

contentAPI

query API

result (XML)

document (XML)annotations (binary)

MQL query (text)

contentsubmission/processing

contentdistribution

Index

Index

Index

querycompiling

resultformatting

queryexecution

query graph

result sets

Search core overview

32

System Manager

System nodes

Architecture Views

Management consoleManagement nodes

33

Deployment View: Hosts and Nodes

System deployed across several Hosts

A host may contain several Nodes

Node 1 Node 2

HostController

Architecture Views

34

FAST ESP 5.0 indexing performance

INDEXING LATENCY REDUCED BY

LATENCY IS NOW FLAT

INDEXING THROUGHPUT

GAIN40%

70-90%

DISK FOOTPRINTREDUCTION

50%

4.1.2 Previous Maintream release

5.0 Current Maintream release

35

Next generation FAST ESP Search core: MARS

• Average indexing latency: 20 ms

• Average query latency while sustaining>100 QPS (scoped 2-200 term queries): 12 ms

• Disk footprint vs. ESP: -70%

• Disk – Memory configuration flexibility

• In-place Updatable index

•>80MB/s continuous index disk

• Beta deployments in Q3 ’06• Supported by $70M research grants

• 200 FAST developers

• Extensive academic network

36

“The scientific theory I like best is that the rings of Saturn are composed entirely of lost airline luggage.”

- Mark Russell.