Download - FAST Search & Transfer
2
Todays agenda
– Information discovery and metadata extraction for scientific documents
– Data Cleansing
– Ad hoc analysis and visualization. Use cases: CERN administration, CDS, EDMS
– Processing and search solutions for complex structured data
– Extreme scalability
3
Search
The Emerging “OSI Layer Model”
ECM
Data Security
Data Management
Storage
Network / Protocols
Web Services
ContentManagement
ConsumptionManagement
Service Layer
VirtualizedInfrastructure
MCE RetailInvestment
BankingSecurities
ConsumerFinancialServices
GovermentLife
SciencesHealth Care
SearchSolutions
4
The Shift that follows
Users
Content
5 What are enabling content & services?
5 How to hard-wire services in portal?
3 Key user and business drivers?
3 What type of relevant analysis?
2 What “jobs” do I serve?
2 How do I store and manage content?
1 Who are my users?
1 What content do I have?
4 How to build uniqueness?
4 How to expose services?
6 Content refinement advantages?
6 “I hope someone like it...”
5
ContentRefinement
QueryAnalysis
ALERT
CONTENTPIPELINE
QUERYPIPELINE
RESULTPIPELINE
ResultAnalysis
SEARCHC
OLL
EC
TIO
NS
VIE
WS
…
…. . . . .
D
Q
CO
NTE
NT
API
QU
ERY
API
MANAGEMENT & APPLICATION SERVICES
Deployment Studio Business Studio
News
Intranet
Xyz…
FAST ESP Architecture
DecisionsInsightInformationData & Content
6
The Data processing Pipeline;Real-Time Content Refinement
FormatConversion
LanguageDetection
Lemmas(tenses, forms)
Synonyms
OntologiesCustomPLUG-IN
SpellChecking
indexContexts
TaxonomyClassification Speech2text
GeographyCompaniesPeopleNews
10
LOCATION NAVIGATORSPRODUCED FROM FULL DOCUMENT
Search for COUNTERTERRORISMNavigate on locations
CanadaWestern EuropeMassachusettsPittsburghGermanyGuantanamo BayMalibu
LOCATION NAVIGATORSPRODUCED FROM LOCAL CONTEXT
Search for COUNTERTERRORISMNavigate on locations
Guantanamo BayGermany
RELEVANTRELEVANT
IRRELEVANTIRRELEVANT
IRRELEVANT
IRRELEVANT
Contextual Insight – ‘The new Search’
IRRELEVANT
RELEVANTRELEVANT
15
’Search’ is mastering the challenge of relaxed matching
488941 britney spears40134 brittany spears36315 brittney spears24342 britany spears7331 britny spears6633 briteny spears2696 britteny spears1807 briney spears1635 brittny spears1479 brintey spears1479 britanny spears1338 britiny spears1211 britnet spears1096 britiney spears991 britaney spears991 britnay spears811 brithney spears811 brtiney spears664 birtney spears664 brintney spears664 briteney spears601 bitney spears601 brinty spears544 brittaney spears544 brittnay spears364 britey spears364 brittiny spears329 brtney spears269 bretney spears269 britneys spears244 britne spears244 brytney spears220 breatney spears220 britiany spears199 britnney spears163 britnry spears147 breatny spears147 brittiney spears147 britty spears147 brotney spears147 brutney spears133 britteney spears133 briyney spears121 bittany spears
121 bridney spears121 britainy spears121 britmey spears109 brietney spears109 brithny spears109 britni spears109 brittant spears98 bittney spears98 brithey spears98 brittiany spears98 btitney spears89 brietny spears89 brinety spears89 brintny spears89 britnie spears89 brittey spears89 brittnet spears89 brity spears89 ritney spears80 bretny spears80 britnany spears73 brinteny spears73 brittainy spears73 pritney spears66 brintany spears66 britnery spears59 briitney spears59 britinay spears54 britneay spears54 britner spears54 britney's spears54 britnye spears54 britt spears54 brttany spears48 bitany spears48 briny spears48 brirney spears48 britant spears48 britnety spears48 brittanny spears48 brttney spears44 birttany spears44 brittani spears44 brityney spears
44 brtitney spears39 brienty spears39 brritney spears36 bbritney spears36 briitany spears36 britanney spears36 briterny spears36 britneey spears36 britnei spears36 britniy spears32 britbey spears32 britneu spears29 britent spears29 brittnany spears29 britttany spears29 btiney spears26 birttney spears26 breitney spears26 brinity spears26 britenay spears26 britneyt spears26 brittan spears26 brittne spears26 btittany spears24 beitney spears24 birteny spears24 brightney spears24 brintiny spears24 britanty spears24 britenny spears24 britini spears24 britnwy spears24 brittni spears24 brittnie spears21 biritney spears21 birtany spears21 biteny spears21 bratney spears21 britani spears21 britanie spears21 briteany spears21 brittay spears21 brittinay spears21 brtany spears
21 brtiany spears19 birney spears19 brirtney spears19 britnaey spears19 britnee spears19 britony spears19 brittanty spears19 britttney spears17 birtny spears17 brieny spears17 brintty spears17 brithy spears17 brittanie spears15 brinney spears15 briten spears15 briterney spears15 britheny spears15 britneny spears15 brittamy spears15 brittmey spears15 brytnei spears15 btirney spears15 rittney spears14 brinet spears14 britneyy spears14 britten spears12 beritney spears12 bretiny spears12 briatny spears12 brieney spears12 brinany spears12 britaany spears12 britan spears12 britine spears12 britnea spears12 britnes spears12 britnez spears12 britnny spears12 brittenay spears12 brittneys spears12 brittony spears12 brteny spears12 btitany spears12 btittney spears
12 btriney spears12 btritney spears12 rittany spears11 braitney spears11 brettney spears11 britamy spears11 britery spears11 britnary spears11 brittent spears11 bruttney spears11 pritny spears10 bitaney spears10 brenty spears10 bristney spears10 britay spears10 britinny spears10 brittaany spears10 brittanys spears10 brittini spears10 brittniy spears10 brtieny spears10 brutany spears9 bitteny spears9 brinttany spears9 britanay spears9 britinany spears9 britn spears9 britnew spears9 britneyn spears9 britrney spears9 brtiny spears9 brtittney spears9 brtny spears9 brytny spears9 rbitney spears8 birtiny spears8 bithney spears8 brattany spears8 breitny spears8 breteny spears8 brightny spears8 brintay spears8 brinttey spears8 briotney spears
8 britanys spears8 britley spears8 britneyb spears8 britnrey spears8 britnty spears8 brittner spears8 brottany spears7 baritney spears7 birntey spears7 biteney spears7 bitiny spears7 breateny spears7 brianty spears7 brintye spears7 britianny spears7 britly spears7 britnej spears7 britneyu spears7 britniey spears7 britnnay spears7 brittian spears7 briyny spears7 brrittany spears7 brttiney spears7 btiteny spears7 btrittany spears6 beritny spears6 bhritney spears6 birthney spears6 breathney spears6 breaty spears6 bretany spears6 briatany spears6 brint spears6 britenney spears6 britian spears6 britinty spears6 brititney spears6 britnsy spears6 britrey spears6 britsny spears6 brittine spears6 brittnry spears6 brittsny spears
6 brtanny spears6 brtittany spears6 bruttany spears6 brytany spears6 btitny spears6 pretny spears6 ritany spears5 bbritany spears5 bbrittney spears5 biitney spears5 bitny spears5 breathny spears5 breney spears5 brethney spears5 brettny spears5 bridny spears5 brigney spears5 briittney spears5 brinaty spears5 brinay spears5 brinetty spears5 brinitty spears5 brinny spears5 brintery spears5 britary spears5 britbney spears5 brithany spears5 britieny spears5 britiey spears5 britnt spears5 britnys spears5 britrny spears5 brittenny spears5 brittnee spears5 brittnt spears5 brney spears5 broitney spears5 brotny spears5 bruteny spears5 btiyney spears5 btrittney spears5 gritney spears5 spritney spears4 bittny spears
4 bnritney spears4 brandy spears4 brbritney spears4 breatiny spears4 breetney spears4 bretiney spears4 brfitney spears4 briattany spears4 brieteny spears4 briety spears4 briitny spears4 briittany spears4 brinie spears4 brinteney spears4 brintne spears4 britaby spears4 britaey spears4 britainey spears4 britinie spears4 britinney spears4 britmney spears4 britnear spears4 britnel spears4 britneuy spears4 britnewy spears4 britnmey spears4 brittaby spears4 brittery spears4 britthey spears4 brittnaey spears4 brittnat spears4 brittneny spears4 brittnye spears4 brittteny spears4 briutney spears4 briyeny spears4 brnity spears4 brtteny spears4 brttiany spears4 bryney spears4 brythney spears4 brytne spears4 brytni spears4 brytnie spears
20
Cleansing Process :: Birdsview
XML
TELCO listings
GOVT. registers
Master database /persistant storage
Lookup to FAST for Matching
Logic for cleansing
clean data
manual inspection, correction, storage/learning
Ambigousdata
XMLXML
WEB Crawling
21
ABU BUK UKA KAB ABL BLA LAN AN, N, , K KH KHA HAL ALI LID ID D H HU HUS USS SSE SEI EIN IN N O OC OCE CEA EAN AN N B BL BLV LVD
3-Gram
Score
131 776
ABUKABLAN, KHALID HUSSEIN OCEAN BLVD 33062 POMPANO BEACH
Sample de-duping assessment
KABLAN, KHALID HUSEIN OCEAN BULEVARD POMPANO BEACH
3-Gram
KAB ABL BLA LAN AN, N, , K KH KHA HAL ALI LID ID D H HU HUS USE SEI EIN IN N O OC OCE CEA EAN AN N B BUL ULE LEV EVA VAR ARD
22
- N-Gram Score to compare strings on similarity- Lai Nancy, 501 N. Clinton St. ≈ Nancy J. Lai, 501 North Clinton
- Levenshtein edit distance algorithms- Calculates the number of edits that separates two strings - John Meyers ≈ John P. Meiers; Britney Spears ≈ Britney Spaers
- Phonetic match using Soundex, Kölner fonetic, Chapman and Jarowinkler- Calculates an integer from the strings, and matches on ’sounds like’- Christin ≈ Kristin; Kierkegaard ≈ Kierkegård
- Weighted Multi-field Compare / Context hits- Scores hits differently based on field importance
- Dictionaries to support ’proper spelling’ (names and addresses)
- Jaro and Jarowinkler- Looks for similarity around a string position, scoring starting of strings higher
- ’Charset’ normalization- For multinational chars and name combinations- André = Andre, Jürgen = Jurgen etc.
Features in FAST Data CleansingAccuracy Approaching 99.5%
25
ContentRefinement
QueryAnalysis
ALERT
CONTENTPIPELINE
QUERYPIPELINE
RESULTPIPELINE
ResultAnalysis
SEARCHC
OLL
EC
TIO
NS
VIE
WS
…
…. . . . .
D
Q
CO
NTE
NT
API
QU
ERY
API
MANAGEMENT & APPLICATION SERVICES
Deployment Studio Business Studio
News
Intranet
Xyz…
Current FAST Architecture
Better ConnectorsPush & Pull
Operational ControlPerformance
A GUI for orchestration
CCR MARS
26
CCR content universe
REFINEMENTCONNECTORSCOLLECTIONS
MANAGEMENT
CONTENTREFINEMENT
STR
UC
TUR
EDD
ATA
RIC
H M
EDIA
UN
STR
UC
TUR
EDD
ATA
John Smith, George Johnsen13th of January, 2006
<sentence>1 GEOFFREY BIBLE, recently retired CEO of Philip Morris International. Born in Sydney in 1937, he now lives in New York.</sentence>
<person>GEOFFREY BIBLE</person>
29
MARS: Vision and objectives
– Schema flexibility; no a priori schema definitions– Contextual search driving extreme precision and analytics– Advanced query operators including join, groupby, etc.– End-to-end XML support including XPath and XQuery– Improved performance, availability, and security– Flexibility in development and deployment – Complete disk – cache – pure memory flexibility– Support new HW and technology developments (multi-core,
64-bit, SAN/NAS)
«Project Mars designs the next generation search core that will strengthen FAST’s position as the most compelling vendor of advanced, mission critical search solutions for the next 10 years.»
Key Design Objectives:
30
Mars Node Module Decomposition
JRE 1.5/1.6
OSGi R4
IndexingESP Services
Core Services
Plug-In Engine
Java Runtime
Search ● ● ●
Platform
Node
Log Storage Config ● ● ●
Architecture Views
31
Content Processing and Querying
contentAPI
query API
result (XML)
document (XML)annotations (binary)
MQL query (text)
contentsubmission/processing
contentdistribution
Index
Index
Index
querycompiling
resultformatting
queryexecution
query graph
result sets
Search core overview
33
Deployment View: Hosts and Nodes
System deployed across several Hosts
A host may contain several Nodes
Node 1 Node 2
HostController
Architecture Views
34
FAST ESP 5.0 indexing performance
INDEXING LATENCY REDUCED BY
LATENCY IS NOW FLAT
INDEXING THROUGHPUT
GAIN40%
70-90%
DISK FOOTPRINTREDUCTION
50%
4.1.2 Previous Maintream release
5.0 Current Maintream release
35
Next generation FAST ESP Search core: MARS
• Average indexing latency: 20 ms
• Average query latency while sustaining>100 QPS (scoped 2-200 term queries): 12 ms
• Disk footprint vs. ESP: -70%
• Disk – Memory configuration flexibility
• In-place Updatable index
•>80MB/s continuous index disk
• Beta deployments in Q3 ’06• Supported by $70M research grants
• 200 FAST developers
• Extensive academic network