http file shares sharepoint user profiles lotus notes documentum exchange folders custom - bcs...

Search Architecture in SharePoint 2013Rune Zakariassen Thomas MolbachProgram Manager Architect, Search COE

SPC202

HTTPFile sharesSharePointUser profilesLotus Notes DocumentumExchange foldersCustom - BCS

SharePointSP AppsDevicesNon-SP UX

SharePoint 2013 Search Architecture

SearchAdmin

Content UXCrawl

ContentProcessing Index

QueryProcessing WFE

API

AnalyticsProcessing

FAST Search Index

Public API

Unit of scale/role boundary

Feeding ChainIndex Core

Query Chain

Analytics Service

What is SharePoint 2013 search?

SharePoint 2013 Search

SharePoint 2010 Search

FAST Search

For SharePoi

nt

FAST Technolog

y

Office Search Scenarios

Search 2013 Platform

Exchange 2013 platform SharePoint 2013 platform

Search 2013 platform

`

Exchange

SharePoint

Custom solution

CSOM/REST API

Outlook Web App (OWA) search eDiscovery

Intranet search

Custom app (Internet/intranet)

Marketplace app

All

resu

lts

Peop

le

Sit

e s

earc

h

Vid

eo

Cust

om

searc

h

vert

ical

Internetsearch

Content sources

Public API

SharePoint 2013 Search Architecture

SearchAdmin

Content UXCrawl

ContentProcessing Index

QueryProcessing WFE

API

AnalyticsProcessing

FAST Search Index

Public API

Unit of scale/role boundary

Feeding Chain

HTTPFile sharesSharePointUser profilesLotus Notes DocumentumExchange foldersCustom - BCS

Crawl Component• OOB connectors

• Extensible through BCS

• Local disk cache

• Crawled items tracked in Crawl database

• Configurations stored in Admin database

• Crawl modes• Full Crawl• Incremental Crawl• Continuous Crawl

Crawl

ContentProcessing Index

Crawl

FAST Search Index

HTTP

File Shares

SharePoint

User Profiles

Exchange

Lotus Notes

Documentum

Custom (BCS)

...SearchAdmin

Admin

mssearch.exe

Content Processing Component• Stateless node

• Analyzes content for indexing

• Processing flow

• Dictionaries

• Schema mapping

• Stores links and anchors in Link database (analytics)

• Extensible through web service call-outs

• Configurations stored in admin database

Crawl

ContentProcessing Index

Crawl

FAST Search Index

SearchAdmin

Admin

AnalyticsProcessing

Link

ContentEnrichmentWeb Service

Content Processing Component

Content Processing Flow (conceptual)

Detect language

Document

summary

Map to managed propertie

s

Parse documents and extract metadata

Custom Entity

Extraction

Phonetic name

variations

Word breaking

Web Service Callout

Ifilter sandbo

x

Security Descriptor

s

Delete item

Update security

Crawler

Index

Delete Links

Security Descriptor

s

Insert new or updated item

Each CrawlerFlow processes one item at time.

Demo 1

Pipeline – smart title/author extractor

SharePoint 2013 Search Architecture

SearchAdmin

Content UXCrawl

ContentProcessing Index

QueryProcessing WFE

API

AnalyticsProcessing

FAST Search Index

Public API

Unit of scale/role boundary

Index Core

• Partitions and replicas• No rows and columns

• Index system• n Partitions• 1 primary replica• ≥ 0 secondary replica

• All nodes performs indexing• Journal shipping from primary to replicas

Index Partition details

Index

Index

Index

Index

Index

Content Processing

Index

Index

Index

Index

Index

Index

Index

Append sectionCheckpoint section

Merging Index PartsMaster partLevel

M

Level 0

Level 1

Level 2

New Master part

+

Disk

Memory

Checkpoint

Master merge

= 1 generation

Demo 2

Index Schema and Managed Properties

• Crawled property managed property• Site collection schema mapping allows• Custom/managed property mapping• Aliasing of managed properties• Enhanced functionality with managed property:• Data type• Queryable• Searchable• Retrievable• Refinable• Sortable• Safe for anonymous access

Understanding Index Schemas

With Schema Mapping

SearchAdmin

Content UXCrawl

ContentProcessing Index

QueryProcessing WFE

API

AnalyticsProcessing

FAST Search IndexDocId Partition

Content Title Author Modified

RefinableText1

111 0 Xyz... My lab ... Rune ... Some value

222 0 Abc... My Word ... Rune ... ...

333 1 .... ....

HTTPFile sharesSharePointUser profilesLotus Notes DocumentumExchange foldersCustom - BCS

SharePointSP AppsDevicesNon-SP UX

SharePoint 2013 Search Architecture

SearchAdmin

Content UXCrawl

ContentProcessing Index

QueryProcessing WFE

API

AnalyticsProcessing

FAST Search Index

Public API

Unit of scale/role boundary

Query Chain

SharePointSP AppsDevicesNon-SP UX

Web Front-End

SearchAdmin

UXQueryProcessing WFE

API

Index

FAST Search Index

Public API

Unit of scale/role boundary

• REST/OData API• Client-Side object model

(CSOM)• Server-Side object

model (SSOM)• Search center• Portals and publishing• ContentWebPart• RefinerWebPart• Result templates

Search

SharePoint 2013

SharePoint 2013 Remoting API

CSOM

http://<spsite>/_api

JavaScriptLibrary

SilverlightLibrary

.NET CLRLibraryREST

ODataJSON

Custom Client Code

Server

Client

SharePointSP AppsDevicesNon-SP UX

Query Processing Component

SearchAdmin

UXQueryProcessing WFE

API

Index

FAST Search Index

Public API

Unit of scale/role boundary

• Stateless node• Processing flows• Query analyzer• Linguistics• Dictionaries• Result sources• Schema mapping• Query rules• Query federation • Configurations

stored in admin database

Admin

Query Execution Flow (conceptual)

WFE

Query Processing Component

Index

Index

IndexP0

Index

Index

IndexP1

Index

Index

IndexP2

Index

Index

IndexP3

Search Provider Flows

Best Bets

Exchange Search

People Search

Personal Favorites

Remote SharePoint

Local SharePoint

Keyw

ord

Query

FlowInput

QueryRuleMatching

QueryTransformer

QueryRouter

FlowOutput

ResultTableAssembler

LayoutSelector

Query Rules

Captures Search IntentReplaces search keywords in 2010Create query rules at Farm, Site Collection, and Site levels

Composed of 3 top level elementsQuery Conditions

What queries should be handled?

Query Actions

What happens when a rule matches?

PublishingOptions

Is the rule active and for how long?

Demo 3

The query sent to the index

Search

INTERNAL ITEM CATALOGSITE COLLECTION

Electronics

DESKTOP WEB CHANNEL MOBILE CHANNEL

REST API

NATIVE APP

CAMPAIGNPROMOTION

Editorial Control with Query Rules

HTTPFile sharesSharePointUser profilesLotus Notes DocumentumExchange foldersCustom - BCS

SharePointSP AppsDevicesNon-SP UX

SharePoint 2013 Search Architecture

SearchAdmin

Content UXCrawl

ContentProcessing Index

QueryProcessing WFE

API

AnalyticsProcessing

FAST Search Index

Public API

Unit of scale/role boundary

Analytics Service

Analytics Processing Component

Crawl

ContentProcessing Index

QueryProcessing WFE

API

AnalyticsProcessing

FAST Search Index

Link

Analytics Reporting

• Map-reduce• Learns by usage• Search Analytics• Usage Analytics• Enriches index by

updating index items

• Usage reports in Analytics database

Event Stream

Up sell / cross sellConsumers

Light-up with Recommendations

SharePoint 2013 Analytics Processing

Purchased Products

Viewed Products Liked Products

Trending Products

Recommendations

Built-in recommendationsEvent stream analysisItem-to-item recommendationsStored in “recommendedfor:” managed propertyUp to 12 configurable, weighted, events

Demo 4

Analytics and Recommendations

SharePoint 2013 Search Architecture

SearchAdmin

Content UXCrawl

ContentProcessing Index

QueryProcessing WFE

API

AnalyticsProcessing

Crawl

Admin

Link

Analytics Reporting

FAST Search Index

ContentEnrichmentWeb Service

CustomConnectors

Public API

Unit of scale/role boundary

Extensibility Points

Search HOLs and events @ SPCHOL031 – Introduction to Search in SharePoint 2013HOL034 – Exploring Search Query Rules in SharePoint 2013HOL032 – Extending the Search experience in SharePoint 2013HOL033 – People Search in SharePoint 2013HOL035 – SharePoint Server 2013 Search Connectors and Using BCS

Meet a Search SME

Ask questions, meet the community and share knowledge!

Mon-Thu @ Exhibit Hall

Hands on Labs

Daily 10:30am-6:30pm @ HOL Lab Lounge

Ask the Experts

Discuss search!

Wed 6:15PM @ Ask the Experts Lounge

Related Search Sessions @ SPCTue 10:30am - SPC044 - Crawl and Index all Enterprise Content with SharePoint 2013 Search - Speaker: Vaidy Raghavan

Tue 5:00pm - SPC172 - Overview of Capacity Planning, Sizing and High Availability for Search in SharePoint 2013 - Speakers: Barry Waldbaum, Olaf Birkeland

Wed 9:00am - SPC143 - Making Great Search Based Applications with Query Rules in SharePoint 2013 - Speaker: Pedro DeRose

Tue 5:00pm - SPC203 - Search Content Enrichment and Extensibility in SharePoint 2013Speakers: Runar Olsen, Kathrine Hammervold

Wed 3:15pm - SPC049 - Custom Security Trimming for Search in SharePoint 2013Speakers: Morgan Larsson, Sveinar Rasmussen Wed 5:00pm - SPC145 - Optimize Search Relevance in SharePoint 2013Speakers: Jan Inge Bergseth, Victor Poznanski

Questions?

Evaluate this session now on MySPC using your laptop or mobile device: http://myspc.sharepointconference.com

MySPC

© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.