portal.just.roportal.just.ro/test doclib/1663fastsearchatmstwp.docx  · web viewhowever, many...

44
Improving Enterprise Search at Microsoft How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft Technical White Paper Published: February 2012

Upload: ngotuyen

Post on 04-Feb-2018

213 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Improving Enterprise Search at Microsoft

How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft

Technical White PaperPublished: February 2012

Page 2: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Contents

Executive Summary............................................................................4

Background on Enterprise Search at Microsoft....................................5MSIT Implementation of Enterprise Search That Used SharePoint Server 2007 5

Requirements for an Improved Search Solution 8

FAST Search Server 2010 for SharePoint 8

Microsoft Enterprise Search Teams.........................................................................................................10

Enterprise Search Positions in MSIT 10

Enterprise Search Product Group 12

Migration to FAST Search Server 2010 for SharePoint.........................................................................................................13

FAST Search Server 2010 for SharePoint Solution: Environment and Topology.........................................................................................................14

SharePoint Server 2010 Server Farm 15

FAST Search Server 2010 for SharePoint Server Farm 15

Database Servers 15

Measurable Gains from Implementing FAST Search Server 2010 for SharePoint.........................................................................................................16

Administration at the Search Service Application Level.........................................................................................................18

Configuring Crawls 18

Content Sources.....................................................................................................................18

Full Crawls..............................................................................................................................18

Incremental Crawls.................................................................................................................19

Crawl Rules.............................................................................................................................20

Dedicated Crawl Targets........................................................................................................20

Page 3: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Crawler Impact Rules..............................................................................................................20

Configuring Other Search Settings 20

Global Scopes.........................................................................................................................20

Removal of Items from the Content Index..............................................................................21

Time-out Settings, IFilters, and File Types.............................................................................21

Thesaurus Files and Stop Word Files.....................................................................................21

Backups..................................................................................................................................21

Monitoring Search 21

Search Health Monitoring.......................................................................................................22

Search Usage Monitoring.......................................................................................................23

Collaboration with Site Owners.........................................................................................................24

Creating Managed Properties 24

Configuring Relevance 24

Rank Profiles...........................................................................................................................24

Custom Full-Text Indexes.......................................................................................................25

Enterprise Search Center.........................................................................................................26

Getting to the Enterprise Search Center 26

Enterprise Search Center Tabs 27

Intranet Tab.............................................................................................................................27

People Tab..............................................................................................................................28

Video Tab................................................................................................................................28

Best Bets and URL Promotion 29

Lessons Learned and MSIT Best Practices.........................................................................................................30

Conclusion.........................................................................................................32

Page 4: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

For More Information.........................................................................................................33

Page 5: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

EXECUTIVE SUMMARYMicrosoft employees create and store large amounts of information on the Microsoft corporate intranet every day. Employees require a powerful enterprise search solution that enables them to find items on the intranet quickly and easily amid the huge amount of stored data.

The Microsoft information Technology (MSIT) group manages all aspects of the enterprise search implementation at Microsoft. In 2006, MSIT deployed Microsoft SharePoint Server 2007 to provide a new enterprise search solution for employees. The solution was geographically distributed to provide fast crawl and query performance. It also provided an enterprise Search Center that all employees could access to perform queries on all intranet content.

By 2009, the content index for the enterprise Search Center approached the SharePoint Server 2007 maximum capacity of 50 million items, and the large crawl load caused crawl and query performance to decrease. At that point, MSIT evaluated options for migrating to an enterprise search solution that would provide benefits such as centralized management, greater scalability for large capacity, improved crawl and query performance, high availability, and advanced features such as fine adjustments for ranking of search results. MSIT chose Microsoft FAST Search Server 2010 for SharePoint because it satisfied all of these demands.

MSIT upgraded to FAST Search Server 2010 for SharePoint in late 2009, some months before the public release of the software. This paper describes the experience of MSIT as an early adopter of FAST Search Server 2010 for SharePoint. The paper describes how MSIT deployed and configured FAST Search Server 2010 for SharePoint, how MSIT administers the search system, and how the solution meets the need for an advanced enterprise search solution at Microsoft by providing fresh, complete, and useful search results in a robust, highly customizable deployment.

In comparison to a SharePoint Server 2007 search solution, MSIT found that it takes additional hardware, skills, and people to plan, deploy, operate, and support a large deployment such as the MSIT search solution that uses FAST Search Server 2010 for SharePoint. However, automation of certain maintenance tasks, together with careful planning and governance, helped MSIT simplify and expedite deployment and management of the FAST Search Server 2010 for SharePoint solution.

This paper is not a procedural guide. Because each IT environment is unique, each organization should implement enterprise search according to its particular needs. However, many of the considerations, practices, and lessons that this paper describes can be relevant to the implementation of enterprise search in other large-scale IT environments.

The primary audience for this paper includes enterprise-level business decision makers, technical decision makers, FAST Search Server 2010 for SharePoint server farm administrators, and search-service application administrators. SharePoint site collection administrators and site owners might also find this paper useful. It is assumed that readers of this paper are familiar with the purpose and business value of enterprise search, SharePoint search technologies, and features and functionality of FAST Search Server 2010 for SharePoint and SharePoint Server 2007.

Note: For security reasons, names in this paper that refer to Microsoft-internal resources are for illustration purposes only and do not represent actual names.

Improving Enterprise Search at Microsoft Page 5

SituationTo perform their day-to-day responsibilities, Microsoft employees frequently use enterprise search to find information on the corporate intranet. Microsoft IT used Microsoft Office SharePoint Server 2007 to provide enterprise search for employees until 2009, when the content index for the enterprise Search Center approached the software capacity of 50 million items and crawl and query performance decreased. To improve performance, MSIT configured the search system so that it no longer crawled certain content repositories, but then employees were no longer able to get complete search results from the enterprise Search Center.

SolutionMicrosoft IT migrated to Microsoft FAST Search Server 2010 for SharePoint to provide a more powerful enterprise search solution.

Benefits Centralized management Content index completeness Huge index capacity Faster crawls for fresher search

results Faster query responses More customizable search

experiences More relevant search results High availability Increased user satisfaction

Products & Technologies FAST Search Server 2010 for

SharePoint SharePoint Server 2007

Page 6: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

BACKGROUND ON ENTERPRISE SEARCH AT MICROSOFTEvery day, thousands of Microsoft employees, contractors, and vendors create and store large amounts of digital information on servers in many locations across the worldwide Microsoft corporate enterprise. Content is in the form of Microsoft Office documents, graphics, videos, and other formats. Because Microsoft workers use the collaborative features of SharePoint Server extensively, MSIT must manage more than 34 terabytes of content on SharePoint intranet sites. This includes content such as SharePoint Server libraries, lists, blogs, wikis, and My Sites.

MSIT hosts and manages all intranet sites that consume the enterprise search service. The following table provides examples of these sites.

Table 1. Examples of Sites That Use the Enterprise Search Service

Site or sites Description

MSW MSW is one of the primary enterprise portals at Microsoft. It provides daily news, executive updates, and other information. In addition, employees frequently go to this site to get to other intranet sites, such as sites for human resources, legal affairs, and technical research. The MSW farm also hosts the enterprise Search Center.

Division-level portals Examples of division-level portals that use the enterprise search service include the following:

ITWeb, the portal for technical support FinWeb, the finance portal LCAWeb, the portal for the Legal and Corporate Affairs group Infopedia, the portal for sales and marketing information, which

provides content related to engaging with customers and partners Office, the portal for employees in the Microsoft Office Division

Department-level portals Employees can use these portals to create collaboration sites or portal sites that host business-critical information.

Team sites Employees can use these portals to create team sites for collaboration. On team sites, employees can also use the latest SharePoint features and test proof-of-concept solutions.

My Sites Each employee has a My Site to store business and personal information and documents. Employees can restrict access to items on their My Site.

However, many other groups and individuals in the company create and host their own sites. MSIT does not permit these sites to consume the enterprise search service, but the site owners can ask MSIT to configure the enterprise search service to crawl their sites. Other important corporate data repositories include file shares and structured data sources in line-of-business applications. MSIT must provide an enterprise search solution that enables the members of the Microsoft work force to quickly and easily find up-to-date, relevant information in all of these large and diverse repositories.

MSIT Implementation of Enterprise Search That Used SharePoint Server 2007MSIT has used SharePoint technologies to provide enterprise search capability to Microsoft employees for a number of years. MSIT used Microsoft SharePoint Portal Server 2003 to

Improving Enterprise Search at Microsoft Page 6

Page 7: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

provide enterprise search from 2003 until late 2006, at which point that solution no longer scaled to company requirements. MSIT then upgraded to an enterprise search solution that used SharePoint Server 2007.

As with the previous solution, in the SharePoint Server 2007 solution, MSIT maintained three separate Shared Services Providers (SSPs), each in a separate Microsoft data center. One SSP was in Singapore, the second SSP was in Dublin, Ireland, and the third SSP was in Redmond, Washington. The search service in the Dublin SSP crawled only content in the Europe, Middle East, and Africa (EMEA) region (and therefore provided search results for that region only), while the search service in the Singapore SSP crawled content in the Asia-Pacific region (and therefore provided search results for that region only). For a time, however, the Redmond SSP search service crawled content in all three regions and provided worldwide search results that all users could access by going to the enterprise Search Center.

Figure 1 indicates the region or regions that the search service of each SSP crawled in the initial state of the SharePoint Server 2007 search solution.

SSP in Redmond data center:Initially crawled content worldwide

SSP in Singapore data center:Crawled Asia-Pacific regional content

SSP in Dublin data center:Crawled EMEA regional content

Figure 1. Content crawled by each search servicein the initial SharePoint Server 2007 search solution

MSIT took this regional approach to enterprise search because search services were not fully supported over the corporate wide area network (WAN). Therefore, each region required a separate server-farm deployment for search. The deployment that hosted the Redmond search service used one dedicated index server, three dedicated query servers, and two database servers. The EMEA and Asia-Pacific search deployments each used one dedicated index server, two servers configured with both the query server role and the web server role, and one database server. Thus, a total of 14 computers in three different data centers handled the worldwide enterprise search solution.

An advantage of this approach for users in the EMEA and Asia-Pacific regions was that search results were relatively fresh, because crawling and indexing were relatively fast on the

Improving Enterprise Search at Microsoft Page 7

Page 8: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

regional scale. Query responses were also relatively fast in those regions for a similar reason.

The main disadvantage of this approach was that there were three separate search services for MSIT to manage, so there were three separate content indexes, one corresponding to each search service. By searching from a site in the Asia-Pacific region, users could get search results for sites in that region only. Similarly, by searching from a site in the EMEA region, users could get search results for sites in that region only. However, because the Redmond search service was crawling all three regions, users in any region could still get search results for content worldwide by searching from the enterprise Search Center.

By late 2009, the Redmond SSP search service was crawling and indexing more than 13 terabytes of data that corresponded to more than 40 million items. Incremental crawl times for some content sources had increased to seven days or more, so that search results were frequently not fresh. Moreover, query latency in the Redmond search service had increased to eight seconds or more in some cases. Therefore, MSIT needed to reduce the amount of content that the Redmond search service was crawling.

For this reason, MSIT configured the Redmond search service so that it crawled only the Americas region and no longer crawled the EMEA and Asia-Pacific regions. The search service in each region was then crawling only regional data. However, users became dissatisfied because they could no longer get search results for content worldwide by searching from the enterprise Search Center. Figure 2 indicates the region that the search service of each SSP crawled in the final state of the SharePoint Server 2007 search solution.

SSP in Redmond data center:Crawled Americas regional content

SSP in Singapore data center:Crawled Asia-Pacific regional content

SSP in Dublin data center:Crawled EMEA regional content

Figure 2. Content crawled by each search servicein the final state of the SharePoint Server 2007 search solution

To improve performance of the Redmond search service even more, MSIT removed two other content sources from that service: Team (the content source for crawling SharePoint team sites) and Office (the content source for crawling the site collection of the Microsoft

Improving Enterprise Search at Microsoft Page 8

Page 9: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Office Division). The drawback of this change was that the only way to get search results from either of those site collections after that was to search directly from that site collection.

Another issue was that users in various locations were sometimes reporting that they were not getting helpful search results. Search results were sometimes out of date or not ranked appropriately as to relevance. In some cases, the search service took a long time to report that there were no search results at all.

In summary, due to the increased number of users, the distribution of users in different geographical locations, and the growth in the amount of content, the MSIT deployment of SharePoint Server 2007 no longer provided an adequate enterprise search solution to Microsoft workers. Thus, the need arose for a more powerful enterprise search solution.

Requirements for an Improved Search SolutionMSIT identified the following business, technical, and administrative requirements for an enterprise-level search service:

Centralized management. The search solution must be hosted and manageable in a single deployment, rather than through deployments in multiple locations.

Completeness of content index. There must be a single content index for all crawled content, including all site content in each of the three geographic regions (Americas, Asia-Pacific, and EMEA), so that all users can get search results from the same set of content.

High capacity. The content index must be able to accommodate projected growth to 200 million items in 2013 or later.

Significantly reduced crawl times. Incremental crawls of all high-priority content sources must take less than 24 hours, so that all search results are as fresh as possible for the most important content.

Significantly reduced query latency. Users must be able to get search results in less than two seconds.

Increased ability to find relevant content. Search results must be ranked according to relevance—for example, based on authority of sites and pertinence of metadata—so that users can quickly find information that they are looking for.

Uniform search experience. All users worldwide must have access to an enterprise search center that provides the same search experience.

Significantly increased user satisfaction. Users must be pleased with the increase in performance and effectiveness of the search system and with the overall search experience.

FAST Search Server 2010 for SharePointIn 2008, Microsoft acquired FAST Search and Transfer, a leading enterprise search vendor. Microsoft merged search technologies from that acquisition with SharePoint search technologies to create a new product, Microsoft FAST Search Server 2010 for SharePoint.

MSIT evaluated the search features and functionality of SharePoint Server 2010 and FAST Search Server 2010 for SharePoint to determine which product would best meet the enterprise search requirements. FAST Search Server 2010 for SharePoint satisfied all of the demands. MSIT adopted FAST Search Server 2010 for SharePoint primarily because it is highly scalable and can support high capacity and performance for a large enterprise

Improving Enterprise Search at Microsoft Page 9

Page 10: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

deployment. FAST Search Server 2010 for SharePoint also provides many advanced options, such as the ability to tune the ranking of search results.

Improving Enterprise Search at Microsoft Page 10

Page 11: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

MICROSOFT ENTERPRISE SEARCH TEAMSThis section provides information about the responsibilities of MSIT and the Microsoft enterprise search product group with respect to the implementation of enterprise search at Microsoft.

Enterprise Search Positions in MSITAmong the responsibilities of MSIT are the deployment and operation of SharePoint shared services for the enterprise. These shared services include the Managed Metadata service, the User Profile service, Business Connectivity Services, and the enterprise search service.

MSIT manages all aspects of the enterprise search implementation at Microsoft, which include planning, deployment, operation, and support. MSIT administers the search service, deploys and maintains related hardware and software, provides the search experience on the enterprise Search Center site, and helps other site owners provide a search experience that meets their requirements and the requirements of their users.

The following positions in MSIT have responsibility for enterprise search:

Search service manager. This is a program manager position that has the following responsibilities:

Business ownership, which includes gathering business requirements from site owners and end users, specifying solutions for the end-user experience, managing the budget, and acting as liaison to the search service operations team and the enterprise search product group.

Planning and deployment of the FAST Search Server 2010 for SharePoint solution.

Configuration of the search service to ensure that published content is readily discoverable in end-user searches, such as by configuring managed properties and global scopes for the search service and by managing Best Bets in the enterprise Search Center.

Management and support of search service operations.

Training and support for owners of sites that consume the enterprise search service. This includes working with site owners to help them create the search experience that they want for their users. It also includes helping site owners make their content more readily discoverable by using search features such as scopes, managed properties, and Best Bets.

Collecting and analyzing end-user feedback and updating the configuration of the search service to meet end-user needs.

Training for Helpdesk personnel who provide end-user support.

Planning and vision for the MSIT implementation of enterprise search and the user search experience beyond the current deployment.

Search service engineer. Search service engineers plan, design, test, deploy, operate, and support the search solution. They have primary responsibility for ensuring the continuity and responsiveness of the enterprise search service. This includes server infrastructure and topology (logical and physical components), operations monitoring, incident management, configuration changes, application of updates, and documentation of certain search issues.

Improving Enterprise Search at Microsoft Page 11

Page 12: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Search analyst. Search analysts create, maintain, and review Best Bets and associated keywords for the enterprise Search Center every day based on user feedback and based on their analysis of search usage reports such as query logs and click-through logs. As the analysts identify new queries and new trends, they can create Best Bets for content that is not appearing high enough in search results or for content that is not in the content index.

Software development engineer. This developer migrated customizations from the SharePoint Server 2007 search solution and customizes certain aspects of the end-user search experience on the enterprise Search Center.

UI designer. The designer creates standards for MSW and other sites and is responsible for the visual design of the search experience.

Support engineer. This person responds to technical or operational issues with the search service. For example, a site owner might report that new content is not yet showing in search results as expected, or that content that has been removed from a site is still appearing in search results. In such cases, the support engineer can check whether a crawl has finished since the given content was added or removed. As another example, a site owner might report that search is not working on a particular site, and the support engineer might verify that search is not available and can notify a search service engineer of the problem. The support engineer also responds to end-user issues and feedback, such as questions about query syntax or content availability in search results.

The following table shows how MSIT allocated these resources during the planning and deployment phases of FAST Search Server 2010 for SharePoint, and how the resources are allocated during the operations and support phase. To provide an indication of the kinds of duties of each position, the table includes information about SharePoint permissions and local server permissions.

Improving Enterprise Search at Microsoft Page 12

Page 13: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Table 2. Search Team Member Permissions and Resource Allocations by Phase

Team member position

Permissions for the enterprise search

deployment

Number of full-time resources allocated for each

life-cycle phase

Planning and deployment

Operations and support

Search service manager

Farm administrator

Search service application administrator

1 1

Search service engineer

Farm administrator

Search service application administrator

Member of the local Administrators group on each farm server

Microsoft SQL Server 2008:

db_owner fixed database role

dbcreator fixed server role

1.5 2

Search analyst Site collection administrator for the enterprise Search Center

0 1.5

Software development engineer

Site collection administrator for the enterprise Search Center

.5 1

UI designer Site collection administrator for the enterprise Search Center

.5 1

Support engineer Farm administrator

Search service application administrator

0 1

Enterprise Search Product GroupThe enterprise search product group at Microsoft is responsible for program management, development, and testing of SharePoint Server 2010 and FAST Search Server 2010 for SharePoint features and functionality. MSIT collaborates with this group to exchange information and recommendations regarding the MSIT enterprise search solution.

MSIT deployed FAST Search Server 2010 for SharePoint to production at Microsoft in late 2009, before public release of the product in the second quarter of 2010. MSIT provided feedback to the enterprise search product group at that time about the MSIT experience with product deployment and operations. This included feedback that MSIT received from end

Improving Enterprise Search at Microsoft Page 13

Page 14: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

users. This helped the product group ensure that the software was thoroughly tested in an enterprise production environment before it was delivered to customers.

MSIT continues to provide feedback to the product group about its use of FAST Search Server 2010 for SharePoint. This feedback helps improve the product. The product group in turn provides occasional guidance to MSIT regarding configuration and use of certain features and functionality of the software.

Improving Enterprise Search at Microsoft Page 14

Page 15: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

MIGRATION TO FAST SEARCH SERVER 2010 FOR SHAREPOINTTo migrate from the SharePoint Server 2007 search solution to the FAST Search Server 2010 for SharePoint solution, MSIT performed the following steps:

1. Built a new FAST Search Server 2010 for SharePoint deployment dedicated to search.

2. Migrated SSP-level settings from the Redmond SharePoint Server 2007 search solution to the new FAST Search Server 2010 for SharePoint deployment as follows:

Reproduced content sources from the SharePoint Server 2007 search solution by using an internal script

Migrated managed properties by using SharePoint Search Property Creator, an internal tool

Migrated global scopes by using Windows PowerShell scripts

3. From the FAST Search Server 2010 for SharePoint deployment, completed a full crawl of all content sources. The full crawl took approximately 10 days and indexed about 90 million items.

4. Selected a SharePoint Server 2007 consumer farm (a farm such as http://teamsites that was consuming a SharePoint Server 2007 search service) that had to be upgraded to SharePoint Server 2010.

5. For the SharePoint Server 2007 consumer farm that was selected in step 4, did the following:

A. Ensured that all local search settings (such as Best Bets and local scopes) in the farm were exported to a file.

B. Ensured that the farm was upgraded to SharePoint Server 2010.

6. From the new SharePoint Server 2010 farm (the farm mentioned in step 5), set up a federated connection to the FAST Search Server 2010 for SharePoint deployment (so that the new SharePoint Server 2010 farm was consuming search from the FAST Search Server 2010 for SharePoint deployment.)

7. Ensured that all local search settings that were exported to a file from the SharePoint Server 2007 consumer farm were imported into the new SharePoint Server 2010 consumer farm (by using the SharePoint Enterprise Search Migration Tool for SharePoint Server 2010).

8. Repeated steps 4 through 7 as necessary to connect SharePoint Server 2010 farms to the FAST Search Server 2010 for SharePoint deployment.

At this point, all participating SharePoint sites were consuming search from the FAST Search Server 2010 for SharePoint deployment, rather than from a SharePoint Server 2007 search service. The entire migration process took about 20 weeks—6 weeks for planning and 14 weeks to set up the new deployment, migrate and configure search settings, and deploy the search service.

MSW was one of the SharePoint Server 2007 consumer farms that was upgraded to SharePoint Server 2010 as part of this procedure. After the upgrade, MSIT created a new enterprise Search Center and customized the search experience for that new Search Center.

Improving Enterprise Search at Microsoft Page 15

Page 16: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

FAST SEARCH SERVER 2010 FOR SHAREPOINT SOLUTION:ENVIRONMENT AND TOPOLOGYMSIT operates and maintains the FAST Search Server 2010 for SharePoint deployment as a centralized search solution. This solution provides enterprise search capability for all of the SharePoint Server 2010 sites that MSIT hosts and manages.

Enterprise search is a resource-intensive service, especially when it is deployed on a large scale. Therefore, to provide high capacity and performance, and to expedite system administration and maintenance, the FAST Search Server 2010 for SharePoint deployment is dedicated to search. The deployment does not host any content and does not host any SharePoint services other than the search service.

The FAST Search Server 2010 for SharePoint deployment has the following topology:

A SharePoint Server 2010 farm that contains four servers

A FAST Search Server 2010 for SharePoint farm that contains 30 servers

Two database clusters that contain a total of four servers

Figure 3 shows the FAST Search Server 2010 for SharePoint deployment, and farms and sites that consume the search service from the deployment.

Figure 3. MSIT enterprise search topology in theFAST Search Server 2010 for SharePoint solution

The following subsections describe the topology of the FAST Search Server 2010 for SharePoint deployment in more detail.

Improving Enterprise Search at Microsoft Page 16

Page 17: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

SharePoint Server 2010 Server FarmThe SharePoint Server 2010 farm consists of the following four servers:

Two servers that host crawl components that crawl all content other than user profiles. Each server hosts two search service application (SSA) crawl components for content. The four crawl components can crawl content at the same time.

Two servers that host crawl components that crawl only user profiles. One server is primary; the second server is a backup for high availability. Each server hosts one query SSA crawl component. Only one query SSA crawl component crawls user profiles at a time. In addition, these servers connect to the consumer farms to serve search queries.

FAST Search Server 2010 for SharePoint Server FarmThe associated FAST Search Server 2010 for SharePoint farm consists of the following 30 servers:

Six servers that perform one or more tasks such as the following: distribute crawled items to the index, extract properties, maintain managed properties, perform indexing, and match queries.

24 servers in a configuration of two rows with 12 columns each. The primary task of the servers in the index row is to index content; the primary task of the servers in the search row is to query the index. To provide high availability, if one of the servers in a column becomes unavailable, the other server in the column can perform both the index and the query tasks. The content index is distributed across the 12 columns so that each column has a partition of approximately equal size. Each column or partition has a capacity of 15 million items, so that the total capacity of the content index is 200 million items. (The solution is designed to meet enterprise search requirements at Microsoft until the index grows to 200 million items in about 2013 or later.) For indexing efficiency, each of the 12 index partitions has four sub-partitions.

Database ServersThere are two database clusters associated with the FAST Search Server 2010 for SharePoint deployment. Each cluster contains two database servers. The clusters store the following databases:

Cluster 1:

Content crawl database 1

Content property database

People property database

Content SSA database

Cluster 2:

Content crawl database 2

SharePoint configuration database

People crawl database

FAST administration database

Query SSA database

Improving Enterprise Search at Microsoft Page 17

Page 18: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

MEASURABLE GAINS FROM IMPLEMENTINGFAST SEARCH SERVER 2010 FOR SHAREPOINTThe new search solution immediately provided measurable gains in many areas, such as those shown in tables 3, 4, and 5.

Table 3 shows how the scale of the FAST Search Server 2010 for SharePoint solution is much greater than the scale of the search solution that the SharePoint Server 2007 Redmond SSP provided.

Table 3. Scale Comparison

Characteristic SharePoint Server 2007 solution (approximate,

2009)

FAST Search Server 2010 for SharePoint solution

(approximate, August 2011)

Amount of content crawled on sites hosted by MSIT

13 terabytes 34.5 terabytes

Number of sites and subsites crawled

350,000 683,050

Number of items in content index

43 million 90 million

Size of content index

300 gigabytes (GB) 6 terabytes

Database sizes Redmond SSP search database: 340 GB

Redmond SSP profiles database: 70 GB

Crawl database 1: 427.3 GB

Crawl database 2: 237.1 GB

People crawl database: 17.3 GB

Search administration database: 2.15 GB

Content (FAST connector) property database: 0.19 GB

People (SharePoint connector) property database: 5.42 GB

Query volume per month

Query volume per second

500,000

.1929

5.45 million

2.0333

Table 4 shows the amount of SharePoint content that MSIT hosted for each of the three regions, and that the FAST Search Server 2010 for SharePoint solution crawled, as of August 2011.

Improving Enterprise Search at Microsoft Page 18

Page 19: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Table 4. SharePoint Content Crawled by Region

Region Number of sites and subsites crawled

Corpus size (terabytes)

Americas (content hosted in Redmond data center)

128,435 sites with 333,831 subsites 23.34

EMEA (content hosted in Dublin data center)

35,932 sites with 89,591 subsites 6.6

Asia (content hosted in Singapore data center)

33,360 sites with 61,901 subsites 4.53

Total 197,727 sites with 485,323 subsites 34.47

In addition to the 34.47 terabytes of hosted SharePoint content, the search service crawls a large amount of other content, which includes:

Sites that are not hosted by MSIT.

File shares.

Structured data sources.

Table 5 shows average incremental crawl times for some content sources with the SharePoint Server 2007 search solution compared to average incremental crawl times with the FAST Search Server 2010 for SharePoint solution.

Table 5. Crawl Performance Comparison

Content source Average incremental crawl time with SharePoint

Server 2007 solution (late 2009)

Average incremental crawl time with FAST Search

Server 2010 for SharePoint solution

MSW 21 hours 40 minutes

SharePoint sites,not including My Sites

145 hours 9 hours

My Sites 18 hours 2 hours

Redmond portals 35 hours 4 hours

User profiles 7 hours 10 minutes

Host headers 19 hours 1 hour

Team Deleted in 2009to reduce crawl load

2 hours

Office Deleted in 2009to reduce crawl load

4 hours

Improving Enterprise Search at Microsoft Page 19

Page 20: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

ADMINISTRATION AT THE SEARCH SERVICE APPLICATION LEVELThe following sections describe the management and current configuration of the search service in the FAST Search Server 2010 for SharePoint deployment.

Configuring CrawlsIn addition to setting up content sources for crawls, MSIT uses crawl rules and dedicated crawl targets.

Content SourcesMSIT reviews content sources on a quarterly schedule to ensure that crawls are appropriately configured. Also, MSIT is often asked to add or remove a site from crawls.

To reduce the complexity of scheduling and managing crawls, MSIT consolidates content sources and start addresses based on similarity and priority whenever possible. For example, MSIT decreased the number of content sources from 25 in the SharePoint Server 2007 search solution to 14 in the FAST Search Server 2010 for SharePoint solution.

Table 6 shows the number of each kind of content source in the previous and the current enterprise search solutions. Some of the content sources contain multiple start addresses. For example, a content source that contains sites that MSIT does not host (SharePoint sites such as http://sqlserversites and http://devdivsites, and other websites) has more than 150 start addresses.

Table 6. Comparison of Content Sources

Type of content source Number of content sources

SharePoint Server 2007

search solution

FAST Search Server 2010 for SharePoint solution

SharePoint sites hosted by MSIT (including the user profile store

13 11

Sites not hosted by MSIT (SharePoint sites and other websites)*

6 1

File shares** 1 1

Microsoft Exchange public folders 1 0

Business Data Catalog 3 1

Custom 1 0

*These sites host content that the search service crawls, but the sites do not consume the search service.**The search service currently crawls nine file shares.

Full CrawlsMSIT performs a full crawl of all content sources only when it is necessary, as in the following situations:

After deployment of a crawl-related hotfix or a schema configuration change.

Improving Enterprise Search at Microsoft Page 20

Page 21: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

After addition of managed properties to the search system. In this case, it is only necessary to perform a full crawl of content sources to which the managed properties might apply.

MSIT tries to minimize the impact of full crawls on search performance by crawling content sources in order of priority, so that crawls are performed in a staggered manner.

As of August 2011, the search service was crawling about 90 million items on the Microsoft intranet worldwide. Table 7 shows the number of items that each of the major content sources was crawling at that time.

Table 7. Items Crawled During Full Crawl of Each Content Source

Content source Number of items crawled

Academy portal 72,619

Asia content (includes team sites, My Sites, and portal sites)

9,344,940

EMEA content (includes team sites, My Sites, and portal sites)

15,798,592

File shares 55,407

Infopedia portal 69,967

MSLibrary portal 91,202

MSW portal 292,751

My Sites 6,131,717

Sites not hosted by MSIT (SharePoint sites and other websites)

2,715,466

Office portal 8,717,300

Primus 25,168

Redmond custom portals 5,099,814

SharePoint 29,495,749

Team 6,367,784

Incremental CrawlsMSIT determines the priority of a content source primarily according to the degree of dependence that consumer sites have on the content source for search, and according to how frequently the associated content is updated. MSIT schedules incremental crawls according to four levels of priority of content sources. The content sources that have priorities 1, 2, and 3 are for repositories that MSIT hosts and manages. Table 8 shows how the frequency of incremental crawls depends on the priority of content sources.

Improving Enterprise Search at Microsoft Page 21

Page 22: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Table 8. Incremental Crawl Frequencies

Contentsourcepriority

Crawl frequency

Content sources (examples)

1 Four times a day MSW, Infopedia, user profiles

2 Two times a day Academy, Team, My Sites

3 Once a day SharePoint, MSLibrary

4 Three times a week Sites that MSIT does not manage

Crawl RulesThe current search solution uses about 120 crawl rules. When MSIT builds a new deployment, it uses a custom Windows PowerShell script to create some of these crawl rules. Some of the crawl rules specify how to crawl complex URLs. Other crawl rules specify certain sites or site collections to exclude from crawling because they contain confidential information or information that should not be included in the content index. For example, although the FAST Search Server 2010 for SharePoint deployment can crawl content on the Internet, MSIT uses a crawl rule that excludes the path http://*.*. This rule prevents crawlers from going outside the firewall.

MSIT tries to minimize the number of crawl rules that it uses, and it encourages site owners to set permissions on content locally to include or exclude content from crawls.

Dedicated Crawl TargetsMSIT uses at least one web server as a dedicated crawl target in each farm that it crawls. MSIT uses two web servers as dedicated crawl targets for each farm that hosts a large amount of content (relative to the amount of content in other farms) and for each farm that hosts content that must be crawled frequently due to requirements for freshness of search results.

Crawler Impact RulesMSIT used crawler impact rules at one point as a temporary solution to reduce the load on two sites that had performance issues. The issues were not related to crawler impact and were resolved. Currently, MSIT is not using any crawler impact rules.

Configuring Other Search SettingsMSIT configures other search settings such as global scopes and removal of items from the content index, and it manages other global operations such as backups.

Global ScopesTable 9 shows the global scopes that the FAST Search Server 2010 for SharePoint deployment provides. Site owners can use these scopes on their sites.

Improving Enterprise Search at Microsoft Page 22

Page 23: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Table 9. Global Scopes

This global scope Provides results from this content

Intranet All SharePoint sites (preconfigured scope)

MSLibrary http://mslibrary

People User profiles and My Sites (preconfigured scope)

EnterpriseMedia All SharePoint content that has content type Video. (This scope is used by the Video tab in the enterprise Search Center.)

News MSW

Webcasts MSW

MSArchives MSW

ImageOnly MSW

Removal of Items from the Content IndexIn rare cases, MSIT must immediately remove items from the content index due to business or legal requirements. To do this, MSIT uses Windows PowerShell. MSIT then creates a crawl rule to exclude the content from future crawls.

Time-out Settings, IFilters, and File TypesMSIT uses the default time-out settings for connection time and request-acknowledgement time. In addition, MSIT uses only the default IFilters and the default file-type inclusions and exclusions list.

Thesaurus Files and Stop Word FilesMSIT uses the default configuration for thesaurus files and stop word files. During product development, MSIT provided input to the product team about the default configuration for these files, and helped validate the usefulness of the default settings before the software was released.

BackupsMSIT performs weekly backups of the FAST Search Server 2010 for SharePoint configuration. This includes the administration database, which contains settings such as managed properties, Best Bets, and global scopes.

As part of regular SQL Server backups, MSIT also backs up the SharePoint configuration database and the other SQL Server databases that are part of the FAST Search Server 2010 for SharePoint deployment.

Monitoring SearchMSIT uses various tools to monitor the health and use of the search system.

Improving Enterprise Search at Microsoft Page 23

Page 24: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Search Health MonitoringTo monitor the health of the search system, MSIT uses the following FAST Search Server 2010 for SharePoint tools:

Crawl logs. MSIT reviews the crawl logs every day to find errors and troubleshoot issues. For example, site owners or end users sometimes report that content is missing from search results. By reviewing the crawl logs, MSIT can determine the time of the last successful crawl of each content source, and can determine whether crawled content was successfully added to the index, whether it was excluded because of a crawl rule, or whether indexing failed because of an error. In the case of a complex issue, MSIT sometimes works with the enterprise search product group, such as when an item is in the content index but does not appear as it should in search results.

Search administration reports. MSIT reviews search administration reports for the following details about crawl and query performance:

Crawl rate per content source

Crawl rate per type of crawl transaction

Overall query latency

SharePoint back-end query latency

Crawl processing per activity in the crawl pipeline

Crawl processing per component in the crawl pipeline

Crawl queue

Query latency trend

High query latency might indicate delays in server rendering, for example, or in query matching or other back-end processing.

MSIT also uses the following tools to monitor the search system:

Microsoft System Center Operations Manager 2007 R2 with the FAST Search Server 2010 for SharePoint management pack. MSIT uses System Center Operations Manager to monitor the status of each server and service in the FAST Search Server 2010 for SharePoint deployment. For example, System Center Operations Manager obtains monitoring data and performance counters from the FAST Search Server 2010 for SharePoint Monitoring Service, which runs on each server in the farm. System Center Operations Manager helps identify issues and can provide alerts when performance bottlenecks occur.

Custom SQL Server tool for monitoring crawls. MSIT developed a custom tool that helps monitor crawl progress. The tool runs a SQL Server 2008 job on each computer that hosts a crawl database. This helps monitor crawl latencies and can identify when a crawl hangs. The tool writes an entry with a time stamp to a SQL Server table for each item that is crawled. The SQL Server job sends an alert if more than a 30-minute interval appears between successive time stamps.

Windows Server 2008 R2 performance counters. MSIT uses Windows Server 2008 R2 performance counters to monitor the following:

Crawl and query progress

Disk, memory, and application performance

Improving Enterprise Search at Microsoft Page 24

Page 25: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Search Usage MonitoringTo understand the effectiveness of the enterprise search service and provide the best experience for users, MSIT regularly analyzes use of the service. MSIT uses the results of this analysis to make adjustments to improve search results.

To monitor search usage, MSIT uses the following FAST Search Server 2010 for SharePoint tools:

Click-through logs. These logs provide information about click-through rates on search results, which indicate how users browse through results.

Query logs. Query logs provide the following information about search queries that users submit:

Queries submitted during the previous 30 days

Queries submitted most frequently during the previous 30 days

Queries submitted most frequently from each site collection during the previous 30 days

Queries submitted per scope during the previous 30 days

Queries submitted during the previous 12 months

Queries that returned zero results

Destination pages reached most frequently from search results

This information helps MSIT configure the search service to provide more useful search results, such as by adjusting Best Bets.

MSIT constantly strives to improve the search experience for employees, and it uses the following resources to understand the user experience:

Feedback Tool for search. The Feedback Tool is a Microsoft-internal tool that was developed with the help of the enterprise search product team. This tool provides a Feedback button on the enterprise Search Center results page that invites the search user to explain the goal of the most recent search, rate the search experience and usefulness of the results, report search problems, and suggest improvements. When the user clicks Send, the tool sends an email message to an alias that monitors the feedback. The message captures the user's query, the URL where the user submitted the query (which indicates the scope that was used), and the search results, with an indication of which results were Best Bets, if any. If information that a user was looking for did not appear in the search results, MSIT works with site owners to determine where the associated content is available, so that links to the content can be returned in search results. If the information that a user is looking for is in the search results but the ranking is not appropriate, MSIT can adjust the Best Bets or use URL promotion or demotion accordingly.

Annual user survey regarding the MSW enterprise portal. This survey includes several questions to help measure user satisfaction with the enterprise search service. Users can provide feedback on the effectiveness of the search service and on what kind of content has been difficult to find.

MSIT also uses a third-party tool that monitors and tracks website usage. The tool collects and reports data on the number of visits to MSW and the queries that were performed there.

Improving Enterprise Search at Microsoft Page 25

Page 26: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

COLLABORATION WITH SITE OWNERSUpon request, MSIT collaborates with site owners of sites that consume the search service to help them customize the search experience and ensure that content on their sites is discoverable. For example, in response to requests from site owners, MSIT creates managed properties and helps site owners configure relevance.

Creating Managed PropertiesThe FAST Search Server 2010 for SharePoint deployment includes more than 600 managed properties. For the deployment, MSIT started with the default managed properties and the managed properties that MSIT migrated from the SharePoint Server 2007 search solution by using the internal SharePoint Search Property Creator tool.

Addition of new managed properties is one of the most common change requests that MSIT receives, and therefore MSIT has a governance policy to keep the process orderly. In accordance with the policy, MSIT tracks requests for new managed properties, requires requests to be in a certain format, automates the creation of managed properties to help ensure that the properties have certain attributes, and creates the managed properties on a regular schedule.

To request a new managed property, a site owner fills out a template that MSIT provides. When site owners use the template, it encourages them to plan thoroughly before submitting a request. This helps MSIT avoid having managed properties with different names that serve the same purpose, for example. The request template is a comma-separated value (CSV) file that includes the following attributes of the proposed new managed property: Name, Description, Type (text, decimal, or date/time), StemmingEnabled, SummaryType, MergeCrawledProperties, SortableType, Queryable (whether the property can be used in queries), Refinement (whether the property can be used as a refiner), Crawled Properties, and IsExisting. If MSIT approves the request, it uses a Windows PowerShell script that it created that takes the CSV file as input, creates the managed property with the appropriate attribute values, and maps the managed property to one or more crawled properties.

In general, MSIT adds new managed properties only once a month, because adding a new managed property requires a full crawl of all content sources to which the managed property might apply.

Configuring RelevanceFor most intranet sites that it hosts, such as the enterprise Search Center, MSIT uses the default rank profile and the default content index. Following are examples of the way MSIT works with site owners to adjust search relevance by providing custom rank profiles and custom full-text indexes.

Rank ProfilesIn response to a request from the Infopedia site owner to rank results differently than the default rank profile, MSIT provided a custom rank profile for the site. The new profile caused topic pages to be displayed first, and then blogs, documents, and document sets. The topic pages (usually .aspx pages) are articles with named owners to which other users can contribute notes or attach other documents.

To implement a new rank profile, MSIT created a script that copied the default rank profile into a new rank profile, added new ranking attributes and settings, and updated weights of

Improving Enterprise Search at Microsoft Page 26

Page 27: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

various criteria. MSIT then ran the script in a pre-production environment that was crawling production content. The site owner created test cases to determine how the new rank profile affected the search results. MSIT and the site owner compared the search results visually, because there were no tools to automate the comparison. The site owner provided feedback to MSIT regarding the results, and MSIT updated the script based on the feedback. MSIT then ran the script again. MSIT and the site owner repeated this procedure until they achieved the desired search results. At that point, MSIT ran the script in the production deployment. Although the process was iterative and time-consuming, in the end it yielded better search results for end users.

Rank profiles are global to a search service application, so any site in the web application can use any custom rank profile. After MSIT created the custom rank profile, the Infopedia site owner added it to the sort drop-down list for the site's search results page. The name of the custom rank profile then appeared among the managed properties, and the site owner was able to set it as the default sort option for that search results page.

Custom Full-Text IndexesAnother requirement from certain site owners was to provide keyword-based search, that is, to provide a search experience based on metadata associated with items in the content index. MSIT satisfied this requirement by using a Windows PowerShell script that it created to provide custom full-text indexes.

Each custom full-text index takes advantage of content classification through the Managed Metadata service to provide more relevant search results by targeting keyword searches to a limited set of properties. For example, for a full-text search (the default) on the term "FAST," a search returned several thousand results. However, for the same term, a tag-based search (which uses a custom full-text index) returned only slightly more than 100 results—only items that were tagged with the term "FAST." In either case, the user was able to narrow the results by using custom refiners (such as Industries, Partners, and Products) that MSIT created.

Improving Enterprise Search at Microsoft Page 27

Page 28: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

ENTERPRISE SEARCH CENTERAfter the migration to FAST Search Server 2010 for SharePoint, MSIT created a new enterprise Search Center that used the FAST Search Center template. The enterprise Search Center in FAST Search Server 2010 for SharePoint includes many default features that were not available by default in SharePoint Server 2007, such as search refiners, document previews for Microsoft PowerPoint presentations, and a Web part for related searches.

For the new Search Center, MSIT provided customizations to match the visual design of the MSW enterprise portal, and provided customizations for the search tabs and search results page. MSIT uses the default full-text index and the default rank profile for search results on the enterprise Search Center, but adjusts relevance by using Best Bets and URL promotion.

MSIT also upgraded custom search features from the SharePoint Server 2007 search solution, such as the Glossary Web part, so that the features worked with FAST Search Server 2010 for SharePoint. To reduce page rendering and refresh times, MSIT consolidated multiple cascading style sheets (CSS) files into one CSS file, and configured the Glossary Web part to render and refresh asynchronously with respect to the rest of the search results page.

Getting to the Enterprise Search CenterEmployees often start their searches for information on the MSW enterprise portal, which is one of the most heavily used intranet sites at Microsoft. The portal is shown in Figure 4.

Figure 4. MSW enterprise portal

When a user types a query in the search box on MSW, the search service displays the results on the enterprise Search Center results page. Employees can also get to the enterprise Search Center by typing a search query in a search box on certain other intranet

Improving Enterprise Search at Microsoft Page 28

Page 29: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

sites. Employees can even get to the enterprise Search Center from Windows Explorer on a computer that is running Windows 7 or Windows Server 2008.

Enterprise Search Center TabsThe FAST Search Center template includes two search tabs: All Sites and People. MSIT modified this configuration so that the enterprise Search Center contains three tabs: Intranet, People, and Video.

Intranet TabMSIT renamed the default All Sites tab to Intranet to help clarify the purpose of the tab. The Intranet tab uses the default All SharePoint Sites search scope. When employees conduct a general search, they often use this tab because it returns the most comprehensive search results. However, the Intranet tab does not include people search results or results from structured data sources.

The Intranet tab includes the following preconfigured refiners when applicable: result type, site, author, modified date, and company. (In FAST Search Server 2010 for SharePoint, company is a preconfigured managed property for property extraction.) Web parts in the right zone include Related Searches, Glossary, People Matches, and MSDN/TechNet Results. Related Searches, People Matches, and MSDN/TechNet Results are default Web parts. The MSDN/TechNet Results Web part displays links to federated results from MSDN or TechNet and a View More Results link that starts the Bing search engine.

The Glossary custom Web part draws on a corporate glossary that MSIT created by using the Managed Metadata service. When a user types a term in the search box on the Intranet tab, the Glossary Web part accesses the corporate glossary by sending a query to the Managed Metadata service via the Managed Metadata web service. The Glossary Web part then displays a list of terms and definitions that the web service returns. For example, if a user searches on "DOS", the Web part displays definitions for "denial of service attacks" and "disk operating system."

Figure 5 shows a portion of the enterprise Search Center results page after a user searches on "windows phone 7" on the Intranet tab. The first search result is a Visual Best Bet graphic that MSIT configured as a hyperlink.

Figure 5. Search results page for the Intranet tab

Improving Enterprise Search at Microsoft Page 29

Page 30: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

People TabPeople is a default tab. A search on the People tab provides search results based on information that is in employee user profiles and My Sites. (MSIT imported user profiles from Active Directory Domain Services, and SharePoint Server uses the information for the creation of My Sites.) The People tab includes the following refiners where applicable: job title, organization, ask me about, interests, skills, past projects, and schools. The job title and organization refiners use managed properties that MSIT created based on fields in the user profiles.

Video TabMSIT added the Video tab in response to popular demand. For the Video tab, MSIT created a "video" content type. The Video tab uses a global scope that displays only results that have that content type. (A search from the Video tab does not extract data from audio or video.)

The Video tab includes custom refiners in the left zone based on metadata for Channel, Published Date, and Tags. Channel refiners enable users to sort results by subject matter, and Tags refiners enable users to sort results according to metadata tags that site owners have applied to the items. Other metadata such as title, description, and length are displayed in the search results under a thumbnail of a frame of the associated video. To customize the search experience for the Video tab so that it is different from the Intranet tab and the People tab, MSIT used Extensible Stylesheet Language Transformations (XSLT) and CSS.

The Video tab search results page can display results in a grid view or a list view. Figure 6 shows a list view of search results on the Video tab.

Figure 6. Search results page for the Video tab (list view)

Improving Enterprise Search at Microsoft Page 30

Page 31: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Best Bets and URL PromotionFor Best Bets for the enterprise Search Center, MSIT selects keywords and synonyms from the Microsoft corporate vocabulary that MSIT created by using the Managed Metadata service. Addition of new Best Bets is another one of the most common change requests that MSIT receives from site owners and end users. In addition, MSIT search analysts regularly review user feedback, query logs, and click-through logs to ensure that Best Bets are available, up-to-date, and relevant. Based on their reviews, the search analysts can create new Best Bets as appropriate.

In the MSIT SharePoint Server 2007 search solution, some keywords had as many as 10 Best Bets. However, in general, the fewer the Best Bets, the more authoritative and effective they are. Therefore, for the enterprise Search Center in the FAST Search Server 2010 for SharePoint solution, MSIT tries to maintain fewer than five Best Bets per keyword and uses the URL promotion feature to boost the rank of any additional results that are highly relevant. With the URL promotion feature, the appearance of a particular keyword in a search query causes a specified site or document to rank higher in search results, regardless of other factors.

MSIT provides Visual Best Bets for the most popular search results of the most common queries, and for search results regarding some Microsoft events that have worldwide participation. The Visual Best Bets are graphical hyperlinks.

Improving Enterprise Search at Microsoft Page 31

Page 32: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

LESSONS LEARNED AND MSIT BEST PRACTICESIn comparison to a SharePoint Server 2007 search solution, MSIT found that it takes additional hardware, skills, and people to plan, deploy, operate, and support a large deployment such as the MSIT search solution that uses FAST Search Server 2010 for SharePoint. Accordingly, MSIT followed these best practices to avoid issues and keep the environment available and performing well:

Plan and design the search architecture carefully. MSIT considered many factors when planning its FAST Search Server 2010 for SharePoint solution. This included the location and amount of content to be crawled, the number of users, the anticipated query load, user requirements for results freshness, requirements for search availability, and the expected growth in the amount of content and number of user queries in connection with the anticipated longevity of the farm. Understanding these factors helped MSIT determine the number of physical and logical components that the new farm required. Certain aspects of a FAST Search Server 2010 for SharePoint farm cannot be easily changed after the initial deployment, so it was important to plan carefully to avoid having to deploy the farm again if the initial design was not viable or optimal.

Plan content types and managed properties. MSIT had to rapidly deploy FAST Search Server 2010 for SharePoint to production at Microsoft due to a requirement to test the software before it was released to the public. Therefore, MSIT did not have time to plan all content types and managed properties before it deployed the search service, and it had to create some content types and managed properties after the deployment was in operation. In some cases, site owners then had to adjust search refiners or rank profiles to accommodate changes in content types and managed properties. In addition, end users had to make adjustments in the way they used search. For example, when a managed property changed, a user might not be able to use it in searches anymore, or might get different results by using it. This experience confirmed to MSIT that it is best to plan and create content types and managed properties before an organization deploys a search solution. This enables site owners to create and configure more permanent search refiners and rank profiles at the beginning of a deployment, and it results in a more consistent search experience for users.

Automate search configuration changes. MSIT automates search configuration changes whenever possible by using Windows PowerShell. Automation of changes, such as addition of managed properties to the system, saves time and provides uniformity in the change process. This helps MSIT ensure that it makes changes in similar ways each time.

Provide governance policies for search configuration changes. MSIT receives many requests for changes to the search configuration, such as requests for new managed properties. Without governance policies, change processes in such a large deployment could quickly become unmanageable. Therefore, MSIT enforces governance policies for certain configuration changes, which provides order and helps in change tracking.

Integrate with other SharePoint services. To maximize the benefits of using FAST Search Server 2010 for SharePoint, MSIT invested resources in configuring and using other SharePoint services on which search has dependencies. For example, the search service has the following dependencies:

User Profile service and My Sites for results for people searches

Improving Enterprise Search at Microsoft Page 32

Page 33: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

Managed Metadata service for managed properties that can be used for refiners, ranked results, custom full-text indexes, and property queries

Business Connectivity Services for the operation of connectors to crawl repositories of structured data

Investing in these other areas helped MSIT create richer search experiences and improve information discovery.

Gather user feedback. User feedback has been invaluable to MSIT for understanding user problems and improving the enterprise search system.

Improving Enterprise Search at Microsoft Page 33

Page 34: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

CONCLUSIONWhen the enterprise search solution that used SharePoint Server 2007 was no longer viable, MSIT reviewed business, technical, and user requirements for an improved solution and chose to migrate to FAST Search Server 2010 for SharePoint. The new deployment met all of the requirements and continues to provide the following benefits:

Centralized management. The search solution is hosted in a single, central deployment.

Content index completeness. All users can get search results from a single content index.

Huge index capacity. The content index holds all content items from the crawls of all repositories for which MSIT wants to provide search coverage. The content index can accommodate projected growth to 200 million items.

Faster crawls for fresher search results. An incremental crawl of each high-priority content source takes much less than 24 hours.

Faster query responses. For most queries, users get search results in less than two seconds.

More customizable search experience for more relevant search results. Search results are ranked according to relevance, so that users can quickly find information that they need.

High availability. FAST Search Server 2010 for SharePoint redundancy and failover configuration provides continuous search availability.

Uniform search experience. Through the enterprise Search Center, all users worldwide have access to the same search experience.

Significantly increased user satisfaction. As measured by the annual survey on MSW, user satisfaction with the performance and effectiveness of the search system and with the overall search experience increased 20 percent in 2010 and 26 percent in 2011.

The FAST Search Server 2010 for SharePoint deployment provides a robust enterprise search solution that empowers employees to find up-to-date, relevant information that is essential for performing their day-to-day job functions. The solution helps connect employees with the work of colleagues all over the world, enabling them to collaborate more easily and work more productively than ever before.

Improving Enterprise Search at Microsoft Page 34

Page 35: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

FOR MORE INFORMATIONFor more information about FAST Search Server 2010 for SharePoint, see the following resources:

FAST Search Server 2010 for SharePoint at http://office.microsoft.com/en-us/fast-search-server-help/

"FAST Search Server 2010 for SharePoint Evaluation Guide" at http://office.microsoft.com/en-us/fast-search-server-help/redir/XT101861353.aspx?queryid=6c8d8c66%2Dfe8d%2D4731%2D9f03%2D24aabef5c3de&respos=0&CTT=1

"Plan site permissions (SharePoint Server 2010)" at http://technet.microsoft.com/en-us/library/cc262778.aspx

"SharePoint Enterprise Search Migration Tool for SharePoint Server 2010" at http://msdn.microsoft.com/en-us/library/ff828776.aspx

"Connect to a service application on a remote farm (SharePoint Server 2010)" at http://technet.microsoft.com/en-us/library/ee704558.aspx

"Manage crawl load (SharePoint Server 2010)" at http://technet.microsoft.com/en-us/library/dd335962.aspx

"Manage crawler impact rules (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/ff381257(office.14).aspx

"Remove URLs from search results (SharePoint Server 2010)" at http://technet.microsoft.com/en-us/library/ff621095.aspx

"Configure search time-out settings (SharePoint Server 2010)" at http://technet.microsoft.com/en-us/library/ee808862.aspx

"IFilter and file type reference (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/gg471168.aspx

"Manage thesaurus files (SharePoint Server 2010)" at http://technet.microsoft.com/en-us/library/dd361734.aspx

"Manage stop word files (SharePoint Server 2010)" at http://technet.microsoft.com/en-us/library/dd361733.aspx

"Configuration backup and restore (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/ff460224.aspx

"Best practices for using crawl logs (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/ff756029.aspx

"Use search administration reports (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/ff756028.aspx

"View search administration reports (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/ff756031.aspx

"Monitor FAST Search Server 2010 for SharePoint with SCOM" at http://technet.microsoft.com/en-us/library/ff383319.aspx

"Manage rank profiles (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/ff453904.aspx

"Dynamic rank (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/gg982948.aspx

Improving Enterprise Search at Microsoft Page 35

Page 36: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

"Manage full-text indexes by using Windows PowerShell (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/ff191230.aspx

"Search the enterprise from Windows 7" at http://technet.microsoft.com/en-us/library/ff899315.aspx

"Add, remove and display Visual Best Bets for a keyword by using Windows PowerShell (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/ff191236.aspx

"Tune keyword rank (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/ff453900.aspx

"Promoting or demoting sites" at http://office.microsoft.com/en-us/fast-search-server-help/promoting-or-demoting-sites-HA010381927.aspx

"Manage FAST Search keywords" at http://office.microsoft.com/en-us/fast-search-server-help/manage-fast-search-keywords-HA010381994.aspx

"Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/ff599530.aspx

"Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint)" at http://technet.microsoft.com/en-us/library/ff381247.aspx

"User Profile service application overview (SharePoint Server 2010)" at http://technet.microsoft.com/en-us/library/ee662538.aspx

Managed Metadata and Taxonomy in SharePoint Server 2010 at http://technet.microsoft.com/en-us/sharepoint/ff924923.aspx

Business Connectivity Services in SharePoint Server 2010 at http://technet.microsoft.com/en-us/sharepoint/ee518675

For more information about Microsoft products or services, call the Microsoft Sales Information Center at (800) 426-9400. In Canada, call the Microsoft Canada Order Centre at (800) 933-4750. Outside the 50 United States and Canada, please contact your local Microsoft subsidiary. To access information via the World Wide Web, go to:

http://www.microsoft.com

http://www.microsoft.com/technet/itshowcase

This document may be updated and changed substantially following initial publication.

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places, and events depicted herein are fictitious, and no association with any real company,

Improving Enterprise Search at Microsoft Page 36

Page 37: portal.just.roportal.just.ro/Test Doclib/1663FASTSearchAtMSTWP.docx  · Web viewHowever, many other groups and individuals in the company create and host their own sites. MSIT does

organization, product, domain name, e-mail address, logo, person, place, or event is intended or should be inferred.

2012 Microsoft Corporation. All rights reserved.

Microsoft, Active Directory, Bing, FAST, MSDN, PowerPoint, SharePoint, SQL Server, Windows, Windows PowerShell, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.

All other trademarks are property of their respective owners.

Improving Enterprise Search at Microsoft Page 37