london amazon cloudsearch meetup jon handler

© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

London Amazon CloudSearch Meetup

Jon Handler


AgendaCloudSearch technical overview (Jon Handler, Amazon CloudSearch Solution Architect)

NakedWines and CloudSearch (Matt Reid, Developer at NakedWines)

Searching Wikipedia with Amazon CloudSearch (Iain Fletcher, Search Technologies)

Building UI with CloudSearch (Stefan Olafsson, Co-Founder, Twigkit)


What is SearchShoes


Do You Want Search With That?

Build your own – database, home-rolled, site search

Open source

Legacy enterprise search


Search Challenges

Complex, expertise required

Costly, often with up-front expenditure

Long time to market, innovation and experimentation are slowed

Operational overhead is undifferentiated work


Amazon CloudSearch

Pay for infrastructure you need when you need itLow costNo need to guess capacityExperiment fast with low riskWe do the undifferentiated heavy liftingGo global in minutes


Amazon CloudSearch ArchitectureDNS / Load Balancing

Search API Console

SEARCH SERVICE

DocSvc API

CommandLine Tools

Console

DOCUMENT SERVICE

AWS Query

ConfigAPI

CommandLine Tools

Console

CONFIG SERVICE

Search Domain


Automatic Scaling

SEARCH INSTANCEIndex Partition n

Copy 1

SEARCH INSTANCEIndex Partition 2

Copy 2


Copy 2


Copy n

SEARCH INSTANCE

DATA Document Quantity and Size

TRAFFICSearch Request Volume and Complexity

Index Partition nCopy n


Copy 1


Copy 1


Copy 2


Copy n



Copy 1


Copy 2


Copy 2


Copy n


Copy n


Copy 1


Copy 1


Copy 2


Copy n

ComputeStorage

Load BalancingSecurity


Text Search


Highly Relevant Results


Faceted Drilldown


Integer Range Searching


Complex Queries


Query

564

726

123

Ranking

564

726

123

SortingFilteringMatching

Search Query Processing


Reference Architecture


Create An Amazon CloudSearch Domain


Text fields for matching user terms

Result enabled to retrieve source data


Literal fields for Faceting

Facet enabled to retrieve facets

Search enabled for narrowing


Integer fields for ranking, narrowing


Configure the Domain


Data Preparation and Upload

Search Documents

ExtractSDF Batch

Amazon CloudSearch

POST


CloudSearch SDF[{"type":"add", "id": "b007oznzg0", "version": 1, "lang": "en", "fields": { "title":"Kindle Paperwhite", "description":"World's most advanced e-reader", "category": ["Electronics","eBook Readers"], "price":11900} }, ...]


Document Service API

http(s)://< document service endpoint >/2011-02-01/documents/batchAccept: application/json Content-Length: 1176 Content-Type: application/json Host: doc.imdb-movies-rr2f34ofg56xneuemujamut52i.us-east-1.cloudsearch.amazonaws.com

[{"type": "add","id":"b007oznzg0","version": 1,"lang": "en","fields": {"title":"Kindle Paperwhite","description":"World's most advanced e-reader","category":["Electronics","eBook Readers"],"price":11900} },{ "type": "delete", "id": "tt0434409", "version": 1337648735 } ]


Search Service APIhttp(s)://< search service endpoint>/2011-02-01/search?

Simple searches• q= text

Boolean combination of fields• bq= (or field:'value1' (and field:'value2' field:'value3'))

Faceting• facet= comma separated list of facet fields

Pagination• start=, size=

Customized ranking• rank= sort results based on the rank expression provided


Search Results{"rank": "-text_relevance","match-expr": "(label 'kindle paperwhite')","hits": { "found": 204, "start": 0, "hit": [ { "id": "sontsst12cf5f88b42" }, { "id": "sopvopr12ab017f082" }, { "id": "sorzrpw12ac468a13b" }, ] },...}


Customizing Ranking

Rank expressions• Compute a score for each document• &rank=<function>

E.g. recency based


Customizing Ranking With Queries

Define rank expressions in your query• &rank-recency=text_relevance + (1 / (2012 - year)) * 100• &rank=-recency

Uses• A/B testing• User-customized searches• Geo-searching


IMDB DATA DEMO


Pricing

Get started for just $2.40/day; $75/month

AWS Calculator http://calculator.s3.amazonaws.com/calc5.html

Free Trial


Wrap Up

Powerful search is a critical component of today's applications

Amazon CloudSearch makes adding search easy

Create a domain, POST documents, GET search results


Resources and Q&AAmazon CloudSearch Overview Pagehttp://aws.amazon.com/cloudsearch/• FAQs• Community Forum• Documentation & Getting Started Tutorial (IMDb)

Contact our EU business development team• http://aws.amazon.com/contact-us

http://aws.amazon.com/cloudsearch/

http://aws.amazon.com/cloudsearch/

http://aws.amazon.com/contact-us


Thank You

Jon Handler / [email protected]


Searching Wikipedia with Amazon CloudSearch

Iain [email protected]

mailto:[email protected]


Search Engine ExpertiseMicrosoft SharePoint/FASTGoogle Search ApplianceSolrAmazon CloudSearch LucidWorksAttivioExaleadAutonomyMarkLogicelasticsearchVivisimoSinequaHadoopSphinx…..

37


400+ Customers

http://www.cisco.com/en/US/hmpgs/index.html

http://www.lenovo.com/

http://www.chick-fil-a.com/Home.asp

http://commons.wikimedia.org/wiki/Image:US-FederalTradeCommission-Seal.svg

http://www.petco.com/

http://www.teleflora.com/


Searching Wikipedia with Amazon CloudSearch




Agenda

Project BackgroundHigh-level ArchitectureSummary & Observations


Project Background

Amazon contracted with Search Technologies to help with beta-testing, prior to the launch of Amazon CloudSearchDecision to use Wikipedia as a convenient data set for testing purposes

41


High-level Architecture42


Indexing

Wikipedia provides content in a series of large xml filesAmazon CloudSearch ingests xml in a specified formVarious content processing tasks to perform• Splitting into individual documents• Date normalization• Metadata extraction & mapping• Cleanup, etc.

We used Aspire for these tasks

43


Aspire in BriefBased on Apache Felix / OSGi• Thread-safe, multi-threaded, distributable• Any number of pipelines, conditional branching• Plug-in components individually testable & upgradable• In use with FAST ESP, FS4SP, Solr, Amazon CloudSearch, GSA.• Tested with Elasticsearch and SP 2013

44


XML Input45


Indexing

Streaming Wikipedia Dump Files directly into CloudSearch500 docs/second achieved without much effort• Using 4 x XL instances of CloudSearch• 1 x XL EC2 instance for Aspire

46


Searching

Amazon CloudSearch provides a RESTful/XML interface for search purposesFor the Wikipedia project, we needed a UI• Chose to use Twigkit• Wrote a Java API for CloudSearch • The Java API is freely downloadable (with source) at http://

www.searchtechnologies.com/java-api-amazon-cloudsearch.html

47

http://www.searchtechnologies.com/java-api-amazon-cloudsearch.html




SearchingSupports navigators and relevancy customization• E.g. a “PageRank” style link analysis

was performed

Limits set high: E.g. retrieve 500,000 results in a single list, delivered in just a few seconds• Hugely useful for analysis applications

So, what does it look like?

48


wikipedia.searchtechnologies.com 49

http://wikipedia.searchtechnologies.com/



wikipedia.searchtechnologies.com50



Summary & Observations

A capable and scalable “raw” engine• xml in, RESTful/xml out• Easy to set up – much the same as an EC2 instance• Elastic scalability

51


Summary & Observations

Cost effective• From $75 per month, including management /

maintenanceExtremely convenient• Switch on / off at leisure• Promotes experimentation & agility

52



For further details, see Paul Nelson’s blog at www.searchtechnologies.com


http://www.searchtechnologies.com/

london amazon cloudsearch meetup jon handler

Documents

search challengescomplex

search stuff

search engine

express consent of

isiama powerful search

amazon cloudsearchpay

information need

cloudsearch matt reid