how to tune search requests using amazon cloudsearch

© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

How to Tune Search Requests

Tom Hill / [email protected]


Agenda

!   Query Processing !   Common Issues !   Benchmarking !   Analytics


…query processing


Query

564

726

123

Ranking

564

726

123

Sorting Filtering Matching

Query Processing


Query Processing !   Matching

•  Text Fields •  Literal Fields

!   Filtering •  Numeric terms, ranges

!   Ranking •  Score computation for each document

!   Sorting •  Based on rank computation, alphabetic, numeric

!   Faceting


Matching !   The starting point for results !   Full text matching with “text” fields !   Exact matching with “literal” fields


Filtering !   UINT fields

•  Numbers •  Numeric ranges

!   After all of these you get results: <hits found="2432" start="0">


Ranking !   Score can be

•  Text Relevance •  Rank expression

!   Done for every document that makes it past filtering


Sorting !   Last step !   Again, cost a function of match set size !   Sort by

•  Text Relevance •  Rank Expression •  Field Value


Performance: Match Set Size

AND color:black AND age:0..6 cat|dog <all documents>

Increasing Performance


…common issues


Literals vs Uints

!   Literals will tend to improve performance ! Uints will tend to take less space


Query Restriction: Literals Vs. Uints

GeoMethod TextRel Limits Queries Seconds QTimeMS Threads CompletedQ AveHits NONE false 10 6.2255 622 1 10 8345450.00 CARTESIAN false 10 15.6064 1560 1 10 8345450.00 EQUI false 10 19.7106 1971 1 10 8345450.00 COSINES false 10 27.4968 2749 1 10 8345450.00 HAVERSINE false 10 31.2595 3125 1 10 8345450.00 NONE false Numeric 10 9.1758 917 1 10 3807.00 CARTESIAN false Numeric 10 9.0255 902 1 10 3807.00 EQUI false Numeric 10 9.1158 911 1 10 3807.00 COSINES false Numeric 10 9.8321 983 1 10 3807.00 HAVERSINE false Numeric 10 9.1272 912 1 10 3807.00 NONE false literal 10 0.8254 82 1 10 3781.00 CARTESIAN false literal 10 0.5936 59 1 10 3781.00 EQUI false literal 10 0.6173 61 1 10 3781.00 COSINES false literal 10 0.5916 59 1 10 3781.00 HAVERSINE false literal 10 0.6289 62 1 10 3781.00

No Restriction

UINT Restriction

Literal Restriction


Negative Queries

!   CloudSearch supports negative queries •  &q=-amazon

!   Can match all documents •  if "doesntmatchanything" matches 0 docs, then

-doesntmatchanything will match all docs.

!   Matching all docs means lots of computations •  Sorting, rank expressions, etc.


Implicit Limits

!   If the user doesn't give you restrictions, add some!

!   Top items for their category !   Add implicit limits to broad queries

•  &bq=important:1 •  &bq=population:10000..

!   select the best


Wildcard Queries

!   Expands the query terms •  a* => aardvark, aaron, … azimuth •  Limited to first 2000 terms.

•  But that's still 2000 terms! •  Match set is all docs that contain any one of the terms

!   Match set gets big!

a* a aaron after apple

doc1 doc3

doc99 doc3

doc9 doc50 doc110 doc5

doc17 doc87 doc117 doc18

doc80 doc111

doc111

doc85 doc90


Wildcard Queries

!   Stemming •  cats* doesn't match cats •  cats is stemmed to "cat", but wildcards are not stemmed

!   Avoid negative wildcard queries •  -cat works fine

•  but may take a while to execute •  -cat123 may confuse you. It becomes:

•  -cat 123


Boolean Queries

!   Used to restrict your searches •  Faceting (e.g. category) •  Date •  Security

!   &bq=(and title:'potter' author:'rowling') !   Can improve performance !   Can slow performance


Boolean Queries

!   Effects of additional AND !   Effects of Additional OR

Color

Style Size

Color

Style Size

Color

Size Style


Rank Expressions Review

!   Used to enhance search results !   Include non-text factors in scoring

•  Popularity (likes/upvotes) •  Distance •  Price/Profit

&rank-‐pop=((0.3*popularity)/10.0)+(0.7*text_relevance)&rank=pop


Rank Expressions

!   Execute once for each document in match set •  Reduce your match set with text matches, literals, uints •  Done after filters applied

Faster! Slower


Rank Expressions

!   Avoid queries that match all (or many) documents •  -foo •  state:CA

! Precompute static parts of rank expression •  sqrt(rating)+200*log10(doc.votes) + 20*log10(doc.sales) =>

"precomputed"

!   Add implicit limits to broad searches


Default Search Field

!   Searches all text fields by default •  Usually the right thing •  If not, remove unneeded fields from it

!   Changeable via the API !   Alternative: explicitly select one field to search

•  title:'harry potter'


Target Slow Queries

!   Slow searches affect other searches !   Optimizing the slowest search requests may speed up all

of your searches. !   Slow queries can produce timeouts (507s)


…benchmarking


Testing Latency

!   Hard to estimate •  Depends on queries, usage patterns, data…

!   Build a domain & test •  It's the cloud, spin up & down as needed! •  Use your own data •  As close to real usage as you can

•  Log replay is good!

!   A/B Test your changes


Two Approaches

!   Testing •  Just run some queries

!   Benchmarking •  Run enough to have some statistical validity


Testing Approaches

!   Statistics provided

!   Browser tools •  Chrome •  Firebug for Firefox


Benchmarking

!   Apache JMeter (or similar) •  Well documented •  Well tested •  General

•  This is good, and bad !   Custom Code

•  Usually looks a lot like your application •  So, not as much code as you might think •  Flexible


Custom Code

!   Multithread it for realistic results •  If it's simulating more than one client

!   Make sure the clients (tester) isn't the bottleneck •  Benchmark it with searches stubbed •  Avoid languages with global interpreter locks

!   Personally, I use: •  Java •  Apache Http Client


Custom Code Example LinkedList<Thread> threads = new LinkedList<Thread>(); int nQ = queue.size(); long time = System.nanoTime(); List<Consumer> consumers = new LinkedList<Consumer>(); for (int i = 0; i < threadCount; i++) { Consumer c1 = new Consumer(queue); consumers.add(c1); Thread t = new Thread(c1); threads.add(t); t.start(); } // Wait for signal that we have processed all queries. for (Thread thread : threads) { thread.join();


Sample Data & Queries

!   Sample Data – must be realistic •  BAD: a123 b456 xyzzyx •  Better: use Wikipedia, project Gutenberg •  Best: Your own data

!   Queries •  BAD: random words •  Better: read words from test data •  Best: Replay log files •  Always check your number of responses in benchmarks.

•  It's easy to get fast queries, if you get no hits.


…analytics


Analytics & Metrics

!   You have to know what to tune !   CloudSearch Metrics !   Custom Logging


CloudSearch Metrics

!   Top Queries !   Zero result queries

•  Lack of data? •  Or query issues?


Custom Logging

!   Log all requests on your end !   Watch

•  Longest running queries •  Failed Queries •  HTTP error codes

!   Track Changes over time


Warning Signs

!   Know your http error codes •  500 series can be retried •  May indicate server is overloaded •  Long queries can tie up threads until timeout

•  More of an issue on small servers.


…wrap up


Wrap Up

!   1) Limit match set size !   2) Limit Match set size J !   Be aware of the cost of features

•  Test/Benchmark !   Resources

•  Slides will be on meetup group •  http://jmeter.apache.org/ •  https://getfirebug.com/ •  http://en.wikipedia.org/wiki/Category:Load_testing_tools


Thanks!

Tom Hill [email protected]

how to tune search requests using amazon cloudsearch

Documents

express consent of

text matching

query processing

query restriction

exact matching

rank computation

increasing performance

ranking score computation