geospatial search with amazon cloudsearch

© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

GeoSpatial Search in

Amazon CloudSearch

Tom Hill January 30, 2013


Agenda

!   What is GeoSpatial search? !   Why do we care? !   Computing distance !   Geospatial Search in CloudSearch


What is GeoSpatial Search?

!   Using location & distance as factors in search


What's this "Geo", anyway?

!   Geographic •  On the earth

!   Spatial •  Simple distance


How do we use distance?

!   Limit to an Area •  Box •  Circle •  Polygon*

!   Sort by Distance !   Include distance in score

*not yet!


Why Do You Care?

!   People care about things near them. •  Pizza, Classified Ads, etc. •  Find a Doctor, Lawyer,…

!   Mobile is a key driver


What can you do with CloudSearch?


Computing Distance

!   Many Formulas •  Rectangular distance •  Equirectangular projection •  Spherical Law of Cosines •  Haversine Formula •  Vincenty's Formula

!   Speed Vs. Accuracy •  Speed: Rectangular distance •  Accuracy: Vincenty's Formula


Why So Many Ways to Compute Distance?

The earth isn't flat! It isn't a sphere either.


Is the Earth Flat?

!   If it's flat •  distance = sqrt((lat1-lat2)^2 + (lon1-lon2)^2)

!   If it's not


How Much Accuracy?

!   "Pizza, 1 Mile" ! Haversine is more accurate

•  If you are a bird

!   Any distance is approximate


Comparing Distance Computations

!   Four computations: Haversine, Cosines, EquiRrect, Rect. !   Different computations – Different Results !   How accurate do you need to be?

Haversine Cosines EuqiRect Rect - CosErr EquErr RecErr 994.79893 994.79893 995.25921 1044.40926 - 0.000 0.000 0.050 Fort Lauderdale, FL to Anniston, AL 624.04339 3624.04339 3642.98321 4163.41737 - 0.000 0.005 0.149 Fort Lauderdale, FL to San Diego, CA1812.54997 1812.54997 1814.38516 1871.77660 - 0.000 0.001 0.033 Fort Lauderdale, FL to New Haven, CT8175.93897 8175.93897 8817.96563 11107.90729 - 0.000 0.079 0.359 Fort Lauderdale, FL to Adak, AK7244.45661 7244.45661 8008.74698 12015.96646 - 0.000 0.106 0.659 Bangor, ME to Adak,AK


GeoSpatial Search in

Amazon CloudSearch


How to Compute Distance in CloudSearch?

!   Rank Expressions •  Computations run for each matching document

!   Can be used for •  Sorting •  Influencing Scoring


Two Types of Rank Expressions

!   Static Rank Expressions •  Computation based on values in index

!   Query Time Rank Expressions •  Allow including parameters at run time.

•  e.g. latitude, longitude.


Query Time Rank Expressions

!   Define Rank Expression •  &rank-NAME=EXPRESSION •  &rank-geo=sqrt(pow(lat-userlat),2)+pow(lon-userlon),2))

!   Select Rank Expression •  &rank=NAME •  &rank=geo

http://searchendpoint?q=creek&rank=geo&rank-geo=sqrt(pow(la1-123),2)+pow(lo1-456),2))


Influencing Scoring

!   Include both text relevance and distance:

&rank-geo=(1000-text_relevance) + sqrt(pow(la1-ulat),2)+pow(lo1-ulon),2))

!   Relative Weight •  N * text_relevance + M * distance •  That's where the art comes in. •  Will vary by your application. •  Test & tune, and test again.


Storing Data

!   CloudSearch supports unsigned integers !   Have to convert latitude, longitude to positive ranges

•  latitude + 90 •  longitude + 180

!   Have to store as integers; need to scale •  latitude = (latitude + 90) * 100 •  longitude= (longitude+ 180) * 100


Performance

!   Don't query the whole world •  Can limit by literals or numeric fields. •  Literals are more efficient for limits.

!   Limit Options •  Literal

•  &bq=state:'CA' •  &bq=zip:'94402'

•  Numeric •  &bq=(and latitude:40..50 longitude:80..85)

•  Geohash


Performance Measures

!GeoMethod!!!!TextRel!!!!!Limits!!!!Queries!!Seconds!!QTimeMS!!!!Threads!CompletedQ!!!!!!AveHits!!!!!!!NONE!!!!!!false!!!!!!!!!!!!!!!!!!!!10!!!6.2255!!!!!!622!!!!!!!!!!1!!!!!!!!!10!!!8345450.00!!CARTESIAN!!!!!!false!!!!!!!!!!!!!!!!!!!!10!!15.6064!!!!!1560!!!!!!!!!!1!!!!!!!!!10!!!8345450.00!!!!!!!EQUI!!!!!!false!!!!!!!!!!!!!!!!!!!!10!!19.7106!!!!!1971!!!!!!!!!!1!!!!!!!!!10!!!8345450.00!!!!COSINES!!!!!!false!!!!!!!!!!!!!!!!!!!!10!!27.4968!!!!!2749!!!!!!!!!!1!!!!!!!!!10!!!8345450.00!!HAVERSINE!!!!!!false!!!!!!!!!!!!!!!!!!!!10!!31.2595!!!!!3125!!!!!!!!!!1!!!!!!!!!10!!!8345450.00!!!!!!!!NONE!!!!!!false!!!!Numeric!!!!!!!!!10!!!9.1758!!!!!!917!!!!!!!!!!1!!!!!!!!!10!!!!!!3807.00!!CARTESIAN!!!!!!false!!!!Numeric!!!!!!!!!10!!!9.0255!!!!!!902!!!!!!!!!!1!!!!!!!!!10!!!!!!3807.00!!!!!!!EQUI!!!!!!false!!!!Numeric!!!!!!!!!10!!!9.1158!!!!!!911!!!!!!!!!!1!!!!!!!!!10!!!!!!3807.00!!!!COSINES!!!!!!false!!!!Numeric!!!!!!!!!10!!!9.8321!!!!!!983!!!!!!!!!!1!!!!!!!!!10!!!!!!3807.00!!HAVERSINE!!!!!!false!!!!Numeric!!!!!!!!!10!!!9.1272!!!!!!912!!!!!!!!!!1!!!!!!!!!10!!!!!!3807.00!!!!!!!!NONE!!!!!!false!!!!literal!!!!!!!!!10!!!0.8254!!!!!!!82!!!!!!!!!!1!!!!!!!!!10!!!!!!3781.00!!CARTESIAN!!!!!!false!!!!literal!!!!!!!!!10!!!0.5936!!!!!!!59!!!!!!!!!!1!!!!!!!!!10!!!!!!3781.00!!!!!!!EQUI!!!!!!false!!!!literal!!!!!!!!!10!!!0.6173!!!!!!!61!!!!!!!!!!1!!!!!!!!!10!!!!!!3781.00!!!!COSINES!!!!!!false!!!!literal!!!!!!!!!10!!!0.5916!!!!!!!59!!!!!!!!!!1!!!!!!!!!10!!!!!!3781.00!!HAVERSINE!!!!!!false!!!!literal!!!!!!!!!10!!!0.6289!!!!!!!62!!!!!!!!!!1!!!!!!!!!10!!!!!!3781.00!!

Why you don't want query the whole world!


Geo-Spatial Demo Application


Demo Application Structure

HTML Page

Javascript Server (Tomcat)

CloudSearch Ajax


Demo Implementation

!   JavaScript •  Ajax •  JQuery •  Google Maps API

!   Twitter Bootstrap •  css

!   Tomcat Server •  Java •  Just for forwarding of requests

•  Because XSS, that's why.


Querying CloudSearch $.ajax({(

(url(:("searchx",((data(:({(( ('q'(:(currentQuery,(( ('domain'(:("geoname25",(( ('return@fields'(:(returnFields.join(),(( ("rank"(:("geo",(( ("rank@geo"(:("Math.sqrt(Math.pow(Math.abs(doc.latitude_90@(12539),2)(+(

Math.pow(Math.abs(doc.longitude_180@(5784),2))"((},((dataType(:("json",((success(:(function(data)({(( (var(hits(=(data['hits'];(( (displaySearchResults(hits['hit'],(hits['found']);(( (populateMap(hits['hit']);((}(

});(


Wrap Up


Thanks for Coming!

!   Data •  http://www.geonames.org/export/

!  Slides •  On the meetup group soon

!   Sample Code •  Talk to me. ([email protected])

!   Computations •  http://www.movable-type.co.uk/scripts/latlong.html •  wikipedia


Querying Data !   Java/Javascript

•  Fields are latitude_90, longitude_180 •  user location is userlat, userlon

!   simple distance

rank!=!"Math.sqrt(Math.pow(Math.abs(latitude_90W("!+!userlat!+!")),2)+Math.pow(Math.abs(longitude_180W("!+!userlon!+!")),2))";!!

!   Spherical Law of Cosines

rank = "6371*Math.acos(Math.sin(" + userlat + ") * Math.sin(lat_rad/" + scale + ") + Math.cos(" + userlat + ") * Math.cos(lat_rad/" + scale + ") * Math.cos((lon_rad/" + scale + ") - " + userlon + ") )";!


What CloudSearch Doesn't Do


Issues with assuming the earth is flat.

geospatial search with amazon cloudsearch

Technology

express consent of

distance computations

flat distance

rank expression rank

rights reserve

computing distance

location distance

rank expressions computations