solr, lucene and hadoop @ etsy

Solr, Lucene & Hadoop@

Thursday, May 10, 12

4 Years Lucene and Solr @ Etsy

[email protected]


mailto:[email protected]


History of Search @ Etsy

Hadoop + HBase Indexing(in development)

Replication


AboutUs


13MM Listings

39MM Unique Visitors880K Shops / 150 Countries

100+ Engineers


Architecture Overview


OverviewSearch

+n slaves

Memcached+n caches

Web+n webs

Database+n db shards


Thrift

slave

slave

Search

+n slaves

Web

web

web

+n webs

query = hats for cats

result = 402, 283, 837


Web

web

web

+n webs

Database

shard

+n shards

shard

Memcached

cache

cache

+n caches

Hydration


The Results


History of Searchat Etsy


History of Search2007

•1 Million Listings •A Single “Master” Postgres Database•PHP > Twisted > Stored Proc > TSearch•18 “Baby” Postgres Databases•Baby Replicator



•2 Million Listings •A Single “Master” Postgres Database•PHP > Solr•4 Solr Slaves + 2 Masters•Baby Replicator + DIH for Reindexing



•4 Million Listings •A Single “Master” Postgres Database•PHP > Solr•6 Solr Slaves + 2 Masters•Webs >ActiveMQ > Solr



•7 Million Listings •A Single “Master” Postgres Database•PHP > Thrift > Solr•10 Solr Slaves + 1 Master•Custom Import Handler



•10 Million Listings •“Master” Postgres Database + DB SHARDS!•PHP > Thrift > Solr•24 Solr Slaves + 1 Master•Custom Import Handler


Future of Search2012

•?? Million Listings•MORE DB SHARDS!•PHP > Thrift > Solr•?? Solr Slaves + 1 Master•HBase + Hadoop Indexers


What Did We Learn?


Lucene + Solr > TSearch

http://www.depesz.com/2010/10/17/why-im-not-fan-of-tsearch-2/


https://github.com/etsy/ttorrent


Love Lucene + Solr Trunk!


Run, Don’t Walk...


Fork it: https://github.com/etsy/deployinator

Deployinator




Smoker


StatsD, Graph Everything!

Fork it: https://github.com/etsy/statsd




95th Percentile


start · build_query · perform_search · receive_search_ads · search_side_response · create_event_logger · set_tpl_vars · tpl_render · receive_search_ads_post_render


Solr Top Level Cache > Memcached


etsy-index.properties

$ cat /search/data/person/index/etsy-index.properties#Tue Mar 27 13:05:51 EDT 2012max_update_time=2012-03-27T17\:05\:51.955Z


Check Index SizeDon’t Install if < 50% Current Size


Check if Index is Too OldDon’t Update if > 10 Days Old


What Did We Learn?

Store Nothing


Keep Denormalized Data


SearchDatabase

DB Shard

DB Shard

DB Shard

PHPDenormalizer

JSON


FullReindex

ApplyIncremental

Install


FullReindex

ApplyIncremental

InstallApply

Incremental


Indexer

Database


HBase + HadoopIndexing


HBase + Hadoop Indexing

Why HBase?



HBase

DB Shard

DB Shard

DB Shard

PHPDenormalizer

JSON


{NAME => 'listings_denormalized', FAMILIES => [{NAME => 'listing_data', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'SNAPPY', VERSIONS => '1', TTL => '-1', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}

listings_denormalized



{NAME => 'listings_denormalized_modified_index', FAMILIES => [{NAME => 'pks', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'SNAPPY', VERSIONS => '1', TTL => '-1', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}

listings_denormalized_modified_index




SOLR-1301

https://issues.apache.org/jira/browse/SOLR-1301




HDFS

Disk

SolrOutput Format

•Solr Document Converter•Solr Requires Posix Disk•Index Copied Back to HDFS



•Not Great with Multi-Core Configs•Added Solr Multi-Core Support• Solr Config Issues•Added ENV support for Configs•Uses “new” style Hadoop API•Added Support for both Old and New



SolrInputDocumentWritable


public class SolrInputDocumentWritable extends SolrInputDocument implements org.apache.hadoop.io.Writable {



Oozie



Oozie + HBase?


ScanStringGenerator


http://blog.ozbuyucusu.com/2011/07/21/using-hbase-tablemapper-via-oozie-workflow/




Hadoop Indexer

StartOozie

Map HBase

HDFSReduce

DiskSolr

Output

Copy

Merge

Install



IndexerActionMain




Deployinator


IndexCompare



$ ./compare ERROR: please provide two index directories example: ./compare -p 0.1 -i user_id ./index ./index-1332867952588options: -p --percent= percent of the index to check -i --id= primary key id field in the index -h --hash= comparison or hash field in the index <index> <index>



$ ./compare \/search/data/person/index-1332867952588/ \/search/data/person/index-1335378487672

id field: user_idhash field: hashpercentage: 0.0010 files: /search/data/person/index-1332867952588/ /search/data/person/index-1335378487672

/search/data/person/index-1332867952588 contains 1515512 docs/search/data/person/index-1335378487672 contains 14837972 docs1516 of 1516 documents are the same



Copy and Merge



Open Source



Replication


Slaves

+n slaves

Master

Replication


BitTorrentReplication


Bit Torrent

Using BitTornado:


ReplicationBit Torrent + Solr


Fork of TTorent: https://github.com/etsy/ttorrent

Multi-File SupportLarge File Support

Fork BitTorrent: Comming Soon

Replication




Need a job?


Thanks!


[email protected]




solr, lucene and hadoop @ etsy

Technology