solr, lucene and hadoop @ etsy

Solr, Lucene & Hadoop@

Thursday, May 10, 12

4 Years Lucene and Solr @ Etsy

david@etsy.com

History of Search @ Etsy

Hadoop + HBase Indexing(in development)

Replication

AboutUs

13MM Listings

39MM Unique Visitors880K Shops / 150 Countries

100+ Engineers

Architecture Overview

OverviewSearch

+n slaves

Memcached+n caches

Web+n webs

Database+n db shards

Thrift

Search

+n slaves

+n webs

query = hats for cats

result = 402, 283, 837

+n webs

Database

+n shards

Memcached

+n caches

Hydration

The Results

History of Searchat Etsy

History of Search2007

•1 Million Listings •A Single “Master” Postgres Database•PHP > Twisted > Stored Proc > TSearch•18 “Baby” Postgres Databases•Baby Replicator

•2 Million Listings •A Single “Master” Postgres Database•PHP > Solr•4 Solr Slaves + 2 Masters•Baby Replicator + DIH for Reindexing

•4 Million Listings •A Single “Master” Postgres Database•PHP > Solr•6 Solr Slaves + 2 Masters•Webs >ActiveMQ > Solr

•7 Million Listings •A Single “Master” Postgres Database•PHP > Thrift > Solr•10 Solr Slaves + 1 Master•Custom Import Handler

•10 Million Listings •“Master” Postgres Database + DB SHARDS!•PHP > Thrift > Solr•24 Solr Slaves + 1 Master•Custom Import Handler

Future of Search2012

•?? Million Listings•MORE DB SHARDS!•PHP > Thrift > Solr•?? Solr Slaves + 1 Master•HBase + Hadoop Indexers

What Did We Learn?

Lucene + Solr > TSearch

http://www.depesz.com/2010/10/17/why-im-not-fan-of-tsearch-2/

Love Lucene + Solr Trunk!

Run, Don’t Walk...

Fork it: https://github.com/etsy/deployinator

Deployinator

Smoker

StatsD, Graph Everything!

Fork it: https://github.com/etsy/statsd

95th Percentile

start · build_query · perform_search · receive_search_ads · search_side_response · create_event_logger · set_tpl_vars · tpl_render · receive_search_ads_post_render

Solr Top Level Cache > Memcached

etsy-index.properties

$ cat /search/data/person/index/etsy-index.properties#Tue Mar 27 13:05:51 EDT 2012max_update_time=2012-03-27T17\:05\:51.955Z

Check Index SizeDon’t Install if < 50% Current Size

Check if Index is Too OldDon’t Update if > 10 Days Old

What Did We Learn?

Store Nothing

Keep Denormalized Data

SearchDatabase

DB Shard

PHPDenormalizer

FullReindex

ApplyIncremental

Install

FullReindex

ApplyIncremental

InstallApply

Incremental

Indexer

Database

HBase + HadoopIndexing

HBase + Hadoop Indexing

Why HBase?

DB Shard

PHPDenormalizer

{NAME => 'listings_denormalized', FAMILIES => [{NAME => 'listing_data', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'SNAPPY', VERSIONS => '1', TTL => '-1', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}

listings_denormalized

{NAME => 'listings_denormalized_modified_index', FAMILIES => [{NAME => 'pks', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', COMPRESSION => 'SNAPPY', VERSIONS => '1', TTL => '-1', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}

listings_denormalized_modified_index

SOLR-1301

https://issues.apache.org/jira/browse/SOLR-1301

SolrOutput Format

•Solr Document Converter•Solr Requires Posix Disk•Index Copied Back to HDFS

•Not Great with Multi-Core Configs•Added Solr Multi-Core Support• Solr Config Issues•Added ENV support for Configs•Uses “new” style Hadoop API•Added Support for both Old and New

SolrInputDocumentWritable

public class SolrInputDocumentWritable extends SolrInputDocument implements org.apache.hadoop.io.Writable {

Oozie + HBase?

ScanStringGenerator

http://blog.ozbuyucusu.com/2011/07/21/using-hbase-tablemapper-via-oozie-workflow/

Hadoop Indexer

StartOozie

Map HBase

HDFSReduce

DiskSolr

Output

Install

IndexerActionMain

Deployinator

IndexCompare

$ ./compare ERROR: please provide two index directories example: ./compare -p 0.1 -i user_id ./index ./index-1332867952588options: -p --percent= percent of the index to check -i --id= primary key id field in the index -h --hash= comparison or hash field in the index <index> <index>

$ ./compare \/search/data/person/index-1332867952588/ \/search/data/person/index-1335378487672

id field: user_idhash field: hashpercentage: 0.0010 files: /search/data/person/index-1332867952588/ /search/data/person/index-1335378487672

/search/data/person/index-1332867952588 contains 1515512 docs/search/data/person/index-1335378487672 contains 14837972 docs1516 of 1516 documents are the same

Copy and Merge

Open Source

Replication

Slaves

+n slaves

Master

Replication

BitTorrentReplication

Bit Torrent

Using BitTornado:

ReplicationBit Torrent + Solr

Fork of TTorent: https://github.com/etsy/ttorrent

Multi-File SupportLarge File Support

Fork BitTorrent: Comming Soon

Replication

Need a job?

Thanks!

david@etsy.com

solr, lucene and hadoop @ etsy

Technology

using lwe/solr/lucene for ecom

solr lucene revolution 2014 - solr compute cloud - nitin

solr @ etsy - apache lucene eurocon

a static rank framework for lucene / solr

understanding the solr security framework - lucene solr...

introduction to lucene & solr and usecases

a collections searching center using lucene – solr

apache solr/lucene: looking ahead

lessons from sharding solr at etsy: presented by gregg...

apache solr cms integration @ lucene/solr revolution san...

boosting documents in solr (lucene revolution 2011)

solr at etsy - giovanni fernandez-kincade

trihug: lucene solr hadoop

grouping and joining in lucene/solr

solr lucene conference 2014 - nitin presentation

nyc lucene/solr meetup: spark / solr

semantic & multilingual strategies in lucene/solr

randomized continuous testing: solr & lucene use case

solr and lucene search revolution

lucene/solr 3.1