sf elasticsearch meetup - how hipchat scaled to 1b messages

Wednesday, November 20, 13

Heavy Lifting

ZUHAIB S IDDIQUE • @ZUHAIB • ENGINEER


1. Persistent chat rooms


2. File sharing & Chat history


3. @mentions + notifications


4. Integrations and API


5. Runs on everything


Group chat and IM for teams.


1,000,000,000

messages sent

1,200,000,000


{ "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" }}

What is a HipChat Message?


In the beginning there was …


Then we added...

https://github.com/rnewson/couchdb-luceneWednesday, November 20, 13

https://github.com/rnewson/couchdb-lucene

https://github.com/rnewson/couchdb-lucene

CouchDB + CouchDB-Lucene

•Started to tip over at 200M Docs

•m2.4xlarge


CouchDB + CouchDB-Lucene

m2.4xlarge m2.4xlarge

Worked well for the first 500MWednesday, November 20, 13

m2.4xlarge hs1.8xlarge


But it still wasn’t big enough.


Dont F*ck the Customer


First Attempt at Scaling


Why ElasticSearch for HipChat



{ "query_string" : { "default_field" : "content", "query" : "this AND that OR thus" }}


Search Error Rate


Don’t Forget about CouchDB

AWS Max EBS size


Stop the CouchDB bleeding


CouchDB end of the road


(allthethings) ElasticSearch

{ "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } }}

{ "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" }}

ElasticSearch CouchDB


CouchDB vs ElasticSearchCouchDB Spike

to 800ms!

Avg 100ms with ES


master

Running ElasticSearch

clientsHipChat Customer

Thanks Philip O’Toole - Loggly


$ fab create_esindex:hostname=esindex-d4


•Spin up EC2 Instance• Install Oracle Java 7•Create Provision IOPs EBS• Install and configure ElasticSearch•???•Profit!

Behind the Scenes


# Elasticsearchnodes = partial_search(:node, "role:elasticsearch_server AND chef_environment:#{node.chef_environment}", :keys => { 'hostname' => [ 'hostname' ] })if nodes.empty? Chef::Log.error "site.json - No Elasticsearch servers found"else es_nodes = nodes.map {|n| { 'host' => n['hostname'], 'port' => 9200 } } data["elasticsearch"] = { "connections" => es_nodes.sort_by {|n| n['host'] } }end


site.json"elasticsearch": { "connections": [ { “host": "esindex-c1", "port": 9200 }, ... { "host": "esindex-e3", "port": 9200 } ]}


You


Running Elasticsearch


Running ElasticSearch

Single Node OOM!


Monitor, Monitor, Monitor!


Backup


100,000,000,000?To infinity and beyond!


Questions?


sf elasticsearch meetup - how hipchat scaled to 1b messages

Technology