Wednesday, November 20, 13
Heavy Lifting
ZUHAIB S IDDIQUE • @ZUHAIB • ENGINEER
Wednesday, November 20, 13
Wednesday, November 20, 13
1. Persistent chat rooms
Wednesday, November 20, 13
2. File sharing & Chat history
Wednesday, November 20, 13
3. @mentions + notifications
Wednesday, November 20, 13
4. Integrations and API
Wednesday, November 20, 13
5. Runs on everything
Wednesday, November 20, 13
5. Runs on everything
Wednesday, November 20, 13
5. Runs on everything
Wednesday, November 20, 13
Group chat and IM for teams.
Wednesday, November 20, 13
1,000,000,000
messages sent
1,200,000,000
Wednesday, November 20, 13
{ "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" }}
What is a HipChat Message?
Wednesday, November 20, 13
In the beginning there was …
Wednesday, November 20, 13
Then we added...
https://github.com/rnewson/couchdb-luceneWednesday, November 20, 13
CouchDB + CouchDB-Lucene
•Started to tip over at 200M Docs
•m2.4xlarge
Wednesday, November 20, 13
CouchDB + CouchDB-Lucene
m2.4xlarge m2.4xlarge
Worked well for the first 500MWednesday, November 20, 13
m2.4xlarge hs1.8xlarge
Wednesday, November 20, 13
But it still wasn’t big enough.
Wednesday, November 20, 13
Dont F*ck the Customer
Wednesday, November 20, 13
First Attempt at Scaling
Wednesday, November 20, 13
Wednesday, November 20, 13
Why ElasticSearch for HipChat
Wednesday, November 20, 13
Why ElasticSearch for HipChat
{ "query_string" : { "default_field" : "content", "query" : "this AND that OR thus" }}
Wednesday, November 20, 13
Why ElasticSearch for HipChat
Wednesday, November 20, 13
Search Error Rate
Wednesday, November 20, 13
Wednesday, November 20, 13
Don’t Forget about CouchDB
AWS Max EBS size
Wednesday, November 20, 13
Stop the CouchDB bleeding
Wednesday, November 20, 13
CouchDB end of the road
Wednesday, November 20, 13
Wednesday, November 20, 13
(allthethings) ElasticSearch
{ "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } }}
{ "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" }}
ElasticSearch CouchDB
Wednesday, November 20, 13
(allthethings) ElasticSearch
{ "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } }}
{ "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" }}
ElasticSearch CouchDB
Wednesday, November 20, 13
CouchDB vs ElasticSearchCouchDB Spike
to 800ms!
Avg 100ms with ES
Wednesday, November 20, 13
master
Running ElasticSearch
clientsHipChat Customer
Thanks Philip O’Toole - Loggly
Wednesday, November 20, 13
$ fab create_esindex:hostname=esindex-d4
Wednesday, November 20, 13
•Spin up EC2 Instance• Install Oracle Java 7•Create Provision IOPs EBS• Install and configure ElasticSearch•???•Profit!
Behind the Scenes
Wednesday, November 20, 13
# Elasticsearchnodes = partial_search(:node, "role:elasticsearch_server AND chef_environment:#{node.chef_environment}", :keys => { 'hostname' => [ 'hostname' ] })if nodes.empty? Chef::Log.error "site.json - No Elasticsearch servers found"else es_nodes = nodes.map {|n| { 'host' => n['hostname'], 'port' => 9200 } } data["elasticsearch"] = { "connections" => es_nodes.sort_by {|n| n['host'] } }end
Wednesday, November 20, 13
site.json"elasticsearch": { "connections": [ { “host": "esindex-c1", "port": 9200 }, ... { "host": "esindex-e3", "port": 9200 } ]}
Wednesday, November 20, 13
You
Wednesday, November 20, 13
Running Elasticsearch
Wednesday, November 20, 13
Running ElasticSearch
Single Node OOM!
Wednesday, November 20, 13
Monitor, Monitor, Monitor!
Wednesday, November 20, 13
Backup
Wednesday, November 20, 13
100,000,000,000?To infinity and beyond!
Wednesday, November 20, 13
Questions?
Wednesday, November 20, 13