mastering elasticsearch with ruby (+tire)
DESCRIPTION
At RubyConf 2013, Co-founder and Chief Product and Technology Officer, Luca Bonmassar, covered how to build an ElasticSearch using the Ruby Tire Library. Here are the slides from the presentation.TRANSCRIPT
![Page 1: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/1.jpg)
MasteringElasticSearch w/ Ruby (+ Tire)
RubyConf 2013
Luca Bonmassar, Gild
Tuesday, 5 November 13
![Page 2: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/2.jpg)
Who am I?I’m Luca Bonmassar (@openmosix) # 31 # Italian living in San Francisco (and Stockholm)
I work at Gild
I love building products [for fun, profit and boredom]
Tuesday, 5 November 13
![Page 3: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/3.jpg)
Search - use case You’re building a product
User generated content
Let (other) users find or discover this content
Tuesday, 5 November 13
![Page 4: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/4.jpg)
Search is NOT easy It usually starts as
but then you want to support AND, OR, NOT, double quotes on multiple fields so
and then it goes like
Tuesday, 5 November 13
![Page 5: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/5.jpg)
And so it goes...
Tuesday, 5 November 13
![Page 6: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/6.jpg)
AgendaLet’s define a “pet project”
Boilerplate (download, install, scaffold, config, bla bla bla, yadda, yadda, yadda)
Build a website w/ simple search
Build a more advanced search
What next (homework)
Tuesday, 5 November 13
![Page 7: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/7.jpg)
Project! (for fun and no profit)
Tuesday, 5 November 13
![Page 8: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/8.jpg)
Project: build a full text-search for RubyGems.org
Tuesday, 5 November 13
![Page 9: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/9.jpg)
RubyGems Search Architecture
RubyGems.org
MongoDB
Rails App (Tire)
Elastic Search
Web Spider
Tuesday, 5 November 13
![Page 10: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/10.jpg)
RubyGems crawler
github.com/openmosix/rubygems-crawler
1. Download all ruby gem names from https://rubygems.org/gems2. Use gems API to download each gem info: read JSON - write JSON (MongoDB)
Tuesday, 5 November 13
![Page 11: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/11.jpg)
RubyGems Crawler II
Tuesday, 5 November 13
![Page 12: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/12.jpg)
RubyGems Search v1
Code: github.com/openmosix/rubygems-searchSpoiler: http://rubyconf.bonmassar.it
Tuesday, 5 November 13
![Page 13: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/13.jpg)
ElasticSearch
Tuesday, 5 November 13
![Page 14: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/14.jpg)
ElasticSearch II Open source search+analytics engine
Distributed[near] Realtime searchMulti tenantBuilt on Apache Lucene
REST APIsJSON documents
Tuesday, 5 November 13
![Page 15: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/15.jpg)
1. Download / set up a ES cluster
4. Query Index3. Index Data2. Define settings and data mapping (opt)
ElasticSearch III
Elastic Search MongoDB
Curl>
curl -‐X GET 'h.p://localhost:9200/ruby_gems/_search?from=0&size=25&pre.y'
{ "ok" : true, "status" : 200, "version" : { "number" : "0.90.5", "lucene_version" : "4.4" }, "tagline" : "You Know, for Search"}
Tuesday, 5 November 13
![Page 16: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/16.jpg)
ES download + setup
> wget http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.6.tar.gz
> tar zxvf elasticsearch-0.90.6.tar.gz
> sudo mv elasticsearch-0.90.6 /usr/local/
Hint #1: you need JavaHint #2: you need Oracle Java
Tuesday, 5 November 13
![Page 17: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/17.jpg)
ES config > ls elasticsearch-0.90.6/config/
Logging.yml: where to log, how much to logElasticsearch.yml: all server config. Defines:Name of the cluster (change it!!!)Node parameters (master/slave, store data/router)Sharing and # replicasPathsPluginsMemory (JVM, heap, memory locking)Network config“Gateway” (cluster backup)RecoveryDiscoverySlow log + GC log
Default options are good enough for dev env
Tuesday, 5 November 13
![Page 18: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/18.jpg)
ES boot + test Start: bin/elasticsearch
Test: curl http://localhost:9200/{ "ok" : true, "status" : 200, "name" : "Iron Fist", "version" : { "number" : "0.90.6", "lucene_version" : "4.5.1" }, "tagline" : "You Know, for Search"}
Stop: curl -XPOST 'http://localhost:9200/_shutdown'
{"cluster_name":"elasticsearch","nodes":{"jm2Z3J4dSzebjJ7Px2fAJg":{"name":"Iron Fist"}}}
Tuesday, 5 November 13
![Page 19: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/19.jpg)
Profit(you are now an ElasticSearch expert - go and tell the world)
Tuesday, 5 November 13
![Page 20: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/20.jpg)
ElasticSearch operations
Create a “RubyGem” IndexDefines a “RubyGem” Index data mapping
Index data (e.g. upload data from MongoDB to ES index = POST)
Query (= GET)
Tuesday, 5 November 13
![Page 21: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/21.jpg)
Tire now Re-Tire ;(
A ruby gem wrapping ElasticSearch REST APIs into a powerful ruby DSL
ActiveModel integration
Rake tasks and utilities to load and query ElasticSearch
Tuesday, 5 November 13
![Page 22: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/22.jpg)
Tire setupcat “gem ‘tire’” > Gemfile && bundle install
> cat config/initializers/elasticsearch.rb...Tire::Configuration.url('http://localhost:9200')Tire.configure { logger "#{Rails.root}/log/elasticsearch-queries.log" } if ENV['ES_LOG']
Tuesday, 5 November 13
![Page 23: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/23.jpg)
Define an ES index (with Tire DSL)
Tuesday, 5 November 13
![Page 24: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/24.jpg)
Indexing Get a recordConvert it to JSON format (to_indexed_json)Push it to Elastic Search (.update_index)
...under the hood...
Tuesday, 5 November 13
![Page 25: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/25.jpg)
Index (all data) Naive (POST on index for each record):
Use bulk updates:
...under the hood...
Tuesday, 5 November 13
![Page 26: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/26.jpg)
Search I
Tuesday, 5 November 13
![Page 27: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/27.jpg)
Search II
Tuesday, 5 November 13
![Page 28: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/28.jpg)
Simple Search
Tuesday, 5 November 13
![Page 29: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/29.jpg)
Highlight matches
Text
...add some CSS...
Tuesday, 5 November 13
![Page 30: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/30.jpg)
Advanced Search
Tuesday, 5 November 13
![Page 31: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/31.jpg)
Advanced Search II
Tuesday, 5 November 13
![Page 32: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/32.jpg)
Advanced Search III
Tuesday, 5 November 13
![Page 33: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/33.jpg)
Facets
Tuesday, 5 November 13
![Page 34: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/34.jpg)
ES facets
Tuesday, 5 November 13
![Page 35: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/35.jpg)
ES facets (running)
Tuesday, 5 November 13
![Page 36: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/36.jpg)
Facets - UI
Tuesday, 5 November 13
![Page 37: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/37.jpg)
Bonsai CoolI Search Suggesters (Did you mean... ?)
Tuesday, 5 November 13
![Page 38: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/38.jpg)
Bonsai Cool II
“Similar to this” (aka “More Like This” API)
Tuesday, 5 November 13
![Page 39: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/39.jpg)
Bonsai Cool III
Percolate API
Tuesday, 5 November 13
![Page 40: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/40.jpg)
DeploymentI Run your own cluster
Some learnings: at least 3 nodesmemory profiling / GCinstall very good monitoring (github.com/karmi/elasticsearch-paramedic)more RAM is (always) betterCheck IOPS (if on AWS)
Pros:Total controlCheaper (lot cheaper)
Cons:Can be a nightmare / Require dedicated devop
Tuesday, 5 November 13
![Page 41: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/41.jpg)
Deployment II ElasticSearch as a service
http://found.nohttp://searchly.comhttp://bonsai.io
Pros:Get cluster up & running in a minuteFocus on dev, not troubleshootingProfessional support
Cons:ExpensiveCan be in the wrong region / hosting providerExpensiveDid I say expensive?
Tuesday, 5 November 13
![Page 42: Mastering ElasticSearch With Ruby (+Tire)](https://reader034.vdocuments.us/reader034/viewer/2022051612/54bdbfd54a79598b408b4599/html5/thumbnails/42.jpg)
Thanks!Code: github.com/openmosix/rubygems-searchgithub.com/openmosix/rubygems-crawler
Demo (will be down by the end of rubyconf): http://rubyconf.bonmassar.it
Say “hi”:Luca Bonmassar - [email protected]
twitter.com/openmosix
github.com/openmosix
linkedin.com/in/lucabonmassar
Tuesday, 5 November 13