![Page 1: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/1.jpg)
You know, for search
11. května 2011, EurOpen - Pavlov
![Page 2: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/2.jpg)
Outline
● Talking about ElasticSearch and giving some demos
● What you should take away from this talk?
![Page 3: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/3.jpg)
About me
● Lukáš Vlček ( @lukasvlcek )● Java developer since 2001● Joined Red Hat (JBoss division) in 2010● Member of JBoss.org team, focusing on search
![Page 4: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/4.jpg)
What is ElasticSearch ?
● Open Source (ASL2)● Distributed (cloud friendly)● Highly-available● RESTful search engine (on top of Lucene)● Designed to speak JSON (JSON in, JSON out)● Author: Shay Banon ( @kimchy )
![Page 5: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/5.jpg)
Where ?
https://github.com/elasticsearch/elasticsearch
http://www.elasticsearch.org/
![Page 6: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/6.jpg)
Demo #1
Searching in emails
REST API: Faceted search, Highlighting
![Page 7: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/7.jpg)
RESTful
● Network interface for data indexing, searching and administration.
curl XGET 'http://localhost:9200/index1,index2/typeA,typeB/_search' d '{ “query“ : { “match_all“ : {} }}'
You can query one or more indices. Indices can have aliases, you can also use _all for all indices.
Each index have one or more types, something like columns in DB table.
![Page 8: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/8.jpg)
Talking to the cluster
Node 3Node 2Node 1
Client
● REST client
● Many clients built on top of REST API● Perl, PHP, Python, Ruby, Erlang, ... etc
![Page 9: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/9.jpg)
Demo #2
RESTful JSON teaser
![Page 10: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/10.jpg)
Talking to the cluster
Node 3Node 2Node 1
Client
● Native client in Java and Groovy
● Client type:● Node client● Transport Client
![Page 11: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/11.jpg)
Highly available
● For each index you can specify:● Number of shards
– Each index has fixed number of shards● Number of replicas
– Each shard can have 0-many replicas, can be changed dynamically
![Page 12: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/12.jpg)
ZEN Discovery
Node 1 Node 4Node 3Node 2
A B C
A1
A2
A3
A1
A2
A3
A1
A2
A3
B1
B2
B1
B2
B1
B2
B1
B2
C1
C: { shards: 1, replicas: 0 }B: { shards: 2, replicas: 3 }A: { shards: 3, replicas: 2 }
Gateway (longterm persistency of cluster data & metadata)
![Page 13: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/13.jpg)
ZEN Discovery
Node 1 Node 4Node 3Node 2
A B C
A1
A2
A3
A1
A2
A3
A1
A2
A3
B1
B2
B1
B2
B1
B2
B1
B2
C1
C: { shards: 1, replicas: 4 }B: { shards: 2, replicas: 3 }A: { shards: 3, replicas: 2 }
C1 C1 C1 C1
Can not allocate all replicas!Check Health API
Gateway (longterm persistency of cluster data & metadata)
![Page 14: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/14.jpg)
Nodes do not have to be equal
● Can be a master● Can be a data node● Can allow for REST transport interface
● Http, memcached, thrift
● Index store (file, memory)● ...
Node 3Node 2
Node 1
![Page 15: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/15.jpg)
Gateway
● Long time persistency allows for whole (and partial) cluster backup and recovery.
Types:● Local (default)● NFS● HDFS● AWS: S3
![Page 16: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/16.jpg)
Demo #3
Dynamic allocation of indices,shards, replicas and Health API
![Page 17: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/17.jpg)
Admin API
● Indices– Status– CRUD operation– Mapping, Open/Close, Update settings– Flush, Refresh, Snapshot, Optimize
● Cluster– Health– State– Node Info and stats– Shutdown
![Page 18: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/18.jpg)
Demo #4
Admin API: getting JVM and OS stats
![Page 19: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/19.jpg)
Rich query API
● There is rich Query DSL for search, includes:● Queries
– Boolean, Fuzzy, MLT, Prefix, DisMax, ...● Filters
– And/Or/Not, Boolean, Geo, Missing, Exists, ...● Highlighting● Sort● Facets
– on a next slide...
![Page 20: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/20.jpg)
Facets
● Facets allows to provide aggregated data for the search request.● query● filter● terms● range● histogram● statistical● geo distance
![Page 21: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/21.jpg)
Scripting support
● There is a support for using scripting languages in many places (for example for custom scoring, script fields, script key in facets ...)● mvel (default)● JS● Groovy● Python
![Page 22: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/22.jpg)
Parent / Child
● The parent/child support allows to define a parent relationship from a child to a parent type.● has_child (query, filter)● top_children (filter)
![Page 23: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/23.jpg)
River
● Let's listen on stream of changes and index the data...● CouchDB● RabbitMQ● Twitter● Wikipedia
![Page 24: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/24.jpg)
Versioning
● “update if current” functionality● ie: I can get a document, change it and then put
it back in (referencing the version ID I fetched) and it will either index or fail (if the document has been modified in the interim)
● Completely real-time
![Page 25: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/25.jpg)
Percolator
● The percolator API allows to register queries against an index, and then send a percolate request which includes a document, and getting back the queries that match on that document out of set of registered queries.
![Page 26: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/26.jpg)
Q&A
![Page 27: You know, for search · About me Lukáš Vlček ( @lukasvlcek ) Java developer since 2001 Joined Red Hat (JBoss division) in 2010 Member of JBoss.org team, focusing on search](https://reader036.vdocuments.us/reader036/viewer/2022070808/5f077f8f7e708231d41d47e5/html5/thumbnails/27.jpg)
Thank you!