be elastic: leapset innovation session 06-08-2015
TRANSCRIPT
![Page 1: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/1.jpg)
“Where in the world is Elastic”
Innovation SessionData & Analytics Team
1
![Page 2: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/2.jpg)
Try SQL??? SELECT *,IF(`discount`>10%,1,0) `has_discounts`,(3959 * acos ( cos ( radians(cur_lat) ) * cos( radians( lat ) ) * cos( radians( lng ) - radians(cur_lng) ) + sin ( radians(cur_lat) ) * sin( radians( lat ) ) ) ) AS distanceFROM `Restaurants`WHERE `type`=’Pizza’AND `price` <= 1500AND `distance` < 500ORDER BY `WiFi` DESC,`has_discounts` DESC
2
● Requirementso Pizzaso Under 1500o Within 500m
● (Optional)o Wifio Discount
![Page 3: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/3.jpg)
What If
3
501m
1501
500m
![Page 4: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/4.jpg)
ElasticSearch Approach{"bool": {
"must": {"multi_match": {"query": "pizza", "fields":
["type^2", "restaurant"]},},"must_not": {},"should": {
"term": {"features": "wifi"},"range": {"discounts": {"gt": 10}}
}}}
4
● Requirementso Pizzaso Under 1500o Within 500m
● (Optional)o Wifio Discount
![Page 5: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/5.jpg)
Via ElasticSearch
5
{"gauss": {
"location": {"origin": "<lat>,<lon>","offset": "0.5km","decay": 0.5
}"price": {
"origin": 0,"offset": 1500,"decay": 0.5
}}}
501m
1501
500m
![Page 6: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/6.jpg)
Outline● Introducing ElasticSearch● Naive comparison● Lucene: The Architecture● Plug & Play● How we use ElasticSearch● Summary
6
![Page 7: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/7.jpg)
Introducing ElasticSearch“Your data, your search”
7
![Page 8: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/8.jpg)
ElasticSearch● Open-Source Search & Analytics engine
o Structured & Unstructured Datao (Near) Real Timeo Analytics capabilities (facets)o REST based
● Distributedo Designed for the Cloudo Designed for Big Data
● Lightweight● Popular: ~200K dl/month
8
![Page 9: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/9.jpg)
Naïve ComparisonElasticSearch vs. Solr.
9
![Page 10: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/10.jpg)
10
● Based on Lucene● Full-text search● Structured and Unstructured● Queries, filters, caches,
Facets● Cloud-ready
● Download size ● Zookeeper Vs ES own algorithm● Release process (LGTM vs
Apache)● Config vs Magic
Commonalities
Differences
![Page 11: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/11.jpg)
Architecture behind ElasticSearch
Inverted Index at Apache Lucene
11
![Page 12: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/12.jpg)
Elasticsearch Storage Architecture ● Analysis process
12
![Page 13: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/13.jpg)
Elasticsearch Storage Architecture ● Inverted Indexes
13
![Page 14: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/14.jpg)
Plug & Play
14
![Page 15: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/15.jpg)
ElasticSearch Cluster
15
Cluster StateIndex Mappings
Shard Routing TablesNodes’ metadata
Node 1 Node 2 Node 3
Shard C
Shard A Shard B
Shard A
Shard C
Shard B
ElasticSearch Index
Lucene Index
Segments
Inverted Index Replicas
Zone A Zone B
![Page 16: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/16.jpg)
Index Request
16
![Page 17: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/17.jpg)
Search Request
17
![Page 18: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/18.jpg)
How we use ElasticSearch
Use-cases at Leapset
18
![Page 19: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/19.jpg)
19
“Can you check the CouchDB errors from last Monday
between 1.00 and 1.05 a.m?”
You gotta be kidding me,
Ale!!!!
![Page 20: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/20.jpg)
Centralized Logs
20
![Page 21: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/21.jpg)
Log Monitor with Kibana
21 *GrayLog by LPG is also powered by Elasticsearch
![Page 22: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/22.jpg)
“Can you detect the slow queries on last Sunday
night?”
22
![Page 23: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/23.jpg)
Network Monitoring - PacketBeat + Kibana
● A packet sniffer that collects and scans packets
● ElasticSearcho As the storage backendo As the index/search
backend
Up & Running at TST2 master DB
23
![Page 24: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/24.jpg)
“Can you compare restaurant sales within X
distance in a given hour?”
24
![Page 25: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/25.jpg)
Graph Model
25
![Page 26: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/26.jpg)
Inside story: Comparison ReportsTechnology Stack● Search Backend: Elasticsearch
o Vertex centric indices▪ Sort & index edges per
vertexo Enables efficient focused
traversals▪ Only retrieve edges that
mattero Uses pushdown predicates
for quick index driven retrieval
26
![Page 27: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/27.jpg)
Summary
27
![Page 28: Be Elastic: Leapset Innovation session 06-08-2015](https://reader033.vdocuments.us/reader033/viewer/2022042819/55cf6717bb61ebba1d8b47c8/html5/thumbnails/28.jpg)
28