elasticsearch in production (london version)
DESCRIPTION
Elasticsearch in production, or an overview of things you want to know about before happening upon them in production.TRANSCRIPT
![Page 3: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/3.jpg)
Who?
Co-founder of Found AS 8+ years search, 3+ Elasticsearch
Herding hundreds of Elasticsearch clusters
![Page 4: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/4.jpg)
Agenda
![Page 5: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/5.jpg)
Agenda• Anti-patterns
• Memory / Resource Usage
• Distributed problems
• Security
• Client concerns
• Changing a cluster
![Page 6: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/6.jpg)
found.no/foundation
Elasticsearch in Production Elasticsearch as a NoSQL Database
Intro to Function Scoring All About Analyzers
Securing your Elasticsearch Cluster
![Page 7: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/7.jpg)
![Page 8: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/8.jpg)
![Page 9: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/9.jpg)
![Page 10: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/10.jpg)
![Page 11: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/11.jpg)
Snapshot / Restore
Circuit breakersDocument values
Aggregations
Distributed percolation
Suggesters
…
![Page 12: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/12.jpg)
Anti-Patterns
![Page 13: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/13.jpg)
Arbitrary Keys
• “Schema Free”
• One field per value
• Ever-growing cluster state
acls: 1234: READ 42: WRITE
![Page 14: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/14.jpg)
Heavy Updating
• Update = Delete + Reindex
• Be careful with counters
![Page 15: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/15.jpg)
Slow queries
• WHERE foo ILIKE ‘%bar%’
• {“query_string”: {“query”: “foo:*bar*”}}
• Don’t ask for 3300 results :)
![Page 16: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/16.jpg)
Arbitrary searchesquery: filtered: filter: term: user_id: 42 query: [user’s query here]
![Page 17: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/17.jpg)
Memory
![Page 18: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/18.jpg)
Memory• Field caches
• Filter caches
• Page caches
• Aggregations
• Index building
![Page 19: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/19.jpg)
Page Cache
• Keeping index pages in memory
• Can’t have too much
• Outgrow: Gradual slowdown
![Page 20: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/20.jpg)
Heap Space
• Memory used by Elasticsearch process
• Field / Filter caches
• Aggregations
![Page 21: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/21.jpg)
![Page 22: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/22.jpg)
Time Bomb
![Page 23: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/23.jpg)
Time Bomb
![Page 24: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/24.jpg)
OutOfMemoryError
Woah there I ate all the memories
Your cluster may or may not work any more
![Page 25: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/25.jpg)
OutOfMemory
• Growing too big
• Selecting too big timespan in Kibana
• Document ingestion peak
![Page 26: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/26.jpg)
Preventing OOMs• Have enough memory :-)
• Understand your search’s memory profile
• Bulk / Circuit breaker settings
• Monitoring
• Document values
![Page 27: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/27.jpg)
Marvel( /_stats )
![Page 28: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/28.jpg)
![Page 29: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/29.jpg)
![Page 30: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/30.jpg)
![Page 31: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/31.jpg)
"my_field": { "type": "string", "fielddata": { "format": "doc_values" } }
![Page 32: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/32.jpg)
Document Values
• Rely on page cache
• Only caches doc values actually used
![Page 33: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/33.jpg)
Sizing
![Page 34: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/34.jpg)
Sizing
• Test, don’t guess
• Start big, scale down
• Index, search, monitor
![Page 35: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/35.jpg)
![Page 36: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/36.jpg)
![Page 37: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/37.jpg)
![Page 38: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/38.jpg)
Glitch Meltdown
![Page 39: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/39.jpg)
Glitch Meltdown
![Page 40: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/40.jpg)
![Page 41: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/41.jpg)
![Page 42: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/42.jpg)
• Tie-breaker can be a cheap master-node
• Applies to data centers / availability zones too
![Page 43: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/43.jpg)
Data-only nodes
Master-only nodes
![Page 44: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/44.jpg)
![Page 45: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/45.jpg)
Jepsen
![Page 46: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/46.jpg)
Jepsen
• Kyle Kingsbury’s series on distributed systems
• Distributed systems are hard
• aphyr.com
![Page 47: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/47.jpg)
Security
![Page 48: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/48.jpg)
Security
• “Not my job!” – Elasticsearch
• That’s fine!
![Page 49: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/49.jpg)
Dynamic Scripts
!
• Scoring
• Aggregations
• Updating
![Page 50: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/50.jpg)
Dynamic Scripts
Runtime.getRuntime().exec(…)
![Page 51: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/51.jpg)
Dynamic Scripts
Runtime.getRuntime().exec(…)
<script src=“http://127.0.0.1:9200/_search?callback=capture&…
![Page 52: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/52.jpg)
Security
!
• Disable dynamic scripts (On by default in ≤1.1)
• Mind index patterns
• Even then, don’t accept arbitrary requests
![Page 53: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/53.jpg)
Client Concerns
![Page 54: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/54.jpg)
Client Concerns
• Connection pools
• Idempotent requests
• Have sane syncing/indexing strategies
![Page 55: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/55.jpg)
![Page 56: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/56.jpg)
# BOOM !
![Page 57: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/57.jpg)
Cluster changes
![Page 58: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/58.jpg)
Cluster changes
• Make new nodes join existing cluster
• No rolling restarts
• Easy rollback if things go bad
![Page 59: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/59.jpg)
v1.0.0 v1.0.1
![Page 60: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/60.jpg)
Cluster changes
• Test first
• Mind recover_*-settings
![Page 61: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/61.jpg)
Multi-Cluster Workflows
• Snapshot/Restore
• Operations across clusters
• Swap clusters!
• Works well with good syncing strategy
![Page 62: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/62.jpg)
• Rolling restarts: Risky, fast
• Grow and shrink: Less risky, copies lots of data
• Multiple clusters: Least risky, copies lots of data
![Page 63: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/63.jpg)
Misc
• Same JVM
• ulimits
• Unicast
• Kernel-settings like IO-scheduler
![Page 64: Elasticsearch in Production (London version)](https://reader033.vdocuments.us/reader033/viewer/2022042813/53fdea558d7f72a81c8b4bcc/html5/thumbnails/64.jpg)
?
@foundsays