elasticsearch & docker
TRANSCRIPT
Elasticsearch &Docker
RafałKuć– Sematext Group,Inc.@kucrafal @sematext sematext.com
Running HighPerformanceFault Tolerant
ElasticsearchClusters OnDocker
Next 20minutes
You Are Probably Familiar WithThat
Development
You Are Probably Familiar WithThat
Development Test
You Are Probably Familiar WithThat
Development Test QA
You Are Probably Familiar WithThat
Development Test QA
Production enviroment
AndProblems That Come WithIt
Resources notutilized
AndProblems That Come WithIt
Resources notutilized
OverprovisonedServers
AndProblems That Come WithIt
Resources notutilized
OverprovisonedServers
≠ ≠
Thesolution
Development Test QA Production
Why Docker?
Light weight
Based onOpenStandards
Secure
Containers vsVirtualMachines
Hardware
Traditional VirtualMachine
Containers vsVirtualMachines
Hardware
HostOperatingSystem
Traditional VirtualMachine
Containers vsVirtualMachines
Hardware
HostOperatingSystem
Hypervisor
Traditional VirtualMachine
Containers vsVirtualMachines
Hardware
HostOperatingSystem
Hypervisor
Guest OS Guest OS
Traditional VirtualMachine
Containers vsVirtualMachines
Hardware
HostOperatingSystem
Hypervisor
Guest OS Guest OS
Libraries Libraries
Traditional VirtualMachine
Containers vsVirtualMachines
Hardware
HostOperatingSystem
Hypervisor
Guest OS Guest OS
Libraries Libraries
Application1 Application2
Traditional VirtualMachine
Containers vsVirtualMachines
Hardware
HostOperatingSystem
Hypervisor
Guest OS Guest OS
Libraries Libraries
Application1 Application2
Hardware
HostOperatingSystem
Traditional VirtualMachineContainer
Containers vsVirtualMachines
Hardware
HostOperatingSystem
Hypervisor
Guest OS Guest OS
Libraries Libraries
Application1 Application2
Hardware
HostOperatingSystem
Docker Engine
Traditional VirtualMachineContainer
Containers vsVirtualMachines
Hardware
HostOperatingSystem
Hypervisor
Guest OS Guest OS
Libraries Libraries
Application1 Application2
Hardware
HostOperatingSystem
Docker Engine
Libraries Libraries
Application1 Application2
Traditional VirtualMachineContainer
Why Elasticsearch?
Distributedbydesign
http://www.dailypets.co.uk/2007/06/17/kittens-rest-at-half-time/
Indices Aggs Admin
Monitor Search
Index
Running Offical ElasticsearchContainer
$docker run-delasticsearch
Running Offical ElasticsearchContainer
$docker run-delasticsearch:latest
Running Offical ElasticsearchContainer
$docker run-delasticsearch:latest$docker run-d--name es1elasticsearch
Running Offical ElasticsearchContainer
$docker run-delasticsearch:latest$docker run-d--name es1elasticsearch$docker run-d--name es1-eES_HEAP_SIZE=1gelasticsearch
Running Offical ElasticsearchContainer
$docker run-delasticsearch:latest$docker run-d--name es1elasticsearch$docker run-d--name es1-eES_HEAP_SIZE=1gelasticsearch$docker run-d--name es1elasticsearch -Dnode.name bbuzz
Container Constraints
$docker run-d-m2G elasticsearch
$docker run-d-m2G--memory-swappiness=0 elasticsearch
http://docs.docker.com/engine/reference/run/
Container Constraints
$docker run-d-m2G elasticsearch
$docker run-d-m2G--memory-swappiness=0 elasticsearch
$docker run-d--cpuset-cpus="1,3"elasticsearch
http://docs.docker.com/engine/reference/run/
Container Constraints
$docker run-d-m2G elasticsearch
$docker run-d-m2G--memory-swappiness=0 elasticsearch
$docker run-d--cpuset-cpus="1,3"elasticsearch
http://docs.docker.com/engine/reference/run/
$docker run-d--cpu-period=50000--cpu-quota=25000elasticsearch
Constraints - GoodPractices
Limitcontainermemory
Constraints - GoodPractices
Limitcontainermemory
Account forI/Ocachewhen givingmemory
Constraints - GoodPractices
Limitcontainermemory
Account forI/Ocachewhen givingmemory
Limitamount ofCPUcores
Constraints - GoodPractices
Limitcontainermemory
Account forI/Ocachewhen givingmemory
Limitamount ofCPUcores
Remember about JVMGC
Creating Optimized Image
Dockerfile:FROM elasticsearchADD ./elasticsearch.yml /usr/share/elasticsearch/config/
Creating Optimized Image
Dockerfile:FROM elasticsearchADD ./elasticsearch.yml /usr/share/elasticsearch/config/
$ docker build -t bbuzz/example .
Creating Optimized Image
Dockerfile:FROM elasticsearchADD ./elasticsearch.yml /usr/share/elasticsearch/config/
$ docker build -t bbuzz/example .
Sending build context toDockerdaemon 5.12kBStep1:FROMelasticsearch--->1e23f30a3667Step2:ADD./elasticsearch.yml /usr/share/elasticsearch/config/--->015f12adfd2aRemoving intermediate container de560c6ae0d1Successfully built 015f12adfd2a
Dealing WithNetwork
$docker run -d -p9200:9200-p9300:9300elasticsearch
Dealing WithNetwork
$docker run -d -p9200:9200-p9300:9300elasticsearch$docker run -d --linkes1 elasticsearch
-Ddiscovery.zen.ping.unicast.hosts=es1
Dealing WithNetwork
$docker run -d -p9200:9200-p9300:9300elasticsearch
Add network.publish_hostwhen buildingown container
$docker run -d --linkes1 elasticsearch-Ddiscovery.zen.ping.unicast.hosts=es1
Dealing WithNetwork
$docker run -d -p9200:9200-p9300:9300elasticsearch
Add network.publish_hostwhen buildingown containerAdd discovery.zen.ping.unicast.hostswhen buildingown
container
$docker run -d --linkes1 elasticsearch-Ddiscovery.zen.ping.unicast.hosts=es1
Network- Good Practices
Separate networkforElasticsearchcluster
Network- Good Practices
Separate networkforElasticsearchcluster
Common hostnames forcontainers$docker run -d -hesnode1elasticsearch
Network- Good Practices
Separate networkforElasticsearchcluster
Common hostnames forcontainers$docker run -d -hesnode1elasticsearch
Exposeonly needed ports
Network- Good Practices
Separate networkforElasticsearchcluster
Common hostnames forcontainers$docker run -d -hesnode1elasticsearch
Exposeonly needed ports
Elasticsearchdata &client nodes pointtomasters only
Dealing WithStorage
Bydefault in/usr/share/elasticsearch/data
Dealing WithStorage
Bydefault in/usr/share/elasticsearch/data
Bydefault notpersisted
Dealing WithStorage
Bydefault in/usr/share/elasticsearch/data
Bydefault notpersisted
$docker run -d-v /opt/elasticsearch/data:/usr/share/elasticsearch/dataelasticsearch
Dealing WithStorage
$docker run -d-v /opt/elasticsearch/data:/usr/share/elasticsearch/dataelasticsearch
Bydefault in/usr/share/elasticsearch/data
Bydefault notpersisted
Use dataonly Dockervolumes
Permissions
DataOnly Docker Volumes
Bypasses UnionFileSystem
DataOnly Docker Volumes
Bypasses UnionFileSystem
Can beshared between containers
DataOnly Docker Volumes
Bypasses UnionFileSystem
Can beshared between containers
Datavolumes persist if thecontainer itself is deleted
DataOnly Docker Volumes
Bypasses UnionFileSystem
Can beshared between containers
Datavolumes persist if thecontainer itself is deleted
$docker create -v/mnt/es/data:/usr/share/elasticsearch/data--name esdata elasticsearch
Permissions
DataOnly Docker Volumes
Bypasses UnionFileSystem
Can beshared between containers
Datavolumes persist if thecontainer itself is deleted
$docker create -v/mnt/es/data:/usr/share/elasticsearch/data--name esdata elasticsearch
$docker run --volumes-fromesdata elasticsearch
HighlyAvailable Cluster
Masteronly
Masteronly
Masteronly
Dataonly
Dataonly
Dataonly
Dataonly
Dataonly
Dataonly
Clientonly
Clientonly
HighlyAvailable Cluster
Masteronly
Masteronly
Masteronly
Dataonly
Dataonly
Dataonly
Dataonly
Dataonly
Dataonly
Clientonly
Clientonly
minimum_master_nodes=N/2+1
MasterNodes &Docker
$docker run-d elasticsearch-Dnode.master=true-Dnode.data=false-Dnode.client=false
ClientNodes &Docker
$docker run-d elasticsearch-Dnode.master=false-Dnode.data=false-Dnode.client=true
DataNodes &Docker
$docker run-d elasticsearch-Dnode.master=false-Dnode.data=true-Dnode.client=false
Multiple Tiers
node.tag=hot node.tag=cold node.tag=cold
Multiple Tiers
$docker run-d elasticsearch -Dnode.tag=hot
Multiple Tiers
curl -XPUT 'localhost:9200/data_2016-06-05' -d '{ "settings": { "index.routing.allocation.include.tag" : "hot"
}}'
Multiple Tiers
node.tag=hot node.tag=cold node.tag=cold
data_2016-06-05
data_2016-06-05
Multiple Tiers
node.tag=hot node.tag=cold node.tag=cold
data_2016-06-05
data_2016-06-05
TopMetrics – Health &Shards
https://sematext.com/spm/
TopMetrics - CPU
https://sematext.com/spm/
TopMetrics – MemoryUsage
https://sematext.com/spm/
TopMetrics – I/OUsage
https://sematext.com/spm/
TopMetrics – JVMHeap
https://sematext.com/spm/
TopMetrics – Garbage Collector
https://sematext.com/spm/
TopMetrics – Request Rate &Latency
https://sematext.com/spm/
TopMetrics - Caches
https://sematext.com/spm/
TopMetrics – Indexing
https://sematext.com/spm/
TopMetrics – Refresh Time
https://sematext.com/spm/
Top10Metrics – Merge Time
https://sematext.com/spm/
Short summary
http://www.soothetube.com/2013/12/29/thats-all-folks/
WeAre Hiring !
Dig Search ?Dig Analytics?Dig BigData?Dig Performance?Dig Logging ?Dig working withandinopen– source ?We’re hiring world – wide !
http://sematext.com/about/jobs.html