flink in a container - inovex...2017/02/21 · docker run -p 8081:8081 -p 6123:6123 -d --name...
TRANSCRIPT
Flink in a Container
Learn it the Caipi-Way
Nico Kreiling Karlsruhe, 21.2.2017
2
Flink in a Container
3
The Ingridients
4
If you just mix it all...
5
But bundled in a nice container...
6
And Containers can be used for many drinks
7
Why not use a VM? Source:http://stackoverflow.com/a/33132462/1806392
8
Rezept = Dockerfile
9
Let’s make a it…
Create Dockerfile
docker build –t <image> .
10
If you are to lazy to build it yourself...
docker pull <image>
11
Let’s make a it…
Create Dockerfile
docker build –t <image> .
docker run <image>
12
Thats all?
13
Scale out!
But how?
14https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/cluster_setup.html
Flink Cluster: Standalone Mode
› docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager
› docker run --expose 6121,6122 --link jobmanager1 krlng/flink taskmanager
› docker run --expose 6121,6122 --link jobmanager2 krlng/flink taskmanager
15
Create the pure way
› YAML Syntax
› Similar to classic Docker Commands
› Custom Networking
› Easy to re-run
16
Docker-Compose
version: "2"services:flink-jobmanager:image: krlng/flinkports:- "8081:8081"- "6123:6123"
command: jobmanager
17
Docker-Compose
flink-taskmanager:image: krlng/flinkexpose:- "6121"- "6122”
links:- flink-jobmanager
command: taskmanager
18
It‘s kind of a messBut wait, something is missing...
There we are!
19
Don‘t try that...
20
We need something to consume it
› Service discovery: Let’s talk containers to each other
› Monitoring: track containers’ and cluster health
› Configuration scripting: Load application configuration into containers
› Establish container policies: Configure resources limits
› Provisioning: spin up containers depending on their requirements
› Rolling upgrades and rollback
21Source: https://insights.hpe.com/content/hpe-nxt/en/articles/2017/02/the-basics-explaining-kubernetes-mesosphere-and-docker-swarm.html
Container Management
22
Container Orchestration
23
Kubernetes Terminoligy
Pod(==Container)
ReplicationController
Service
Namespace
24
Namespace: flink
RC: Jobmanager
Pod: Jobmanager
Pod: Taskmanager
RC: Taskmanager
Service: WebUI
Service: jobmanager
› More Applications: Using real-world problems
› More Tools: Spark, Kafka, Cassandra, ElasticSearch…
› More Automation: Build-Pipeline
25
Whats next?
26
Bringing it all together in an automated way
Kubernetes
ReplicationController
Pod
Service(+LB)
PodGitlab
WebGUI
DockerImage
RESTAPIgit push
kubectl
dockerpull
Thanks for the Attention!Time for Questions!
27
Vielen Dank
Nico Kreiling
Big Data Scientist
inovex GmbH
Ludwig-Erhard-Allee 6
76131 Karlsruhe
0173 3181 168