flink in a container - inovex...2017/02/21  · docker run -p 8081:8081 -p 6123:6123 -d --name...

Flink in a Container

Learn it the Caipi-Way

Nico Kreiling Karlsruhe, 21.2.2017

2

Flink in a Container

3

The Ingridients

4

If you just mix it all...

5

But bundled in a nice container...

6

And Containers can be used for many drinks

7

Why not use a VM? Source:http://stackoverflow.com/a/33132462/1806392

8

Rezept = Dockerfile

9

Let’s make a it…

Create Dockerfile

docker build –t <image> .

10

If you are to lazy to build it yourself...

docker pull <image>

11

Let’s make a it…

Create Dockerfile

docker build –t <image> .

docker run <image>

12

Thats all?

13

Scale out!

But how?

14https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/cluster_setup.html

Flink Cluster: Standalone Mode

› docker run -p 8081:8081 -p 6123:6123 -d --name jobmanager krlng/flink jobmanager

› docker run --expose 6121,6122 --link jobmanager1 krlng/flink taskmanager

› docker run --expose 6121,6122 --link jobmanager2 krlng/flink taskmanager

15

Create the pure way

› YAML Syntax

› Similar to classic Docker Commands

› Custom Networking

› Easy to re-run

16

Docker-Compose

version: "2"services:flink-jobmanager:image: krlng/flinkports:- "8081:8081"- "6123:6123"

command: jobmanager

17

Docker-Compose

flink-taskmanager:image: krlng/flinkexpose:- "6121"- "6122”

links:- flink-jobmanager

command: taskmanager

18

It‘s kind of a messBut wait, something is missing...

There we are!

19

Don‘t try that...

20

We need something to consume it

› Service discovery: Let’s talk containers to each other

› Monitoring: track containers’ and cluster health

› Configuration scripting: Load application configuration into containers

› Establish container policies: Configure resources limits

› Provisioning: spin up containers depending on their requirements

› Rolling upgrades and rollback

21Source: https://insights.hpe.com/content/hpe-nxt/en/articles/2017/02/the-basics-explaining-kubernetes-mesosphere-and-docker-swarm.html

Container Management

22

Container Orchestration

23

Kubernetes Terminoligy

Pod(==Container)

ReplicationController

Service

Namespace

24

Namespace: flink

RC: Jobmanager

Pod: Jobmanager

Pod: Taskmanager

RC: Taskmanager

Service: WebUI

Service: jobmanager

› More Applications: Using real-world problems

› More Tools: Spark, Kafka, Cassandra, ElasticSearch…

› More Automation: Build-Pipeline

25

Whats next?

26

Bringing it all together in an automated way

Kubernetes

ReplicationController

Pod

Service(+LB)

PodGitlab

WebGUI

DockerImage

RESTAPIgit push

kubectl

dockerpull

Thanks for the Attention!Time for Questions!

27

Vielen Dank

Nico Kreiling

Big Data Scientist

inovex GmbH

Ludwig-Erhard-Allee 6

76131 Karlsruhe

nico.kreiling@inovex.de

0173 3181 168