yarn and the docker container runtime

1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

YARN and the Docker Container RuntimeDublin, April 2016Sidharta S


Context

LinuxContainerExecutorWhy docker with YARN?


Container Runtimes in LinuxContainerExecutor

⬢Support for Container Runtimes in YARN was added as part of – YARN-3611 (umbrella). Multiple container types are supported in the same executor.

⬢The current mechanism of handling container lifecycle is moved into its own runtime

⬢A new docker container runtime is introduced that manages docker containers

⬢LinuxContainerExecutor can delegate to either runtime on a per application basis

⬢Clients specify which container type they want to use – currently via environment variables but eventually through well-defined client APIs.

⬢We could support more container types in the future.


DockerContainerRuntime in LinuxContainerExecutor

⬢Exposes a subset of Docker container lifecycle functionality

⬢Docker v1.10.x required for some of the work being planned (e.g. user namespaces)

⬢A recent linux kernel is required (3.10+) for basic functionality. Some features (e.g. overlay fs) require an even more recent kernel.


DockerContainerRuntime : Resource Isolation

⬢Support added in YARN-4553

⬢ LinuxContainerExecutor still manages resource isolation and enforcement .

⬢Docker uses the cgroup specified by LCE ( --cgroup-parent introduced in docker v 1.6 , net_cls support added to libcontainer recently – support added in v1.9)


DockerContainerRuntime : Linux Capabilities

⬢Support added in YARN-4258

⬢Based on linux capabilities

⬢Admin controlled – cluster administrator can control which capabilities docker containers have on the cluster (yarn.nodemanager.runtime.linux.docker.capabilities)

⬢Default set is based on what docker uses by default.


DockerContainerRuntime : Privileged Containers

⬢ Support added in YARN-4262

⬢ Allows certain applications to run in docker containers – e.g. oracle

⬢ This could be a security hazard so access to this needs to be controlled :–Disabled by default (yarn.nodemanager.runtime.linux.docker.privileged-containers.allowed)–Admin controlled whitelist (yarn.nodemanager.runtime.linux.docker.privileged-containers.acl)–This whitelisted set of users allowed to launch privileged containers – but they must explicitly

request for it (YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER)


DockerContainerRuntime : Users

⬢Docker runs the container as the specified user (-u)

⬢This user needs to be available in the image being used.

⬢Depending on capabilities (CAP_SETUID), privileged escalation could occur.

⬢Docker v1.10 added support for user namespaces – requires daemon re-configuration. Support for this in DockerContainerRuntime needs to be planned.


DockerContainerRuntime : Networking

⬢Defaults to --net=host in the docker container runtime.–This is not secure – but this is the only way some applications can run. –We need to switch the default to bridged mode or an admin specified network

plugin

⬢Network plug-in support was added in v1.9 .–This should allow for more sophisticated networking scenarios–YARN doesn’t have to do anything except delegate to the specified network

plugin when launching the container.–Support for this is a work in progress (YARN-4007)

10

© Hortonworks Inc. 2011 – 2016. All Rights Reserved

DockerContainerRuntime : Images

⬢Localization via HDFS? : We could localize images from HDFS and load them using ‘docker load’.

⬢This approach has the advantage of using an existing HDFS instance for storage/distribution at scale.

⬢However : –we lose some of the optimizations/functionality that using a full-fledged docker

registry might provide.–We have to figure out security implications. What if users clobber each others

images when ‘loading’ ?

11


Demos

Spark on DockerYARN on YARN

12


Q&A

yarn and the docker container runtime

Technology