New Docker Network Drivers: Macvlan & Ipvlan

Brent Salisbury - @networkstaticJohn Willis - @botchagalupeDocker Inc. at #ONS2016 - 3/16/2016

Macvlan Bridge & Ipvlan L2

• Very practical. No Unicorns required but cats welcome.• Great for both existing and new networks.• Native to Linux• Lightweight• Extremely Fast• No NAT/PAT

• Docker Macvlan and Ipvlan Experimental Readme:github.com/docker/docker/blob/master/experimental/vlan-networks.md

• Kernel docs on Macvlan and Ipvlan:kernel.org/doc/Documentation/networking/ipvlan.txt

Getting Started• Download the experimental binary$ wget https://experimental.docker.com/builds/Linux/x86_64/docker-latest$ chmod +x ./docker-latest

# Start the Docker engine daemon$ ./docker-latest daemon

# Verify running version$./docker-latest -vDocker version 1.11.0-dev, build ..., experimental

• Build from source$ git clone https://github.com/docker/docker.git$ cd docker$ DOCKER_EXPERIMENTAL=1 make binary

• Note on VirtualBox: If using, the bridge mode interfaces can be flaky. VBox NAT mode interface is the path of least promiscuous pain

• Vmware Fusion: works out of the box with both modes.

Bridge/L2 Modes

$ ip routedefault via 172.16.86.2 dev eth0192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.251172.16.0.0/16 dev eth0 proto kernel scope link src 172.16.86.151

$ ip a show eth02: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP link/ether 00:50:56:2b:29:40 brd ff:ff:ff:ff:ff:ff inet 172.16.86.151/16 brd 172.16.255.255 scope global eth0 valid_lft forever preferred_lft forever

Pre-Requisites Subnet+Gateway• For Macvlan Bridge Mode and Ipvlan L2 modes, get some

details about the existing network.

Macvlan Bridge Mode

# Create a Docker Network Using the Macvlan Driver

$ docker network create -d macvlan \ --subnet=172.16.86.0/24 \ --gateway=172.16.86.2 -o \ parent=eth0 mcv

# Ping the Internetz.

$ docker run --net=mcv -it --rm alpine ping -c 4 8.8.8.8

PING 8.8.8.8 (8.8.8.8): 56 data bytes64 bytes from 8.8.8.8: seq=0 ttl=128 time=3.455 ms64 bytes from 8.8.8.8: seq=1 ttl=128 time=15.909 ms64 bytes from 8.8.8.8: seq=2 ttl=128 time=7.843 ms

--- 8.8.8.8 ping statistics ---3 packets transmitted, 3 packets received, 0% packet lossround-trip min/avg/max = 3.455/9.069/15.909 ms

Macvlan Bridge Mode

Ipvlan L2 Mode

# Create a Docker Network Using the Macvlan Driver

docker network create -d ipvlan \ --subnet=192.168.1.0/24 \ --gateway=192.168.1.1 \ -o ipvlan_mode=l2 \ -o parent=eth0 db_net

# Start a container on the db_net networkdocker run --net=db_net -it --rm alpine /bin/sh

Ipvlan L2 Mode

$ docker run --net=mcv --ip=172.168.86.10 -it --rm alpine /bin/sh

Do Whatever You Want

As of Docker v1.10 users can set container IP addresses explicitly.

IPAM

### Network macvlan with --ip-range

$ docker network create -d macvlan \ --subnet=192.168.32.0/24 \ --ip-range=192.168.32.128/25 \ --gateway=192.168.32.254 \ -o parent=eth1 mcv

$ docker run --net=mcv -it --rm alpine /bin/sh

# View the address in the container$ ip a | grep 192 inet 192.168.32.128/24 scope global eth0

# View the gateway you explicitly set$ ip routedefault via 192.168.32.254 dev eth0192.168.32.0/24 dev eth0 src 192.168.32.128

• There are a lot of features in the default IPAM plugin, here are a couple.

Note: The addresses are not NATed. All addresses whether RFC 1918 or publicly routable addresses are sent as the src_ip out the parent interface.

Moar IPAM# Network exclude eth0 192.168.41.2 # address from IPAM with --aux-address# eth0 in --aux-address=exclude1=192.168.41.2 # key/IP ${key} can be named anything# Example: —aux-address=“favorite_ip_ever_ever=192.168.31.2”

$ docker network create -d macvlan \ --subnet=192.168.41.0/24 \ --aux-address="favorite_ip_ever=192.168.41.2" \ --gateway=192.168.41.1 \ -o parent=eth0 macnet41

# First address is the specified gateway, second is aux$ docker run --net=macnet41 -it --rm alpine /bin/sh

# Check the IP$ ip a show eth0 | grep 192 inet 192.168.41.3/24 scope global eth0

int gig 0/1 switchport trunk encapsulation dot1q switchport trunk allowed vlan 10,20,30 switchport mode trunk

:-)

802.1Q Trunking

VLANs

Manually Creating IP Links# create a new sub interface tied to dot1q vlan 40ip link add link eth0 name foo type vlan id 40

# enable the new sub-interfaceip link set foo up

# now add networks and hosts as you would normally by # attaching to the master (sub)interface that is taggeddocker network create -d ipvlan \ --subnet=192.168.40.0/24 --gateway=192.168.40.1 \ -o parent=foo ipvlan40

# in two separate terminals, start a Docker container# and the containers can now ping one another.docker run --net=ipvlan40 -it --name ivlan_test5 --rm alpine /bin/shdocker run --net=ipvlan40 -it --name ivlan_test6 --rm alpine /bin/sh

Automated 802.1q Trunk Provisioning# View Links prior to network create `ip link`$ ip link

# Create multiple macvlan bridge subnets using a sub-interface eth0.215 and VLAN ID 215docker network  create  -d macvlan \    --subnet=192.168.215.0/24 \    --subnet=192.168.217.0/24 \    --gateway=192.168.215.1  \    -o parent=eth101 \     -o macvlan_mode=bridge macnet215

# View Links after to network create `ip link`$ ip link

# Test 192.168.215.0/24 connectivitydocker run --net=macnet215 --ip=192.168.215.10 -itd alpine /bin/shdocker run --net=macnet215 --ip=192.168.215.9 -it --rm alpine ping -c 2 192.168.215.10

# Test 192.168.217.0/24 connectivitydocker run --net=macnet215 --ip=192.168.217.10 -itd alpine /bin/shdocker run --net=macnet215 --ip=192.168.217.9 -it --rm alpine ping -c 2 192.168.217.10

# Delete All Containers$ docker rm -f `docker ps -qa`

# Delete all Networks$ docker network rm $(docker network ls -q)

# Run ip links again and verify the links are cleaned up$ ip link

Ipvlan L3 Mode

Really, Whatever You Want# Dual Stack Ipvlan L3 mode with an interface # specified using a dummy interface# gateways IPs are ignored: (default dev eth0)# no ARP/Broadcasts allowed

$ docker network create -d ipvlan \ --subnet=192.168.8.0/24 \ --subnet=192.168.9.0/24 \ --subnet=fded:7a74:dec4:5a18::/64 \ --subnet=fded:7a74:dec4:5a19::/64 \ -o ipvlan_mode=l3 \ dualstack

Start Some Targets# Start containers on 192.168.8.0/24 & 7a74:dec4:5a18::/64docker run --net=dualstack --ip6=fded:7a74:dec4:5a18::81 -itd alpine /bin/shdocker run --net=dualstack --ip=192.168.8.80 -itd alpine /bin/shdocker run --net=dualstack --ip=192.168.8.81 --ip6=fded:7a74:dec4:5a18::80 -itd alpine /bin/sh

# Start containers on 192.168.9.0/24 & 7a74:dec4:5a19::/64docker run --net=dualstack --ip6=fded:7a74:dec4:5a18::91 -itd alpine /bin/shdocker run --net=dualstack --ip=192.168.9.90 -itd alpine /bin/shdocker run --net=dualstack --ip=192.168.9.91 --ip6=fded:7a74:dec4:5a18::90 -itd alpine /bin/sh

# Start containers on a mix of the v4/v6 networks createdocker run --net=dualstack --ip=192.168.9.100 --ip6=fded:7a74:dec4:5a18::100 -itd alpine /bin/shdocker run --net=dualstack --ip=192.168.8.100 --ip6=fded:7a74:dec4:5a19::100 -itd alpine /bin/sh

Ipvlan L3 things it shouldn't be able to do

# Ping from one v6 subnet to another enabled by L3 modedocker run --net=dualstack --ip6=fded:7a74:dec4:5a19::25 -it --rm alpine ping6 -c 2 fded:7a74:dec4:5a18::81docker run --net=dualstack --ip6=fded:7a74:dec4:5a19::25 -it --rm alpine ping6 -c 2 fded:7a74:dec4:5a18::100# Ping from one v6 subnet to another enabled by L3 modedocker run --net=dualstack --ip6=fded:7a74:dec4:5a18::25 -it --rm alpine ping6 -c 2 fded:7a74:dec4:5a18::91docker run --net=dualstack --ip6=fded:7a74:dec4:5a18::25 -it --rm alpine ping6 -c 2 fded:7a74:dec4:5a19::100

# Ping from one v4 inside a subnet and to another enabled by L3 modedocker run --net=dualstack --ip=192.168.8.25 -it --rm alpine ping -c 2 192.168.8.80docker run --net=dualstack --ip=192.168.8.25 -it --rm alpine ping -c 2 192.168.9.91

# Ping from one v4 inside a subnet and to another enabled by L3 modedocker run --net=dualstack --ip=192.168.9.25 -it --rm alpine ping -c 2 192.168.9.91docker run --net=dualstack --ip=192.168.9.25 -it --rm alpine ping -c 2 192.168.8.80

Create 50+ networks & 125+ Containers in < 60 seconds

- Requires an interface named eth0 or set the ENV for $ETH or- modify script ETH=${ETH:-eth0}

$ curl -o vlan-tests.sh \ https://raw.githubusercontent.com/nerdalert/dotfiles/master/ipvlan-macvlan-it.sh && \ chmod +x vlan-tests.sh

$ ./vlan-tests.sh

Networks are created twice to validate add/del functionality

Really Fast!

• Skunkworks repo to Dockerize network tools, all welcome to contribute!https://github.com/gopher-net/dockerized-net-tools

$ docker run -it --rm gophernet/nmap -sT 192.168.1.1

Unable to find image 'gophernet/nmap:latest' locallylatest: Pulling from gophernet/nmap7268d8f794c4: Pull completea3ed95caeb02: Pull completeb45e16452ecd: Pull completeDigest: sha256:de08ac219d9d665beaad55f8796c85aba44dafcfc64ba4cbf3d53e8e62b2d95aStatus: Downloaded newer image for gophernet/nmap:latest

Starting Nmap 6.47 ( http://nmap.org ) at 2016-03-16 23:43 UTC

Network Tooling

# nmap in a container# A couple of example usages:# $ docker run -it --rm networkstatic/nmap --help# Scan for open ssh (tcp/22) ports on a range of IPs# $ docker run -it --rm networkstatic/nmap -sT 192.168.1.1-100 -p 22 #FROM debian

MAINTAINER Brent Salisbury <brent.salisbury@gmail.com>

# build initial cache | install binary | remove cacheRUN apk update && apk add \

nmap \&& rm -rf /var/cache/apk/*

ENTRYPOINT ["nmap"]

Network Tooling w/ Docker on HW Switches

• Do you know what your network is doing? • Run and manage apps on switches without dependency nightmares

• drill is a tool from lens that is a replacement of dig.• fping - tool for measuring latency, status and all around ping on steroids. • hping is useful for both scanning networks and crafting packets.• iperf - extremely versatile tool for measuring network bandwidth and performance. • mz Mausezahn is a fast traffic generator which allows you to send nearly any kind of packet.• nmap - security scanner, port scanner and network discovery tool• netcat - security scanner, port scanner and network discovery tool• netflow generator - generate generic NetFlow data and send it to the specified IP/Port of the NetFlow collector.• sflowtool - sFlow collector• traceroute print the route that IP packets traverse going to a remote host.• traceroute6 print the route IPv6 packets will take to a network node.

Network Tooling

Questions?