agenda - iscte-iulsoftware-defined-networks.iscte-iul.pt/presentations/microsoft_sdn.pdf · azure...
TRANSCRIPT
![Page 1: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/1.jpg)
![Page 2: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/2.jpg)
Agenda
![Page 3: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/3.jpg)
![Page 4: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/4.jpg)
Azure Hyper-Scale Global Infrastructure100+ Datacenters Across 46 Regions +4 new regions announced
![Page 5: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/5.jpg)
The world is divided into geographies
A region is defined by a bandwidth and latency envelope
Region 1
Region 2
![Page 6: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/6.jpg)
10s of Km
Zone 1
Zone 2
Zone 3
Rng 1
Rng 2
![Page 7: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/7.jpg)
Quincy, WA
![Page 8: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/8.jpg)
Quincy, WA
![Page 9: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/9.jpg)
Amsterdam, NL
![Page 10: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/10.jpg)
Cheyenne, WY
![Page 11: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/11.jpg)
Cheyenne, WY
![Page 12: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/12.jpg)
Gen 2
Processor 2 x 6 Core 2.1 GHz
Memory 32 GiB
Hard
Drive
6 x 500 GB
SSD None
NIC 1 Gb/s
Gen 3
Processor 2 x 8 Core 2.1 GHz
Memory 128 GiB
Hard
Drive
1 x 4 TB
SSD 5 x 480 GB
NIC 10 Gb/s
HPC
Processor 2 x 12 Core 2.4 GHz
Memory 128 GiB
Hard
Drive
5 x 1 TB
SSD None
NIC 10 Gb/s IP, 40 Gb/s IB
Gen 4
Processor 2 x 12 Core 2.4 GHz
Memory 192 GiB
Hard Drive 4 x 2 TB
SSD 4 x 480 GB
NIC 40 Gb/s
Godzilla
Processor 2 x 16 Core 2.0 GHz
Memory 512 GiB
Hard Drive None
SSD 9 x 800 GB
NIC 40 Gb/s
Gen 5.1
Processor 2 x 20 Core 2.3 GHz
Memory 256 GiB
Hard Drive None
SSD 6 x 960 GB PCIe Flash
and 1 x 960 GB SATA
NIC 40 Gb/s + FPGA
GPU Gen 5
Processor 2 x 8 Core 2.6 GHz
Memory 256 GiB
Hard Drive 1 x 2 TB
SSD 1 x 960 GB SATA
NIC 40 Gb/s
GPU 2 x 2 Compute GPU
Beast
Processor 4 x 18 Core 2.5 GHz
Memory 4096 GiB
Hard Drive None
SSD 4 x 1920 GB NVMe and
1 x 960 GB SATA
NIC 40 Gb/s
Gen 6
Processor 2 x Skylake 24 Core 2.7GHz
Memory 192GiB DDR4
Hard Drive None
SSD 4 x 9600 GB M.2 SSDs
and 1 x 960 GB SATA
NIC 40 Gb/s
FPGA Yes
![Page 13: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/13.jpg)
Intel, AMD, ARM64 CPUs
High density GPU expansion for HPC/AI
NVM (DRAM+battery) and 3DXP for low-latency
High density HDD and Flash expansion
Microsoft custom designed SSDs
50 Gbps networking
Accelerated VMs using FPGAs
![Page 14: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/14.jpg)
New industry standard design on Project Olympus for machine learning
Extreme performance scalability -Interconnectivity for up to 32 GPUs
High-density
8 – GPUs
Powered by NVIDIA Pascal and NVLINK
![Page 15: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/15.jpg)
Storage
2 servers, PCIe bus stretched rack-wide
3.5 kW/rack
2 x 40GigE to datacenter network
![Page 16: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/16.jpg)
![Page 17: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/17.jpg)
Azure
region C
Azure
WAN backbone
Azure
region B
Internet
Azure region A
DCDCDCDC
Intra-region
Regional Network Gateway
Inter-region
Backbone
SWAN – Custom 100Gb Optical
Global optical
MSFT dedicated optical network
Pure dark fiber in regions and between large regions
Private waves
![Page 18: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/18.jpg)
United States
United States
Canada
Mexico
Venezuela
Colombia
Peru
Bolivia
Brazil
Argentina
Atlanta OceanAlgeria
MaliNiger
Nigeria
Chad
Libya Egypt
Sudan
Ethiopia
Dr Congo
AngolaZambia
Nambia
South
Africa
Greenland
Svalbard
Sweden
Norway
United
Kingdom
France
PolandUkraine
Turkey
Saudi
Arabia
Iran
Kazakistan
India
Russia
Russia
China
Myanmar
(Burma)
Indian Ocean
Indonesia
Australia
Pacific Ocean
Pacific Ocean
Data centerOwned capacity
New capacity
Leased capacity
Edge site
DCs and network sites not exhaustive
![Page 19: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/19.jpg)
Microsoft
backbone
DC Region
RNG RNG
DC DC DC DC
DC DC
Regional network gateway
Massively parallel, hyper scale
DC interconnect
Space and power protected
RNG data centers
Small, Medium, or Large (T-shirt sizes)
Only contains server racks, DC network
RNGs are sized to support growing the region by adding data centers
![Page 20: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/20.jpg)
High-cost high-power Coherent:
8 Tb/s per fiber pair
Low-cost low-power Madison:
4 Tb/s per fiber pair
Allows us to cost effectively deploy 1.689 Petabits/sec of inter datacenter switching
![Page 21: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/21.jpg)
![Page 22: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/22.jpg)
Large L2
Domains
HW-based
service
modules
Simple Tree
Design
L3 at all
layers
Services in
software
Clos*-based
design
L3
L2
LB/FW LB/FW LB/FW LB/FW
Low due to diversity and manual
provisioning process
Low due to complex hardware
and lack of automated operations
Low due to high complexity and
human error
Resilient design, automated
monitoring and remediation,
minimum human involvement
Simplify requirements, optimize
design, and unify infrastructure
Automated network provisioning,
integrated processAgility
Efficiency
Availability
* Charles Clos, 1952
![Page 23: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/23.jpg)
Compute RPNetwork RP
Regional Network
Manager
Network State
Manager
Software load
balancer
Directory
Service
Compute
Controller
Physical Host
Ap
plicati
on
Co
nta
iners
Ap
plicati
on
Co
nta
iners
Network Manager
Agent
Load balancer
agent
Ap
plicati
on
Co
nta
iners
Node Agent
VM
Virtual
Filtering
Platform
VM VM
![Page 24: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/24.jpg)
Virtual NETwork
(contains subnets,
DHCP and DNS)
Network Security Group
(ACL, for NICs or VNETs)
NIC
(owns IPs, is
assigned to VNET)
Load Balancer
(Internal/External)
DNS
(Private or Public)VPN Gateway
Network Virtual Appliance
(owns NICs)
User-Defined Routes
(applied to VNETs)
Distributed processing,
Pure SDN
Guaranteed resources,
NFV-like
![Page 25: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/25.jpg)
![Page 26: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/26.jpg)
Physical Host
Virtual MachineSpecifically Optimized To Run a Container
System Processes
Application Processes
TCP/IPSoftware
System Processes
Application Processes
Host TCP/IPSoftware
Hyper-V Container
System Processes
Application Processes
TCP/IPSoftware
Physical Network
![Page 27: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/27.jpg)
Physical Host
Host TCP/IPSoftware
Docker Container
Application Processes
Physical Network
Docker Container
Application Processes
Docker Container
Application Processes
![Page 28: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/28.jpg)
Performance
Default Docker networking is slow and introduces 30-70% overhead depending on OS/kernel/versions, due to bottlenecks, repeated transitions between kernel/userspace, etc.
Transparency
TCP proxying through dockerd masks origin IP addresses, and NAT/overlay networks introduce further complications (MTUs, IP address allocation, etc.).
Scalability
Managing connectivity between multiple hosts via port mapping or NAT is just… insane.
Orchestration
Real world deployments require well-defined, open APIs that tie in to orchestrators like Swarm, Mesos and Kubernetes
CNM
(Container
Network Model)
CNI
(Container
Network
Interface)
IPAM
(address
management)
VS
![Page 29: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/29.jpg)
https://github.com/Azure/azure-container-networking
![Page 30: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/30.jpg)
Azure VNET (172.16.0.0/16)
Virtual Host 2NIC
Virtual Host 1NIC
Direct mapping to
multiple subnets:
• 172.16.10.0/24
• 172.16.20.0/24
• 172.16.30.0/24
Docker/rktCNI
Pod/Docker Container
App
Pod/Docker Container
App
Pod/Docker Container
App
Pod/Docker Container
App
Pod/Docker Container
App
Pod/Docker Container
App
kubelet kubeletDocker/rktCNI
![Page 31: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/31.jpg)
![Page 32: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/32.jpg)
Hosts are Scaling Up: 1G 10G 40G 50G 100G …?
Reduces COGS of VMs (more VMs per host) and enables new workloads
Need the performance of hardware to implement policy without CPU
Need to support new scenarios: BYO IP, BYO Topology, BYO Appliance
We are always pushing richer semantics to virtual networks
Need the programmability of software to be agile and future-proof
“How do we get the
performance of hardware
with programmability
of software?
![Page 33: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/33.jpg)
FPGA
DRAM Controller
USB Controller
Ethernet Controller
DSPSlice
RAM
RAM
DSPSlice
CPU
CPU
![Page 34: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/34.jpg)
Azure SmartNIC
Use an FPGA for reconfigurable functions
Roll out Hardware features as we do software
Programmed using Generic Flow Tables (GFT)
Language for programming SDN to hardware
Uses connections and structured actions as primitives
Deployed on all new Azure compute servers since late 2015
SmartNIC is also doing Crypto, QoS, storage acceleration, and more…
Host
SmartNIC
FPGA
Top of Rack
Switch
NIC ASIC
SmartNIC
CPU
![Page 35: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/35.jpg)
Performance
Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM by default
Transparency
Containers as first-class SDN citizens (already 2/3 of the way there)
Scalability
Kubernetes DNS/IPv6 for service discovery/connectivity across datacenter regions (already possible via VNET peering, we want to make it simpler as K8s evolves)
Orchestration
Full integration with Azure Network Resource Provider/SDN management through Kubernetes network policy APIs
![Page 36: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/36.jpg)
![Page 37: Agenda - ISCTE-IULsoftware-defined-networks.iscte-iul.pt/presentations/Microsoft_SDN.pdf · Azure Kubernetes Service (AKS) and Azure Container Instance (ACI) already use CNI and IPAM](https://reader030.vdocuments.us/reader030/viewer/2022041016/5ec7f1bc6ac82244b105bc22/html5/thumbnails/37.jpg)