erez cohen & aviram bar haim, mellanox - enhancing your openstack cloud with advanced network...

© 2014 Mellanox Technologies 1- Mellanox Confidential -

Erez Cohen, Sr. Director, Cloud ProgramAviram Bar-Haim, Cloud Solutions Engineer, SW

CloudX: The most efficient and scalable cloud solutionOpenStack Berlin, June 2015


We Live in a World of Data

Data Needs to be Accessible Always and in Real-Time

Exponential Data Growth – The Best Platforms Are Needed

More DataMore ApplicationsMore Devices


Exponential Data Growth – Few Facts

▪More than a 1.44B active Facebook users ~ 1/5 of the earth

population.

▪Every second 1157 people start watching YouTube videos

(100,000,000 videos a day).

▪People currently create 1.2 trillion GB of data a year (Equals to

10 iPODs for each person on earth).

▪Average US household has 5.7 internet connected devices (More

than human bodies).


Exponential Data Growth – Best Interconnect Required

0.8 Zettabyte

200935 Zettabyte

2020

44X

Source: IDC


The Future Depends on the Fastest and Efficient Interconnects

10Gb/s 40/56/100Gb/s1Gb/s


Exponential Data Growth – Conclusion

▪We can infer two main conclusions from this data growth:

1. Fast network interconnect is critical for efficient and scalable cloud

2. Efficient storage solutions are necessary to handle all future data


Agenda

▪Interconnect technology evolution in virtualized environments

• User-space networking overview (TCP/UDP -> RDMA)

• Virtual interfaces overview (emulation -> paravirt -> SRIOV)

- Using SR-IOV in Openstack Neutron project

• Storage protocols and performance (FC -> iSCSI -> iSER)

- Using iSER in Openstack Cinder project

• Overlay network scalability and performance (VLAN -> VXLAN)

- Using VXLAN offloading in Openstack Neutron project


Agenda










TCP/UDP -> Remote Direct Memory Access (RDMA)

RDMA over IB / Ethernet


Application / Middleware

Thread

User-space Networking using RDMA - Overview

▪HW managed by kernel• Applications create resource via system calls, e.g.- Queues (descriptor rings)- Registering memory buffers

▪Data-path bypasses kernel• Shared queues between application and HW• HW accesses registered buffers directly• Direct signaling mechanism (“doorbells”)• Direct completion detection- In memory polling- Can also register for event (interrupt)

▪Multiple HW resources• No need for locking if resources are accessed by a

single thread

▪Efficient• Asynchronous progress• Zero copy

HW kernel driver

User

Kernel

NIC

RegisteredMemory

ThreadS

end

Q

Recv Q

Co

mp

Q

HW user-space driver

Access library


User-space Networking using RDMA – Packet Interfaces

▪Verbs Raw Ethernet Queues as an example▪Basic data path operations

• Post packet buffers to be sent out• Post buffers to receive packets• Poll for completions

▪Checksum offloads for TCP/UDP• Insert checksum on TX• Validate checksum on RX

▪VLAN insertion/stripping▪Receive-side scaling (RSS)

• Distribute incoming packets into multiple queues• Distribution is semi-random (hash based)

▪Flow steering• Deterministic steering of specific flows to specific RQs

▪Deliver very high packet rate to the application• E.g., 25Mpps for 64b packets

Application / Middleware

Thread

HW kernel driver

User

Kernel

NIC

RegisteredMemory

ThreadS

end

Q

Recv Q

Co

mp

Q

HW user-space driver

Access library


User-Space Networking – RDMA Infinband / Ethernet Interfaces

▪Pass messages instead of packets• Up to 2GB in size

▪Semantics• Channel (Message passing)- Requestor provides source buffer- Responder provides receive buffer

• Remote Direct Memory Access (RDMA)- Requestor provides both source and target buffers- Both RDMA-read and –write are supported

▪Advanced RDMA operations• Atomics- Compare & swap- Fetch & add- Multi-field

• Data integrity

▪Extreme performance• 700ns one-way latency between applications• 40GE BW at negligible CPU utilization• Packet rate > 35Mpps

Send Queue

Receive Queue

Send Queue


Agenda










Host

Virtual Interfaces – Device Emulation

▪Host emulates a complete HW device• E.g., Intel e1000 NIC

▪Guest runs unmodified driver

▪Pros• No need to install special drivers in guests• Transparent migration• Unlimited virtual interfaces

▪Cons• Slow• Emulation exists only for very simple devices• High overhead

Qemu process

VM

e1000 driver

e1000 emulator

SW Switch

macvtapnetdev

physnetdev

User

Kernel

User

Kernel

NIC


Virtual Interfaces – Para-Virtualization

▪Host exposes a virtual “SW-friendly” device• E.g., virtio-net

▪VM runs special device driver

▪Host emulates device back-end

▪Pros• Decent performance• Transparent migration• Unlimited virtual interfaces

▪Cons• Simple devices only

Host Qemu process

VM

virtio-net

virt-io emulator

SW Switch

macvtapnetdev

physnetdev

User

Kernel

User

Kernel

NIC


Virtual Interfaces – Accelerated Para-Virtualization

▪Same para-virtual control interface

▪Fast path offloaded to host kernel• vhost_net

Host Qemu process

VM

virtio-net

virt-io control

SW switch

macvtapnetdev

physnetdev

User

Kernel

User

Kernel

vhost-net

NIC


Virtual Interfaces – Physical Device Pass-Through

▪Hosts grants guest direct access to a physical device• Security and isolation still maintained- PCI configuration space is virtualized- IOMMU governs DMA access

▪VM runs standard device driver

▪Pros• Near-native performance• VMs can use any device that is passed to them

▪Cons• No transparent migration• Very limited scalability (physical devices are not

shared)

Host Qemu process

VM

HW driver

User

Kernel

User

Kernel

HW driver

NIC 1 NIC 2

Switch


NIC

Virtual Interfaces – Virtual Device Pass-Through (SR-IOV)

▪Single Root I/O Virtualization (SR-IOV)

▪Hosts grants guest direct access to a virtual device• Security and isolation still maintained- PCI configuration space is virtualized- IOMMU governs DMA access

▪VM runs device driver for virtual function

▪Pros• Near-native performance• High scalability (128-256 VFs)

▪Cons• No transparent migration

Host Qemu process

User

Kernel

PF driver

Phyiscal Function

Virtual Function

Embedded switch

Switch

VM

VF driver

User

Kernel


Agenda










▪The Modular Layer 2 (ML2) Plugin is a framework allowing OpenStack Neutron to simultaneously utilize the variety of layer 2 networking technologies found in complex real-world data centers

▪Mellanox ML2 Mechanism Driver• Adds support for SRIOV in a transparent manner• Upon port creation and binding, the driver will allocate VF and connect it to the VM• The driver will also configure the internal embedded switch

▪The Mellanox Mechanism Driver can also support InfiniBand network for OpenStack in a transparent manner • Convert MAC/VLAN to GUID/PKEY

Mellanox Neutron Plugin – Icehouse Release


Architecture Block Diagram – ML2 based neutron

OpenStackCloud Manager

OpenvSwitch Agent

ConnectX

Controller

Compute

Mellanox neutron Agent

RPC

eswitchDOVS

ConnectX

Mellanox neutron Agent

eswitchD

RPC

Neutron

ML2

MLNXOVS

Compute


SR-IOV in Openstack Juno/Kilo

▪Starting from Juno, a standard SRIOV NIC switch neutron mechanism driver is supported

upstream

• Offers a standard way to create a direct SRIOV neutron port from a virtual function

▪Two ways to configure SR-IOV in OpenStack environment:

• SRIOVNICSwitch Mechanism Driver: https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking

• MLNX mechanism Driver (support Eth & IB): https://wiki.openstack.org/wiki/Mellanox-Neutron-ML2

https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking

https://wiki.openstack.org/wiki/Mellanox-Neutron-ML2


OpenStack SRIOV demo with 100G or similar


Agenda










Storage Area Networks – Fibre Channel (1997) VS. iSCSI (2000)

▪Fibre Channel provides low latency and high availability. But Fibre Channel can be expensive and complex to manage.

▪iSCSI (IP based SCSI) uses Ethernet as its underlying communications fabric, so it is much less expensive than Fibre Channel.

▪As Ethernet continues to advance, iSCSI advances right along with it.

▪In 2007, Fibre Channel support for Ethernet networks (FCoE), proposed a more standard way to use Fibre Channel (with reduced performance..).


What is iSER?

▪iSER – iSCSI Extensions for RDMA▪Maps the iSCSI protocol over RDMA fabrics ▪Leverages iSCSI management infrastructure▪The transport layer iSER and/or iSCSI/TCP are transparent

to the user. Just need a simple configurable to decideEthernet (RoCE) / InfiniBand

SCSI Application Layer

iSCSI

iSER

SCSI Layer

Verbs APINet IF (Ethernet/IPoIB)

iSCSI

TCPiSNS/SLP

IB HCARNICNIC

Management

TOE iSER

Datamover API


iSER Protocol Overview (Read)

▪ SCSI Reads • Initiator Send Command PDU (Protocol data unit) to Target • Target return data using RDMA Write• Target send Response PDU back when completed transaction• Initiator receives Response and complete SCSI operation

iSC

SI

Init

iato

r

iSE

R

HC

A

HC

A

iSE

R T

arge

t

Tar

get

Sto

rage

Send_Control (SCSI Read Cmd)

RDMA Write for Data

Send_Control + Buffer advertisement

Control_Notify

Data_Put (Data-In PDU) for Read

Control_NotifySend_Control (SCSI Response)

Send_Control (status, sense data)


iSER Protocol Overview (Write)

▪SCSI Writes

• Send Command PDU (optionally with Immediate Data to improve latency)

• Map R2T to RDMA Read operation (retrieve data)

• Target send Response PDU back when completed transaction

iSC

SI

Init

iato

r

iSE

R

HC

A

HC

A

iSE

R T

arge

t

Tar

get

Sto

rage

Send_Control (SCSI Write Cmd)

RDMA Read for Data (Optional)

Send_Control + Buffer advertisement Control_Notify

(SCSI Command)

Get_Data (R2T PDU)

Control_NotifySend_Control (SCSI Response)

Send_Control (status, sense data)


Performance


Conclusion: iSER Is the FASTEST Networked Block Protocol

▪What it is: iSCSI With RDMA Transport• Runs over Ethernet or InfiniBand at 10, 40, 56Gb/s &100Gb/s

• Works with all applications that support SCSI/iSCSI

▪Benefits• Most bandwidth, Highest IOPs, Lowest Latency, Lowest CPU utilization

• iSCSI storage features, management and tools (security, HA, discovery...)

• Faster than iSCSI, FC, FCoE; Easier to manage than SRP

▪Ideal For• SSD / nvmE

• Latency-sensitive workloads; Small, random I/O

• Databases, Virtualization (especially VDI)


Need FibreChannel Protocol

Need Highest Performance

Want Ethernet

Protocol / Transport Comparison

InfiniBand FCoE Fibre ChannelRoCE

Transport InfiniBand RoCE Std. Ethernet FCoE FC

Speed Up to 100 Gb/s Up to 100 Gb/s Up to 100 Gb/s Up to 100 Gb/s 8/16 Gb/s

RDMA Yes Yes No No No

Routable Yes Yes Yes No No

SMB Direct

iSER

SMB, NFS, or iSCSI on TCP

FCP

NFSoRDMA


Storage Protocol Comparison

Storage Protocol

iSER FC FCoE iSCSI TCP

SMB Direct NFSRDMA

NVMe over fabrics

Access Block Block Block Block File File Block

Transports RoCE, IB FC DCBx Ethernet Ethernet RoCE, IB RoCE, IB RoCE, IB

RDMA Yes No No No Yes1 Yes2 Yes3

1. SMB Direct is an in-box option for SMB 3.0 when the network supports RDMA with InfiniBand, RoCE, or iWARP2. NFS over RDMA has limited ecosystem support but is improving rapidly. It supports InfiniBand and is expected to support RoCE and iWARP3. NVMe over Fabrics is a proposed standard (expected to be finalized in 2015) that will support InfiniBand, RoCE, and iWARP

BlockFile

RDMA

FC

SMB Direct

NFSoRDMA

SMB/CIFS

NFS iSER

NVMe over Fabrics

iSCSI over TCP

FCoE


Agenda










Mellanox Accelerate OpenStack Cinder Storage

▪iSER driver using TGT / LIO based targets

integrated in Cinder since Havana release

Hypervisor (KVM)

OS

VMOS

VMOS

VM

Adapter

Open-iSCSI w iSER

Compute Servers

Switching Fabric

iSCSI/iSER Target (tgt)

Adapter Local Disks

RDMA Cache

Storage Servers

OpenStack (Cinder)

Utilizing OpenStack Built-in components and management Tools to accelerate storage access

>4X Faster


Requirements to Deploy iSER

▪Application(s) that can use SCSI/iSCSI• All applications that use SCSI-based block storage work with iSER

▪with OS or Hypervisor that Supports an iSER initiator• Today: Linux & VMware ESXi, Oracle Solaris

• Expected soon: Windows, FreeBSD

▪iSER Storage Target• Oracle ZFS, Violin Memory, Zadara, Saratoga Speed, HP SL4500 (More coming!)

• Create in Linux using LIO, TGT, or SCST target

▪Network that supports RDMA• Adapters support InfiniBand or RoCE

• Switches support InfiniBand or DCBx with PFC

Mellanox Switches support DCBx Ethernet (with PFC) and/or InfiniBand

Mellanox ConnectX-3 Pro and ConnectX-4 Adapters support DCBx Ethernet (with PFC) and/or InfiniBand


[DEFAULT]enabled_backends = default, iser, iser_oldiscsi_helper = tgtadm # This is the default..

[default]volume_group = stack-volumes-defaultvolume_driver = cinder.volume.drivers.lvm.LVMVolumeDrivervolume_backend_name = default

[iser]iscsi_ip_address = 1.1.1.1 #supports RDMAiscsi_protocol = iservolume_group = stack-volumes-defaultvolume_driver = cinder.volume.drivers.lvm.LVMVolumeDrivervolume_backend_name = iser

[iser_old]iser_ip_address = 1.1.1.1 #supports RDMAvolume_driver = cinder.volume.drivers.lvm.LVMISERDrivervolume_group = stack-volumes-defaultvolume_backend_name = iser_old

iSER configurations in Openstack - TGT


[DEFAULT]enabled_backends = default, iser..

[default]iscsi_helper=lioadmvolume_group = stack-volumes-defaultvolume_driver = cinder.volume.drivers.lvm.LVMVolumeDrivervolume_backend_name = default

[iser]iscsi_ip_address=1.1.1.1iscsi_protocol = iseriscsi_helper=lioadmvolume_group = stack-volumes-defaultvolume_driver = cinder.volume.drivers.lvm.LVMVolumeDrivervolume_backend_name = iser

iSER configurations in Openstack - LIO


Cinder Demo Video – iSER Performance


Agenda










Server

VM1 VM2 VM3 VM4

Overlay Networks (VXLAN/NVGRE/GENEVE) Acceleration

Overlay Network Virtualization: Isolation, Simplicity, Scalability

Virtual Domain 3

Virtual Domain 2

Virtual Domain 1

Physical View

Server

VM5 VM6 VM7 VM8

Mellanox SDN Switches & Routers

VirtualView

NVGRE/VXLAN Overlay Networks Virtual Overlay Networks Simplifies Management and VM Migration

ConnectX-3 Pro Overlay Accelerators Enable

Bare Metal Performance


VxLAN = Virtual Extensible LAN

▪What is VxLAN• “Virtual Extensible LAN (VXLAN) is a network virtualization technology that attempts to ameliorate the

scalability problems associated with large cloud computing deployments. It uses a VLAN-like encapsulation technique to encapsulate MAC-based OSI layer 2 Ethernet frames within layer 3 UDP packets.” Wikipedia

▪Enable creating millions of virtual L2 networks over traditional IP networks• Can serve tenants in a cloud provider infrastructure

▪Can span local or wide area networks • Can migrate the entire network between cloud providers/sites and in case of a disaster • Can create logical L2 networks which span multiple locations (like VPNs)

▪Can run over routers • Leverage L3 network scalability and protocols (OSPF, BGP, ECMP)


The VxLAN and NV-GRE overlay challenge

▪Hypervisor IP stack and standard NICs are not aware of the client TCP/IP traffic

▪Common offload techniques such as hardware segmentation/re-assembly, checksum offload, and CPU core scaling (RSS/TSS) do not operate on the VM TCP/IP packets (inner payload)

▪Leading to significant CPU overhead and much lower performance

▪Solution: Overlay aware Network Interface Cards• Overlay Network Accelerators• Penalty free overlays at bare-metal speed

Generated by the VM

Generated by the Hypervisor

VXLAN Packet Format


Turbocharge Overlay Networks with ConnectX-3/4 NICs

“Mellanox is the Only Way to Scale Out Overlay Networks”

Saving 35% of total cores while doubling the throughput!


Agenda










VXLAN in Openstack

▪In order to use VXLAN in Openstack, the OVS Mechanism drivers has to be used (one of them),

tunneling type has to be set to VXLAN and L2 population enabled.

▪VXLAN is usually being used in environments with large amount of tenants or as a tunneling

solution.

▪When using VXLAN in Openstack, neutron creates tunnels between every two hypervisors (Mesh),

and creates a virtual endpoint with the remote IP and VXLAN Network Identifier with OVS/LB.


Mellanox NIC VXLAN offload in Openstack

▪VXLAN offload is done in Hardware (Mellanox NIC).

▪In order to enable VXLAN offload, the following configurations has to be used:

https://community.mellanox.com/docs/DOC-1446

Offloading VXLAN Overhead

https://community.mellanox.com/docs/DOC-1446


Edit /etc/neutron/plugins/ml2/ml2_conf.ini:

[ovs]bridge_mappings = default:br-eth5enable_tunneling = Truelocal_ip = 192.168.215.1

[agent]vxlan_udp_port = 4789tunnel_types = vxlanl2_population = Trueroot_helper = sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf

VXLAN configurations in Openstack


VXLAN Demo


Mellanox high performance features in Openstack

Summary


Comprehensive OpenStack Integration for Switch and Adapter

Neturon-ML2 support for mixed environment with

SR-IOV

Hardware support for performance and

security(VXLAN offload, PV,

SRIOV)

Accelerating storage access by

up to 5x

Neutron ML2 plugin


Vast OpenStack distributions support

Ethernet

Integrated with Major OpenStack

Distributions

Provided In-Box Since Openstack Havana

release

Can be used transparently over

Ethernet or Infiniband Networks


Accelerating Openstack Cloud Performance - Summary

6X Faster

Storage

Overlay Networks

Virtualization

2.5X

20X

6X

20X Faster

Fibre Channel 8Gb

iSER 40GbE VMs Write

2.5X Faster


▪CloudX is a group of reference architectures which allow

building the most efficient, high performance and scalable

Infrastructure As A Service (IaaS) clouds based on

Mellanox superior interconnect and off the shelf building

blocks

▪Supports the most popular cloud software

• Windows Azure Pack (WAP)

• OpenStack

• VMware

http://www.mellanox.com/solutions/cloud/reference.php

CloudX: Optimized Cloud Platform

http://www.mellanox.com/solutions/cloud/reference.php


Mellanox Advantages - Summary

• Support more VMs per server

• Offload hypervisor CPU

• Overlay networks

• Unlimited scalability

• Record braking throughput

• Record braking IOPS

• Higher storage density

• Centralized management (e.g. SDN)

• I/O consolidation (one wire)

• 100Gb/s per port with RDMA

• 2us for VM to VM connectivity

• Low CPU utilization

Higher Performance

Cost EffectiveStorage

Simplified and Integrated Solutions

Higher Infrastructure

Efficiency

Maximizing Cloud Return on Investment


Thank YouThank You

erez cohen & aviram bar haim, mellanox - enhancing your openstack cloud with advanced network...

Technology