Virtual Machine Queue Architecture Review

Ali DabaghArchitectWindows Core Networking

alid@microsoft.com

Don StanwyckSr. Program ManagerNDIS Virtualizationdon.stanwyck@microsoft.com

Agenda

• Introduction• Network device sharing in virtualization environment• Performance bottlenecks• Virtual Machine Queue (VMQ) Architecture• NIC embedded switch• Coexistence with offload technologies• Call to action• Q&A

Why VMQ?

• Virtualization is now

• Networking optimization supports virtualization

• Virtualization impacts performance – VMQ helps overcome those effects

• VMQ supports Live Migration

• VMQ coexists with other optimizations

• Broad industry support

Terminology

• Virtual Service Provider (VSP)• A driver running in the parent partition• Provides a specific functionality to Virtual Machines• Network VSP (NVSP) implemented as an NDIS intermediate driver

• Virtual Service Client (VSC)• A driver running in the child partition• Usually provides a device class functionality in child partitions• Communicates with its VSP counterpart in the parent partition• Network VSC (NVSC) implemented as an NDIS virtual miniport driver

• Virtual Machine Bus (VMBus) • A communication bus for passing control and data messages between child

and parent partition

• VMBus Channel• A communication channel created between a VSC in child partition and VSP

in parent partition

Synthetic Device Data Path

Parent Partition VM1 VM2

Ethernet

VM BUS

TCP/IP TCP/IP

VM NIC 1 VM NIC 2

Network Virtual Service Provider

NIC

MiniportDriver

RoutingVLAN Filtering

Data Copy Port 1Port 2

VMQ Concepts: Receive Queues

• Receive Queues• Are allocated by Network Virtual Service Provider• Collect network packets that pass filter tests

• Properties of a receive queue• Unique queue ID per miniport adapter• Interrupt processor affinity• Filters• Receive buffers

• Default Queue• Always exists• Collects packets that are not collected by any other queue

VMQ Concepts: Receive Filters

• Receive filters are set on receive queues• One or more “network header field tests”• Unique filter ID per miniport adapter

• Specifying network headers in a filter• VMQ uses MAC header

• Specifying fields within the headers• VMQ uses destination MAC address and VLAN tag

• Setting multiple tests in one receive filter• A packet must pass all the network header field tests

• Setting multiple filters on the same queue• Packet is accepted if it passes any of those filters

• Specifying test operation and test value• VMQ uses “Equal” test operation

Receive Filters and Receive Queues

Packet Filtering Data Flow

Virtual Machine Queue Overview

• Classifying received packets• Parsing the packets in hardware• Setting received packet queue ID

• Grouping received packets• All packets in a receive indication call are for the same VM NIC

• Applying VLAN filtering in hardware• Packets with invalid VLAN IDs are dropped by the NIC

• Concurrent receive processing for multiple VMs• Every queue is serviced by a different processor

• Avoiding copy from NIC receive buffers to VM address space• Using switch on the NIC to do route lookup on transmit

Synthetic Device Data Paths With VMQ

Parent Partition VM1 VM2

Ethernet

VM BUS

TCP/IP TCP/IP

VM NIC 1 VM NIC 2

Network Virtual Service Provider

MiniportDriver

Switch/Routing unit

Q1 DefaultQueueQ2

RoutingVLAN filtering

Data Copy Port 1Port 2

NIC

NIC

Queue 1

Using VM Address Space for Receive Buffers

VM1 VM2

D

D

D

D

D

Parent Partition

H

H

H

H

H

Queue 2D

D

D

D

D

H

H

H

H

HAd

dres

s Sp

ace

Addr

ess

Spac

e

Addr

ess

Spac

e

NIC Embedded Switch

• Receive queues are paired with transmit queues• Each queue pair is a switch port

• No MAC address learning is required• The MAC address of each port is the one set on the receive queue

• Switch inspects destination MAC address +VLAN ID of transmit packets• If they pass the filter set on a receive queue, DMA to that queue• Otherwise send the packet on the wire

• Advantages in VM to VM communication• Avoids route lookup in software• Avoids a packet copy• Offload support in hardware

• Advantages in VM to wire communication• Avoids route lookup in software

Coexistence with Other Offload Technologies

• Transmit and receive checksum• Still needs to be calculated and verified• Should work for both VM-to-VM and VM-to-wire data flow

• Large send offload• Should work for both VM-to-VM and VM-to-wire data flow

• Jumbo frame support• Maximum packet size • Scatter/gather support for received packets

• Chimney offload• If NIC supports both VMQ and Chimney, VMQ is used

Summary

• VMQ improves network throughput• Distributes processing of network traffic for multiple VMs

among multiple processors

• VMQ reduces CPU utilization by: • Offloading packet classification to hardware• Avoiding network data copy• Avoiding route lookup on transmit path

• VMQ is compatible with most task offloads• VMQ is secure• VMQ supports live migration

Call to Action

Call to Action

• Virtualization is NOW

• Network optimizations are crucial for virtualization support

• These vendors are among those committed to supporting VMQ in their NICs:• Alacritech• Broadcom• Intel• Neterion• ServerEngines• Solarflare

• Build/update all your drivers to be NDIS 6.20 compliant

Resources

• Windows 7 WDK will be available as of WinHEC • Windows Logo Program Web site:

http://www.microsoft.com/whdc/winlogo/default.mspx • NDIS 6 Feedback alias:

ndis6fb@microsoft.com

Related Sessions

Session Day / Time

Virtual Machine Queue Driver Development Tues. 4-5

Developing TCP Chimney Drivers for Windows 7 Tues. 11-12

Driver Scalability Mon. 11-12 andTues. 11-12

NDIS 6.20 Overview Mon. 9:45-10:45 andTues. 9:45-10:45

Windows Logo Program Tests for NDIS Mon. 11-12 andWed. 9:45-10:45

Discussion: Porting Drivers to NDIS 6.20 Wed. 8:30-9:30

Lab: How to Debug an NDIS 6.20 Driver Mon. 2:45-3:45 andWed. 9:45-10:45