vxlan - bringing hypervisord2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/brkdcn-2200.pdf · vxlan -...
Post on 17-Jul-2018
239 Views
Preview:
TRANSCRIPT
VXLAN - Bringing Hypervisor & Nexus Together
Wayne Davis – Technical Solutions Architect
BRKDCN-2200
• Ready, Set, Tunnel - VXLAN Refresher
• Design Details - Under the Hood
• Avoid Resume Generating Events
• Best Practices
• Case Study Deployment Scenario's
• Roadmap – Whats Next?
• Wrap It Up
Agenda
BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Session Rat hole’s
• Concentrate on deployment VXLAN – F & L
• How to Jumpstart ESX
• Introduction to Nexus 1000v
• All components of the design(s) choices
• Configuration Installation “Gotha's”
• Deep Dive into ACI
• Security “Line by Line cfg”
• Troubleshooting Deep dive Design
4BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
Which Encapsulation?
BRKDCN-2200
VXLAN NVGRE
MPLS
FabricPath
LISP
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Questions
• Is it a standards based protocol used for traffic flows?
• Would you consider using BGP as a Control Plane in your Data Center ?
• Barriers to Adoption – Configuration Complexity ? Automation help ?
• Importance of being Standards-Based ? Proof of Interoperability
• Reliability and Scale out design *Important*?
• Active/Active Data Center design, is it possible?
6BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 7BRKDCN-2200
Introduction – What is VXLAN ?MAC-in-IP Encapsulation
Outer
MAC
DA
Outer
MAC
SA
Outer
802.1Q
Outer
IP DA
Outer
IP SA
Outer
UDP
VXLAN ID
(24 bits)
Inner
MAC
DA
Inner
MAC
SA
Optional
Inner
802.1Q
Original
Ethernet
Payload
CRC
VXLAN Encapsulation Original Ethernet Frame
CRC
DA
TA
PLA
NE
16 M Segments
A
B
C
Switch
Encap
A
B
C
Switch
DcapIP Network
Ethernet Frames Ethernet Frames
IP/UDP Packets
IP Addr
1.1.1.1
IP Addr
2.2.2.2
NE
TW
OR
K
Tunnel
Endpoints
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 8BRKDCN-2200
VXLAN: Flood-&-Learn vs EVPN Control PlaneFlood-&-Learn EVPN Control Plane
Overlay Services L2+L3 L2+L3
Underlay Network IP network with ECMP IP network with ECMP
Encapsulation MAC in UDP MAC in UDP
Peer Discovery Data-driven flood-&-learn MP-BGP
Peer Authentication Not available MP-BGP
Host Route Learning Local hosts: Data-driven flood-&-learn
Remote hosts: Data-driven flood-&-learn
Local Host: Data-driven
Remote host: MP-BGP
Host Route Distribution No route distribution. MP-BGP
L2/L3 Unicast Forwarding Unicast encap Unicast encap
BUM Traffic forwarding Multicast replication
Unicast/Ingress replication
Multicast replication
Unicast/Ingress replication
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 9BRKDCN-2200
VXLAN - VTEP
VXLAN terminates its tunnels on VTEPs (VXLAN Tunnel End Point).
Each VTEP has two interfaces, one is to provide bridging function for local hosts, the
other has an IP identification in the core network for VXLAN encapsulation / de-
encapsulate.
Local LAN Segment
IP Interface
End SystemEnd System
VTEP
Transport IP Network
Local LAN Segment
IP Interface
End SystemEnd System
VTEP
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 10BRKDCN-2200
VXLAN - BGP-EVPN
Tunnel Endpoints LocationHost Reachability Information
• Mac Address + IP Address
VTEP VTEP VTEP VTEP VTEP
R/R R/RIBGP Route Reflector*
(on spine or different box)
VXLAN OverlayBGP Peers
on VTEPs
Use Multi-Protocol BGP with EVPN Address family for :
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 11BRKDCN-2200
VXLAN EVPN - Solution Advantages
Early ARP
Termination
Distributed Anycast
Gateway
Suppresses flooding for Unknown Unicast
ARP
Authenticate Tunnel Endpoints
Seamless and Optimal vm-mobility
Forwarding in the overlay
Active/Active
Multipathing
Active/Active and Resilient Multipathing
using vPC on Nexus
Ingress Replication Unicast Alternative to Multicast underlay
Security
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 13BRKDCN-2200
Under the Hood – “Choice”
• N1K -Segmentation
• N9K - EVPN • Application Centric Infrastructure
• VxLAN 1.0 / 2.0
• Multiple OS support
• VSG VM and Custom Attributes
• Appliance based option
• Multi-technology Design
• Requires 9k switches
• Can be upgraded (NxOS to ACI)
• VxLAN GW (anycast)
• Jump Data Centers with L2 domains
• Broadcast suppression
• Supports Any Hypervisor
• Stateful Firewall Support
• Single Pane of Glass Mgmt.
• Container design Model
• Security per vNIC
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 14BRKDCN-2200
Under The Hood – Physical
N2K N2K
Chassis
Servers
vPC
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
Cisco Nexus 2148T
1GE Fabric Extender
STAT
ID 45 46 47 48 1 2 3 4
Bay1
Bay9
DSModule
PS1
Bay8
Bay16
PS6
HPBladeSystem
c7000Enclosure
CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
STAT
ID
SLOT
1
SLOT
5
SLOT
3
SLOT
7
SLOT
2
SLOT
6
SLOT
4
SLOT
8
!
UCS 5108
OK FAIL OK FAIL OK FAIL OK FAIL
1/10 GIGABIT ETHERNET 1/2/4/8G FIBRE CHANNEL
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16N55-M16UP
1/10 GIGABIT ETHERNET 1/2/4/8G FIBRE CHANNEL
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16N55-M16UP
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
!
Reset
Console
UCS B200 M3
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
SLOT
1
SLOT
5
SLOT
3
SLOT
7
SLOT
2
SLOT
6
SLOT
4
SLOT
8
!
UCS 5108
OK FAIL OK FAIL OK FAIL OK FAIL
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
!
Reset
Console
UCS B200 M3
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
SLOT
1
SLOT
5
SLOT
3
SLOT
7
SLOT
2
SLOT
6
SLOT
4
SLOT
8
!
UCS 5108
OK FAIL OK FAIL OK FAIL OK FAIL
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
!
Reset
Console
UCS B200 M3
! ResetConsole
UCS B200 M3
! ResetConsole
UCS B200 M3
ACI Leaf ACI Leaf ACI Border
LeafACI Border Leaf
UCS
FW
STS
BCN
ACT
Cisco Nexus 9396PX
1
2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
STS
BCN
ACT
Cisco Nexus 9396PX
1
2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44
Cisco Nexus 2148T
1GE Fabric Extender
STAT
ID 45 46 47 48 1 2 3 4
ADC
ADC
CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
STAT
ID
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
PW
R
SY
S
CO
NS
OL
E
UCS
C240 M3
!
Servers
FW ADC
SEC
VMVM
VM
Adaptive
Security
Appliance
Cisco
ASA 5545-X
1
0
ALARM
VPN
HD1
HD0
BOOT
ACTIVE
PS1
PS0
Adaptive
Security
Appliance
Cisco
ASA 5545-X
1
0
ALARM
VPN
HD1
HD0
BOOT
ACTIVE
PS1
PS0
1.11.1
G~ POWER 2~ POWER 1
1.11.1
G~ POWER 2~ POWER 1
STS
BCN
ACT
Cisco Nexus 9396PX
1
2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
STS
BCN
ACT
Cisco Nexus 9396PX
1
2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
VM VM
STS
BCN
ACT
Cisco Nexus 9396PX
1
2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
STS
BCN
ACT
Cisco Nexus 9396PX
1
2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
40 Gig
10 Gig
VPC
STS
BCN
ACT
Cisco Nexus 9396PX
1
2
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
VLAN 111
VLAN 222
VLAN 10
Nexus 9396 Nexus 9396
Nexus 7000
ACI-9396 Leaf
ACI-9336 Spine
Nexus 9396 Nexus 9396
vlan 1011
V
D
C
40G
10G
RR
vlan 2022
172.16.222.222172.16.111.11110.96.126.17
10.222.222.1610.111.111.50
VEM
VSM
HYPERVISOR
VXLAN Overlay
10G
10G
10.96.126.80
99.99.99.0/30
.1
.2
150.150.150.0/30
.2.1
A B
10.9.9.0/30
.1
.2
BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 16BRKDCN-2200
Under the Hood - Topo
VM
OS
VM
OS
VXLAN L2
Gateway
VXLAN L2
Gateway
VM-A VM-B
VXLAN L2
Gateway
VM-C
L3 – FW,
SLB
LAN Extension
Tunnel
VM
OS
N7k-1 N7k-2
N9k N9k N9k N9k
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 17BRKDCN-2200
Under The Hood - Nexus 7000 - Transport
• interface Ethernet1/1
• no switchport
• ip address 10.9.9.1/8
• ip router eigrp 813
• no shutdown
interface Vlan900
description ACI-VLAN900
no shutdown
bandwidth 80000000
no ip redirects
ip address 99.99.99.1/30
no ipv6 redirects
no ip passive-interface eigrp 813
ip pim sparse-mode
interface Vlan901
description Transit_vlan_901_between_sydney23-
sydney24
no shutdown
ip address 150.150.150.2/30
ip router eigrp 813
no ip passive-interface eigrp 813
ip pim sparse-mode
feature tacacs+
cfs eth distribute
feature pim
feature eigrp
feature udld
feature interface-vlan
feature hsrp
feature lacp
feature dhcp
feature vpc
feature sflow
Nexus 7000 - ABoth N7k’s Transit Networks
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 19BRKDCN-2200
Under the Hood – Nexus 9000
VM
OS
VM
OS
VXLAN L2
Gateway
VXLAN L2
Gateway
VM-A VM-B
VXLAN L2
Gateway
VM-C
L3 – FW,
SLB
LAN Extension
Tunnel
VM
OS
N7k-1 N7k-2
N9k N9k N9k N9k
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 20BRKDCN-2200
Under the Hood - Each Device BeginsEnable VXLAN and MP-BGP EVPN Control Plane
feature nv overlay
feature vn-segment-vlan-based
feature bgp
nv overlay evpn
Enable VXLAN
Enable VLAN-based VXLAN (the currently
only mode)
Enable OSPF if it’s chosen to be the
underlay IGP routing protocol
Enable VLAN SVI interfaces if the VTEP
needs to be IP gateway and route for the
VXLAN VLAN IP subnet.
Enable EVPN control plane for VXLAN
feature ospf
feature pim
feature interface-vlan
Other features may need to be enabled
Enable BGP
Enable IP PIM multicast routing in the
underlay network
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 21BRKDCN-2200
Under the Hood - Tenant CreationVXLAN – virtual routing / forwarding
vrf context vxlan-n1k-vm-a
vni 22200
rd auto
address-family ipv4 unicast
route-target import 22200:22200
route-target export 22200:22200
route-target both auto evpn
Create a VXLAN Tenant VRF
Specify the Layer-3 VNI for VXLAN routing
within the tenant VRF
Define VRF Route Target and import/export
policies in address-family ipv4 unicast
Define VRF RD (route distinguisher)
vrf context vxlan-n1k-vm-b
vni 22210
rd auto
address-family ipv4 unicast
route-target import 22210:22210
route-target export 22210:22210
route-target both auto evpn
Example to create a 2nd tenant VRF
following the above steps
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 22BRKDCN-2200
Under the Hood - Layer-3 (VNI) Routing – VM(A)Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant
vlan 2220name vrf-L3-vm-avn-segment 22200
interface Vlan2220description vrf-L3-vm-a-routingno shutdownvrf member vxlan-n1k-vm-a
vrf context vxlan-n1k-vm-avni 22200rd autoaddress-family ipv4 unicastroute-target import 22200:22200route-target export 22200:22200route-target both auto evpn
Create the VLAN for the Layer-3 VNI.
One Layer-3 VNI per tenant VRF routing
instance
Create the SVI interface for the Layer-3 VNI
Put this SVI interface into the tenant VRF
context
Associate the Layer-3 VNI with the tenant
VRF routing instance.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 23BRKDCN-2200
Under the Hood - Layer-3 (VNI) Routing – VM(B)Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant
vlan 1110
name vrf-L3-vm-b
vn-segment 22210
interface Vlan1110
description vrf-L3-vm-b-routing
no shutdown
vrf member vxlan-n1k-vm-b
vrf context vxlan-n1k-vm-b
vni 22210
rd auto
address-family ipv4 unicast
route-target import 22210:22210
route-target export 22210:22210
route-target both auto evpn
Define Layer-3 VNI for a 2nd tenant
following the same steps in the previous
slide
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 24BRKDCN-2200
Under the Hood - Layer-2 VXLAN Network Identifier Map VLANs to VXLAN VNIs and Configure their MP-BGP EVPN Parameters
vlan 222
vn-segment 20000
vlan 111
vn-segment 21000
Map VLAN to VXLAN VNI
evpn
vni 20000 l2
rd auto
route-target import auto
route-target export auto
vni 21000 l2
rd auto
route-target import auto
route-target export auto
Under EVPN configuration, define RD
and RT import/export policies for each
Layer-2 VNIs
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 25BRKDCN-2200
Under the Hood - Interface SVI – Layer 2Create SVI interface for Layer-2 VNIs for VXLAN routing
interface Vlan111
no shutdown
vrf member vxlan-n1k-vm-a
ip address 10.111.111.1/8
fabric forwarding mode anycast-gateway
interface Vlan222
no shutdown
vrf member vxlan-n1k-vm-b
ip address 10.222.222.1/8
fabric forwarding mode anycast-gateway
Create SVI interface for a Layer-2 VNI.
Associate it with the tenant VRF.
Enable distributed anycast gateway for this
VLAN/VNI
All VTEPs for this VLAN/VNI should have the
same SVI interface IP address as the
distributed IP gateway.
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 26BRKDCN-2200
Under the Hood - Distributed Gateway – Anycast
fabric forwarding anycast-gateway-mac 0000.1111.2222
interface Vlan111
no shutdown
vrf member vxlan-n1k-vm-a
ip address 10.111.111.1/8
fabric forwarding mode anycast-gateway
Configure virtual IP address
All VTEPs for this VLAN should have the same
virtual IP address
Configure distributed gateway virtual MAC
address
One virtual MAC per VTEP
All VTEPs should have the same virtual MAC
address
Enable distributed gateway for this VLAN
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 27BRKDCN-2200
Under the Hood - Network Virtualization Endpoint Configure VXLAN tunnel interface nve1
interface nve1
no shutdown
source-interface loopback0
host-reachability protocol bgp
member vni 20000
suppress-arp
mcast-group 239.1.1.1
member vni 21000
suppress-arp
mcast-group 239.1.1.2
member vni 22200 associate-vrf
member vni 22210 associate-vrf
Specify loopback0 as the source interface
Define BGP as the mechanism for host
reachability advertisement
Add Layer-3 VNIs, one per tenant VRF
Associate tenant VNIs to the tunnel
interface nve1
Define the mcast group on a per-VNI basis
Enable arp suppression on a per-VNI basis
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 28BRKDCN-2200
Under the Hood - VXLAN Tunnel Interface Configuration – Cont’d Configure VXLAN tunnel interface nve1
interface loopback 0
ip address 10.111.222.1/32
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.0
ip pim sparse-mode
The loopback interface to source VXLAN
tunnels
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 29BRKDCN-2200
Under the Hood - BGP – “Yes” in the LANrouter bgp 65535
router-id 10.111.222.1
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
neighbor 10.1.2.1 remote-as 65535
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
neighbor 10.1.2.2 remote-as 65535
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
vrf vxlan-n1k-vm-a
address-family ipv4 unicast
advertise l2vpn evpn
vrf vxlan-n1k-vm-b
address-family ipv4 unicast
advertise l2vpn evpn
Address-family ipv4 unicast for prefix-
based routing
Define MP-BGP neighbors.
Under each neighbor define address-family
ipv4 unicast and l2vpn evpn
Under address-family ipv4 unicast of each
tenant VRF instance, enable advertising
EVPN routes
Send extended community in l2vpn evpn
address-family to distribute EVPN route
attributes
Address-family l2vpn evpn for evpn host
routes
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 30BRKDCN-2200
Under the Hood - Route Reflectorrouter bgp 65535
router-id 10.1.2.1
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
retain route-target all
template peer vtep-peer
remote-as 65535
update-source loopback0
address-family ipv4 unicast
send-community both
route-reflector-client
address-family l2vpn evpn
send-community both
route-reflector-client
neighbor 10.111.222.1
inherit peer vtep-peer
neighbor 10.1.2.12
inherit peer vtep-peer
Address-family ipv4 unicast for prefix-
based routing
iBGP RR client peer template
Send both standard and extended
community in address-family l2vpn evpn
Send both standard and extended
community in address-family ipv4 unicast
Address-family l2vpn evpn for EVPN vxlan
host routes
Retain route-targets attributes
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Nexus 1000V Architecture
Hypervisor Hypervisor Hypervisor
VEM-NVEM-1 VEM-2
VSM: Virtual Supervisor Module
VEM: Virtual Ethernet Module Server
Admin
NX-OS
Data Plane
VSM-1 (active)
VSM-2 (standby)
Virtual ApplianceNX-OS
Control PlaneNetwork
Admin
Modular Switch
…
Linecard-N
Supervisor-1 (Active)
Supervisor-2 (StandBy)
Linecard-1
Linecard-2
Ba
ck P
lan
e
BRKDCN-2200 32
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 33BRKDCN-2200
Under the Hood - Nexus 1000v
VM
OS
VM
OS
VXLAN L2
Gateway
VXLAN L2
Gateway
N7k-1 N7k-2
VM-A VM-B
VXLAN L2
Gateway
10.222.222.50
111.111.111.110.222.222.1
VM-C 10.222.222.49
L3 – FW,
SLB
LAN Extension
Tunnel
111.111.111.x
VM
OS
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 34BRKDCN-2200
Under the Hood - Nexus 1000v transport
n1kv-wayne# sh run
version 5.2(1)SV3(1.5a)
hostname n1kv-wayne
port-profile type ethernet UPLINK
switchport mode trunk
switchport trunk allowed vlan 1-2,100-300
channel-group auto mode on mac-pinning
no shutdown
system vlan 1-2
state enabled
vmware port-group
port-profile type vethernet L3-Control
switchport mode access
switchport access vlan 1
no shutdown
capability l3control
system vlan 1
state enabled
vmware port-group
vrf context management
ip route 0.0.0.0/0 10.96.126.254
vlan 1-2,100-300
port-channel load-balance ethernet source-mac
port-profile default max-ports 32
port-profile type ethernet Unused_Or_Quarantine_Uplink
shutdown
description Port-group created for Nexus 1000V internal usage. Do not use.
state enabled
vmware port-group
port-profile type vethernet Unused_Or_Quarantine_Veth
shutdown
description Port-group created for Nexus 1000V internal usage. Do not use.
state enabled
vmware port-group
interface Vethernet1
inherit port-profile L3-Control
description VMware VMkernel, vmk2
vmware dvport 100 dvswitch uuid "75 3e 37 50 a5 6b ef f6-85 60 6a 7a 7f b6
3d"
vmware vm mac 0050.5671.47DA
interface Vethernet3
inherit port-profile vm-222
description Windows-7-222, Network Adapter 1
vmware dvport 256 dvswitch uuid "75 3e 37 50 a5 6b ef f6-85 60 6a 7a 7f b6
3d"
vmware vm mac 0050.56B7.0108
port-profile type vethernet EVPN-VXLAN
switchport mode access
switchport access vlan 111
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 35BRKDCN-2200
Under the Hood - VXLAN Forwarding Basics - VSM
VEM 1 VEM 2
Forwarding mechanisms similar to Layer 2 bridge: Flood & Learn
VEM learns VM’s Source (MAC, Host VXLAN IP) tuple
Broadcast, Multicast, and Unknown Unicast Traffic
VM broadcast & unknown unicast traffic are sent as multicast
Unicast Traffic
Unicast packets are encapsulated and sent directly (not via multicast) to destination host VXLAN IP (Destination VEM)
VM VM VM VM
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 36BRKDCN-2200
Under the Hood - VM Host - VXLAN Topo
Guest Machine(s)
configured for setup
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
L3 – N9k Enhanced VXLAN – VSM
VLAN CLI Model
vlan 222name – n1k
interface vlan 222ip address 10.222.222.1ip router eigrp 22
interface Ethernet3/2switchportswitchport mode trunkrate-mode dedicated forcechannel-group 222 mode activeno shutdown
VSM config
feature segmentation
segment mode unicast-only
port-profile type vethernet vxlan-n1k
capability l3control
vmware port-group
switchport mode access
switchport access vlan 222
capability vxlan
no shutdown
system vlan 1
state enabled
37
Normal SVI’s
BRKDCN-2200
• VMkernel interface acts as VTEP
• VSM Control Mode should be L3
• Bridge domain is configured as Unicast or
Unicast Mac Distribution
B+U – no “M”
Under the Hood -
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 38BRKDCN-2200
Under the Hood - VMKernel
port-profile type vethernet vxlan-n1k
capability l3control
vmware port-group
switchport mode access
switchport access vlan 222
capability vxlan
no shutdown
system vlan 1
state enabled
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 39BRKDCN-2200
Under the Hood – VSM Bridge Domain
port-profile type vethernet bd-22222
switchport access bridge-domain BD-vxl
no shutdown
state enabled
vmware port-group
port-profile type vethernet vmk-l3-vxlan-vtep
switchport mode access
switchport access vlan 222
capability vxlan
no shutdown
capability l3control
state enabled
vmware port-group
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 40BRKDCN-2200
Under the Hood - Port Profile Attachment
N1K - DVS
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Under the Hood – VTEP status
vsm-vxlan# show bridge-domain bd-22222
Bridge-domain bd-22222 (2 ports in all)
Segment ID: 22222 (Manual/Active)
Mode: Unicast-only (override)
MAC Distribution: Disable (override)
Group IP: NULL
State: UP Mac learning: Enabled
Veth4, Veth18
vsm-vxlan# show bridge-domain bd-22222 vteps
Bridge-domain: bd-22222
VTEP Table Version: 21
Port Module VTEP-IP Address VTEP-Flags
---------------------------------------------------------------------------
Veth1 3 10.111.111.49 (D) <---Designated VTEP (vmk)
Veth2 4 10.111.111.50 (D)
41BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Best Practice(s)
43BRKDCN-2200
What Should We Do?
• Optimization
• Decision Trees
Hit the
EASY BUTTON
LCM
• Backups
• High Availability Options
• Software Repository
Success
Deployment
Virtual Switch Update Manager
Life Cycle Management – VSM / VEM
Performance & Scalability
What Should We Do?
What’s the desired outcome?
Enterprise Architecture Framework – Network, Security, Server, Virtualization
P & S
• HW Limits
• SW Limits
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Performance & Scalability
• ESX – 5.2(1)SV3(1.2)
• 256 VEMs, 12K vEth count
• VxLAN 2.0 (BGP Control Plane)
• VxLAN UDP Port Configurable
• N1K Virtual Switch Update Manager
• Distributed NetFlow
• IGMP Multicast Offload (1k Groups)
• BPDU Guard & Storm Control
• Cisco TrustSec, IPv6 Enhancements
• ESX – 4.2(1)SV2(2.2)
• Dynamic Fabric Automation Leaf
• VDP – VSI Discovery Protocol
• Universal Licensing
• Hyper-V – 5.2(1)SM3(1.1)
• VxLAN 1.0 & HVN
• Hyper-V – 5.2(1)SM1(5.2a)
• SCVMM 2012 SP1 & R2
• Windows Server 2012 & R2
• VSG VM and Custom Attributes
• Universal Licensing
• KVM – 5.2(1)SK3(2.1)
• IceHouse
• RHEL-OSP – OpenStack Platform Inst
• VxLAN GW
• pVLAN
• UUFB blocking44
BP
BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Life Cycle Management - VSM Control Modes
• L3 Mode
• This is not routing
• L3 is the recommended & default• Easier to troubleshoot
• Cross Firewalls & L3 boundaries
• Requires an IP address be assigned to the VEM (vmk)
• Uses UDP 4785 for both source and destination
• Sourced from mgmt0 by default
• L2 mode (Legacy)
• Requires L2 connectivity through control0 interface to all VEM modules
• Deprecated but supported on ESX
• Not supported with Hyper-V or KVM
45BRKDCN-2200
BP
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Life Cycle Management - VSM vMotion
• Manual vMotion/Live Migration is supported
• VMware DRS is NOT recommended for Primary & Secondary VSMs
• Aggressive DRS could lead to excessive VSM-VEM heartbeat packet drops
• Best practice to keep Primary and Secondary VSM outside DRS control
• Use anti-affinity rules where possible
• FT is not supported
46
BP
BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Life Cycle Management - VSM Backups
• A running-config is not enough to restore due to PSS
• VSM on ESXi / HyperV
• Clone to a template
• Restore from an older template + running-config
• Both VSMs must be powered down
• VSM on Nexus 1110
• Export a VSM to a file
• Import the saved VSM to restore
• VSM on ESXi Snapshots
• Not officially supported
• I/O latency cost associated with expanding the differential file
47
BP
BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Life Cycle Management - VSM Interfaces
• Control
• VSM-VSM HA Heartbeats
• VSM-VEM Heartbeats
• VSM-VSM Synchronization
• BGP Control Plane
• Packet
• CDP, IGMP*, SNMP
• Layer3 Mode
• Collapsed ctrl0 & pkt into mgmt0
• VSM-VEM communication on mgmt0
• Dedicated Control:svs mode L3 interface [control | mgmt0]
• Management
• SSH console access
• SNMP, HTTP, XML
• vCenter Communication
• HA Heartbeat Backup
• Interface Order is always the same!
VSM-Peth0: control
eth1: mgmt0
eth2: packet
48
BP
BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Life Cycle Management - VEM Deployment
• L3 control requires a VMKernel NIC on N1K DVS
• We need an L3 interface to forward control traffic
• 200/100/10ms latency between VSM & VEM
• Recommend using the ESXi management VMKernel NIC
• Migrate management vmk behind VEM
• Doesn’t require static routes on ESXi hosts
• Put additional vmks on different subnets (vMotion / Storage)
• UCS “Dynamic vNICs” in Service-Profiles
• VEM and VM-FEX are mutually exclusive
49BRKDCN-2200
BP
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VEM Deployment – VMKs on same subnet
• VMware uses a single TCP/IP stack for all VMK interfaces
• Don’t use multiple VMKs on the same subnet on different virtual switches
• No way to pin traffic to an uplink interface.
• One interface gets picked for all traffic on that subnet
• VMware KB article 2010877
• Only one default gateway per host
VMware ESX
VEM
VMK1
192.168.10.100
VMK0
192.168.10.200
vSwitch
50
BP
BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 51BRKDCN-2200
VEM - Port-Profiles Secret Sauce
port-profile type ethernet uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 10,119
channel-group auto mode on mac-pinning
no shutdown
system vlan 119
state enabled
vmnic0 Eth3/1
vmnic1 Eth3/2
port-profile type vethernet vmk-l3
capability l3control
vmware port-group
switchport mode access
switchport access vlan 119
capability vxlan
no shutdown
system vlan 119
state enabled
VM1
VMK1
VM2port-profile type vethernet vm-vlan10
vmware port-group
switchport mode access
switchport access vlan 10
no shutdown
state enabled
PO1
BP
vEthernet PP (default)
-Virtual Interfaces (vEthernet x)
-Typically Access Ports or Bridge Domains
-Configuration: VLAN, ACLs, VxLAN, QoS
Ethernet PP
-Physical Interfaces (Ethernet x/y)
-Typically Trunk Ports
-Configuration: Port-Channel, ACLs, QoS
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 52BRKDCN-2200
VSUM – Virtual Switch Update Manager
• Install, Migrate, Upgrade, Monitor Nexus 1000V and ACI AVS
• Standalone VM
• Nexus 1000V Binaries are Self-Contained
• Integrated in vSphere Web Client through Plugin
• VMware only today
• Single instance manages all N1k on a vCenter
• Manages existing N1k DVS
No
Charge
BP
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 53BRKDCN-2200
VSUM – Plugin IconBP
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VSUM – Installing Nexus 1000V VSM
54
1 2
3
BP
BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Upgrades - Deployment
• First always read and follow the upgrade guides
• Order matters: VSM then VEM
• Take a backup of the VSMs
• On ESXi use the clone to template option (Powered Down)
• On Nexus 1110s / Cloud Services Platform use the export command
• Backup the running-config
• Generate a tech-support before the upgrade
• If something goes wrong STOP and call TAC
• Use a maintenance window
• VEM upgrades require ESXi hosts to be in Maintenance Mode
• Use N1k Upgrade Utility Matrix to Plan a combined N1k+vSphere Upgrade
BP
55BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 57BRKDCN-2200
ACI Relationship Map
1
3
2 7 4 569
8
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
End-Points end EPG membership
Device connected to network directly or indirectly
Has address (identity), location, attributes (version, patch level)
Can be physical or virtual or container
• Examples:
• End Point Group (EPG) membership defined by:• Ingress physical port (leaf or FEX)
• Ingress logical port (VM port group)
• VLAN ID
• VXLAN (VNID)
• IP address
• IP Prefix/Subnet
• VM-based attributes
• NVGRE (VSID) (future)
• Layer 4 ports (future)
Server
Virtual Machines & Containers
Storage
Client
58BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 59BRKDCN-2200
ACI – Segmentation
Micro-segmentation
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Background• 111.111.111.10 and 111.111.111.11 in CL-VXLAN
• Create CL-VXLAN(useg)
• Put 111.111.111.10/32 in EPG VM-1(useg)
• Both VM – talk to GW + Each other !
LEAF 222
1/17
VLAN 111
111.111.111.11
BaseCL-
VXLAN
Can talk to each other as they are in same EPG
111.111.111.10
BaseCL-VXLAN
BD1
LEAF 222
1/7
VLAN 222
LEAF 222
1/17
VLAN 111
111.111.111.11
CL-VXLAN
111.111.111.10
CL-
VXLAN(useg)
BD1
LEAF 222
1/7
VLAN 222
Still can talk to each other
Configure CL-VXLAN(useg)
bypasses IP classification
BRKDCN-2200 60
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Case Study Deployment: Basic Tunneling
• VSM is a Virtual Machine
• Control plane for the Nexus 1000V switch
• VEM packet forwarding not impacted by reloads
• VSM HA pair distributed across multiple host
• Responsible for:
• Programming and Managing Virtual Ethernet Modules (VEM)
• Communicating with Management Applications (vCenter, SCVMM, Horizon Dashboard, etc.)
62BRKDCN-2200
Hypervisor
VEM
VM VMVM
VSM
#1
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 63BRKDCN-2200
SPINE SPINE
TPA
LEAF
3
LEAF
SVC Block
I-NETINTEXT I-NET INT EXTDFW
FI
FI
SCE
SAP MSFT RAC RHEL
NGFWNGFWFI
FI
SCE
SAP MSFT RAC RHEL
N2kN2kN2kN2k
SERVER ACCESS
N2kN2kN2kN2k
SERVER ACCESS
UNIFIED COMPUTE SYSTEM UNIFIED COMPUTE SYSTEM
Case Study - Deployment
N2k N2k N2k N2k
#2
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 64BRKDCN-2200
DMZ
Case Study - ACI LEVEL
Data Farm
Data Farm
OSPF
VM VMVM
#3
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Now is the future• Migrate Customers from Nexus 1010/1010-X/1110-S/1110-X
• Dedicated Cisco Cloud Services Platform appliance ( CSP 2100 )
• Preparation for Nexus 1000 release 3 – BGP control plane interoperability with Nexus 9000
• Whitepaper to follow – design guidance on VM scale and extended attribute parity
• Look @ ACI – you just might “love it”
66BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.
• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.
67BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
68BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
R&S Related Cisco Education OfferingsCourse Description Cisco Certification
CCIE R&S Advanced Workshops (CIERS-1 &
CIERS-2) plus
Self Assessments, Workbooks & Labs
Expert level trainings including: instructor led workshops, self
assessments, practice labs and CCIE Lab Builder to prepare candidates
for the CCIE R&S practical exam.
CCIE® Routing & Switching
• Implementing Cisco IP Routing v2.0
• Implementing Cisco IP Switched
Networks V2.0
• Troubleshooting and Maintaining
Cisco IP Networks v2.0
Professional level instructor led trainings to prepare candidates for the
CCNP R&S exams (ROUTE, SWITCH and TSHOOT). Also available in
self study eLearning formats with Cisco Learning Labs.
CCNP® Routing & Switching
Interconnecting Cisco Networking Devices:
Part 2 (or combined)
Configure, implement and troubleshoot local and wide-area IPv4 and IPv6
networks. Also available in self study eLearning format with Cisco Learning
Lab.
CCNA® Routing & Switching
Interconnecting Cisco Networking Devices:
Part 1
Installation, configuration, and basic support of a branch network. Also
available in self study eLearning format with Cisco Learning Lab.
CCENT® Routing & Switching
71
For more details, please visit: http://learningnetwork.cisco.com
Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com
BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Design Cisco Education OfferingsCourse Description Cisco Certification
Designing Cisco Network Service Architectures
(ARCH) Version 3.0
Provides learner with the ability to perform conceptual, intermediate, and
detailed design of a network infrastructure that supports desired capacity,
performance, availability required for converged Enterprise network
services and applications.
CCDP® (Design Professional)
(Available Now)
Designing for Cisco Internetwork Solutions
(DESGN) Version 3.0
Instructor led training focused on fundamental design methodologies used
to determine requirements for network performance, security, voice, and
wireless solutions. Prepares candidates for the CCDA certification exam.
CCDA® (Design Associate)
(Available Now)
For more details, please visit: http://learningnetwork.cisco.com
Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com
72BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Data Center / Virtualization Cisco Education OfferingsCourse Description Cisco Certification
Introducing Cisco Data Center Networking (DCICN);
Introducing Cisco Data Center Technologies (DCICT)
Learn basic data center technologies and skills to build a
data center infrastructure.
CCNA® Data Center
Implementing Cisco Data Center Unified Fabric (DCUFI);
Implementing Cisco Data Center Unified Computing (DCUCI)
Designing Cisco Data Center Unified Computing (DCUDC)
Designing Cisco Data Center Unified Fabric (DCUFD)
Troubleshooting Cisco Data Center Unified Computing
(DCUCT)
Troubleshooting Cisco Data Center Unified Fabric (DCUFT)
Obtain professional level skills to design, configure,
implement, troubleshoot data center network infrastructure.
CCNP® Data Center
Product Training Portfolio: DCNMM, DCAC9K, DCINX9K,
DCMDS, DCUCS, DCNX1K, DCNX5K, DCNX7K
Gain hands-on skills using Cisco solutions to configure,
deploy, manage and troubleshoot unified computing, policy-
driven and virtualized data center network infrastructure.
Designing the FlexPod® Solution (FPDESIGN);
Implementing and Administering the FlexPod® Solution
(FPIMPADM)
Learn how to design, implement and administer FlexPod
solutions
Cisco and NetApp Certified
FlexPod® Specialist
73
For more details, please visit: http://learningnetwork.cisco.com
Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com
BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Network Programmability Cisco Education OfferingsCourse Description Cisco Certification
Integrating Business Applications with Network
Programmability (NIPBA);
Integrating Business Applications with Network
Programmability for Cisco ACI (NPIBAACI)
Learn networking concepts, and how to deploy and troubleshoot
programmable network architectures with these self-paced courses.
Cisco Business Application
Engineer Specialist Certification
Developing with Cisco Network Programmability
(NPDEV);
Developing with Cisco Network Programmability
for Cisco ACI (NPDEVACI)
Learn how to build applications for network environments and effectively
bridge the gap between IT professionals and software developers.
Cisco Network Programmability
Developer Specialist Certification
Designing with Cisco Network Programmability
(NPDES);
Designing with Cisco Network Programmability
for Cisco ACI (NPDESACI)
Learn how to expand your skill set from traditional IT infrastructure to
application integration through programmability.
Cisco Network Programmability
Design Specialist Certification
Implementing Cisco Network Programmability
(NPENG);
Implementing Cisco Network Programmability
for Cisco ACI (NPENGACI)
Learn how to implement and troubleshoot open IT infrastructure
technologies.
Cisco Network Programmability
Engineer Specialist Certification
For more details, please visit: http://learningnetwork.cisco.com
Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com
74BRKDCN-2200
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cloud Cisco Education OfferingsCourse Description Cisco Certification
Understanding Cloud Fundamentals
(CLDFND)
Learn how to perform foundational tasks related to Cloud computing, and the essentials
of Cloud infrastructureCCNA Cloud
Introducing Cloud Administration
(CLDADM)
Learn the essentials of Cloud administration and operations, including how to provision,
manage, monitor, report and remediate.
Implementing and Troubleshooting the
Cisco Cloud Infrastructure (CLDINF)
Learn how to implement and troubleshoot Cisco Cloud infrastructure: compute,
network, storage.
CCNP Cloud
Designing the Cisco Cloud (CLDDES)*Learn how to design private and hybrid Clouds including infrastructure, automation,
security and virtual network services
Automating the Cisco Enterprise Cloud
(CLDAUT)*
Learn how to automate Cloud deployments – provisioning IaaS (private, private with
network automation and hybrid) and applications, life cycle management
Building the Cisco Cloud with Application
Centric Infrastructure (CLDACI)*
Learn how to build Cloud infrastructures based on Cisco Application Centric
Infrastructure, including design, implementation and automation
UCS Director Foundation (UCSDF)Learn how to manage physical and virtual infrastructure using orchestration and
automation functions of UCS Director.
75
* Available Q2CY2016
For more details, please visit: http://learningnetwork.cisco.com
Questions? Visit the Learning@Cisco Booth or contact ask-edu-pm-dcv@cisco.com
BRKDCN-2200
top related