aci for network...

ACI for Network Administrators

Steve Sharman – Technical Solutions Architect

• Understand ACI through the eyes of the network administrator

• Understand ACI building blocks

• Understand external and services integration

Session Objectives

• ACI in the market

• Role of the Network Manager

• ACI is all about Applications isn’t it?

• Comparing ACI and Traditional Network Building Blocks

• VMware Integration

• External Connectivity

• Service Graph Integration

• Getting Started

Agenda

Momentum Continues to Grow

6,000+ 501400+Nexus 9K and ACI Customers Globally

EcosystemPartners

ACICustomers

Programmable NetworkProgrammable FabricApplication Centric

Infrastructure

Integrated stack

Or

A-la-carte Automation

Streamlined Workflow Management

Modern NX-OS with enhanced NX-APIs

DevOps toolset used for Network Management(Puppet, Chef, Ansible etc.)

Customer Script based Operations and Workflows

Turnkey integrated solution with security, centralised

management, compliance and scale

Automated application centric-policy model with

embedded security

Broad and deep ecosystem

Turnkey or DIY solution

Fault

Accounting

Performance

Security

Configuration

External

ToolsIntegrated

Tools

VTSCreation Expansion

Fault MgmtReporting

Connection

External

Tools

Integrated

Tools

Enough Marketing, what do networking teams really spend their time doing?

What does ACI typically mean to a Network Admin?

In reality ACI is all about networking and how you deploy applications onto the network!

At a very basic level ACI is really just a CLOS network of Nexus 9k

switches with a management platform

The network management platform (APIC) provides you

with a single place from which to manage the network

ACI is a Software Defined Network which uses VXLAN to transport

packets between switches across an automated IP fabric with end to

end header visibility

IETF Draft

ACI can transport any IP (and non IP) traffic including “Overlay”

networks based on VXLAN*, NVGRE* etc.

* ACI has visibility of the outer header

BRKACI-1002

Understanding ACI Building Blocks

Comparing ACI and “Traditional” Network

Management

Traditional Networking

Management options:• CLI

• Cut/Paste

• Limited automation

• Disparate management platforms

Limitations:• Box by box approach

• Lack of consistent configuration (no

network wide policies)

• Leftover/unknown configuration

• Open “any to any” connectivity*

• Lack of traffic visibility

• Separate virtual and physical networks

• Separate L4-7 device management

ACI Networking

APIC

APIC

APIC

Management options:• GUI (basic/advanced)

• CLI

• XML/JSON

• Scripting

• Open API

• Automation

Benefits:• Distributed, Centralised Management

• Full traffic visibility*

• Self documenting

• Integrated virtual and physical network

• Integrated L4-7 device management

• Policy defined network

A Policy Defined Network –Lighting up switch interfaces

Virtual Machine

Domains

(vSwitches)

vCenter-01-vDS-01

Which vDS do I

want to configure?

VLAN mgmt

(Phy/Out Domain)

UCS-phys-svrs

Outside-Fabric

VLAN/VXLAN

(Pools)

vCenter-01-vDS-01

UCS-phys-svrs

Outside-Fabric

What “function” do

I want to allocate

VLANs for?

Interface Usage

(Policy Groups)

vPC_to_UCS_FI_A

SVI_to_outside

Interface Parameters

(Policies)

CDP_enabled

LACP_Active

Allowed VLANs

(AAEP)

vCenter-01-vDS-01

UCS-phys-svrs

Outside-Fabric

Policy Defined Network

Target Interfaces ID

(Profiles)

vPC_to_UCS_FI_A

SVI_to_outside

Target Switches

(Profiles)

vPC_Leaf_1_and_2

Leaf_3

Concrete Model

Logical Model

APIC

APIC

APIC

Where do I want

to use my VLANs?What interface

settings do I want

to configure?

What type of

interface do I want

to configure, and what

device do I want to

connect to it?

Which interfaces

should be

configured?

Which switches

should be

configured?

Group my VLANs

together to allow

them on an interface

OpenStack Hosts

Ports 21-40

ESX Hosts

Ports 1-20

F5

Port 47

Policy Defined Network – Simple, Consistent Configuration

Concrete Model

Logical Model

APIC

APIC

APIC

ASA

Port 46

Outside_L3

Port 48

OpenStack Hosts

Switch 1-6ESX Hosts

Switches 1,3,5F5

Switches 1,2

ASA

Switches 1,2

Outside_L3

Switches 1,2

Comparing ACI and Traditional Network Building Blocks

Traditional Network – Limited Multi Tenancy

Box by box configuration

VDCs and VRFs configured on a per

switch basis

Manual inter switch configuration

ACI Tenants are Network Wide Administrative Containers

Tenant: Common

Tenant: Production Tenant: Pre-Production

Objects created in “Common” can be

consumed by other Tenants

BD: 01 BD: 02 BD: 03

VRF: A VRF: B VRF: C

AD DHCPDNS

APIC

APIC

APIC

Tenant: ESX-Hosts

BD: 01 BD: 02 BD: 03

VRF: A

Looking under the covers at Tenants

apic1# show tenant

Tenant Tag Description

--------------- --------------- ----------------------------------------

avanker

common

fgandola

hyper-v

infra

mgmt

nickmart

nvermand

nvermand-vRA-01 vRA Tenant

openstack

robvand

rwhitear

ssharman

vmware

apic1#

New NX-OS CLI in 1.2.1i

Traditional L3 Networking

VRF: VRF-01 (HSRP gateway)

VRF configuration is performed on

a switch by switch basis

ACI VRFs (aka Private Networks, aka Contexts) provide the routing function within a given Tenant

VRF: VRF-01 (Anycast gateway)

Tenant: Common

APIC

APIC

APIC

Multiple VRFs allow overlapping IP address space and Integration with External Devices

VRF: VRF-01 (Anycast gateway) VRF: VRF-02 (Anycast gateway)

Tenant: Common

APIC

APIC

APIC

Looking under the covers at VRFs

apic1# show vrf

Tenant Vrf

---------- ----------

common default

common inside_enforced

common inside_unenforced

common outside_ospf

common outside_static

common outside_vlans

fgandola VRF-01

mgmt inb

mgmt oob

nickmart nickmart

nvermand VRF-01

nvermand VRF-02

nvermand VRF-AVS

Leaf-1# show vrf

VRF-Name VRF-ID State Reason

black-hole 3 Up --

common:default 26 Up --

common:outside_ospf 5 Up --

common:outside_vlans 7 Up --

management 2 Up --

mgmt:inb 15 Up --

nickmart:nickmart 8 Up --

nvermand:VRF-01 12 Up --

nvermand:VRF-AVS 9 Up --

nvermand:VRF-int-NSX-EDGE 19 Up --

nvermand:VRF-Mig 13 Up --

nvermand:VRF-NSX 16 Up --

overlay-1 4 Up --

robvand:VRF-01 33 Up --

ssharman:VRF-01 31 Up --

VM-tenant:vcenter_default_pvn 14 Up --

vmware:VRF-01 18 Up --


Traditional L2 Networking

Layer 2 VLAN: VLAN10

VLAN configuration is performed

on a switch by switch basis

ACI Bridge Domains are Pervasive Layer 2 Boundaries with Defined Forwarding Characteristics


Bridge Domain: BD-01

APIC

APIC

APIC

Tenant: Common

BD: 03Hardware Proxy: No

ARP Flooding: Yes

Unknown Unicast Flooding: Yes

IP Routing: No


ARP Flooding: Yes


IP Routing: No


ARP Flooding: Yes


IP Routing: No

The Bridge Domain to VRF

association is always required,

even if the VRF is not routing

Display all Bridge Domains

apic1# show bridge-domain

Tenant Interface MAC Address MTU Description Multi-Dest Action Unknown Mcast Action Unknown MAC Ucast Action

---------- ---------- ------------------ -------- ------------ ----------------- -------------------- ----------------------

VM-tenant BD-02 00:22:BD:F8:19:FF inherit encap-flood flood flood

VM-tenant vcenter_de 00:22:BD:F8:19:FF inherit encap-flood flood flood

fault_bd

common outside_in 00:22:BD:F8:19:FF inherit bd-flood flood flood

fra-

ssharman

common outside_in 00:22:BD:F8:19:FF inherit bd-flood flood flood

fra-

teoyenug

ssharman 192.168.65 00:22:BD:F8:19:FF inherit bd-flood flood proxy

.0


.0


.0


.0


.0


.0


.0


Display Details of a Single Bridge Domain

apic1# show bridge-domain outside_infra-ssharman

Tenant : common

Interface : outside_infra-ssharman

MAC Address : 00:22:BD:F8:19:FF

MTU : inherit

Description :

Multi-Destination Action : bd-flood

Unknown Multicast Action : flood

Unknown MAC Unicast Action : flood

Tenant : ssharman

Interface : Internal_Fabric_02


MTU : inherit

Description :


Unknown Multicast Action : opt-flood

Unknown MAC Unicast Action : proxy


A Bridge Domains use a Locally Significant VLAN ID on each Leaf which Dynamically Maps to a VXLAN ID


Bridge Domain: outside_infra-ssharman

APIC

APIC

APIC

Tenant: Common

Leaf 101Tenant: Common

BD: outside_infra-ssharman

Leaf 102Tenant: Common

BD: outside_infra-ssharman

The Bridge Domain to VRF

association is always required,

even if the VRF is not routing

Layer 2 Bridge Domain

carried over VXLAN

VXLANs Require VTEPs

VRF: 01 (Anycast gateway)

BD: 01Hardware Proxy: Yes

ARP Flooding: No

Unknown Unicast Flooding: No

IP Routing: Yes

APIC

APIC

APIC

Known unicast traffic forwarded directly

between Leaf VTEP’s

Unknown unicast traffic is forwarded to

anycast spine proxy VTEP’s

Logical vPC switch is represented by

anycast Leaf vPC VTEP’s

Multicast and any allowed broadcast

traffic is forwarded to a Group VTEP that

exists on any leaf with membership for

that specific group

VTEP’s may exist in physical or virtual

switches

VTEP VTEP VTEP VTEP

VTEP VTEP VTEP VTEP VTEP VTEP

Tenant: Common

VTEPs are dynamically

created as required

A Bridge Domain uses a Locally Significant VLAN ID Underneathapic1# fabric 101 show vlan

----------------------------------------------------------------

Node 101 (Leaf-1)

----------------------------------------------------------------

VLAN Name Status Ports

---- -------------------------------- --------- -------------------------------

9 infra:default active Eth1/1, Eth1/21, Eth1/22, Po3, Po4

11 common:outside_infra-robvand active Eth1/11, Eth1/21, Eth1/22, Po3,

14 fgandola:www-zone1 active Eth1/33, Po2

15 ssharman:192.168.66.0 active Eth1/21, Eth1/22, Po3, Po4

26 common:outside_infra-ssharman active Eth1/11, Eth1/21, Eth1/22, Po3, Po4, Po8

apic1# fabric 102 show vlan

----------------------------------------------------------------

Node 102 (Leaf-2)

----------------------------------------------------------------


---- -------------------------------- --------- -------------------------------

9 infra:default active Eth1/1, Eth1/21, Eth1/22, Po1, Po2

11 ssharman:L2-to-outside:Group-05 active Eth1/21, Eth1/22, Po1, Po2

14 fgandola:app-zone2 active Eth1/33, Po8

15 -- active Eth1/69, Po7

35 common:outside_infra-ssharman active Eth1/11, Eth1/21, Eth1/22, Po1, Po2, Po4


Leaf 101

Leaf 102

A Bridge Domain uses a VXLAN to Transport Data Between Leaf Switchesapic1# fabric 101 show vlan id 26 extended

----------------------------------------------------------------

Node 101 (Leaf-1)

----------------------------------------------------------------


---- -------------------------------- --------- -------------------------------

26 common:outside_infra-ssharman active Eth1/11, Eth1/21, Eth1/22, Po3,

Po4, Po8

VLAN Type Vlan-mode Encap

---- ----- ---------- -------------------------------

26 enet CE vxlan-15433637

apic1# fabric 102 show vlan id 35 extended

----------------------------------------------------------------

Node 102 (Leaf-2)

----------------------------------------------------------------


---- -------------------------------- --------- -------------------------------

35 common:outside_infra-ssharman active Eth1/11, Eth1/21, Eth1/22, Po1,

Po2, Po4

VLAN Type Vlan-mode Encap

---- ----- ---------- -------------------------------

35 enet CE vxlan-15433637


Leaf 101

Leaf 102

Traditional Networking – SVI



Interface VLAN10

IP Address 192.168.10.1/24

ACI SVIs are Configured on a given Bridge Domain and Instantiated on the Associated VRF


APIC

APIC

APIC

Tenant: Common


ARP Flooding: No


IP Routing: 192.168.10.1/24

ACI Bridge Domains can be Configured with Multiple Subnets/Default Gateways (Secondary)


APIC

APIC

APIC

Tenant: Common


ARP Flooding: No


IP Routing: 192.168.10.1/24

: 192.168.20.1/24

apic1# show bridge-domain outside_infra-ssharman

Tenant : common



MTU : inherit

Description :


Unknown Multicast Action : flood

Unknown MAC Unicast Action : flood

Tenant : ssharman

Interface : Internal_Fabric_02


MTU : inherit

Description :


Unknown Multicast Action : opt-flood

Unknown MAC Unicast Action : proxy

Display Details of a Single Bridge Domain


apic1# show ip interface bridge-domain outside_infra-ssharman

----- IPv4 Bridge-Domain Information: -----

Tenant : common


VRF Member : outside_vlans

IP Addresses : 192.168.29.254/24

192.168.30.254/24

Bridge Domain + SVIBridge Domain + SVI

VRF name

Traditional Networking – Any to Any Communication



Interface VLAN10

IP Address 192.168.10.1/24

192.168.10.11/24

192.168.10.12/24

192.168.10.13/24

192.168.10.14/24

192.168.10.15/24

192.168.10.16/24

192.168.10.17/24

Any to Any Communication on a given segment*

How do devices (Endpoints) communicate on an ACI fabric?

1.

2.

3.

Application Network Profiles and Endpoint Groups

Application Network Profiles are “containers” which group together one or more EPGs and their associated connectivity policies – this is how we can view the “Health” of an application!

Application Network Profiles are used to describe either a Network service or an Application e.g.

• ESX-Hosts• Host-mgmt

• vMotion

• IP-storage

• NSX-transport

• iExpenses• SSO

• Intranet

• Database

New Concept: Application Network Profiles

Are all my ESX Hosts in a

heathy state?

What’s the health of my IP

Storage network?

What’s the health of my

iExpenses application?

Interface Usage

(Policy Groups)

vPC_to_UCS_FI_A

SVI_to_outside

Interface Parameters

(Policies)

CDP_enabled

LACP_Active

Allowed VLANs

(AAEP)

vCentre-01-vDS-01

UCS-phys-svrs

Outside-Fabric

The Lights are on – Let’s add an Application Network Profile

Target Interfaces ID

(Profiles)

vPC_to_UCS_FI_A

SVI_to_outside

Target Switches

(Profiles)

vPC_Leaf_1_and_2

Leaf_3

Concrete Model

Logical Model

Virtual Machine

Domains

(vSwitches)

vCentre-01-vDS-01

ANP: My_AppEPG: Web

EPG: App

EPG: DB

VLAN mgmt

(Phy/Out Domain)

UCS-phys_svrs

Outside_Fabric

VLAN/VXLAN

(Pools)

vCentre-01-vDS-01

UCS-phys-svrs

Outside-Fabric

ANP: ESX-MgmtEPG: Host-Mgmt

EPG: vMotion

EPG: IP-Storage

Endpoint Groups are quite simply groups of endpoints on the network.

The endpoints are identified by their connectivity Domain (virtual/physical/outside) and their connectivity method e.g.

• Virtual machine portgroups (VLAN, VXLAN)

• Physical interfaces / VLANs inc (v)port channels

• External VLANs

• External subnets

Devices within the same Endpoint group can communicate irrespective of their VLAN/VXLAN backing/ID, provided that they have IP reachability.

Communication between Endpoint groups is, by default, not permitted (similar to PVLAN).

New Concept: Endpoint Groups

Static VLANs

• Allocated manually to EPGs

• Bound to an interface

Dynamic VLANs

• Allocated dynamically to EPGs in VMM Domains representing Port Groups

• Allocated dynamically to the (shadow) EPGs representing FW or SLB interfaces as part of a service graph

• Bound to an interface

How do Endpoints (and Groups) use VLANs?

• ACI uses the concept of both Static and Dynamic VLAN Pools

• A single VLAN Pool can contain ranges of both Static and Dynamic VLANs

• VLANs are significant to the switch port meaning they can be reused across the fabric

Secure Networking with ACI End Point Groups


ANP:

ESXi-Hosts

BD: vMotionHardware Proxy: No

ARP Flooding: Yes


IP Routing: No

vPC_to_UCS_a

vlan-8

vPC_to_UCS_b

vlan-8

EPG: Host-Mgmt

Security Zone

vPC_to_UCS_a

vlan-10

vPC_to_UCS_b

vlan-10

EPG: vMotion

Security Zone

vPC_to_UCS_a

vlan-12

vPC_to_UCS_b

vlan-12

EPG: vmk-storage

Security Zone

BD: Host-MgmtHardware Proxy: No

ARP Flooding: Yes


IP Routing: No

BD: storageHardware Proxy: No

ARP Flooding: Yes


IP Routing: No

APIC

APIC

APIC

Tenant: ESXi-Hosts

Communication allowed within EPG Communication allowed within EPGCommunication allowed within EPG

Endpoints in EPG identified by

Interface and VLAN ID

Secure Networking with ACI End Point Groups


ANP:

ESXi-Hosts

BD: ESXiHardware Proxy: Yes

ARP Flooding: No


IP Routing: Yes

vPC_to_UCS_a

vlan-8

vPC_to_UCS_b

vlan-8

EPG: Host-Mgmt

Security Zone

vPC_to_UCS_a

vlan-10

vPC_to_UCS_b

vlan-10

EPG: vMotion

Security Zone

vPC_to_UCS_a

vlan-12

vPC_to_UCS_b

vlan-12

EPG: vmk-storage

Security Zone

APIC

APIC

APIC

Tenant: ESXi-Hosts

Endpoints in EPG identified by


Communication allowed within EPG Communication allowed within EPGCommunication allowed within EPG

The simple answer is, how many Layer 2 Segments do you want to have?

For example, if you have 10x external VLANs you will need 10x Bridge Domains – a Bridge Domain is a Layer 2 Segment.

If you have a Transparent Firewall you will need a 2x Bridge Domains, one either side of the Firewall – it’s just networking!!

Lets have a quick look at EPG to EPG traffic flows

Where are IP/Mac Addresses Stored?



ARP Flooding: No


IP Routing: Yes

APIC

APIC

APIC

Proxy Proxy Proxy Proxy

FIB FIB FIB FIB FIB FIB

Tenant: Common

Leaf Local Station Table

contains addresses of ‘all’

hosts attached directly to the

Leaf

10.1.3.11 Port 9

Leaf Global Station Table

contains a local cache of the

fabric endpoints

10.1.3.35 Leaf 3

Proxy A*

10.1.3.35 Leaf 3

10.1.3.11 Leaf 1Leaf 4

Leaf 6

fe80::8e5e

fe80::5b1a

Spine Proxy Station Table contains

addresses of ‘all’ hosts attached to the

fabric

High Level Packet Walk


ANP:

ESXi-Hosts


ARP Flooding: No


IP Routing: Yes

EPG: Host-Mgmt

Security Zone

Leaf-101/1/10

vlan-8

Leaf-102/1/10

vlan-8

APIC

APIC

APIC

Tenant: ESXi-Hosts

Endpoints identified by


PayloadIP

Packet Sourced from

physical server1

PayloadIPVXLANL1

VTEP

Leaf swaps ingress encapsulation with VXLAN

(EPG) ID and performs any required policy functions2

Leaf-103/1/10

vlan-8

Leaf-104/1/10

vlan-8

Leaf-105/1/10

vlan-8

Leaf-106/1/10

vlan-8

3a

If the ingress Leaf has learned the

destination IP to egress VTEP binding

it will set required destination VTEP

address and forward

PayloadIPVXLANL6

VTEP

If the ingress Leaf has NOT learned the

destination IP to egress VTEP binding

it will set required destination VTEP to

the Spine Proxy VTEP

3b

PayloadIPVXLANS1

VTEP

PayloadIP

Packet Delivered to

physical server5

Communication allowed within EPG

PayloadIPVXLANL6

VTEP

Leaf removes ingress VXLAN (EPG) ID and

performs any required policy functions4

There is no requirement to use

the same VLAN on every Leaf

Host-mgmt EPG –

Access Encap VLAN 8

Alternate command:

show vlan extended

Remember for troubleshooting use

the Internal VLAN ID not the

Access Encap VLAN ID

apic1# fabric 101 show system internal epm vlan all

+----------+---------+-----------------+----------+------+----------+---------

VLAN ID Type Access Encap Fabric H/W id BD VLAN Endpoint

(Type Value) Encap Count

+----------+---------+-----------------+----------+------+----------+---------

9 Infra BD 802.1Q 3967 16777209 11 9 3

10 Ext. BD 802.1Q 2050 15269816 12 10 0

11 Ext. BD 802.1Q 49 15531935 111 11 2

12 Tenant BD NONE 0 15662984 14 12 0

13 FD vlan 802.1Q 2022 8814 15 12 2

14 Ext. BD 802.1Q 2020 14909414 16 14 0

15 Tenant BD NONE 0 15171524 17 15 0

16 FD vlan 802.1Q 33 8324 19 15 1

17 FD vlan 802.1Q 2131 9023 20 15 0

18 Tenant BD NONE 0 15138760 18 18 0

19 FD vlan 802.1Q 2125 9017 21 18 0

20 FD vlan 802.1Q 47 8338 22 18 4

34 Tenant BD NONE 0 15302581 29 34 0

35 FD vlan 802.1Q 14 8305 40 34 4

36 Tenant BD NONE 0 15400873 30 36 0

37 FD vlan 802.1Q 8 8299 41 36 19

38 Ext. BD 802.1Q 115 15269817 31 38 1

Lets look at which VLANs/VXLANs have been used by Bridge Domains and EPGs on a given Leaf

BD_CTRL_VLAN: The infrastructure vlan which was configured during the

APIC setup script.

BD_EXT_VLAN: Bridge Domain to represent external VLAN

BD_VLAN: An internal Bridge Domain construct which is represented by

the grouping of multiple FD_VLANs/VXLANs – i.e many FD_VLANs can

map to one BD_VLAN

FD_VLAN: A VLAN backed EPG identified by the “Access encap” VLAN

ID mapped to the Bridge Domain – a FD_VLAN can only map to a single

BD_VLAN

FD_VXLAN: Used to communicate with hosts behind hypervisors using

VXLAN

Access encap: The Access_enc is significant outside the ACI network as

it is the VLAN that is programmed on a front panel port mapping inbound

frames to an EPG (FD_VLAN)

Fabric Encap: The VXLAN ID for a given EPG/BD

HW_VlanId: The VLAN used to encapsulate incoming traffic from

Access_enc to send to the ALE

VlanId: The VlanId is significant for troubleshooting, most (if not all) show

commands use the VlanId not the Access_enc VLAN ID

Display the Mac Addresses Contained in the EPG

apic1# fabric 101 show mac address-table vlan 37

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link,

(T) - True, (F) - False

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 37 0000.0c07.ac08 dynamic - F F po2

* 37 001a.a2d5.c080 dynamic - F F po2

* 37 02a0.981c.b2be dynamic - F F po2

* 37 0026.0bf1.f002 dynamic - F F po2

* 37 0014.384e.26e1 dynamic - F F po2

* 37 0016.355b.ddda dynamic - F F po2

* 37 0060.1646.97da dynamic - F F po2

* 37 0010.18cf.c318 dynamic - F F po2

* 37 0018.74e2.1540 dynamic - F F po2

* 37 0004.02f6.1f13 dynamic - F F po2

* 37 0025.b506.006d dynamic - F F po2

* 37 001b.21be.fa68 dynamic - F F po2

* 37 0025.b501.04af dynamic - F F po2

* 37 0025.b501.049f dynamic - F F po2

* 37 0025.b501.04bf dynamic - F F po2

* 37 0025.b506.007c dynamic - F F po2

* 37 0025.b501.04df dynamic - F F po2

* 37 0025.b506.0027 dynamic - F F po2

* 37 0025.b506.0068 dynamic - F F po2

Displaying the Endpoints on the Network

apic1# show endpoints

Tenant Application AEPg End Point MAC IP Address Node Interface Encap

---------- ----------------- ---------------------------------------- ---------- ------------------------------ ----------

vmware ESXi- Host-mgmt 00:25:B5:06:00:1F 192.168.29.43 101 102 vpc 1Gbps_vPC_to_ucs-02-b vlan-8

ssharman

vmware ESXi- Host-mgmt 00:25:B5:06:00:3E 192.168.29.44 101 102 vpc 1Gbps_vPC_to_ucs-02-b vlan-8

ssharman

vmware ESXi- Host-mgmt 00:25:B5:06:00:47 192.168.29.46 101 102 vpc 1Gbps_vPC_to_ucs-02-b vlan-8

ssharman

vmware ESXi- Host-mgmt 00:50:56:86:81:1D 192.168.29.102 101 102 vpc 1Gbps_vPC_to_ucs-02-b vlan-8

ssharman

vmware ESXi- Host-mgmt 00:50:56:86:F7:6A 192.168.29.106 101 102 vpc 1Gbps_vPC_to_ucs-02-b vlan-8

ssharman


Displaying the Endpoints on a Leaf

apic1# fabric 101 show endpoint

Legend:

O - peer-attached H - vtep a - locally-aged S - static

V - vpc-attached p - peer-aged L - local M - span

s - static-arp B - bounce

+-----------------------------------+---------------+-----------------+--------------+-------------+

VLAN/ Encap MAC Address MAC Info/ Interface

Domain VLAN IP Address IP Info

+-----------------------------------+---------------+-----------------+--------------+-------------+

common:outside_ospf 101.1.1.1 L

44/common:outside_ospf vxlan-15302582 0000.0c07.ac30 L eth1/96

44/common:outside_ospf vxlan-15302582 0018.74e2.1540 L eth1/96

44/common:outside_ospf vxlan-15302582 001a.a2d5.c080 L eth1/96

13 vlan-2022 0025.b506.0062 LV po3

common:outside_vlans vlan-2022 192.168.22.14 LV

13 vlan-2022 0025.b506.0002 LV po3



32 vlan-22 0000.0c07.ac16 LV po2


32 vlan-22 001a.a2d5.c080 LV po2


32/common:outside_vlans vlan-22 0018.74e2.1540 LV po2

32 vlan-22 0050.5699.9099 LV po2


32 vlan-22 0050.5699.7e05 LV po2

Advanced Query: How to find if/where any VLAN has been used

apic1# moquery -c fvIfConn | grep dn | grep common | grep vlan

dn: uni/epp/br-[uni/tn-common/l2out-outside_infra-robvand/instP-EPG_outside_infra-robvand]/node-102/stpathatt-

[1Gbps_vPC_to_n5548]/conndef/conn-[vlan-47]-[0.0.0.0]

dn: uni/epp/br-[uni/tn-common/l2out-outside_infra-robvand/instP-EPG_outside_infra-robvand]/node-101/stpathatt-


dn: uni/epp/br-[uni/tn-common/l2out-outside_infra-anvanker/instP-EPG_outside_infra-anvanker]/node-102/stpathatt-


dn: uni/epp/br-[uni/tn-common/l2out-outside_infra-anvanker/instP-EPG_outside_infra-anvanker]/node-101/stpathatt-


Interface

Connection

Distinguished

Name

Tenant

NameVLAN

Managed

ObjectClass

How do I control Endpoint Group communication?

Filter: Any-TrafficFilter: 80, 443 etc EPG:

Clients

Contract: Any-to-Any

Contracts are “directional” Access Lists between Provider and Consumer EPGs. They comprise of one or more Filters (ACEs) to identify traffic, e.g:

• Contract: Any-to-Any | Filter: Any-Traffic

• Contract: Web | Filter: 80, 443, 8000

• Contract: DNS | Filter: 53

New concept: Contracts (ACLs)

Provider Consumer

Contract: Clients-to-Web

Filter: none

Flags:

• Apply in both directions (single contract which allows return traffic)

• Reverse filter ports (dynamically permits return flow based on src/dst ports)

Flags:

• IP Protocol

• Ports

• Stateful

• Etc.

EPG: Web

ExternalSubnet

ANP:

My-Web-App

L3out:

Clients

Contracts are Required for Inter EPG Connectivity


ANP:

ESXi-Hosts

Primary Gateway:192.168.10.1/24

Secondary Gateway: 192.168.20.1/24

Contract = Allow Communication No Contract = No Communication

vPC_to_UCS_a

vlan-30

vPC_to_UCS_b

vlan-30

EPG: vmk-storage

192.168.20.11 192.168.20.12

vPC_to_UCS_a

vlan-8

vPC_to_UCS_b

vlan-8

EPG: Host-Mgmt

192.168.10.11 192.168.10.10

APIC

APIC

Tenant: ESXi-Hosts

APIC

vPC Node104_105/1/50

vlan-40

EPG: Shared-storage

192.168.20.10

ANP:

ESXi-Storage


ARP Flooding: No


IP Routing: 192.168.10.1/24

: 192.168.20.1/24

Contracts are “scoped” at:

• Global

• Tenant

• Context (aka Private Network, aka VRF)

Web_to_App

• Application Profile

App_to_DB

Contracts Scope

ANP: 01

EPG: Web

EPG: App

EPG: DB

ANP: 02

EPG: Web

EPG: App

EPG: DB

VRF: 01

Tenant: Web_Hosting


IP Routing: Yes

What happens if I don’t know the required Filter ports?

• Ask the Application Owner – it’s their application, they will (ok should) know

• Ask the Security Admin for the firewall rules

• Use an “any-any” Filter between EPGs Most customers start here

• Use Wireshark

• Configure “Unenforced” mode on the VRF

Filter Discovery

BRKACI-1002

How does ACI integrate with VMware’s virtual

switches?

1. Manually configure the vSwitch/vDS as you do today

2. Dynamically configure the vDS (VMware) by pushing Port Groups (VLAN) from APIC to vCentre

3. Dynamically configure the vDS (Cisco AVS) by pushing Port Groups (VLAN/VXLAN) from APIC to vCentre

4. Build NSX overlay networks (VXLAN) between different hosts –requires additional (costly) NSX licenses from VMware

There are four Choices to Integrate with VMware

Traditional NetworkingSVI | VLAN | Port Group Relationship



Interface VLAN10

IP Address 192.168.10.1/24

vDS-01

VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM VM

Port Group: Web

(VLAN 10)

Host-01 Host-02 Host-03 Host-04

EPG to vDS Port Group Relationship

ANP: My-App-01

vCentre

Service Request:

Create Application

Create vDS Port Groups

Tenant: Tenant-01

APIC

APIC

BD: AppsIP Routing: 192.168.10.1/24

Outside

EPG: Web (Dynamic VLAN 2001)

vDS-01



APIC

Port Group: VMware|My-App-01|Web

(Dynamic VLAN 2001)


Security Groups within a Subnet

ANP: My-App-01

No Contract = No Communication

vCentre

Service Request:

Create Application

Create vDS Port Groups

Tenant: Tenant-01

APIC

APIC

BD: AppsIP Routing: 192.168.10.1/24

Outside

EPG: App (Dynamic VLAN 2002)EPG: Web (Dynamic VLAN 2001) EPG: DB (Dynamic VLAN 2003)

vDS-01


Contract = Allow Communication Contract = Allow Communication


APIC

Port Group: VMware|My-App-01|Web

(Dynamic VLAN 2001)

Port Group: VMware|My-App-01|App

(Dynamic VLAN 2002)Port Group: VMware|My-App-01|DB

(Dynamic VLAN 2003)


PS PS

(Eth1/50, 51 VLAN 3600)

NSX Overlay

ANP: Overlay_Network

vCentre

Tenant: Tenant-01

APIC

APIC

Outside

EPG: NSX_Transport (VLAN 1000)

APIC

vDS-01

(not managed by APIC)

VLAN 1000 VTEP 10.0.0.4VTEP 10.0.0.3VTEP 10.0.0.2VTEP 10.0.0.1

VM VM VM VM VM VM VM

Dedicated Hosts for

“Edge” Functionality

NSX Logical Switch:

Layer 2 segment carried over

VXLAN, carried over a

dedicated VLAN

DLR DLR B/U

NSX ESG Routers Peer

with the Physical Network

NSX Manager

APIC Configures fabric with an NSX

Transport EPG (VLAN) across all hosts

ESG ESG B/U

NSX DLR informs

controllers of learnt routes

VRF: VRF-01

EPG

VM VM VM VM VM

BD: NSXIP Routing: Yes

Controllers push

routes to Hosts

L3outInterface: VLAN 2000

IP: 192.168.30.1

IP: 192.168.30.2

NSX Controller Cluster

• Let’s look at vSphere 6.0 Official Documentation about kernel Virtual Installation Bundles (VIB) - http://vmw.re/1Ta1Zz0

Cisco AVS is a Partner Supported VIB

http://vmw.re/1Ta1Zz0

• Cisco AVS Statement of Support

Customers call Cisco for AVS Support

OpFlex

VM VM VM

VMware ESXi Server

VM VM VM

VMware ESXi Server

OpFlex

VMware vCentreCisco APIC

VMM Domain

AVS AVS

http://www.cisco.com/c/dam/en/us/products/collateral/switches/application-

virtual-switch/avs-support-statement-an.pdf

http://www.cisco.com/c/dam/en/us/products/collateral/switches/application-virtual-switch/avs-support-statement-an.pdf

BRKACI-1002

How do I Provide External Connectivity to the ACI

Fabric?

Layer 2 Connectivity:1 Bridge Domain = 1 Outside VLAN

Option 1: Same VLANs Outside/Inside (No Contract Required)


ANP:

ESXi-Hosts

BD: InsideHardware Proxy: No

ARP Flooding: Yes


IP Routing: No

vPC_to_UCS_a

vlan-10

vPC_to_UCS_b

vlan-10

EPG: Host-Mgmt

192.168.10.11 192.168.10.10

vPC_to_n5ks

vlan-10

vlan-10

APIC

APIC

APIC

Tenant: ESXi-Hosts

Option 2: Different VLANs Outside/Inside(Contract Required)


ANP:

ESXi-Hosts

Contract = Allow Communication


EPG

vPC_to_UCS_a

vlan-100

vPC_to_UCS_b

vlan-100

EPG: Host-Mgmt

192.168.10.11 192.168.10.10

vPC_to_UCS_a

vlan-20

vPC_to_UCS_b

vlan-20

EPG: vMotion

192.168.20.11 192.168.20.10

BD: InsideHardware Proxy: Yes

ARP Flooding: No


IP Routing: 192.168.10.1/24

: 192.168.20.1/24

L2outInterface: vPC_to_n5ks

VLAN: 10

vlan-10

APIC

APIC

APIC

Tenant: ESXi-Hosts

Layer 3 connectivityACI only learns routes via “L3out’s” – these are simply routed interfaces/sub interfaces/SVIs

Layer 3 External


ANP:

ESXi-Hosts



vPC_to_UCS_a

vlan-20

vPC_to_UCS_b

vlan-20

EPG: vMotion

192.168.20.11 192.168.20.10


ARP Flooding: No


IP Routing: 192.168.10.1/24

L3outInterface: 101/102 eth1/96

IP: 192.168.30.1

IP: 192.168.30.5

BD subnet control

Advertise, Private etc

Outside

Security Import Subnet*

i.e which external subnets can

be accessed through this EPG

APIC

APIC

APIC

Tenant: ESXi-Hosts

EPG

vPC_to_UCS_a

vlan-100

vPC_to_UCS_b

vlan-100

EPG: Host-Mgmt

192.168.10.11 192.168.10.10

OSPF Peering

Looking Under the Covers at Routing

apic1# fabric 101 show ip route ospf vrf ssharman:VRF-01

----------------------------------------------------------------

Node 101 (Leaf-1)

----------------------------------------------------------------

IP Route Table for VRF "ssharman:VRF-01"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

10.51.226.0/24, ubest/mbest: 1/0

*via 192.168.48.2, vlan59, [110/1], 02w18d, ospf-default, type-2

10.51.227.0/24, ubest/mbest: 1/0


10.52.204.112/28, ubest/mbest: 1/0

*via 192.168.48.2, vlan59, [110/5], 02w20d, ospf-default, inter

10.52.205.128/27, ubest/mbest: 1/0


10.52.205.160/27, ubest/mbest: 1/0


10.52.207.100/32, ubest/mbest: 1/0


10.52.248.0/26, ubest/mbest: 1/0

*via 192.168.48.2, vlan59, [110/5], 02w20d, ospf-default, inter

External Subnets for the External EPG

Outside Outside

MP BGP

EPG EPG

Subnet 100.1.1.0/24 can be

accessed via EPG

Subnet 60.1.1.0/24 can be

accessed via EPG

L3out L3out

VRF: Production

100.1.1.0/2460.1.1.0/24


Tenant: Common


ARP Flooding: No


IP Routing: 192.168.10.1/24

Transit Routing – Static Routes

Outside Outside

MP BGP

EPG

Static Routes must be

individually exported,

0.0.0.0/0 is not supported

L3out L3out

60.1.1.0/24

VRF: Production

100.1.1.0/24


Tenant: Common

EPG


ARP Flooding: No


IP Routing: 192.168.10.1/24

Static route to 60.1.1.0/24 via

next hop

Static route to 100.1.1.0/24

via next hop

Static route to 100.1.1.0/24

via next hop

Static route to 60.1.1.0/24 via

next hop

Transit Routing – Multiple L3 Out per VRF

Outside Outside

MP BGP

L3out

EPG EPG

L3out


Use a 0.0.0.0/0 subnet with

the ‘aggregate export’ option

checked to export all routes

VRF: Production

70.1.1.0/24

80.1.1.0/24

60.1.1.0/24

Tenant: Common


ARP Flooding: No


IP Routing: 192.168.10.1/24

Import Route Control (BGP only)

Outside Outside

MP BGP

EPG EPGContract = Allow Communication

Which routes should be

imported to the fabric

L3out L3out

VRF: Production

70.1.1.0/24

80.1.1.0/24

60.1.1.0/24

Tenant: Common


ARP Flooding: No


IP Routing: 192.168.10.1/24

BRKACI-1002

Service Graphs and Service Chains

Service Graph Contracts connect two EPGs and optionally provide configuration parameters to the FW and SLB which sit between the EPGs

Note: Normal L2/L3 rules still apply, you still have to direct the traffic to the FW/SLB

In “Managed” mode the APIC pushes the required VLANs and configuration to the FW/SLB


In “Unmanaged” mode the APIC only pushes the required VLANs to the EPG


Service Chains are two L4-7 Devices linked in a series


It is possible to use L4-7 Devices without Service Graphs, in this mode the fabric only provides L2 connectivity

Transparent Firewall – Server’s Default Gateway is the Bridge Domain on the ACI Fabric

EPG: Servers_Inside

L3out

EPG: Servers_Outside

Sta

nd

ard

_C

on

tra

ct

ANP: My-App-01 Service_Graph_Contract

BD: OutsideHardware Proxy: No

ARP Flooding: Yes


IP Routing: Yes


ARP Flooding: Yes


IP Routing: No

Connector type must

be specified as L2

Connector type must

be specified as L2

Tenant: Common

192.168.10.x/24192.168.10.x/24

Servers_Outside can

communicate externally via

the contract to the L3out

Servers_Outside can communicate

with Servers_Inside via the Service

Graph Contract

VRF not used

Server default

gateway

VRF: 01 VRF: 02

Transparent Firewall – Server’s Default Gateway is the Bridge Domain on the ACI Fabric

EPG: Servers_InsideANP: My-App-01

L3out

BD: OutsideHardware Proxy: No

ARP Flooding: Yes


IP Routing: Yes


ARP Flooding: Yes


IP Routing: No

Service_Graph_Contract

Tenant: Common

192.168.10.x/24

Server default

gateway

Connector type must

be specified as L3

Connector type must

be specified as L2

Servers_Inside can communicate to

the “outside world” via the Service

Graph Contract to the L3out

192.168.10.x/24

VRF not used

VRF: 01 VRF: 02

Routed Firewall – Server’s Default Gateway is the Firewall attached to the ACI Fabric

EPG: Servers_InsideANP: My-App-01


ARP Flooding: Yes


IP Routing: No

L3out

L3out

Tenant: Common

Connector type must

be specified as L3

Connector type must

be specified as L2

Servers_Inside can communicate to



Server default

gateway

Static route to firewall

“inside” subnet via

L3out ot Firewall

VRF has Static route to

firewall “inside” subnet

via L3out to Firewall

192.168.10.x/24

10.1.1.0/30

VRF not used


VRF: 01 VRF: 02

Routed Firewall – Server’s Default Gateway is the Bridge Domain on the ACI Fabric

EPG: Servers_Inside

Server default

gateway

ANP: My-App-01


ARP Flooding: No


IP Routing: Yes

L3out

L3out VRF: 01

L3out

VRF: 02

Connector type must

be specified as L3

Connector type must

be specified as L3

Tenant: Common Servers_Inside can communicate to



10.1.1.0/30 10.1.2.0/30

192.168.10.x/24


Static route to firewall

“inside” subnet via

L3out ot Firewall

VRFs peer with Firewall

via L3out

Install a L4-7 device once (e.g the ASA firewall) and deploy it multiple times in different logical topologies

The benefits of the service graph are:

• Reusable configuration templates

• Automatic management of VLAN assignments

• Health score collection from the L4-7 device

• Statistics collection from the L4-7 device

• Automatic ACLs and Pools configuration with endpoint discovery

Service Graph Benefits

ADC Device Package Status (as of 09/02/2016)

Device

Package

Status

Virtual

and

physical

Mode Function

Profile

HA Multi-context on physical appliance Dynamic

Routing

Dynamic

EPG

IPv6 Feature Operational

model

Citrix

NetScaler

FCS Yes Go-To

(one-arm and

two-arm)

Yes No

(manual

OOB)

Yes

Create Virtual instance on SDX

manually

Yes Yes

member of

pool for VIP

Yes ADC Everything via

APIC

F5

BIG-IP LTM

FCS Yes Go-To

(one-arm and

two-arm)

Yes Yes Yes

Create route-domain on physical LTM

automatically or create vCMP

manually (no HA)

No Yes

member of

pool for VIP

No ADC Everything via

APIC

or BIG-IQ

F5

Big-IQ cloud

Q1CY16 Yes - - - - - - - - -

A10

Thunder

FCS Yes Go-To

(one-arm and

two-arm)

No No

(manual

OOB)

No No No No ADC Everything via

APIC

Radware

Alteon

FCS Physical Go-To No No No No No No ADC Everything via

APIC

Avi Networks FCS Virtual

only

Go-To Yes Yes - No No No ADC Avi controller is

required.

FW Device Package Status (as of 09/02/2016)

Device

Package

Status

Virtual

and

physic

al

Mode Functio

n

Profile

HA Multi-context on physical appliance Dynamic

Routing

Dynamic EPG IPv6 Feature Operational

model

Cisco

ASA

FCS Yes Go-To

Go-Through

Yes Yes Yes

Create context on ASA5500X manually

allocate-interface to each context is done

by APIC

Yes Yes

object-group for

ACE

Yes FW,

ACL,NAT

Everything

via APIC

Palo Alto CA Yes Go-To Yes No No No

1HCY16

planning

No No FW Panorama is

required

Cisco

FirePOWER

FCS Oct

2015, in

controlled

introduction

Yes Go-Through Yes No No - - - IPS Everything

via APIC

Checkpoint Q2CY16 Yes Go-To

Go-Through

Yes Yes

(manual

OOB)

Yes No No Yes FW Everything

via APIC

Fortinet Q2CY16 Yes Go-To

Go-Through

Yes Yes Yes No No Yes FW Everything

via APIC

BRKACI-1002

How should I get started with ACI?

Choose your Management Method(s)

Connect the old to the new

APIC

APIC

APIC

Layer 2 vPC to existing

network

Layer 3 (OSPF etc) to

existing network

Connect new workloads

to the ACI fabric and

route out

Separate “border leafs”

shown for clarity

vDS-02vDS-01vDS-01

Separate “border leafs”

shown for clarity

BRKACI-1002

Key Takeaways

Managed Object Hierarchy

EP EP

EPGEPG

EP EP

Bridge Domain

(Flood)

EP EP

EPGEPG

EP EP EP EP

EPGEPG

EP EP

Bridge Domain

(Hardware Proxy)

Tenant “Private”

Private Network

(VRF)

Private Network

(VRF)

Tenant “Common”

Bridge Domain

(Hardware Proxy)

Application Network Profile

OutsideOutside

Requirements Hardware Proxy no ARP flooding IP Routing Subnet Check

Routed traffic, no silent hosts Yes Yes Yes Yes

Routed traffic, silent hosts Yes ARP flooding (optional

since Subnet is present)

(*)

Yes Yes

non-IP switched traffic, silent hosts No N/A No No

non-IP switched traffic, no silent hosts Yes N/A No No

IP L2 switched traffic, silent hosts Yes ARP flooding (optional if

Subnet is present) (*)Yes (for advanced

functions and aging)

Yes (for aging and ARP

gleaning)

IP L2 switched traffic, no silent hosts Yes no ARP flooding (if hosts

send DHCP requests or

gratuitous ARP)

Yes (for advanced

functions and aging)

Yes (for aging and ARP

gleaning)

Bridge Domain Options

(*) if the Subnet is configured ACI can do ARP gleaning so ARP flooding is not strictly needed

1. You must have at least one Tenant or use the Common Tenant

2. VRFs are constrained within Tenants

3. VRFs provide external L3 connectivity (with a contract)

4. You must have at least one Bridge Domain

5. Bridge Domains determine the L2 forwarding characteristics

6. Bridge Domains provide internal L3 connectivity (default gateways)

7. Bridge Domains to outside VLANs must be mapped 1:1

8. Endpoint Groups map to a single Bridge Domain

9. Endpoint Groups are security zones where communication is allowed

10. Communication between Endpoint Groups is allowed through contracts (ACLs)

11. Endpoint Groups must be bound to a virtual, physical, or outside domain

12. Endpoint Groups allow you to mix and match VLANs/VXLANs/interfaces (access, port channel, virtual port channel)

13. Endpoints can only be a member of a single Endpoint Group

14. AAEP’s allow VLANs on interfaces or VMM domains

ACI Networking Rules!

BRKACI-1002

Q & A

MTE Thursday @ 14:00

Complete Your Online Session Evaluation

Learn online with Cisco Live!

Visit us online after the conference

for full access to session videos and

presentations.

www.CiscoLiveAPAC.com

Give us your feedback and receive a

Cisco 2016 T-Shirt by completing the

Overall Event Survey and 5 Session

Evaluations.– Directly from your mobile device on the Cisco Live

Mobile App

– By visiting the Cisco Live Mobile Site http://showcase.genie-connect.com/ciscolivemelbourne2016/

– Visit any Cisco Live Internet Station located

throughout the venue

T-Shirts can be collected Friday 11 March

at Registration

http://www.ciscoliveapac.com/

http://showcase.genie-connect.com/ciscolivemelbourne2016/

Thank you

aci for network...

Documents