hpe mellanox switch training - home | iotlabs...pptv network diagram 3 link aggregation data vlan...
TRANSCRIPT
HPE Mellanox Switch TrainingSuchart BoonpanMASE, CCNP, ACE-A, ACMP
May 2020
Network Diagram
Confidential – For Training Purposes Only 2
PPTV NETWORK DIAGRAM
3
Link Aggregation
Data VLAN 300
ILO
Cisco core L3 #1, #2
SN2410M
Cisco Mgmt/iLO switch
VLAN 400
Mgmt0, 1G
iLO
Qumulo
Servers 1-6
MLAG Mgmt IPMgmt0 SW1: 10.0.99.61Mgmt0 SW1: 10.0.99.62MLAG VIP: 10.0.99.63
MAGP 2:Interface vlan: 400State : MasterVirtual IP : 10.0.102.1SW1 IP : 10.0.102.10/24Sw2 IP : 10.0.102.10/24Virtual MAC : 00:00:5E:00:01:02
inter-peer link (IPL) Management Plane (Active/Standby)
SN2410MMgmt0, 1G
P49-50100GMpo1-6
P1-6 25G
Mpo30P24, 10G
MAGP 1:Interface vlan: 300State : MasterVirtual IP : 10.0.1.254SW1:IP: 10.0.1.252/23SW2:IP: 10.0.1.253/23Virtual MAC : 00:00:5E:00:01:01
4
Cost-optimized unique form factors Optimized for storage performanceFlexible pricing & investment protection
SN2010M SN2100M SN2410bM SN2410M SN2700M/SN2745M
18 Port 8 / 16 Port 24 / 48 Port 24 / 48 Port 16 / 32 Port
Half Width Half Width Full Width Full Width Full Width / Full & Short Depth
18 x 1/10/25G SFP28 + 4 x 40/100G QSFP28
16 x 40/100 QSFP28 48 x 10G SFP + 8 x 40 QSFP
48 x 10/25G SFP28 + 8 x 40/100G QSFP
32 x 40/100G QSFP28
34 x 10/25G SFP28 64 x 10/25 SFP28 64 x 10GbE SFP 64 x 10/25G SFP28 64 x 10/25G SFP28
HPE M-SERIES SWITCHES SPECIFICATION
5
DOWNLINKS (COMPUTE AND STORAGE CONNECT)HPE M‐Series SN2010M
(18) 10/25Gb SFP28 Ports(4) 10/25/40/100Gb QSFP28 Ports
25G
25G
25G SFP+ DAC
25Gb SFP+ Direct Attached Copper (DAC .5m to 3m)HPE 25Gb SFP28 to SFP28 0.5m Direct Attach Copper Cable (844471-B21)
HPE 25Gb SFP28 to SFP28 1m Direct Attach Copper Cable (844474-B21)
HPE 25Gb SFP28 to SFP28 3m Direct Attach Copper Cable (844477-B21)
6
CROSSLINKS (MLAG 100G PREFERRED BEST PRACTICE)
HPE M‐Series SN2100M(16) 10/25/40/100Gb QSFP28 Ports
100GHPE M‐Series SN2010M(18) 10/25Gb SFP28 Ports
(4) 10/25/40/100Gb QSFP28 Ports
100G QSFP28 DAC
100G
100G
100G
100Gb QSFP28 Direct Attached Copper (DAC 1m)HPE X240 100G QSFP28 to QSFP28 1m Direct Attach Copper Cable (JL271A)
7
M-SERIES SN2410M ETHERNET SWITCH
8
SN2410M M-SERIES NETWORK CONNECTIONS
25 Gb DAC cables
1 Gb UTP
1Gb, 10Gb, 25 Gb uplink (with
matching transceiver or adapter) 100 Gb DAC cables
10 Gb DAC cables
Q3
Q1
Q2
Q4
MLAG 1
MLAG 2Edge 2
Edge 1
MLAG 1
MLAG 2
MGMT
MGMT
Q1 iLO
Q2 iLO
Q3 iLO
Q4 iLO
Tor1 Mgmt
Tor2 Mgmt
Q ports go to the 640SFP28 NIC in the Qumulo Nodes.
Comp1
Comp2
Comp ports go to the 640SFP28 NIC in the Compute Nodes.
Comp3 iLO
Comp4 iLO
Comp1 iLO
Comp2 iLO
Comp3
Comp4
Q3
Q1
Q2
Q4
Edge 2
Edge 1
Comp1
Comp2
Comp3
Comp4
Q ports go to the 640SFP28 NIC in the Qumulo Nodes.
Comp ports go to the 640SFP28 NIC in the Compute Nodes.
Q&A
– Question ?
– Break 5 m
9
Mellanox Switch User Interfaces
Confidential – For Training Purposes Only 10
User Interfaces of the MLNX-OS®
1. Command Line Interface (CLI)
The CLI is accessed through: SSH, Telnet sessions, or directly via the console port on the front pane
2. Web Interface (web GUI)
The Web GUI is accessed through: HTTP or HTTPS
11
Connection with MLNX-OS
1. Access the MLNX-OS CLI via Serial Console.
12
User name and password
3. Log in with default credentials.– User name: admin– Password: admin
Confidential – For Training Purposes Only 13
login as: admin
Mellanox MLNX-OS Switch Management
Using keyboard-interactive authentication.Password:admin
Connection with MLNX-OS
2. Access the MLNX-OS WebUI via HTTP/HTTPS
14
Network
Mellanox Switch
PC
Connection with MLNX-OS
3. Access the MLNX-OS CLI via SSH
15
Network
Mellanox Switch
PC
CLI configuration modes—Standard and enable
Standard– Identified by the CLI prompt >.– Most restrictive mode.– Includes commands that query only a restricted set of state information.– Users cannot take any actions that directly affect the system—like rebooting the switch, or changing the
configuration.– Use the enable command to move from standard mode to enable mode.
Enable– Identified by the CLI prompt #.– Offers commands to view all state information, and take actions like rebooting the system.– Does not allow you to change any configurations.
– Use the disable command to move from enable mode to standard mode.
Confidential – For Training Purposes Only 16
switchA [standalone: master] > enable
switchA [standalone: master] # disable
Global configuration mode—Config
Config
– Identified by CLI prompt (config)#.
– Allowed only for user accounts in the “admin” role.
– Has a full, unrestricted set of commands to view anything, take any action, and change any configuration.– Use the configure terminal command to move from enable mode to config mode.
– Use ‘exit’ command to move from config mode to enable mode.
Confidential – For Training Purposes Only 17
g1switchA [standalone: master] # configure terminalg1switchA [standalone: master] (config) #
g1switchA [standalone: master] (config) # exitg1switchA [standalone: master] #
Getting help
– Use ‘?’ from any mode to view available commands.Use the space bar to see more commands, or ‘q’ to quit the display.
– Use the command followed by ‘?’ to view available command parameters.
Confidential – For Training Purposes Only 18
switchA [standalone: master] (config) # ?aaa Configure Authentication, Authorization, and Accountingaccess-list Configure access-list actionbanner Set system bannersboot Configure system boot parametersclear Reset certain statistics or clear cachescli Configure CLI shell optionsclock Set the system clock or timezoneconfiguration Manipulate configuration files
switchA [standalone: master] (config) # show ?aaa Display Authentication, Authorization, and Accounting settingsaccess-list Display IPV4 informationaccess-lists List access listsasic-version Display asic versionbanner Display banner settingsbootvar Display installed system images and boot parameterscli Display CLI optionsclock Display system time and date
CLI commands autocomplete
– Use [Tab] to auto-complete commands.c [Tab] - displays all commands that start with ‘c.’
– co [Tab] - autocompletes to ‘configure.’
– Unique prefix of a command can be used, instead of the full command.Example: ‘co t’ can be used instead of ‘configure terminal.’
Confidential – For Training Purposes Only 19
switchA [standalone: master] # cclear cli configure crypto
switchA [standalone: master] # co tswitchA [standalone: master] (config) #
Saving the configuration
– Save running-config into active-config.
Or
Confidential – For Training Purposes Only 20
switchA [standalone: master] (config) # configuration writeswitchA [standalone: master] (config) # show configuration filesinitial (active)initial.bak
Active configuration: initialUnsaved changes: no
switchA [standalone: master] (config) # write memory
Mellanox Switch Image (Operating System)
Confidential – For Training Purposes Only 21
MLNX-OS images
– Two images are stored in the flash memory: Partition 1 and Partition 2.
– By default, the image from Partition 1 is loaded at reboot.
– Mellanox Operating system = ONYX = MLNX-OS (same thing)
Confidential – For Training Purposes Only 22
switchA [standalone: master] (config interface mgmt1) # show images
Installed images:Partition 1:
version: X86_64 3.8.2204 2019-12-29 16:11:11 x86_64
Partition 2:version: X86_64 3.7.1134 2019-01-24 13:38:57 x86_64
Last boot partition: 1Next boot partition: 1
WebUI Onyx Image Upgrade
Confidential – For Training Purposes Only 23
1. Use the IP address of each Switch of the management interface on the address bar of your browser, example;
(switchA 10.25.19.11)(switchB 10.25.19.12)
2. Type in user name and password default: admin, admin
3. Press Login
WebUI Onyx Image Upgrade
Confidential – For Training Purposes Only 24
1. Choose System
2. Choose Onyx Upgrade
3. Select Install from local file:
4. Select Choose file:
5. Select Install Image
WebUI Onyx Image Upgrade
Confidential – For Training Purposes Only 25
1. Choose System
2. Choose Onyx Upgrade
3. Select Install from local file:
4. Select Choose file: onyx-X86_64-3.8.2204
5. Select Install Image
WebUI Onyx Image Upgrade
Confidential – For Training Purposes Only 26
1. Choose System
2. Choose Onyx Upgrade
3. Select Install from local file:
4. Select Choose file: onyx-X86_64-3.8.2204
5. Select Install Image
WebUI Onyx Image Upgrade
Confidential – For Training Purposes Only 27
1. Choose System
2. Choose Onyx Upgrade
3. Select Install from local file:
4. Select Choose file: onyx-X86_64-3.8.2204
5. Select Install Image
WebUI Onyx Image Upgrade
Confidential – For Training Purposes Only 28
1. Choose System
2. Choose Onyx Upgrade
3. Select Install from local file:
4. Select Choose file: onyx-X86_64-3.8.2204
5. Select Install Image
WebUI Update Status
Confidential – For Training Purposes Only 29
1. Please note file copy and then Install
2. Please note Image Update Status
3. Please select Reboot
WebUI Update Status
Confidential – For Training Purposes Only 30
1. Please note file copy and then Install
2. Please note Image Update Status
3. Please select Reboot
WebUI Update Status
Confidential – For Training Purposes Only 31
1. Please note file copy and then Install
2. Please note Image Update Status
3. Please select Reboot
Q&A
– Question ?
– Break 10 m or Lunch
32
MLAG – Multi Chassis LAG
Confidential – For Training Purposes Only 33
List of Network protocols used in this project.
34
– MLAG L2
– MLAG L3 (MAGP)
– Interface Port-Channel (Link Aggregation)
– Interface MLAG Port-Channel (Multi Chassis Link Aggregation)
– Spanning tree mode RPVST
– Switch port mode Hybridge– Static Route
MLAG – Multi Chassis LAG:
– Physical ports of two separate switches are aggregated in one logical port.
– MLAG switches appear as a single Layer 2 switch.
– A peering device (host or switch) runs a standard LAG, and is not aware of the fact that its LAG is connected to two separate switches.
– MLAG provides:
– High bandwidth and load-balancing
– High availability in case of a link failure
– High availability in case of a switch failure or a switch software upgrade
Layer 3 Network
LAG
MLAG
IPL
Qumulo 1
switchA switchB
Layer 3 Network
The MLAG protocol components.
36
• Keepalive
• Unicast and multicast sync• MLAG port sync
MLAG Keepalive and Failover
37
– Master election in MLAG is based on the highest IPL VLAN interface IPs of the nodes.
– The MLAG pair of switches periodically exchanges a keepalive message (via IPL)
– If the keepalive message fails to arrive for three consecutive intervals the switches break into two standalone switches.
– If IPL fail, the slave shuts down its interfaces to avoid a split brain scenario and the master becomes a standalone switch.
Unicast and Multicast Sync
38
– It prevents unicast asymmetric traffic from loading the network with flood traffic
MLAG Port Sync
39
– Under normal circumstances, traffic from the IPL cannot pass through the MLAG ports (the IPL is isolated from the MLAG ports).
– If one of the MLAG links break, the other MLAG switch opens that isolation and allows traffic from its peer through the IPL to flow via the MLAG port which accesses the destination of the fallen link.
1. Enable IP routing – MLAG may be enabled without IP routing, but without IP routing an IPL VLAN interface cannot be
configured and thus MLAG does not function.2. Enable IGMP snooping
– MLAG may be enabled without IGMP snooping, but if IGMP snooping is disabled, multicast FDBs do not synchronize.
3. Enable LACP – if dynamic LAG is used.4. Enable MLAG protocol commands.
– g1switchA configuration:
– g1switchB configuration:
MLAG Global Configurations
g1switchA [standalone: master] (config) # ip routingg1switchA [standalone: master] (config) # ip igmp snoopingg1switchA [standalone: master] (config) # lacpg1switchA [standalone: master] (config) # protocol mlag
g1switchB [standalone: master] (config) # ip routingg1switchB [standalone: master] (config) # ip igmp snoopingg1switchB [standalone: master] (config) # lacpg1switchB [standalone: master] (config) # protocol mlag
1. Create a port-channel:
– Port-channel indexes on two switches may differ.
2. Set the port-channel as an IPL.
3. Group physical ports to the port-channel.
– LACP or static LAG can be used.
– switchA configuration:
– switchB configuration:g1switchB [standalone: master] (config) # interface port‐channel 1g1switchB [standalone: master] (config interface port‐channel 34) # ipl 1g1switchB [standalone: master] (config interface port‐channel 34) # exitg1switchB [standalone: master] (config) # interface ethernet 1/19‐1/20g1switchB [standalone: master] (config interface ethernet 1/19‐1/20) # channel‐group 1 mode activeg1switchB [standalone: master] (config interface ethernet 1/19‐1/20) # exit
g1switchA [standalone: master] (config) # interface port‐channel 1g1switchA [standalone: master] (config interface port‐channel 1) # ipl 1g1switchA [standalone: master] (config interface port‐channel 1) # exitg1switchA [standalone: master] (config) # interface ethernet 1/19‐1/20g1switchA [standalone: master] (config interface ethernet 1/19‐1/20) # channel‐group 1 mode activeg1switchA [standalone: master] (config interface ethernet 1/19‐1/20) # exit
IPL Port-Channel
IPL VLAN Configuration
1. Create a VLAN and a VLAN interface for the IPL.
2. Set an IP address and a netmask for the VLAN interface.– The switch with highest IP address is elected as the MLAG master.
3. Map the VLAN interface to be used on the IPL and set the peer’s IP address.
–switchA configuration:
– switchB configuration:
g1switchA [standalone: master] (config) # vlan 4094g1switchA [standalone: master] (config vlan 4094) # exitg1switchA [standalone: master] (config) # interface vlan 4094g1switchA [standalone: master] (config interface vlan 4094) # ip address 172.16.34.253 /30g1switchA [standalone: master] (config interface vlan 4094) # ipl 1 peer‐address 172.16.34.254g1switchA [standalone: master] (config interface vlan 4094) # exit
g1switchB [standalone: master] (config) # vlan 4094g1switchB [standalone: master] (config vlan 4094) # exitg1switchB [standalone: master] (config) # interface vlan 4094g1switchB [standalone: master] (config interface vlan 4094) # ip address 172.16.34.254 /30g1switchB [standalone: master] (config interface vlan 4094) # ipl 1 peer‐address 172.16.34.253g1switchB [standalone: master] (config interface vlan 4094) # exit
1. Configure the MLAG cluster:
– Both switches must be configured with an identical unique group name.
– One of the switches is also configured with the VIP –it is the cluster master.
– VIP address must be of the management subnet.
– switchA configuration:
– switchB configuration:
– Wait for a few seconds until prompt is changed to group name and cluster master/standby.
MLAG Cluster Configuration
switchA [standalone: master] (config) # mlag‐vip MLAG‐G1 ip 10.25.19.13 /16switchA [MLAG‐G1: master] (config) #
switchB [standalone: master] (config) # mlag‐vip MLAG‐G1 switchB [MLAG‐G1: standby] (config) #
– Enable MLAG protocol:
– MLAG protocol is disabled by default.
– switchA configuration:
– switchB configuration
switchA [MLAG‐G1: master] (config) # mlagswitchA [MLAG‐G1: master] (config mlag) # no shutdown
Enable MLAG Protocol
switchB [MLAG‐G1: standby] (config) # mlagswitchB [MLAG‐G1: standby] (config mlag) # no shutdown
Verify MLAG VIP Configuration
switchA [MLAG‐ACAD: master] (config) # show mlag‐vipMLAG VIP========MLAG group name: MLAG‐G1MLAG VIP address: 10.25.19.13/16Active nodes: 2
Hostname VIP‐State IP Address‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐switchA master 10.25.19.11switchB standby 10.25.19.12
Cluster master/ standby
switchA [MLAG‐G1: master] (config) # show mlagAdmin status: EnabledOperational status: UpReload‐delay: 30 secKeepalive‐interval: 1 secUpgrade‐timeout: 60 minSystem‐mac: 00:00:5E:00:01:57
MLAG Ports Configuration Summary:Configured: 1Disabled: 0Enabled: 1
MLAG Ports Status Summary:Inactive: 0Active‐partial: 0Active‐full: 1
MLAG IPLs Summary:ID Group Vlan Operational Local Peer
Port‐Channel Interface State IP address IP address‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐1 Po1 34 Up 172.16.34.253 172.16.34.254
MLAG Members Summary:System‐id State Hostname‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐00:02:C9:A8:E2:D8 Up <g1switchA>00:02:C9:83:84:48 Up g1switchB
Verify MLAG Configuration
Switch g1switchA ‐ MLAG master
MLAG virtual MAC
Switch g1switchB ‐ MLAG Standby
1. Create a mlag-port-channel: Host Configuration– ‘mlag-port-channel’ indexes must be identical on both
MLAG switches.
– LACP or static LAG can be used.
2. Qumulo Node physical ports to the mlag-port-channel.
– switchA configuration:
– switchB configurationswitchB [MLAG‐G1: standby] (config) # interface mlag‐port‐channel 16switchB [MLAG‐G1: standby] (config interface mlag‐port‐channel 16) # exitswitchB [MLAG‐G1: standby] (config) # interface ethernet 1/16switchB [MLAG‐G1: standby] (config interface ethernet 1/16) # mlag‐channel‐group 16 mode activeswitchB [MLAG‐G1: master] (config interface ethernet 1/16) # exit
switchA [MLAG‐G1: master] (config) # interface mlag‐port‐channel 16 switchA [MLAG‐G1: master] (config interface mlag‐port‐channel 16) # exitswitchA [MLAG‐G1: master] (config) # interface ethernet 1/16switchA [MLAG‐G1: master] (config interface ethernet 1/16) # mlag‐channel‐group 16 mode activeswitchA [MLAG‐G1: master] (config interface ethernet 1/16) # exit
MLAG Port-Channel Creation
1. Disable STP for the mlag-port-channel:
2. Enable mlag-port-channel:
– Default admin state of mlag-port-channel is disabled.In order to allow administrator to configure bothswitches first, and then enable MLAG.
– switchA configuration:
– switchB configuration
switchA [MLAG‐G1: master] (config) # interface mlag‐port‐channel 16switchA [MLAG‐G1: master] (config interface mlag‐port‐channel 16) # spanning‐tree port type edgeswitchA [MLAG‐G1: master] (config interface mlag‐port‐channel 16) # spanning‐tree bpdufilter enableswitchA [MLAG‐G1: master] (config interface mlag‐port‐channel 16) # no shutdown
MLAG Port-Channel Configuration
switchB [MLAG‐G1: standby] (config) # interface mlag‐port‐channel 16switchB [MLAG‐G1: standby] (config interface mlag‐port‐channel 16) # spanning‐tree port type edgeswitchB [MLAG‐G1: standby] (config interface mlag‐port‐channel 16) # spanning‐tree bpdufilter enableswitchB [MLAG‐G1: standby] (config interface mlag‐port‐channel 16) # no shutdown
– Physical ports flags:– Down - port is down– Up – ports is up
– ‘mlag-port-channel’ flags:– Partial Up – local or remote are down – Up – both local and remote are up– Down – ‘admin’ state is disabled
Verify mlag-port-channel Configuration
switchA [MLAG‐G1: master] (config) # show interfaces mlag‐port‐channel summaryMLAG Port‐Channel Flags: D‐Down, U‐Up
P‐Partial UP, S ‐ suspended by MLAGPort Flags: D ‐ Down, P ‐ Up in port‐channel (members)
S ‐ Suspend in port‐channel (members), I ‐ Individual
GroupPort‐Channel Type Local Ports Peer Ports(D/U/P/S) (D/P/S/I) (D/P/S/I)‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐1 Mpo16(U) LACP Eth1/16(P) Eth1/16(P)
IMPORTANT: Please take note, links will be Down until the Qumulo node adapter ports are set to support LACP
Q&A
– Question ?
– Break 10 m
50
Virtual local area network (VLAN)
Confidential – For Training Purposes Only 51
Virtual local area network (VLAN)
– A virtual local area network (VLAN) is a logical segment of the Ethernet network that defines a broadcast domain.
– A VLAN is identified by a VLAN ID.
– Each VLAN should be assigned with a unique IP subnet.
– Hosts within the same VLAN can communicate with each other in layer 2.
– Hosts in different VLANs can communicate with each other in layer 3.
Confidential – For Training Purposes Only 52
VLAN1
VLAN2 VLAN3
VLAN1
VLAN2VLAN3
VLAN1VLAN2VLAN3
Trunk ports
– Trunk ports carry traffic for multiple VLANs across a single link.
– Hosts in the same VLAN, that are connected to different switches, can communicate with each other over the trunk link.
– When a frame is sent on the trunk port, the sending switch adds a tag that contains the VLAN ID.The receiving switch reads the VLAN ID and removes the tag.
Confidential – For Training Purposes Only 53
VLAN1
VLAN2 VLAN3
VLAN1
VLAN2VLAN3
VLAN1VLAN2VLAN3
trunk port
IEEE 802.1Q trunking protocol
– The IEEE 802.1Q trunking protocol defines the tag added to Ethernet frames carried over a trunk port.
– The 802.1Q tag is 4 bytes in size, including the 12-bit VLAN ID.
Confidential – For Training Purposes Only 54
Original Ethernet frame6 bytes 6 bytes 2 bytes 46-1500 bytes 4 bytes
DestinationMAC
Source MAC
Type/Length
Data FCS
6 bytes 6 bytes 4 bytes 2 bytes 46-1500 bytes 4 bytes
DestinationMAC
Source MAC
802.1Q Tag Type/ Length Data New FCS
802.1Q tagged Ethernet frame
2 bytes 3 bits 1 bit 12 bits
Type Priority CFI VLAN ID
Switch port types
A switch port can be configured in access, trunk, or hybrid mode.
– Access– The port accepts and sends only untagged frames– Frames are assigned to the configured port VLAN ID (PVID)– Usually, an access port is connected to a host
– Trunk– The port accepts and sends only tagged frames– Untagged frames are dropped– Usually, a trunk port is connected to another switch
– Hybrid– The port accepts and sends both tagged and untagged frames– Untagged frames are assigned to the configured port VLAN ID (PVID)– A hybrid port is connected to either a switch or a host
Confidential – For Training Purposes Only 55
Configure new VLAN’s
– switchA Configuration.
* VLANs 1-4094 are supported.
– switchB Configuration
Confidential – For Training Purposes Only 56
switchA [MLAG-G1: master] (config) # vlan 507switchA [MLAG-G1: master] (config vlan 507) # exitswitchA [MLAG-G1: master] (config) # vlan 400switchA [MLAG-G1: master] (config vlan 400) # exitswitchA [MLAG-G1: master] (config) #
switchB [MLAG-G1: standby] (config) # vlan 507switchB [MLAG-G1: standby] (config vlan 507) # exitswitchB [MLAG-G1: standby] (config) # vlan 400switchB [MLAG-G1: standby] (config vlan 400) # exitswitchB [MLAG-G1: standby] (config) #
Configure VLAN Interface and IPV6
– switchA Configuration.
– switchB Configuration
Confidential – For Training Purposes Only 57
switchA [MLAG-G1: master] (config) # interface vlan 507 ip address 192.168.17.3 /24 switchA [MLAG-G1: master] (config) # ipv6 routingswitchA [MLAG-G1: master] (config) # interface vlan 1 ipv6 enableswitchA [MLAG-G1: master] (config) # interface vlan 507 ipv6 enable
switchB [MLAG-G1: standby] (config) # interface vlan 507 ip address 192.168.18.3 /24switchB [MLAG-G1: standby] (config) # ipv6 routingswitchB [MLAG-G1: standby] (config) # interface vlan 1 ipv6 enableswitchB [MLAG-G1: standby] (config) # interface vlan 507 ipv6 enable
Verify new VLANs
– Verify new VLANs configuration.
– VLAN 1 is the default VLAN, and all ports are assigned to it.
Confidential – For Training Purposes Only 58
switchA [MLAG-G1: master] (config) # show vlan----------------------------------------------------------------------VLAN Name Ports----------------------------------------------------------------------1 default Eth1/1, Eth1/2, Eth1/3, Eth1/4, Eth1/5,
Eth1/6, Eth1/7, Eth1/8, Eth1/9, Eth1/10,Eth1/11, Eth1/12, Eth1/13, Eth1/14, Eth1/15,Eth1/17, Eth1/18, Eth1/21, Eth1/22, Mpo16
4005074094
Assign VLAN’s to host Interfaces
– switchA Configuration.
– switchB Configuration
Confidential – For Training Purposes Only 59
switchA [MLAG-G1: master] (config) # interface ethernet 1/16 description Qumulo-Node1 switchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 switchport mode hybridswitchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 switchport access vlan 1switchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 switchport hybrid allowed-vlan 507
switchB [MLAG-G1: standby] (config) # interface ethernet 1/16 description Qumulo-Node1switchB [MLAG-G1: standby] (config) # interface mlag-port-channel 1 switchport mode hybridswitchB [MLAG-G1: standby] (config) # interface mlag-port-channel 1 switchport access vlan 1switchB [MLAG-G1: standby] (config) # interface mlag-port-channel 1 switchport hybrid allowed-vlan 507
Verify VLANs Assignment
– Verify new VLANs configuration.
– VLAN 1 is the default VLAN, and all ports are assigned to it.
Confidential – For Training Purposes Only 60
switchA [MLAG-G1: master] (config) # show vlan----------------------------------------------------------------------VLAN Name Ports----------------------------------------------------------------------1 default Eth1/1, Eth1/2, Eth1/3, Eth1/4, Eth1/5,
Eth1/6, Eth1/7, Eth1/8, Eth1/9, Eth1/10,Eth1/11, Eth1/12, Eth1/13, Eth1/14, Eth1/15,Eth1/17, Eth1/18, Eth1/21, Eth1/22, Mpo16
400507 Mpo164094
Configuring Access Mode and Assigning Port VLAN ID (PVID)
61
switch > enableswitch # configure terminalswitch (config) # vlan 6switch (config vlan 6) #switch (config vlan 6) # exitswitch (config) #switch (config) # interface ethernet 1/22switch (config interface ethernet 1/22) #switch (config interface ethernet 1/22) # switchport mode accessswitch (config interface ethernet 1/22) # switchport access vlan 6switch (config 1/22) # exitswitch (config) #
Configuring Hybrid Mode and Assigning Port VLAN ID (PVID)
62
switch > enableswitch# configure terminalswitch (config) # vlan 6switch (config vlan 6) #switch (config vlan 6) # exitswitch (config) #switch (config) # interface ethernet 1/22switch (config interface ethernet 1/22) #switch (config interface ethernet 1/22) # switchport mode hybridswitch (config interface ethernet 1/22) #switch (config interface ethernet 1/22) # switchport access vlan 6switch (config interface ethernet 1/22) #switch (config interface ethernet 1/22) # exitswitch (config) #
Configuring Trunk Mode VLAN Membership
63
switch > enable
switch # configure terminal
switch (config) # vlan 10
switch (config vlan 10) #
switch (config vlan 10) # exit
switch (config) #
switch [standalone: master] (config) # interface ethernet 1/35
switch [standalone: master] (config interface ethernet 1/35) #
switch [standalone: master] (config interface ethernet 1/35) # switchport mode trunkswitch [standalone: master] (config interface ethernet 1/35) #
Q&A
– Question ?
– Break 10 m
64
Spanning Tree Protocol (STP)
Confidential – For Training Purposes Only 65
Multiple spanning tree (MST)
66
- MST maps multiple VLANs to an instance, reducing the number of spanning-tree instances.
- MST and PVST+ are compatible
- Backwards compatible with RSTP and STP- It is the IEEE standard protocol (802.1s)
Rapid spanning tree (RSTP)
67
- The Rapid Spanning Tree Protocol recovers (converges to a new spanning tree) more quickly than STP
- It is backwards-compatible with MST and STP.- It is the IEEE standard protocol (802.1w)
Rapid per-VLAN spanning tree (RPVST)
68
- Cisco proprietary version of Rapid Spanning Tree Protocol (802.1w)
- It creates a spanning tree for each VLAN, just like PVST.- Rapid-PVST is backward compatible with standard Per-VLAN Spanning Tree (PVST/802.1d)
Ethernet layer 2 loops
– Layer 2 redundant links are required to provide a backup path in case of link or switch failure.
– Redundant links result in layer 2 loops—There are multiple paths between a pair of nodes.
– Layer 2 loops cause “broadcast storms.”– When an Ethernet broadcast frame is sent in the network, it
endlessly circulates in a loop consuming all available bandwidth.
– Broadcast storms deny bandwidth for normal network traffic.
Confidential – For Training Purposes Only 69
Spanning Tree Protocol (STP)
– Spanning Tree Protocol (STP) is an IEEE 802.1D standard.
– STP ensures a loop-free topology for Ethernet networks.
– STP allows a network design to include redundant links and to provide automatic backup paths, if an active link fails.
– STP identifies redundant links and puts redundant ports in blocking state.
– When a topology change occurs, STP reacts and moves blocked ports to the forwarding state.
– Convergence time is 30 to 50 seconds.
Confidential – For Training Purposes Only 70
X X
Rapid Per-VLAN Spanning Tree (RPVST) Configuration
Confidential – For Training Purposes Only 71
Configure Spanning Tree RPVST
– switchA Configuration.
– switchB Configuration
Confidential – For Training Purposes Only 72
switchA [MLAG-G1: master] (config) # spanning-tree mode rpvst
switchB [MLAG-G1: standby] (config) # spanning-tree mode rpvst
Configure Flowcontrol and Jumbo Frames
– switchA Configuration.
– switchB Configuration
Confidential – For Training Purposes Only 73
switchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 flowcontrol receive on forceswitchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 flowcontrol send on forceswitchA [MLAG-G1: master] (config) # interface mlag-port-channel 16 mtu 9216 force
switchB [MLAG-G1: standby] (config) # interface mlag-port-channel 16 flowcontrol receive on forceswitchB [MLAG-G1: standby] (config) # interface mlag-port-channel 16 flowcontrol send on forceswitchB [MLAG-G1: standby] (config) # interface mlag-port-channel 16 mtu 9216 force
Maximum Transmission Unit (MTU) Size
74
– The largest possible frame size of a communications Protocol Data Unit (PDU) on an OSI Model Layer 2 data network.– Default frame size is 1518 bytes– Example of commands to check MTU size;
C:\Users\ScottHogg> ping 192.168.10.1 -l 1500 –f
RedHat# ping -s 1500 -M do 192.168.10.1
Router1# ping 192.168.10.1 size 1500 df-bit
Switch7K# ping 192.168.10.1 packet-size 9216 c 10
RP/0/RP0/CPU0:Router1#ping 192.168.10.1 size 1500 donnotfrag
Junos-root@J4350-1# run ping 192.168.10.1 size 1500 do-not-fragment rapid
Q&A
– Question ?
– Break 10 m
75
Link Layer Discovery Protocol (LLDP)
Confidential – For Training Purposes Only 76
Link Layer Discovery Protocol (LLDP)
– Link Layer Discovery Protocol (LLDP) is a vendor-neutral protocol defined in IEEE 802.1AB.
– LLDP is used by network devices for advertising their identity, capabilities, and neighbors on an IEEE 802 LAN.
– LLDP can be used to discover and verify Ethernet network topology.
– LLDP is by default globally disabled.
– LLDP frames are sent every 30 seconds by all LLDP enabled interfaces.
Confidential – For Training Purposes Only 77
Eth1/1
switchA switchB
Configure LLDP
– switchA Configuration.
– switchB Configuration
Confidential – For Training Purposes Only 78
switchA [MLAG-G1: master] (config) # lldp
switchB [MLAG-G1: standby] (config) # lldp
Show local LLDP information – Show local LLDP information.
– Show interface LLDP information.
Confidential – For Training Purposes Only 79
switchA [MLAG-G1: master] (config) # show lldp localLLDP: enabled
Local global configurationChassis sub type: Mac AddressChassis id: b8:59:9f:70:d6:00System Name: g1switchASystem Description: SN2010M,Onyx,SWv3.8.2204Supported capabilities: B,RSupported capabilities enabled: B
switchA [MLAG-G1: master] (config) # show lldp interfaces ethernet 1/19TLV flags
PD: port-descriptionSN: sys-nameSD: sys-descriptionSC: sys-capabilitiesMA: management-addressETS-C: ETS-Configuration
Interface Receive Transmit TLVs-----------------------------------------------------------------------------------Eth1/19 Enabled Enabled PD, SN, SD, SC, MA, PFC, AP, ETS-C, ETS-R
Eth1/19
switchA switchB
View Cable/Transceiver – Show local Transceivers information.
Confidential – For Training Purposes Only 80
switchA [MLAG-G1: master] (config) # show interfaces ethernet 1/19 transceiverPort 1/19 state
identifier : QSFP28cable/module type : Passive copper cableethernet speed and type: 100GBASE-CR4vendor : Mellanoxcable length : 1mpart number : 845404-B21revision : A1serial number : 6C2749003C
Eth1/19
switchA switchB
Onyx system features
Confidential – For Training Purposes Only 81
Feature Description
Software management – Dual software image– Software and firmware updates
File management – FTP, TFTP, SCP
Logging – Event history log– Syslog support
Chassis management – Monitoring environmental controls– Power management– Auto-temperature control– High availability
Network management interfaces – SNMP v1,v2c,v3– Puppet agent
Security – SSH, Telnet– RADIUS, TACACS+
Date and time – NTP
Cables and transceivers – Transceiver info
Configure Clock and NTP
– switchA Configuration.
– switchB Configuration
Confidential – For Training Purposes Only 82
switchA [MLAG-G1: master] (config) # clock timezone America North United_States CentralswitchA [MLAG-G1: master] (config) # clock set 09:00:00 2020/05/12switchA [MLAG-G1: master] (config) # ntp server 10.187.2.2switchA [MLAG-G1: master] (config) # ntp enable
switchB [MLAG-G1: master] (config) # clock timezone America North United_States CentralswitchB [MLAG-G1: master] (config) # clock set 09:00:00 2020/05/12switchB [MLAG-G1: master] (config) # ntp server 10.187.2.2switchB [MLAG-G1: master] (config) # ntp enable
Show NTP and Clock – Display commands NTP Status.
– Display Clock settings.
Confidential – For Training Purposes Only 83
NTP is administratively : enabledNTP Authentication administratively: disabledNTP server role : enabledClock is synchronized:Reference: 10.187.2.2Offset : -0.620 ms
Active servers and peers:10.187.2.2:Conf Type : servStatus : sys.peer(*)Stratum : 1Offset(msec) : -0.620Ref clock : .GPS.Poll Interval (sec): 64Last Response (sec): 53Auth state : none
switchA [MLAG-G1: master] (config) # show clockTime: 14:54:50Date: 2020/05/12Time zone: America North United_States Central (US/Central)UTC offset: -0500 (UTC minus 5 hours)
Reset factory defaults– Reset the switch to factory defaults.
Confidential – For Training Purposes Only 84
g1switchA [MLAG-G1: master] (config) # reset factory ?<cr>halt Halt system after reset, instead of rebootingkeep-all-config Preserve all configuration files (supercedes keep-basic)keep-basic Preserve licenses in the active configurationkeep-virt-vols Preserve all virtual disk volumesonly-config Reset only configurationmseries3 [standalone: master] (config) # reset factory keep-all-configWarning - confirming will cause system reboot.Type 'YES' to confirm reset:
Configuration Licenses System profile Management interfaces
keep-all-config Unchanged Not deleted Unchanged Unchanged
keep-basic Reset Not deleted Reset Reset
only-config Reset Deleted Reset Unchanged
Reset Factory
– switchA Configuration.
– switchB Configuration
Confidential – For Training Purposes Only 85
switchA [MLAG-G1: master] (config) # reset factory
switchB [MLAG-G1: standby] (config) # reset factory
Q&A
– Question ?
– Break 10 m
86
Multi-active gateway protocol (MAGP)
Confidential – For Training Purposes Only 87
Enable the switch as host’s gateway (Layer 3 MLAG)
88
There are two protocols of the Layer 3 MLAG;- VRRP: Virtual Router Redundancy Protocol.
It’s working as Active/Standby.
- MAGP: Multi-active gateway protocol.
It’s working as Active/Active.Note: MAGP is the recommend protocol for implementing Mellanox’s L3 MLAG.
Multi-active gateway protocol (MAGP)
89
- To solve the default gateway problem when a host is
connected to a set of switch routers via MLAG.
- Each switch routers is an active default gateway router
to the host.
- Directly forwarding IP traffic to the L3 cloud regardless
which SR traffic comes through.
Configuring MAGP Example
90
- switch (config)# ip routing
- switch (config)# vlan 20
- switch (config)# interface vlan 20
- switch (config interface vlan 20)# ip address 11.11.11.11 /8
- switch (config interface vlan 20)# no shutdown
- switch (config)# protocol magp
- switch (config interface vlan 20)# magp 100
- switch (config interface vlan 20 magp 100)# ip virtual-router address 11.11.11.254- switch (config interface vlan 20 magp 100)# ip virtual-router mac-address AA:BB:CC:DD:EE:FF
Verify the MAGP configuration
91
IP Routing
92
– IP Interfaces (L3)
– MLNX-OS supports the following 3 types of IP interfaces:
•VLAN interface
•Loopback interface
•Router port interface
Note: Router port interfaces are not supported on SX10xx-xxxR and SX60xx-xxxR systemsNote: Routing for this project is using VLAN interface with ip route 0.0.0.0 0.0.0.0 10.0.102.2
VLAN interface Attributes
– VLAN interface is a logical IPv4 interface created per subnet over a specific 802.1Q VLAN ID.
– Each interface VLAN has the following attributes:
•Admin state
•Operational state
•MAC address
•IP address and mask
•MTU
•Description•Set of counters
93
Configure a Router Port Interface
94
Q&A
– Question ?
– Break 10 m
95
Basic Troubleshooting
Confidential – For Training Purposes Only 96
What happens if the IPL link goes down?
97
Link Aggregation
Data VLAN 300
ILO
Cisco core L3 #1, #2
SN2410M
Cisco MGMG switch
VLAN 400
mgmt0
iLO
Qumulo
Servers 1-6
MAGP 1:Interface vlan: 300Admin state : EnabledState : MasterVirtual IP : 10.0.1.254Virtual MAC : 00:00:5E:00:01:01
MAGP 2:Interface vlan: 400Admin state : EnabledState : MasterVirtual IP : 10.0.102.1Virtual MAC : 00:00:5E:00:01:02 10.0.99.61 ‐ 63 /24
inter-peer link (IPL) Management Plane (Active/Standby)
SN2410Mmgmt0
1. Split-brain2. Only the master switch will pass traffic.
What happens if no IP communication between the MGMT ports
98
Link Aggregation
Data VLAN 300
ILO
Cisco core L3 #1, #2
SN2410M
Cisco MGMG switch
VLAN 400
mgmt0
iLO
Qumulo
Servers 1-6
MAGP 1:Interface vlan: 300Admin state : EnabledState : MasterVirtual IP : 10.0.1.254Virtual MAC : 00:00:5E:00:01:01
MAGP 2:Interface vlan: 400Admin state : EnabledState : MasterVirtual IP : 10.0.102.1Virtual MAC : 00:00:5E:00:01:02 10.0.99.61 ‐ 63 /24
Management Plane (Active/Standby)
SN2410Mmgmt0
1. CLI prompt is displayed: [:unknown]#2. It Split-brain when IPL down
Verify interface MLAG port channel
99
Link Aggregation
Data VLAN 300
ILO
Cisco core L3 #1, #2
SN2410M
Cisco MGMG switch
VLAN 400
mgmt0
iLO
Qumulo
Servers 1-6
10.0.99.61 ‐ 63 /24
SN2410Mmgmt0
#show interface mlag-port-channel sum
mlag-port-channel
Q&A
– Question ?
100
Thank [email protected]