CIS 188 CCNP TSHOOT (Troubleshooting)Ch. 8 Troubleshooting Converged Networks Part 1Rick GrazianiCabrillo Collegegraziani@cabrillo.edu

Fall 2014

*MaterialsBook:Troubleshooting and Maintaining Cisco IP Networks (TSHOOT) Foundation Learning Guide: Foundation learning for the CCNP TSHOOT 642-832By Amir RanjbarBookISBN-10: 1-58705-876-6ISBN-13: 978-1-58705-876-9eBookISBN-10: 1-58714-170-1ISBN-13: 978-1-58714-170-6

TopicsPart 1Troubleshooting Wireless Issues in a Converged NetworkPart 2Troubleshooting Unified Communications Issues in a Converged NetworkTroubleshooting Video Issues in a Converged Network *

Troubleshooting Wireless Issues in a Converged Network*

Common Wireless Integration IssuesIn a standalone or autonomous solution, autonomous access points (APs) provide all the wireless services. The controller-based architecture splits the processing of the IEEE 802.11 protocol between two devices (split MAC or Lightweight ): AP Centralized Cisco Wireless LAN Controller (Cisco WLC)*

*Traditional WLAN ArchitectureTraditional WLAN each AP serves as the central hub for its own BSS.Each AP must be configured individually for network policies including:Radio frequency (RF)Security policiesAuthentication and associationMonitoring trafficQoSBandwidth policingRogue AP detectionCisco calls this an autonomous mode AP.Traffic patterns for an autonomous AP are completely handled by the AP.BSSRange

*Traditional WLAN ArchitectureAP can support multiple SSIDs (to support multiple VLANs over a trunk link)If you want to offer the same SSIDs from several automous APs, the VLANS must be extended to the APs in a contiguous manner.Problem is that the SSID and its VLAN would have to be extended everywhere the user could possibly roam.This would cause end-to-end or campus-wide VLANs which is not good network design practice.SwitchedRouted

*Cisco Unified Wireless Network ArchitectureCisco Unified Wireless Network Architecture centralizes many traditional capabilities by moving many functions to a central location including:WLAN securityWLAN deploymentWLAN managementWLAN controlReal-time ProcessesReal-time ProcessesManagement ProcessesManagement ProcessesRF Transmit/ReceiveMAC ManagementEncryptionRF Transmit/ReceiveMAC ManagementEncryptionRF ManagementAssociation & RoamingManagementClient AuthenticationSecurity ManagementQoSRF ManagementAssociation & RoamingManagementClient AuthenticationSecurity ManagementQoSLWAPP orCAPWAP TunnelTraditional WLANCisco Unified Wireless Network

*Cisco Unified Wireless Network ArchitectureReal-time processes include:Sending/receiving 802.11 framesAP beaconsprobesdata encryptionManagement processes include:RF managementroaming managementQoSSecurityAnd all association, authentication, power saving tasksReal-time ProcessesReal-time ProcessesManagement ProcessesManagement ProcessesRF Transmit/ReceiveMAC ManagementEncryptionRF Transmit/ReceiveMAC ManagementEncryptionRF ManagementAssociation & RoamingManagementClient AuthenticationSecurity ManagementQoSRF ManagementAssociation & RoamingManagementClient AuthenticationSecurity ManagementQoSLWAPP orCAPWAP TunnelTraditional WLANCisco Unified Wireless NetworkLAPWLC

*Cisco Unified Wireless Network ArchitectureLAP or LWAP (Lightweight Access Point) Performs only the real-time 802.11 operations.Layer 1 and 2 operationsThe IOS image and local intelligence is stripped down compared to autonomous APs.Dependent upon the WLC for all other operations.WLC (Wireless LAN Controller)Performs all management functionsThis is known as split-MAC architectureControl MessagesEncapsulated DataLWAPP or CAPWAPLWAPP or CAPWAPLAPWLC

*Cisco Unified Wireless Network ArchitectureTwo devices have a LWAPP or CAPWAP tunnel to exchange 802.11 messages and client data.LAP and WLC can be in the same VLAN/IP subnet or different ones.The LWAPP or CAPWAP tunnel allows user data to be switched or routed across the campus network.LWAPP (Lightweight Access Point Protocol) Developed by Cisco, submitted as draft RFC 4118 CAPWAP (Control and Provisioning Wireless Access Points protocol)The resulting standard.

*Cisco Unified Wireless Network ArchitectureControl messages are authenticated and encrypted so the LAP is securely controlled by the WLC.This also prevents rogue APs from being introduced into the network.Data Packets to and from wireless clients associated with the LAP are encapsulated with the LWAPP or CAPWAP tunnel but are not encrypted or otherwise secured between the LAP and WLC.Authenticated and EncryptedNot secured by LAP/WLC

*Cisco Unified Wireless Network ArchitectureWLC (Wireless LAN Controller) functions:Dynamic channel assignment Chooses and configures RF channel for each LAP.Transmit power optimization Sets transmit power for each LAP based on size of coverage area needed.Self-healing wireless coverage If a LAP radio dies other LAPs can have their power increased.Flexible client roaming Manages Layer 2 and 3 roaming.Dynamic client load balancing If multiple LAPs are in same coverage area, WLC can associate clients with the least used LAP.RF monitoring Scans channels to monitor RF usage, interference, noise, and signals from rogue APs.Security management Require clients to get their IP address from a trusted DHCP server before allowing them to associate.

*Cisco Unified Wireless Network ArchitectureWLC available on several platforms including WLC module for ISR routers (2800 and 3800). The easiest way to distinguish between a regular AP and a LAP is to look at the part number of the AP.LAP (Lightweight AP Protocol [LWAPP])Part numbers always begin with AIR-LAPXXXX.Autonomous AP (Cisco IOS Software)Part numbers always begin with AIR-APXXXX.

*Cisco Unified Wireless Network ArchitectureCisco Wireless Control System (WCS)Optional - Allows for easier management of several WLCs.Server platform with GUI front-end.Uses floor plans to display dynamic representations of wireless coverage.Can be used with Cisco Wireless Location Appliance to track the location of thousands of clients.WCS

*Cisco Unified Wireless Network ArchitectureLAPs (Lightweight Access Point) are designed to be zero touch configuration.Receives its configuration parameters from the WLCDo not need to configure it through its console port or over the network.LAP OperationsStep 1: LAP obtains an IP address from DHCP serverStep 2: LAP learns IP address of an available WLCsDHCP server adds option 43 to its reply containing a list of WLCs, orLAP broadcasts a join request message (as long as the WLC on the local subnet)Step 3: LAP sends a join request to the first WLC in its list.Step 4: WLC compares IOS image number stored to the one stored on the WLC. If they differ the LAP download the code on the WLC and reboots.Step 5: WLC and LAP build a secure LWAPP or CAPWAP tunnel for management traffic, and LWAPP or CAPWAP tunnel (not secured) for wireless client data.DHCP Server

*HREAPWhen LAP is cut off from WLC client associations are dropped and no data can pass over the WLAN.Cisco Hybrid Remote Edge Access Point (HREAP) is used when LAPs are separated from WLCs over a WAN link.The LAPs can keep operating even while the WAN link is down and the WLC is not available like an autonomous AP.Allows wireless clients to keep communicating within the remote site.WLC

Traffic Patterns

*Single VLANsTraffic patterns differ than traditional WLANs.Client data passes:From Client A to LAPFrom LAP to WLCFrom WLC back to LAPFrom LAP to Client BEncryption is still handled between the LAP and the client.BSSRange

*Multiple VLANs With traditional WLANs the access VLANs must be extended or trunked between APs and multilayer switch.This is not the case with LAPs and WLCs.There are two VLANs A and B with their respective SSIDs A and B.The VLANs exist on the trunk between the WLC and SW2 but go no further.The LAPs and the WLC are connected by VLAN Z which is can be totally isolated from VLANs A and B. VLANs A and B are carried over the LWAPP tunnel so they are logically connected between the LAP and the WLC.

Roaming in a Cisco Unified Wireless Network

*Traditional RoamingA WLAN designer must determine whether clients will require seamless roaming from access point to access point.IEEE 802.11 IAPP (Inter-Access Point Protocol).Initial Association:Probing (Probe Request, Probe Response)Authentication (Authentication Request, Authentication Response)Association (Association Request, Association Response)802.11 does not allow associating with more than one AP.

*Traditional RoamingThe client initiates the roaming (re-association) process.As the client is moving out of range of its associated AP, the signal strength will start to drop off. At the same time, the strength of another AP will begin to increase. The re-association process then occurs, including authentication.IAPP: Please send buffered frames forIAPP: Ok!* AP(B) must update MAC address tables on infrastructure switches to prevent to loss of data.AP(B) sends an Ethernet frame to AP(A) with the source MAC address of the client so all the switches can update their SAT/MAC tables.* Packet - Source MAC of client

*Roaming in a Cisco Unified Wireless NetworkWith autonomous APs when a client roams its association moves from one AP to another.Client must negotiate the move independently and the APs must also make sure any buffered data from the client is passed along with the association.WLC supports both Layer 2 and Layer 3 roaming.

*Intracontroller RoamingBoth LAP1 and LAP2: Use SSID MyWLANJoined to the same WLCClient roams into area covered by AP2.Although the AP has changed the same controller is providing the association with through the LWAPP or CAPWAP tunnel.This is known as intracontroller roaming.

*Intracontroller RoamingIntracontroller roaming.The WLC (controller) simply updates its tables to begin using the LWAPP or CAPWAP tunnel to LAP2. Any leftover data that was buffered for the prior association with LAP1 is easily shifted to new association with LAP2.

*Intercontroller Roaming (same subnet)Intercontroller roaming: When LAPs are supported by multiple WLCs (controllers) for redundancy, load balancing or scalability. Client moves to LAP2s cell the same SSID is found.Client moves it association to WLC2.As long as WLC1 and WLC2 are in the same subnet they can easily hand off the clients association.SameSubnetSame IP address

*Intercontroller Roaming (same subnet)When the mobility exchange occurs the client begins using the LWAPP or CAPWAP tunnel between LAP2 and WLC2.Client IP address has not changed.Roaming process completely transparent to the client.SameSubnetMobilityExchangeSame IP address

*Intercontroller Roaming (different subnet)WLC controllers are in different subnets (VLANs A and B).Do not have to have end-to-end or campus wide VLANs.Client begins in cell with association to WLC1.Obtains an IP address within VLAN ALAP1 offers VLAN A on its SSID MyWLANAll traffic passes between LAP1 and WLC1 onto VLAN A.MobilityExchangeSame IP address

*Intercontroller Roaming (different subnet)Client roams into cell provided by LAP2.LAP2 offers access to a different VLAN, VLAN B.The clients IP address remains the same.But WLC1 and WLC2 are in different subnets (VLANs A and B).Client IP address has moved into a foreign subnet.

Same IP addressMobilityExchange

*Intercontroller Roaming (different subnet)Two WLCs (controllers) begin to work together to provide continue service for the client without the client needing to obtain an new IP address.WLCs (controllers) bring up an Ether-IP tunnel (RFC 3378) between them.Carries some of the clients data trafficEncapsulates Ethernet frame inside an IP packet using protocol 97.WLC1 encapsulates packets and sends them to WLC2.WLC2 unencapsulates the packets into their original form.Same IP addressL3 MobilityTunnelMobilityExchange

*Intercontroller Roaming (different subnet)Traffic leaving the client travels from LAP2 to WLC2 onto the network even though it is on a foreign subnet. Its just a packet in an Ethernet frame.Dest MAC Source MAC [ IP Packet: Source IP Dest IP - ]Traffic coming in towards the client takes a different path.Traffic enters MLS (L3 switch)Because the packet has an IP address on VLAN 3 it is routed/switched out VLAN A to WLC1.WLC1 accepts the traffic and forwards it to the appropriate WLC controller that has the current association with the client.WLC1 sends the traffic to WLC2 through the Ether-IP tunnel.WLC2 forwards the traffic to LAP2 and onto the client.Same IP addressL3 MobilityTunnelMobilityExchange

*Intercontroller Roaming (different subnet)The client originally joined the WLAN on WLC1, so WLC1 is known as the anchor.WLC2 is serving a client on a different subnet so it is known as the foreign agent.As the client continues to roam the foreign agent will change but the anchor will remain the same.To do this WLCs are configured into logical mobility groups.Up to 24 WLCs Number of LAPs vary depending upon LAP platform.If a client must move between mobility groups, its IP address and all of its session information maintained by the WLC will be dropped.Same IP addressL3 MobilityTunnelMobilityExchangeAnchorForeign Agent

Some of the common wireless integration issues are: The boundary between the wireless and the wired network is the Cisco WLC because traffic is tunneled between the AP and the WLC. The WLC is an important point of troubleshooting. Filters (ACLs) may be blocking critical traffic, such as those related to LWAPP tunnels, or perhaps related to wireless security (IEEE 802.1x, EAP, or RADIUS).IP addressing typically needs to be investigated, especially in roaming scenarios or DHCP access for the LAP.Maintaining QoS markings consistently across wireless-to-wired boundaries is a challenge. *10, 20

Troubleshooting Example: Wireless LAN Connectivity Problems Wireless services have suddenly stopped; clients are not able to associate to the AP. Even from the wired PCs that are used for troubleshooting, it is not possible to connect to the AP or the WLC, using either SSH or HTTPS. *X

*SW1# show cdp neighborsCapability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge S - Switch, H - Host, I - IGMP, r - Repeater, P - PhoneDevice ID Local Intrfce Holdtme Capability Platform Port IDap Gig 0/37 128 T I AIR-LAP125 Gig 0521-8 Gig 0/39 135 AIR-LAP521 Fas 0521-7 Gig 0/34 122 AIR-LAP521 Fas 0Cisco_9a:8c:e0 Gig 0/36 175 H AIR-WLC210 Unit - 0 Slot 0 Port - 1Using a bottom-up approach we start with the access switch and look at the interfaces, looking for clues at the physical and data link layers. Use show cdp neighbors to try to identify on the switch:which ports are connected to the controller and which are connected to the access pointBased on the results shown:WLC connects to interface Gig 0/36 AP connects to interface Gig 0/37

Next, we examine the status of the interfaces with the show interface status commandThe Gig 0/36 interface connected to the WLC is configured as trunk The Gig 0/37 interface connected to the AP is associated to VLAN 10 *SW1# show interface statusPort Name Status vlan Duplex Speed TypeGi0/1 notconnect 1 auto auto 10/100/1000BaseTXGi0/2 notconnect 1 auto auto 10/100/1000BaseTX

Gi0/34 connected 1 a-full a-100 10/100/1000BaseTXGi0/35 notconnect 1 auto auto 10/100/1000BaseTXGi0/36 connected trunk a-full a-100 10/100/1000BaseTXGi0/37 connected 10 a-full a-1000 10/100/1000BaseTXGi0/38 notconnect 1 auto auto 10/100/1000BaseTXGi0/39 connected 1 a-full a-100 10/100/1000BaseTX

We need to find out:Which VLANs are used for AP to WLC communicationWhich VLAN is used for client trafficIf the access point is operational and registering to the WLC using LWAPP or CAPWAP. *

The wireless administrator informs us that :The AP has a static IP address: 10.10.10.104 The WLC and the AP should be on the same VLANBut the WLC is not seeing registration (join) requests from the AP The static IP address on the AP allows us rule out DHCP preventing the AP from creating a LWAPP tunnel with the WLC. Layer 1 and layer 2 status of the interfaces are operationalWe should therefore pursue the AP to WLC registration process; if the AP cannot register with the WLC, it will not be able to service client requests. *10, 20Static IP 10.10.10.104Same VLANRegistrationRequests?g0/36g0/37

Use debug ip packet to observe AP's registration process. Filter it to only see the traffic related to LWAPP using access list 100. LWAPP uses UDP port 12223 for control messages. The APs Registration Request originates from interface Gig 0/37 (VLAN 10)rcvd means the packet was received by the switchBut this traffic is not forwarded to the trunk on the Gig 0/36 interfaceNo forward at the end of a similar debug statement This is probably why there is no response from the WLC*12223?

SW1# show interfaces switchport | begin 0/36Name: Gi0/36Switchport: EnabledAdministrative Mode: trunkOperational Mode: trunkAdministrative Trunking Encapsulation: dot1qOperational Trunking Encapsulation: dot1qNegotiation of Trunking: OnAccess Mode VLAN: 1 (default)

Trunking VLANs Enabled: 1Pruning VLANs Enabled: 2-1001Capture Mode Disabled

SW1# show interface switchport | begin 0/36 Verify that VLAN 10 is allowed on the trunk interface (Gig 0/36)The output reveals only VLAN1 is enabled (allowed) on the trunkAll other VLANs such as VLAN 10 are not allowed on the trunk. This is definitely wrong!*?

SW1# conf tEnter configuration commands, one per line. End with CNTL/Z.SW1(config)# interface g0/36SW1(config-if)# switchport trunk allowed vlan add 10,20SW1(config-if)# endSW1#The wireless team tells us that the client VLAN is 10, and that the management VLAN is 20. Modify Gig 0/36 to allow Vlans 10 and 20 on the trunk interfaceAfter a few minutes the wireless team tells us that the problem is solved. The AP is registering again to the WLC, and wireless connectivity has been restored. *?SW1(config)# inter gig 0/36SWI1(config-if)# switchport trunk allowed vlan add 10,20

Troubleshooting Example: Duplex and Trust IssuesThe wireless operations team complains about the reliability and performance of wireless traffic. The symptom they observe is: The AP interface pointing to the wired network goes up and down intermittentlyWhen the port is up there is a substantial slowdown on Voice over WLAN*

SW1# show logging | include 0/3400:12:00: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered onGigabitEthernet0/34 (not half duplex), with 521-7 FastEthernet0 (half duplex)00:13:00: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered onGigabitEthernet0/34 (not half duplex), with 521-7 FastEthernet0 (half duplex)00:14:00: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered onGigabitEthernet0/34 (not half duplex), with 521-7 FastEthernet0 (half duplex)00:15:00: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered onGigabitEthernet0/34 (not half duplex), with 521-7 FastEthernet0 (half duplex)

*The first thing we do is display the log and look for any clues with regards to the interface (Gig 0/34) that apparently goes up and down intermittently. See duplex mismatch!

A plain show logging command tells us that the console logging is disabled, which makes a lot of sense in a production switch. If we enable it, we will see the duplex mismatch commands. We fix the duplex problem by configuring the interface for full duplex 100 mbps. *SW1# show loggingSyslog logging: enabled (0 messages dropped, 1 messages rate-limited, 0 flushes,0 overruns, xml disabled, filtering disabledConsole logging: disabled

SW1# conf tEnter configuration commands, one per line. End with CNTL/Z.SW1(config)# int g0/34SW1(config-if)# duplex fullSW1(config-if)# speed 100SW1(config-if)# end

The wireless team now tells us the AP comes stays up but they continue complaining about performance issues, especially for VoIP traffic coming from the wireless network. Check to see if high CPU utilization is an issue with show processes cpuShows relatively low levels of CPU utilization at this point. Look at documentation and notice these values are close to baseline.VoIP issues could be possible QoS configuration errors

*SW1# show processes CPUCPU utilization for five seconds: 4%/0%; one minute: 6%, five minutes: 5%PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process1 0 5 0 0.00% 0.00% 0.00% 0 Chunk Manager2 0 275 0 0.00% 0.00% 0.00% 0 Load Meter3 0 33 0 0.00% 0.00% 0.00% 0 SpanTree Helper4 1019 149 6838 0.00% 0.07% 0.05% 0 Check heaps5 0 1 0 0.00% 0.00% 0.00% 0 Pool Manager6 0 2 0 0.00% 0.00% 0.00% 0 Timers7 118 845 139 0.00% 0.00% 0.00% 0 ARP Input8 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT9 0 2 0 0.00% 0.00% 0.00% 0 AAA high-capacit

*Perhaps the voice traffic may not be tagged with QoS priorities. In the case of a LWAPP deployment, if the AP is tagging packets with values, it is the differentiated services code point (DSCP) field that gets used. We should check if our switch port is honoring that. show mls qos int gi0/34 Used to display the trust boundary settings. A trust boundary is the point within the network where QoS markings such as DSCP are first accepted. By default, switch ports will reset DSCP values unless you explicitly tell the port to trust those values. If an untagged frame arrives at the switch port, the switch will assign a default CoS/DSCP of 0 to the frame/packet before forwarding it. (How it will be treated within the switch.)SW1# show mls qos int g0/34GigabitEthernet0/34trust state: not trustedtrust mode: not trustedtrust enabled flag: enaCOS override: disdefault COS: 0DSCP Mutation Map: Default DSCP Mutation MapTrust device: Noneqos mode: port-based

*Traffic marking The decision of whether to mark traffic at layers 2 or 3 or both is not trivial and should be made after consideration of the following points:Layer 2 marking of frames can be performed for non IP traffic.Layer 2 marking of frames is the only QoS option available for switches that are not IP awareLayer 3 marking will carry the QoS information end-to-endOlder IP equipment may not understand DSCP Layer 3Layer 2

*Mapping Layer 2 and Layer 3When a frame is marked with DSCP, for example, and it needs to traverse a series of Layer 2 switches or 802.1Q Trunks.How will it be queued in these Layer 2 devices? To accomplish this, there is a mapping that takes place between the Layer 3 mapping field (TOS) and the Layer 2 CoS fields.In part 2 I will show you how this works.Mapping is vendor specific. On Cisco devices, this is taken care of for you through a mapping process.

show mls qos indicates that the switch does not trust anything coming from the AP. This could be a real issue:voice traffic is being prioritized on the wireless network butlosing its priority when crossing over to the wired network

*SW1# show mls qos int g0/34GigabitEthernet0/34trust state: not trustedtrust mode: not trustedtrust enabled flag: enaCOS override: disdefault COS: 0DSCP Mutation Map: Default DSCP Mutation MapTrust device: Noneqos mode: port-based

We must set the switch port to trust DSCP values (following best practices and guidelines). SW1(config)# inter gig 0/34SW1(config-if)# mls qos trust dscpInspect the configuration with the show mls qos command The output tells us that we are now trusting DSCP values. After a while, the wireless network support staff confirms that performance issues are alleviated for VoWLAN traffic. The problem is solved!*SW1(config)# int g0/34SW1(config-if)# mls qos trust dscpSW1(config-if)# endSW1#

SW1# show mls qos int g0/34GigabitEthernet0/34trust state: trust dscptrust mode: trust dscptrust enabled flag: enaCOS override: disdefault COS: 0DSCP Mutation Map: Default DSCP Mutation MapTrust device: Noneqos mode: port-based

Whether the trust is conditional (Cisco Phone)

Troubleshooting Example: New Security ImplementationThe wireless team tells us that wireless operations have stopped and none of the APs are able to register to the Wireless LAN controller (WLC). This problem has been expected, because a security auditor recently performed a security assessment and recommended a few improvements to the network policy. In taking all the necessary precautions, all configurations have been reverted to their pre-audit state, except for the LAN switch *XRegistrationRequests?

*XRegistrationRequests?Focus on the Cisco IOS firewall, without discarding the possibility of other issues. The reported symptom, wireless APs not being able to register to the WLC, gives us another hint as to what to look for: LWAPP traffic may be denied by the firewall. This is a valid hypothesis with a very good likelihood of being accurate, and we need to verify it. Cisco IOS Software allows the firewall to be configured using one of two methods: Access control lists (ACLs) exclusively on interfacesZone-based firewall, more widely used and more flexible for a comprehensive deployment of firewall rules.

We check the zone-based policy first, but after entering the show zone-pair security command we no information (or a error message), effectively informing us that no zone-based policies are configured on this router. Next, we consider interface ACLs on our switch using the show ip interface command for the interface connected to the access point. Output reveals that there is an ACL called FIREWALL applied inbound to R1s Fa0/0 interface.

*R1# show ip interface Fa0/0FastEthernet0/0 is up, line protocol is upInbound access list is FIREWALL

XRegistrationRequests?

We display the access list Notice that it allows routing protocols and management protocols such as telnet. We notice one important thing is missing: Permission for the LWAPP ports Both control (the traffic between AP and WLC) and user traffic traverse through the LWAPP tunnel. However, the firewall is blocking those ports. *XRegistrationRequests?R1# show access-listExtended IP access list 100 10 permit udp 10.10.10.0 0.0.0.255 any eq 12223 20 permit udp any any eq 12223Extended IP access list FIREWALL 10 permit icmp any any echo-reply 20 permit tcp any any eq www 30 permit tcp any any eq ftp 40 permit tcp any any eq ftp-data 50 permit tcp any any eq telnet 60 permit tcp any anyeq smtp 70 permit tcp any any eq pop3 80 permit eigrp any any 90 permit udp any any eq rip

We need to permit: UDP 12222 for user data trafficUDP 12223 for AP-to-WLC control messagesThe wireless team reports that this fix seems to have solved the problem.

*R1(config)# ip access-list extended FIREWALLR1(config-ext-acl)# permit udp any any range 12222 12223

Troubleshooting Examples: DHCP IssueIn this case the AP and the WLC are in different VLANs and the R1 router is performing inter-VLAN routing. We have received a call from the wireless team, stating that none of the APs are able to register to the WLC. All APs are DHCP clients but are not able to obtain their IP address from the DHCP server (at address 10.50.50.100). The wireless group is blaming this problem on the wired network, so it is our job to find the problem and fix it. *DHCP ClientDHCP Server10.50.50.100RegistrationRequests?

R1# clear ip dhcp server statisticsR1#R1# show ip dhcp server statisticsMemory usage 5317Address pools 1Database agents 0Automatic bindings 2Manual bindings 0Expired bindings 0Malformed messages 0

Message ReceivedBOOTREQUEST 0DHCPDISCOVER 0DHCPREQUEST 0DHCPDECLINE 0DHCPRELEASE 0DHCPINFORM 0

Message SentBOOTREPLY 0DHCPOFFER 0DHCPPACK 0DHCPNAK 0A couple of things could be wrong here. The process DHCP process between AP and the DHCP serverThe APs has an IP address but cannot register with the WLCWe will look at both processes to isolate the problem. So we will start with the DHCP server (show ip dhcp server statistics) But cant tell how recent this activity is (may be old data).We use the clear ip dhcp server statistics command, and start from there. show ip dhcp server statistics shows no activity this time. debug ip udp (Not shown) Used to monitor any DHCP client activity such as DHCP DISCOVER messages, but see no reference to UDP port 67 (DHCP client)*RegistrationRequests??

SW1# show running-config interface g0/34Building configuration...Current configuration : 108 bytes!interface GigabitEthernet0/34 switchport access vlan 10 switchport mode access mls qos trust dscpend

SW1# show running-config interface vlan 10Building configuration...Current configuration : 61 bytes!interface vlan10 ip address 10.10.10.1 255.255.255.0endBecause the DHCP clients are in a subnet different than that of the DHCP server, we might be missing the DHCP relay agent. This will have to be configured in the switch connecting the AP to the rest of the network. We use show running interface gi0/34 port that points to the APs: But see no ip-helper address command. This is a switchport interface associated to VLAN 10, so we must inspect interface VLAN 10 instead.There is no ip-helper address configured there either.*

SW1# show running-config | include helperip helper-address 10.100.100.100SW1#SW1# conf tEnter configuration commands, one per line. End with CNTL/Z.SW1(config)# int vlan 10SW1(config-if)# ip helper-address 10.50.50.100SW1(config-if)# endSW1#

R1#02:13:57: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=58402:13:58: DHCPD: assigned IP address 10.10.10.115 to client 0100.1bd5.1324.42.02:13:58: UDP: sent src=0.0.0.0(67), dst=255.255.255.255(68), length=30802:13:58: UDP: sent src=0.0.0.0(67), dst=255.255.255.255(68), length=30802:13:58: UDP: rcvd src=0.0.0.0(68), dst=255.255.255.255(67), length=584

show running | include helper shows us that there is one IP helper address configured on the switch but pointing to an old address of the DHCP serverThe debug ip udp results now show UDP packets arriving at the DHCP server So we need to fix that issue. *

A few minutes later we speak to the wireless support team and they verify the successful IP address assignment. However, there is still no registration into the WLC. The wireless operations team tell us to check the configuration of Option 43 on the DHCP server. The show running | section ip dhcp pool displays No Option 43.Option 43 is used to notify the DHCP client the AP-management IP address of the WLC. *RegistrationRequests?R1# show running-config | section ip dhcp poolip dhcp pool vlan10 network 10.10.10.0 255.255.255.0 default-router 10.10.10.1

Therefore, we need to go into the DHCP pool configuration mode using the ip dhcp pool command, and enter the AP-management IP address of the WLC as part of Option 43. For that we use the command option 43, followed by the right IP address in hex. The hex string is assembled by concatenating Type, Length, and Value.Type is always F1 (hex)Length is the number of controller management IP addresses times 4 in hexIf there is only one WLC management address, the Length is 04 (hex)Value is the IP address of the controller listed sequentially in hexThe IP address is 10.10.10.10 which is 0a0a0a0a (hex)*R1# conf tEnter configuration commands, one per line. End with CNTL/Z.R1(config)# ip dhcp pool vlan10R1(dhcp-config)# option 43 hex f1040a0a0a0aR1(dhcp-config)# endR1#FIXED!

CIS 188 CCNP TSHOOT (Troubleshooting)Ch. 8 Troubleshooting Converged Networks Part 1Rick GrazianiCabrillo Collegegraziani@cabrillo.edu

*******************************The decision of whether to mark traffic at layers 2 or 3 or both is not trivial and should be made after consideration of the following points:Layer 2 marking of frames can be performed for non IP traffic.Layer 2 marking of frames is the only QoS option available for switches that are not IP awareLayer 3 marking will carry the QoS information end-to-endOlder IP equipment may not understand DSCP

**