11

Trace Collection Guidelines

Vik Evans

Systems Engineer

Enterprise Networking and Communications

WiNG 5

22

Troubleshooting Checklist

Mandatory Information

ID Description Response

1 Customer

2 Perceived Problem

3 Problem identified by tier-II, including

underlying issues.

4 Config files for devices, switches, AP’s and

firmware versions

5 Steps used to reproduce problem on test

bench

6 Obtain Syslogs

7 Obtain appropriate wireless & wired traces;

Aeropeek / Omnipeek

8 Network topology (logical / physical layout)

33

Troubleshooting Checklist (cont.)

Useful Information

ID Description Response

1 Customer Disposition

2 Duration of problem thus far

3 Severity of impact on operations

4 Current work-arounds customer may be using

5 Does the problem occur at multiple sites?

6 Can the customer reproduce the problem?

44

Troubleshooting Checklist (cont.)

Optional Information (based on relevance)

ID Description Response

1 Intermittent? If so, what is the frequency?

2 Any changes to network / configurations recently?

3 Additional configuration information for AP’s

4 Multiple ESS’s in use?

5 Trunking?

6 Security info on WLAN(s)

7 Bluetooth Enabled?

8 Network Addresses (MAC & IP)

9 Mobile device types (scanners / VoIP phones /

laptops)

10 Vendors / Models

11 Did customer have a site survey performed?

12 Is there proper cell density for coverage?

13 Are there known areas of poor coverage?

14 Is switch redundancy being used?

15 Can engineer(s) visit site?

16 Environmental temperature of site?

55

SPR Pre-Requisite: Trace Files

This presentation focuses on obtaining relevant trace files, prior to opening an SPR. It will cover traditional methods, using laptop software as well as the capabilities within WiNG 5.

66

Trace Collection Guideline

•Objective: Minimize problem identification by providing guidelines on collection of traces and doing a basic analysis to ensure all the relevant information has been captured

Agenda:

• Overview of Trace Collection Tools

• Trace Collection Procedures

• Basic Trace Analysis

77

Section I: Packet Analysis Utilities

88

Trace Collection & Analysis Utilities

WildPackets – OmniPeek (formerly AiroPeek)

Netscout – Sniffer Analysis (formerly Network

General Sniffer Pro)

Wireshark

Other Wireless Analysis Utilities

Riverbed / Cace Technologies - WifiPilot

AirMagnet

ECRT uses Wildpackets OmniPeek and Wireshark as the standard capture analysis tools. OmniPeek and Wireshark will be the focus of this presentation.

99

Host Setup

Wildpackets has many device drivers at:

http://www.wildpackets.com/support/omni/omnipeek_enterprise/wireless

Wireshark will depend on adapter and host OS:

*nix OS will use the libpcap library, included

Windows OS will us winpcap, which will install during Wireshark installation

AirPcap is a link-layer library and adapter used to perform wireless capture in Wireshark on a Windows host

Ensure proper drivers / libraries are installed for your host device.

10

OmniPeek Dashboards

In Network Dashboard mode, OmniPeek displays key statistics like utilization, wireless signal and recently saved files.

11

OmniPeek Capture Window Elements

Some common interface elements are shown.

1212

Performing OmniPeek Captures

Click the “New Capture” icon

Click “Adapter” in the Capture Options dialog and select the desired capture interface.

Click “General”, name the capture and specify continuous or not.

Note that wireless capture will not be possible without a supported adapter and drivers.

1313

Wireshark

Wireshark (formerly Ethereal) is a free, open-source utility that, over the years, has developed into a very robust packet analysis application.

Wireshark runs on many platforms, including Windows, Mac OS X and Linux.

14

Wireshark Startup Screen & Elements

1515

Wireshark Notes:

Wireshark is able to save in / work with formats recognized by OmniPeek, so there is no concern for incompatibilities.

Any adapter that shows up in the “Interface List” is available for capture.

Promiscuous mode will capture wireless packets of the SSID the adapter is joined to only.

Monitor mode will allow capture of all 802.11 packets heard, however will not allow membership to any WLAN – it is purely for capture.

1616

Performing Wireshark Captures

Click the “Capture Options” icon

Select the desired adapter from the drop-down menu.

At “Interface” you can specify whether the capture is local, or a remote-host . This would be another Wireshark machine configured to listen for incoming requests

Click “Start”.

1717

Planning and Validation Applications

Motorola LANPlanner

Motorola AirDefense Mobile

Predictive planning and site survey validation for both

AirMagnet Survey

Site survey / coverage validation

Other utilities exist for planning, validation and troubleshooting that should be used initially for proper implementation of a wireless network.

1818

Section II: WiNG 5 Packet Capture

1919

The packet capture features of WiNG 5 enable one to collect traces from almost any point in a network.

Traces can be captured in real-time or off-line for less impact on the network and stored locally to flash, TFTP, FTP or in real time to a TZSP host running OmniPeek or Wireshark.

For details on utilizing WiNG 5 capture features, please refer to the feature guide at: http://compass.mot-

solutions.com/doc/375558309/How_To_WiNG5_pktcap_v1.4_final.pdf

2020

WiNG 5 Trace Collection Overview

WiNG 5 provides several physical and logical points at which trace collection can take place.

The diagram is representative of a WiNG 5 access point and shows the many local interfaces from which captures can be collected.

Additionally, the “remote-debug” feature of WiNG 5 allows for remote capture at a specified device, like a distributed sniffer.

Router

Bridge Bridge

VLANs

Ethernet Radio

WLAN’s

VPN

Interface

2121

Section III: Trace Collection Procedures

2222

You should synchronize the clocks of all capture PCs to the correct time!

2323

Trace Collection Considerations

Understanding relative time of a trace and occurring problems is important when troubleshooting. It is good practice to sync the time on all capture machines and to reflect the time in the capture

In OmniPeek, click once on the column headings to bring up the Packet List Options dialog, then select “Absolute Time”.

2424

Trace Collection Considerations

In Wireshark, right-click on the column bar, and select Column preferences in the menu. This will bring up the Wireshark preferences dialog.

2525

Trace Collection Considerations

You can high-light the default “Time” column and then change it to “Absolute Time” from the drop-down.

2626

Collection Considerations: Ethernet

Some adapters will strip vlantags by default, when processing traffic. In order to make sure this information is included in your trace, ensure the driver allows for and VLAN processing is enabled.

2727

Wireless Capture Placement

When troubleshooting an AP, capture as close to the target AP as possible.

In WiNG 5 this can be done at the target AP or a neighboring AP.

Wireless capture

Troubleshooting the AP

MU

When troubleshooting MU(s), place the capture device as close to the MU as possible or capture at the AP the MU is trying to associate to.

Wireless Capture

Troubleshooting the MU

2828

Wireless Channel Considerations

When collecting traces, many utilities will scan and capture on all available channels. This may cause some packets to be missed. It is best to lock onto a channel, matching that of an AP clients are trying to associate to.

However, there are times when capturing on multiple channels is necessary in order to get traffic from all MU’s in an area. With WiNG 5’s “remote-debug” command, this can easily be accomplished.

When capturing from multiple hosts using remote-debug, the device at which the command is performed (typically a controller) will automatically collate the captures from multiple devices (AP’s) into one stream for analysis. The following example initiates a capture at two access points on radio 1 for each. These may represent two AP’s in a specific area, on different channels:

remote-debug live-pktcap hosts ap7131-970408 ap7131-

9313CC radio 1

29

Wired Capture Placement

Traffic should be captured as close as possible to all devices related to the specific data conversation. This can be accomplished using switch span ports and / or capturing on WiNG 5 device interfaces

Server with issues

Server with no issues

Wired Capture PC on span port

Wired Capture PC on span port

WiNG 5 AP’s

WiNG 5 RF Switch

3030

Principles for Trace Collection

A trace is only as good as the context in which is was captured.

Give the trace a descriptive name

Include date, customer name, MAC addr (if possible), SPR #. If wired trace, include location (srvr / ap / etc.), trunk #

Example: 0506RamaSPR11008Ch6.apc

0710BellCanadaAPreset.pkt

Should trace be L2 or L3?

Do not use capture filters; filtering can be performed later.

If possible, capture in continuous mode

Problems may take time to manifest

3131

Troubleshooting Tips

Required information for debugging:

Syslog Messages

Syslog server connected to problem LAN

Wired trace of all traffic into and out of the RF switch, in line with suspect traffic.

Wireless traces taken at AP(s), or as near as possible to problem clients / AP’s.

Time of failure, assuming time synchronization

Mac addresses of failed clients, servers, etc.

Network topology diagram

Narrative of problem, how and where the trace(s) was taken.

3232

Trace Collection – Wireless Issue

Definition: Problem only occurs with MU / client.

Examples Include

Wireless Association failure

Roaming issue

Proxy ARP

Traces to collect

Wireless trace at client or AP

Collection laptop at location of client

Collection at AP radio interface or wireless interface

Wired trace on segment client is on

Spanned switch port

Ge1 interface of AP using “pktcap” or “remote-debug” commands

33

Trace Collection – Wireless Issues

Server with issues

Server with no issues

Wired Capture PC on span port

Wireless Capture PC w/ compatible adapter

WiNG 5 AP’s

WiNG 5 RF Switch

Wireless Client device (MU)

Capture here

Or Capture here on radio or wireless interface (using “pktcap” WiNG 5 command).

3434

Trace Collection – Firewall / Routing Issue

Definition: Involves two endpoint in separate IP domains.

Examples Include

Can’t access Internet (LAN →WAN)

VPN not working (LAN →WAN)

Outside can’t access internal server (WAN →LAN)

Traces to collect

Wired traces on each IP segment

Span port on each subnet to capture traffic from wireless AP (client traversal) and wired destination (server / voice gateway, etc.)

Wireless trace not needed

3535

WiNG 5 Command Summary

Simple capture to flash memory of WiNG 5 device:

rfs4000-22D26E#service pktcap on interface ge1 write pktcap-

test.pcap

Capture / send to TZSP host for real-time analysis:

TZSP host is running Wireshark and iperf.exe in server mode*

rfs4000-22D26E#remote-debug live-pktcap hosts ap7131-970408

ap7131-9313CC write tzsp 192.168.150.10

Tazmen Sniffer Protocol is an encapsulation protocol used to “wrap” other protocols; typically in UDP and is used for wireless captures.

WiNG 5 implementation of TZSP sends on UDP port 37008.

*Iperf.exe is a free Windows CLI tool used for performance testing. You can initiate iperf in server mode, listening on UDP port 37008, so you don’t receive ICMP destination port unreachable messages in your trace.

iperf.exe -s -u -p 37008

3636

WiNG 5 Command Summary

When using the remote-debug command to capture on multiple hosts, the independent captures will be collated into a single stream at the initiated device (usually a controller).

Because the actual trace is distributed among multiple devices, there will not be a significant load the controller / initiating device.

The exception to this is if a capture is done using the “rf-domain” option, which then captures on all hosts in the rf-domain. This may cause too many packets too quickly for the initiating device to collate without dropping some packets.

3737

Basic Trace Analysis

Everyone taking traces should, at the least, be able to

look at the capture file and see if there is data to and

from the MU and the host app. in the trace.

Traces of one-way communication do not aid

in determining the problem. Make sure entire

conversation is captured.

Traces that do not capture the failure taking place also

are of no use. Make sure the failure takes place and is

captured in your trace file.

3838

Basic Trace Analysis

Perform quick filtering by right-clicking on packet and choosing “Select Related Packets →”

Group by source / destination Mac address

Group by source / destination IP address

Group by protocol

Traces to collect

Wired traces on each IP segment

Span port on each subnet to capture traffic from wireless AP (client traversal) and wired destination (server / voice gateway, etc.)

Wireless trace not needed

3939

OmniPeek – Quick Filters

4040

OmniPeek – Quick Filters

OmniPeek will highlight all packets related to your selection and then you can choose what to hide.

Filtering can always be done later, so when performing a capture, capture everything.

4141

Wireshark – Quick Filters

4242

Wireshark – Quick Filters

Wireshark has the ability to build quick filters based on specific parts of the packet headers.

Simply right-click the data to filter and select Apply as or Prepare as filter.

The display filter box will be instantly populated with the filter syntax.

4343

Additional Resources

WiNG 5 Packet Capture Feature Guide:

http://compass.mot-solutions.com/doc/375558309/How_To_WiNG5_pktcap_v1.4_final.pdf

Packet Capture Screencasts:

http://compass.mot-solutions.com/web/wlan/How%20To%20Videos

iperf.exe for Windows

https://publishing.ucf.edu/sites/itr/cst/Pages/IPerf.aspx