“ip network troubleshooting“ · " ip network troubleshooting – part 2" advertised...
TRANSCRIPT
“IP Network Troubleshooting“ Part 2
Wayne M. Pecena, CPBE, CBNE Texas A&M University
Educational Broadcast Services - KAMU
November 2015
" IP Network Troubleshooting – Part 2"
Advertised Presentation Scope:
Successful troubleshooting of any system relies upon applying a logical
approach with knowledge of the technology involved, and maybe a little luck
when experience is lacking. This webinar will focus upon applying a logical
approach to troubleshooting an IP network in a LAN environment with a basic
understanding of IP networking principals by following the data flow layers of
the OSI model. A focus will be placed upon utilization of techniques and use
of common open-source tools to identify and isolate network connectivity and
performance abnormalities.
Webinar Goals:
Continue Structured Troubleshooting Approach (from part 1)
Understand the Basics the OSI Model Layers 3-4 in an IP Network
Understand & Apply Techniques to Verify Layer 3-4 Functionality
Understand How to Get Started with Wireshark Protocol Analysis
2
Today’s Outline:
• Refresh Takeaways from Part 1
• Network & Session Layer Functions & Verification
• Understanding ping & traceroute
• Getting Started with Wireshark
• Taping Into the Network
• Wireshark Capture & Filtering
• Takeaways, Questions, and Maybe Some Answers
3
Takeaways Points from Part 1
4
5 Things Required To Build a Network
• Send Host
• Receive Host
• Message or Data to Send Between Hosts
• Media to Interconnect Hosts
• Protocol to Define How Data is Transferred
5
Protocols
Send Host Receive Host
MediaMedia
DATA
The Structured Troubleshooting Process
6
Problem Identification
Problem Diagnosis
Problem Resolution
Develop a
Structured Approach
To
Resolution
Avoid
an
Unstructured Approach
ProblemIdentification
ProblemRe-Creation
Localize & Isolate
Problem
FormulateResolution
Plan
DocumentResolution
Provide FeedbackTo Users
ImplementResolution
Plan
VerifyResolutionResolved
The OSI Model Open Systems Interconnection (OSI) Model
7
Networking
Focus
Open Systems Interconnection “OSI” Model
8
Part
2
Focus
Transport
Physical
Data Link
Network
4
1
2
3
Manages End-End Connections:
TCP, UDP, & Flow Control
Interfaces to Physical Network, Moves Bits Onto &
Off Network Medium
Provides Network Access Control, Physical
Address (MAC), & Error Detection
Provides Internetwork Routing (path)
Provides Virtual Addressing (IP)
Network Abnormalities
• Categories of Abnormalities: – No Connectivity
• Cable Fault
• Blocked or Failed Switch Port
• Failed Host NIC
– Intermittent Connectivity
• Cable Fault
• Failed Switch Port
• Failed Host NIC
• Duplex Mismatch
– Poor Performance
• Cable Fault
• Failed Switch Port
• Failed Host NIC
• Duplex Mismatch
9
Common Layer 1 Faults • Copper Cabling Aspects:
– Continuity Problems (open, shorts, crossed)
– Wrong Cable for Application
– Improper RJ-45 Installation
– Excessive “Un-Twist”
– Excessive Connections (connector blocks)
– Excessive Segment Length
• Fiber Cabling Aspects: – Damaged Fiber
– Improper Connector Installation
– Dirty Connectors
– Component Aging
10
11
12
Optical Power Guidelines:
Stay Within +/- 4 dB
Receive Power Generally Within:
-27 to -8 dB
Design “Sweet Spot”: (66%)
-17 to -23
Launch Power (Transmit) Selected
For the Required Optical Budget
Common Layer 2 Faults
• Failed or Intermittent Host NIC
• Failed or Intermittent Switch Port
• Duplex Mismatch
• Excessive Errors
13
Use
“Managed”
Switch
Port
Capabilities
To
Verify
Operation
Cisco Switchport Metrics
14
Network Documentation “Network Cartography”
15
Documentation Excuses
• “I don’t need to put anything in writing, I’ve done this kind of system a million times.”
• “I don’t need a roadmap. I can just tell where things need to go.”
• “My company doesn’t have the money to invest in creating those documents.”
16
Source: Summarized from InfoComm Blog 11/6/15
Takeaway Points & Concepts – Part 1
• Establish a “Structured” Troubleshooting Approach
• Use the OSI Model as a Guide – Verify Layer 1 Physical Connectivity
– Verify Layer 2 Connectivity is Error Free
• 80% of True Network Problems is Physical Infrastructure Based – Standards Not Properly Applied
– Guidelines Not Adhered To
• Don’t Loose Sight of 100m Ethernet Segment Limit!
• Network Documentation & Baseline Performance Metrics Are Essential to Efficient Network Problem Resolution!
17
Network & Session Layer Functions & Verification
18
IPv4 Packet – Layer 3 RFC 791
19
Version
(4)
Header
(4)
Precedence / Type
(8)
Length
(16)
Identification
(16)
Flag
(3)
Offset
(13)
Time to Live
(8)
Protocol
(8)
Header Checksum
(16)
Source IP Address
(32)
Options & Padding
(0 or 32)
Destination IP Address
(32)
Packet Payload
(Transport Layer Data)
32 bits
20
Bytes
2-Part IPv4 Address
20
192
32 bit IP Address
1100000010101000110010011111110
168 100 254
11000000 10101000 1100100 11111110
Subnet
Mask
Determines
Network
Address
Host
Address
Octet 1 Octet 2 Octet 3 Octet 4
4 Bytes
VLSM • Allows Mask to Be Determined on a “Bit Basis”
– Classful Addressing Specifies Network / Host Boundary
– Classless Addressing Allows Network / Host Boundary to Be Specified at an Individual Bit
21
Octet 1 Octet 2 Octet 3 Octet 4
Octet 1 Octet 2 Octet 3 Octet 4
A B C
19 Subnet Mask Bits = 255.255.224.0
Network Host
Network Host
IPv4 Address Subnet Mask Example “VLSM” - Each IP Address Must Have a Subnet Mask to Define the Network and the Host
32 Bit Subnet Mask
Expressed in Decimal as (4) 8-bit Octets using “Doted Decimal Notation”
IP Address: 192.168.100.254 /19
192.168.100.254 /19 or 255.255.224.0
11000000.10101000.00000001.01100100
11111111.11111111.11100000.00000000Network Host
Is My IP Address Correct? Reverse Engineering an IP Network
23
You Need to Know: Useable IP Address Range?
See .pdf Handout for Further Study
TCP Basics Transmission Control Protocol
• “Connection – Oriented” Protocol – Connection Establishment
– Segmentation & Sequencing
– Acknowledgement
– Flow Control or Windowing
• Guaranteed Or Reliable Data Delivery – Acknowledgment of Packet Receipt
– Retransmission Occurs if Packet Not Received
• High Overhead
• Requires Establishment of a “Session”
• TCP Windowing Feature – Dynamic Window Sizing
– “Slow-Start”
24
TCP 3-Way Handshake
25
Host 1 Host 2
SYN
SYN + ACK
ACK
Host 1 Sends
Synchronize Message
to Host 2
Host 2 Responds With
Acknowledgement
Plus Sends It’s Own
Synchronization
Message to Host 1Host 1 Completes the
3-Way Handshake By
Sending
Acknowledgement to
Host 2
Host 1 Initiates
Connection to Host 2
The TCP Session Summary
26
SYN + ACK
Time
Network
SYN
ACK
FIN
FIN
ACK
ACK
ACK
Connection
Closed
Listen
SYN Sent
SYN Received
Connection
Established Connection
Established
Connection
Closed
FIN Wait 1
FIN Wait 2
CLOSE Wait
Last ACK
ACK
ACK
Data Segment 1
Data Segment 2
Data Segment 3
TCP Sequencing
27
Host 1
Host 2
Sequence Number 1
Sequence Number 1501
Receive ACK
Sequence Number 3001
Sequence Number 4501
Receive ACK
Receive 1 – 1500
Receive 1501 – 3000
Send ACK 3001
Receive 3001 – 4500
Receive 4501 – 6000
Send ACK 6001
1500 bytes
1500 bytes
1500 bytes
1500 bytes
TCP Connection
Established
Window Size = 3000
UDP Basics User Datagram Protocol
• “Connectionless” Protocol
• Simple or Lightweight, but Inherently Unreliable
• “Best Effort” Data Delivery
• Low Overhead, Thus Low Latency
• Why Use?
– Required for Real-Time Applications: • VOIP or “Video Over IP” or “Voice Over IP”
• AOIP or Audio Over IP”
– Latency More Detrimental Than Data Loss
28
UDP Session
29
Network
SYN
SYN + ACK
ACK
Data
Data
Data
Time
Data
Data
TCP Used to
Establish UDP
Session
TCP and UDP Headers
30
Connection Termination
31
Host 1 Host 2
FIN
FIN + ACK
ACK
Host 1 Sends Finish
Message to Host 2
Host 2 Responds With
Finish Plus Sends It’s
Own Synchronization
Message to Host 1
Host 1 Completes the
Termination By
Sending
Acknowledgement to
Host 2
Host 1 is Ready to
Terminate Connection
Understanding ping & traceroute
32
ICMP Internet Control Message Protocol
• Network Layer Based – RFC 1256 – The “Tattle Tale” Protocol
• Unique IP Based “Message” (IP protocol #1)
• Message Originated by a Layer 3 Device “Router”
• Message Sent to a Host or Another Router
• Common Messages: – Destination Unreachable
– Buffer Full
– Hops or Time Exceeded (TTL)
• Common Use by Network Utilities: – Ping
– Traceroute
ICMP in Detail
34
“ping” Packet Internet Groper
35
Send Hosts Sends ICMP “echo request”
Destination Host Replies ICMP “echo reply”
Round-Trip Times Returned
Be Aware of Command Line Options
“traceroute” RFC 1812
• The Most Widely Used Network Diagnostic Tool
• The Most Widely Misunderstood Network Diagnostic Tool
• How? – Send Host Transmits 3 UDP Packets to Receive Host With TTL = 1 (port typical 33434)
– First Hop Router Sends icmp TTL Exceeded
– Send Host Transmits 3 UDP Packets to Receive Host With TTL = 2
– Second Hop Router Sends icmp TTL Exceeded
– Send Host Transmits 3 UDP Packets to Receive Host With TTL = 3
– Third Hop Router Sends icmp TTL Exceeded
– Send Host Transmits 3 UDP Packets to Receive Host With TTL = 4
36
Send Host
Receive Host
icmp TTL exceeded
icmp TTL exceeded
icmp TTL exceeded
icmp destination port unreachable
TTL=1 TTL=4 TTL=3 TTL=2
Understanding “traceroute”
37
Windows
Command
Screen
“tracert”
PingPlotter
What Is
Up?
Limitations of “ping” & “traceroute” • ICMP May Be Blocked Within Networks
• Routers May Limit ICMP Processing (interfaces limited)
• Realize Layer 2 Devices Will Not Be Seen
• Protocol Utilized by traceroute Can Impact Results (UDP, ICMP, TCP)
• Understand: – traceroute Forward Path Route is Displayed (return path may be different)
– traceroute returns Round-Trip Latency
• Understand Traceroute Latency: – Latency Increase May Not Be Significant
– Latency Increase Must Continue Increasing for Additional Hops To Be of Concern
38
Cisco Routing Verification
39
Useful “Cisco” IOS Troubleshooting Commands:
R1# show ip protocols
R1# show ip route
R1# show ip route 66.39.27.70
R1# ping 66.39.27.70
Getting Started with Wireshark
40
What is “Wireshark?” • “Open Source” Protocol Analyzer Application
• Often Referred to as a “Sniffer”©
• Developed in 1998 as “Ethereal”
• Renamed in 2006 Due to Trademark Issues
• Analyses of “Live” & “Recorded” Network Activity
• Useful To: – Isolate performance issues
– Understand application interaction
– Benchmarking
Gerald Combs
Wireshark Developer
Obtain & Install “Wireshark”
• Available for Windows, Mac OSx, & Linux
• Download at: www.wireshark.org
• Include Libraries:
– WinPcap
– libpcap
NIC “Promiscuous” Mode
Network
Interface
Controller
“Listens” for
It’s MAC Address
& Broadcasts
“Promiscuous” Mode
Processes All Frames Received
Transport
Network
Data Link
Physical
Upper Layers
Network
Packet Capture“pcap”
Packet Analyzer Ap“Wireshark”
All TX & RXFrames
pcap ”packet capture”:
API (application programming
interface) for capturing network
traffic:
libpcap for Unix/Linux
WinPcap for MS Windows
Taping Into the Network
44
Where to Tap?
• Problem Nature Often Determines:
– At Problem Host
– At Destination Host
– Mid-Network Locations
• Accessibility May Also Drive Tap Point
Taping Into Ethernet
• Can Be Challenging!
• Where to Tap?
• How to Tap?
– Physical Passive Tap
– Active Tap
– Ethernet Switch Port Mirror
47
Application
Session
Presentation
Transport
Physical
Data Link
Network
Application
Session
Presentation
Transport
Physical
Data Link
Network
Physical
Data Link
Physical
Data Link
Physical
Data Link
Physical
Data Link
Physical
Data Link
Physical
Data Link
Network Network
Layer 2
Device Layer 2
Device Layer 3
Device
Tap Devices
UTP Taps
Optical Taps
HostA
HostB
HostD
HostC
Wireshark Host
Ethernet Switch Port Mirror or “SPAN Port” Switched Port Analyzer Port
Goal – Observe Traffic Between
Host A & Host B on Wireshark Host
“Span” Port
Ops! – Ethernet Switch
Isolates Network
Traffic
Define Source Port(s)
Designate a “Span Port”
config t
monitor session 1 source interface fa0/1
monitor session 1 source interface fa0/23
monitor session 1 destination interface fa0/14
exit
Wireshark Capture & Filtering
50
Capturing Network Traffic
51
Captured Packet(s) List
Selected Header
Data Decoded
Payload Data Decoded Hexadecimal & ASCII
Wireshark Views
packet 192 selected
Header Details Displayed
Payload Data Decoded
Filtering
• Capture Filters – Selectively Capture Packets
– Pre-Capture Configuration
– Minimizes Captured Data
• Analysis Filters – Applied When Viewing
– Allows Focusing on an Attribute(s)
– All Data is Retained
Filter Example
Wireshark Example “Benchmark Network Activity”
56
Wireshark Example “ping www.sbe.org”
57
Wireshark Example
58
“audio stream example – udp filter & modify column headings”
Wireshark Example “TCP/IP Window”
59
100 101 102 103 104 105 106
Bytes Receive
Is Ready to Accept
107 108 109 110 111 112
TCP Receive Window
BYTES Sent
NOT AcknowledgedBYTES Sent &
AcknowledgedBytes Receiver
Is NOT Ready to Accept
RFC 1072 & RFC 1323
Takeaways, Questions, and Maybe Some Answers
60
Takeaway Points & Concepts – Part 2
• Understand Limitations of “ping” & “traceroute”
• Protocol Analysis Is Essential to “See” Network Activity
• “Wireshark” Is The Most Popular Protocol Analyzer
• Understanding the OSI Model & TCP/IP Protocol Action is Essential to Understanding Wireshark Results
• Pre or Post Capture “Filtering” is Essential to Find the Needle in the Haystack
– Capture Filters
– Display Filters
• Wireless Is Also Supported by Wireshark – Think “Layers 1 & 2”
• “Flat Broke” is Often Easier to Fix Than Performance Issues!
• The Network is Commonly Blamed for Performance Issues: In Reality < 5% – You Are Often Proving the “Network is Innocent” (Gartner Research)
61
There is Always More to Know!
62
Don’t Miss IP Network Troubleshooting Part 3!
February 23, 2016 – 2pm ET
Focused on Protocol Analysis in the Broadcast
Plant
References – Further Study
https://wiki.wireshark.org/
Graphical “Traceroute” Utility
http://www.pingplotter.com Free & Paid Versions
64
65
Thank You for Attending! Wayne M. Pecena Texas A&M University [email protected] 979.845.5662
66
? Questions ?
4