Troubleshooting

Tools & Tactics

CPTE 433 – Chapter 15

John Beckett

What is the Real Problem?

• Be clear about distinguishing between symptoms and problems

• “Solving” a symptom will either:– Position it to return, possibly worse (Social Security)– Move it

• Sometimes merely moving a symptom is a good thing– Queue to restaurant: Give them pagers and let them

stroll about the grounds– Give people appointments so they are free to do

other things

Methods of Troubleshooting

1. Successive Refinement (binary search)1. Addition2. Elimination

2. Follow the Path1. Establishes context (same as Successive

Refinement)2. If you understand the context, FTP handles

more complex problems3. Knowledge Base

– Used often for first attempt to solve problem

The User

• Challenge is to communicate clearly.• Cultivate confidence.

Reducing Variables

• Use minimal system possible to verify what you are checking.

“Wire” tests

• Battery and volt-meter– Avoids cross-talk issues

• Tone tester– Increasingly ineffective due to better

wire• Time-Domain Reflectometer

– Save time finding a break• Flashlight

– Works for short (<1000’) fiber links

TCP/IP

• host/nslookup/dig– Looks up hostnames

• ping– Verifies connectivity

• traceroute/tracert– Shows the path

• telnet– Tests a port

• arp –a– Shows Ethernet addresses

Loop-Back

• Old method used with serial ports• Using a network appliance like a

router may enable you to do many of the same types of test.

• “Ping” tests use ICMP packets to do the same thing.

• Using a “dumb” person as transponder – can adapt to many contexts

In Vitro

• Can you reproduce the problem?– On their workstation– On your workstation– In a separate environment

Network Analyzers

• Available both as hardware and software.– Ethereal is a free software analyzer often used for

wireless links.• Increasingly, hardware network analyzers are

looking and acting like computers.• Key issue is ability to analyze packets.

– Contents– Source & Destination– Timing & Size issues

• Of absolutely no use unless you learn how to use!

When Does It Happen?

• If absolutely nothing else helps, carefully analyze the situation timing– How often does it occur?– What times of day is the cycle active?– Who is where when it happens?

• Cases– Network timing issue related to periodic

refreshes– Magnetic office at SAU– Recordings at CUC– Audio interference from cell phones

Is It Really Digital?

• Video fiber units with odd specifications:– No breaks in the fiber (must be fused

connections).– Unusual intolerance for db loss.

• VGA is actually analog, which explains shadows to the right of vertical lines

• These were really analog devices, with electronics able to compensate somewhat for variations in the link.– Digital troubleshooting techniques did not work

with them!

Collect Techniques & Tools

• Each technique or tool has its place.• Your understanding of the technology

and system design allows you to use a tool effectively.

• Example: difference between TCP and UDP

Basic Cabling Issues

• How is the cable routed (managed)?– Need a plan that will survive without

creating a mess.• Credibility• Troubleshooting

• Strain Relief• Electrical or Optical connection

Power

• IEC Connectors– Some work loose over time.– Occasional bad cord (cheapest commodity)

• “Surge” strips– Failure mode is to continue passing voltage– Rarely: bad connection within strip

• Voltage/frequency/etc.– Some power supplies are incompatible with

some “regulating transformers”.• Reason: transformer modifies waveform

Telco Color Code• Blue• Orange• Green• Brown• Slate

• White• Red• Black• Yellow• Violet

1 Wh/Blu 2 Wh/Org 3 Wh/Grn 4 Wh/Brn 5 Wh/Slt

6 Red/Blu 7 Red/Org 8 Red/Grn 9 Red/Brn 10 Red/Slt

11 Blk/Blu 12 Blk/Org 13 Blk/Grn 14 Blk/Brn 15 Blk/Slt

16 Yel/Blu 17 Yel/Org 18 Yel/Grn 19 Yel/Brn 20 Yel/Slt

21 Vio/Blu 22 Vio/Org 23 Vio/Grn 24 Vio/Brn 25 Vio/Slt

http://www.answers.com/topic/25-pair-color-code?cat=technology

BellOperatorsGiveBetterService

WhyRunBackwardsYou’llVomit

Setting Up For Failure…

• Insufficient bandwidth ordered– Failure to partition (get 2 links!)

• Not knowing the resources you’d have

• Stupid misteaks• People who don’t want to pay

– Let them choose the outcome– “We can do it whichever you wish.

Option 1 will cost $65,000, option 2 is $80,000…”

RJ-45for Ethernet

• Essential that wires go to the same pins (duh)– There is good reason for using the right wire for each

pin – reduces crosstalk and improves speed. May not work if wrong.

– Use proper pairing or speed may suffer.– Some switches will swap “mouth and ear” connections,

and some don’t: so you can get confused.– There are testers which will diagnose a cable.

• They either require both ends in one location or a transponder for the “other end”.

• See http://pinouts.ws/rj-45-pinout.html• Speed varies. Speed is important. Speed is

usually auto-negotiated.• Install wiring for the next-higher speed

– Save a rewire sequence

T-1

• Connectivity can be copper or fiber.– Fiber is better – probably because you aren’t re-

using old copper (which is the reason the ISP may lean toward copper).

• Same standing wave issues as ISDN (see next slide), only worse.

• Requires CSU, DSU, and router. CSU and DSU are normally combined. Some routers even include a CSU/DSU.

• Connectors may be the strangest things you’ve seen!

• Tested using a “T-Berd” or similar device.

ISDN

• Uses Cat-2 (old telephone twisted) wires.– But…RJ-45 is usually used!

• Information is transmitted digitally.• Subject to interference from standing waves.

– Don’t run the same wire to several jacks and expect to be able to plug into any of them (e.g. mobile videoconferencing unit).

– Do make the connection straight to the end, and stop there.

• Some proprietary digital phone systems use ISDN or ISDN-like technology.

The Standing-Wave Issue

Source

Destination

AlternateDestination

Two Types of ISDN

• ISDN-BRI– Intended for home or SOHO use– 2-64kbps “bearer” channels– 1-16kbps “data” channel, used only for signaling– Has evolved into “DSL Lite”

• ISDN-PRI– Actually uses a T-1 link, but ISDN protocols– 23-64kbps “bearer” channels– 1-64kbps “data” channel, used only for signaling

Testing ISDN

• T-Berd• Other ISDN testing devices

• At this level you probably have TDR capabilities

• DSL (and Cable Modem): Modem has some diagnostic capability in it

Cable Modem

• Uses one or more channels in a CATV system• Modem has internal DHCP and Web servers

for setup and diagnostics• Speed is set by modem

– TFTP session at power-up allows cable company to download speed limit

– Going over the TFTP-mandated limit invokes severe legal penalties such as refusal to serve you for life!

– Newer technologies such as “speed boost” are more dynamic, and move control to network

Cable Modem Diagnostics

192.168.100.1

Which channel“Locked” meansit is working

Downstream Poweris how good a signalwe are getting fromcableSNR is anotherQuality indicator

Web Page on this

Serial Connections

• Standard is EIA RS-232 (b,c, whatever)• Designed for 9600 bps (that’s 9.6 kbps) for

50 ft.• Commonly run much longer and faster

than that.• Difficult to configure

– Becoming a lost art. • Subject to damage from lightning

– Transmit chip: 75188 / 1458 (more likely to blow)

– Receive chip: 75189 / 1459– Later chips combine both functions, or combine

RS232 translation with other functions such as UART.

Parallel Connections

• Originally described by Centronics so various computer mfgrs could use their printers.

• Epson added a feature (reliability), and documented the “standard” in their manuals.

• HP needed more speed and reliability– IEEE finally defined a standard, which

later printers required.• Non-IEEE cables still on the market

($1 stores)

USB

• Power in the cable– Does your equipment provide enough power?

• Hot-plugs well• USB-1 and USB-2 are vastly different in

speed.– USB-3 emerging, will USB-4 be fiber?

• Used only for workstation wiring, not networking.– …Although networking devices such as

modems and routers may connect via USB using driver software.

Multiple Signals on One Wire

• Space Division Multiplexing: a wire for each purpose.– Local telephones, final wires to desktops

• Frequency Division Multiplexing: Think of channels on the radio dial. No longer used for networking.– Deprecated method used by telcos

• TDM: “Time” for each signal, rotating.– Replaced FDM

• Packets: The entire bandwidth is available, but anyone can use it.– e.g. TCP/IP

Troubleshooting Challenges

• Space Division Multiplexing: Crosstalk, management.

• Frequency Division Multiplexing: Crosstalk is possible if filtering doesn’t work

• TDM: The clock is king.• Packets: You can “cheat” on

bandwidth. Management issues arise.

VoIP

• Uses packet-switched networks.• The issue is one of priority.• Voice cannot tolerate delayed or

dropped packets without sounding degraded.

• Quality of Service (QOS) must be end-to-end to be effective.– Residential “QOS” decreases data

ceiling when phone is active, has no effect on Internet QOS

VoIP State of the Market

• Vendors are under-estimating costs because they do not recognize what it takes to provide the service.

• “Wild West” market, with a new vendor each week coming in at lower price point.– Goal is to kill the others, then raise prices– Then somebody else tries the same thing

• Once service is established, it tends to work well.• No vendor has a good reputation for solving

problems, because they are focusing on market penetration.

• Huge issue: net neutrality

So Why Use Home VoIP?

You must have all of the following qualifications:

1. High-bandwidth connection, lightly loaded.

2. Tolerance for long outages.3. Determination to use the service a

lot when it is working.4. Alternative phone line for essential

calls.Or…5. Use the same vendor as your Cable

ISP

Some possible WAN VoIP Applications

• You can rapidly move your phone presence from one location to another– Wherever you can get an Internet

connection• Could be a good way to provide

backup service at minimal cost• Could be a way to provide the same

phone number when you cycle from one location to another

What About Corporate VoIP?• You must implement end-to-end

QOS.• Your network must be as reliable as

you want your phones to be.• Need VoIP terminals.

– Fortunately, “SIP” phones are available.• Big advantage: Run all

communication through the same infrastructure.– Save an entire infrastructure!– Note: SAU’s telephone person is getting

Cisco CCNA. All telephone managers need to do this.