presenter: steve litt · diagnostic tests: quick preliminary high level diagnostic tests test...
TRANSCRIPT
![Page 1: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/1.jpg)
Troubleshooting djbdnsSystematically Fix It Quickly
Presenter: Steve LittPresented at Greater Orlando Linux User Group
Wednesday, November 2, 2011
Licensing: Copyright c©2011 by Steve Litt. All rights reserved.You are allowed to copy and distribute this document ONLY if you copy it exactly and completely.
All other rights reserved. Version 1.0.1
NO WARRANTY: USE AT YOUR OWN RISK
A Troubleshooters.Com R© Publication.
See http://www.troubleshooters.com for more detailed information ontroubleshooting and the Universal Troubleshooting Process.
Slide 1 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 2: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/2.jpg)
What is djbdns?
djbdns is a DNS server
Translates IP addresses to domain names and vice versa
Like Bind, but different
Separates resolver from authoritative serverMore modularIn my opinion, more reliable
Only malfunctions when upgraded, installed or changed
Slide 2 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 3: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/3.jpg)
At This Course’s End You’ll Be Able To:
Name all elements of the Troubleshooter’s MindsetName all major components of a simple djbdns setupNarrow the root cause to a major component in a few minutesRun any major component by itselfUnderstand the subcomponents of common major componentsIsolate to the subcomponent in a few minutesSolve the whole problem within an hour, 99% of the time
Slide 3 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 4: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/4.jpg)
Troubleshooter’s Mindset
Success depends on how you operate your brain
It’s not how much you know, it’s how you organize it
Block diagram (sometimes called Mental Model)
Look for the root cause
Don’t try to fix it, just try to narrow it down
Always ask, "How can I narrow it down just one more time?"
Stay rational...
Choose diagnostic tests using the quadruple tradeoff...
Use the 10 step universal troubleshooting process...
Slide 4 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 5: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/5.jpg)
Troubleshooter’s Mindset: Stay Rational
Don’t panic
Don’t get angry
No emotion: Hunt the root cause like a cold, deadly predator.
Feel free to ask for help.
Be willing to give help.
Don’t fall victim to mental blocks.
Don’t be arrogant.
Slide 5 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 6: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/6.jpg)
Quadruple Tradeoff
Even divisions
Ease
Likelihood
Safety
Slide 6 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 7: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/7.jpg)
The Ten Step Universal Troubleshooting Process
1 Prepare
2 Make a damage control plan
3 Get the symptom description
4 Reproduce the symptom
5 Corrective Maintenance
6 Narrow it down the root cause
7 Repair or replace the root cause
8 Test
9 Take Pride
10 Prevent future occurrence
Slide 7 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 8: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/8.jpg)
Diagnostic TestsAssumptions
Local subnet is 192.168.100.0/24, local domain is domain.cxm
djbdns installation is on mydesk.domain.cxm at 192.168.100.2
You’re working at mydesk.domain.cxm at 192.168.100.2Domain/subnet also has wincli.domain.cxm at 192.168.100.5
DNS resolver (dnscache) is at 192.168.100.2
Auth DNS (tinydns) for domain.cxm and subnet is at 127.0.0.1The directory scanned by svscan is /service
The dnscache directory is at /var/djb/service/dnscache
Enable with: ln -s /var/djb/service/dnscache /serviceThe tinydns directory is at /var/djb/service/tinydns
Google has a public DNS resolver at 8.8.8.8
All commands should be done logged in as root. BE CAREFULwhen working as root!!!
Slide 8 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 9: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/9.jpg)
djbdns High Level Mental Model
Slide 9 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 10: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/10.jpg)
Diagnostic Tests:Quick Preliminary High Level Diagnostic Tests
TEST MEANINGid dnscacheid tinydnsid dnslog
Check for valid standard djbdns accounts.
ping 8.8.8.8 Ping Google public DNS to test Internet connectivity.dig @8.8.8.8 -x 8.8.8.8 Verify that Google DNS at 8.8.8.8 reverse-resolves itself.dig @192.168.100.2 -x 8.8.8.8 Test dnscache: Reverse-resolve Google’s public DNS
server at 8.8.8.8, using your dnscache at 192.168.100.2.dig @192.168.100.2 google.com Test dnscache: Resolve google.com using your dnscache.
dig @127.0.0.1 wincli.domain.cxm Test tinydns: Query for wincli.domain.cxm on your au-thoritative DNS server at 127.0.0.1.
dig @127.0.0.1 -x 192.168.100.5 Test tinydns: Reverse-query for 192.168.100.5 on yourauthoritative DNS server at 127.0.0.1.NOTE: If the forward authoritative query works but notthe reverse, or vice versa, it’s probably a misconfiguredtinydns (/service/tinydns/root/data) rather than a non-functional one.
dig @192.168.100.2 wincli.domain.cxm Test dnscache to tinydns linkps ax | grep svscan On standard installations this command should produce
a line of output containing "svscan /service".If the directory after the word svscan is anything otherthan /service, then you have a non-standard installation,and you need to carefully evaluate everything.
ls -ldF /service/* Verify tinydns and dnscache are symlinked to /service.
Slide 10 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 11: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/11.jpg)
dnscache Mental Model
Slide 11 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 12: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/12.jpg)
Diagnostic Tests:dnscache Alone
SITUATION TEST MEANINGIf dnscachedidn’t work
ps ax | grep dnscache
Look for "supervise dnscache" and"/usr/local/bin/dnscache"
If no /usr/local/bin/dnscache,your dnscache software isn’t run-ning. If also no "supervisednscache", probable daemon-tools problem.
"supervisednscache"there but not"/usr/local/bin/dnscache"
svc -u /var/djb/service/dnscachesleep 5ps ax | grep dnscachedig @192.168.100.2 google.com
Turn on dnscache. It might havejust been turned off.
If it now works, probably all iswell — retest.
Preceding testfailed to fixproblem, or ifno "supervisednscache"process
cd /var/djb/service/dnscache./run## Check for error messagesps ax | grep /usr/local/bin/dnscachedig @192.168.100.2 google.com
Directly run dnscache, no dae-montools, no logging.
dnscache stilldoesn’t work
ps ax | grep dnscache Check for multiple instancesof "supervise dnscache" or"/usr/local/bin/dnscache", fixif found, repeat previous step.
If you gotdnscacheworking
If you had to do the ./run com-mand, dnscache works OK, prob-able daemontools problem.
Slide 12 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 13: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/13.jpg)
tinydns Mental Model
Slide 13 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 14: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/14.jpg)
Diagnostic Tests:tinydns Alone
SITUATION TEST MEANINGIf tinydnsdidn’t work
ps ax | grep tinydns
Look for "supervise tinydns" and"/usr/local/bin/tinydns"
If no /usr/local/bin/tinydns, yourtinydns software isn’t running. If alsono "supervise tinydns", probable dae-montools problem.
"supervisetinydns"there but not"/usr/local/bin/tinydns"
svc -u /var/djb/service/tinydnssleep 5ps ax | grep tinydnsdig @127.0.0.1 wincli.domain.cxm
Turn on tinydns. It might have justbeen turned off.
If it now works, probably all is well —retest.NOTE: If dig gives question sectionbut no answer, check root/data info.
Preceding testfailed to fixproblem, or ifno "supervisetinydns" pro-cess
cd /var/djb/service/tinydns./run## Check for error messagesps ax | grep /usr/local/bin/tinydnsdig @127.0.0.1 wincli.domain.cxm
Directly run tinydns, no daemon-tools, no logging.NOTE: If dig gives question sectionbut no answer, check root/data info.
tinydns stilldoesn’t work
ps ax | grep tinydns Check for multiple instancesof "supervise tinydns" or"/usr/local/bin/tinydns", fix iffound, repeat previous step.
If you gottinydnsworking
If you had to do the ./run command,tinydns works OK, probable daemon-tools problem.
Slide 14 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 15: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/15.jpg)
Diagnostic Tests:tinydns root/data File Troubleshooting
Make and test trivial root/datacd /service/tinydns/rootmv data data.orgtouch data./add-ns domain.cxm 127.0.0.1./add-ns 100.168.192.in-addr.arpa 127.0.0.1./add-host mydesk.domain.cxm 192.168.100.2./add-host wincli.domain.cxm 192.168.100.5makesvc -t /service/tinydns; sleep 5dig @127.0.0.1 wincli.domain.cxmdig @127.0.0.1 -x 192.168.100.5
If preceding worked, troubleshoot/test data.orgcd /service/tinydns/rootmv data data.simplecp data.org data## Troubleshoot as necessarymakesvc -t /service/tinydns; sleep 5dig @127.0.0.1 wincli.domain.cxmdig @127.0.0.1 -x 192.168.100.5
Slide 15 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 16: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/16.jpg)
Diagnostic Tests:Special Cases
dnscache answers queries from 192.168.100.2 but not192.168.100.5
Add empty file /service/dnscache/root/ip/192.168.100svc -t /service/dnscache
dig @127.0.0.1 wincli.domain.cxm succeeds but [email protected] wincli.domain.cxm fails
Create following files in /service/dnscache/root/servers:domain.cxm containing 127.0.0.1100.168.192.in-addr.arpa containing 127.0.0.1
svc -t /service/dnscacheThis tells dnscache to query tinydns at 127.0.0.1 for domain.cxm
Slide 16 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 17: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/17.jpg)
daemontools Mental Model
Slide 17 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 18: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/18.jpg)
daemontools Mental ModelDirectory Structure
Slide 18 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 19: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/19.jpg)
Diagnostic Tests:Daemontools Preliminary Diagnostic Procedures
Slide 19 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 20: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/20.jpg)
Diagnostic Tests:Daemontools Final Diagnostic Procedures
Slide 20 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 21: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/21.jpg)
Summary
Success at any task depends on how you operate your brain
Organize your system knowledge as block diagrams
Don’t try to fix it, just try to narrow it down
Always ask, "How can I narrow it down one more time?"
Narrow down with tools such as
ps ax | grep whatever
ps axjf | less
dig @serverIP
./run
svc -d path or svc -u path or svc -t path
Remember the Quadruple Tradeoff and the 10 Step UniversalTroubleshooting Process
Slide 21 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 22: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/22.jpg)
Now You Can:
Name all elements of the Troubleshooter’s Mindset
Name all major components of djbdns
Narrow the root cause to a major component in a few minutes
Run any major djbdns component by itself
Understand the subcomponents of each major components
Isolate to the subcomponent in a few minutes
Solve the whole problem within an hour 99% of the time
Slide 22 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.
![Page 23: Presenter: Steve Litt · Diagnostic Tests: Quick Preliminary High Level Diagnostic Tests TEST MEANING id dnscache id tinydns id dnslog Check for valid standard djbdns accounts](https://reader036.vdocuments.us/reader036/viewer/2022081411/60a5cda73a0d8a03d43efa6c/html5/thumbnails/23.jpg)
Troubleshooters.Com
Your source for books on:
TroubleshootingRapid LearningHuman Performance
Your source for Troubleshooting courses and courseware
Home of the Universal Troubleshooting Process
http://www.troubleshooters.com
Slide 23 of 23 Copyright (C) 2011 by Steve Litt. All rights reserved.See title page for license information.
See http://www.troubleshooters.com for complete info on troubleshooting, including books and courseware.