cucm traces
Post on 27-Oct-2015
280 Views
Preview:
TRANSCRIPT
Understanding CUCM TracesCUCM Traces
AKHIL BEHLNCE, GDC UCCCIE (Voice, Security)
© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces 1
Session Objective
CUCM Traces – IntroductionSDI t SDI traces
SDL traces Explain types of tracesp yp Trace configuration Collecting trace files from CUCM Common trace file locations and collection Trace file reading and analysis Best practices for collecting and configuring Best practices for collecting and configuring
traces
2© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CUCM Traces –
Introduction
3© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CUCM Trace Introduction
CUCM has its own logging system. These are in form of txt files
These txt files are called Traces. They help in troubleshooting and are a very effective way to solve an issue
Th t fil id i d th d t il f i t th The trace files can provide in depth details of various events as they occur with time stamp, event id, trace handles (TCP handle) etc
4© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
System Diagnostic Interface (SDI) Traces
CUCM (also known as System Diagnostic Interface or SDI) traces are the user friendly call processing trace files we have available
Traces can be set at different levels to capture different activities
D t il d P id d t il d d b i f ti d hi hl Detailed : Provides detailed debug information and highly repetitive messages that are primarily used for debugging, including Keepalives and responses
Arbitrary : Provides low-level debug traces. This level is best suited for debugging difficult problems. This level includes nearly everything that is included in Detailed with the exception ofeverything that is included in Detailed with the exception of KeepAlives
5© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
Error : Provides traces generated in abnormal conditions, such as coding errors or other errors that normally should not occur
S i l P id t f ll i f ti l titi Special : Provides traces for all informational, non-repetitive messages such as process startup messages, registration messages, and so on. All system and device initialization traces are at this levelare at this level
State Transition : Provides traces for call processing events or normal events traced for the subsystem (signaling layers)normal events traced for the subsystem (signaling layers)
Significant : Provides trace for Media Layer events
SCCPSignaling
RTP
6© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
RTPVoice Stream
System Distribution Layer : SDL Traces An SDL Trace log file contains call-processing information from services such as Cisco CallManager, Cisco CTIManager, and Cisco TFTPTFTP
Its important to understand how the SDL files are named –
SDL trace filenames will be in the format: SDLxxx_yyy_zzzzzz.txt
x is the node id , y is the application id, z is the trace file number
The node id is unique to each node in the cluster and is assigned at the time the server is added to the System->Communications Manager page in CCMAdmin
100 - CallManager (ccm), 200 - CTI Manager (ctimanager) etc
SDL
7© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
SDL
SDL Trace ContentsSDL trace contents
Line Number | date + timestamp | node id | type of signal | signal name | state ofLine Number | date + timestamp | node id | type of signal | signal name | state of receiving process | receiving process | sending process | tag field | signal queue depth | additional info
Line Number starts at 000000000 when the service starts It keeps incrementingLine Number starts at 000000000 when the service starts. It keeps incrementing until the service is restarted and the value is preserved across trace files
The date/timestamp is the date and time when the trace file was written
Node id is simply a reference within each SDL trace line indicating the id of the node that wrote the trace line
The type of signal has multiple valuesThe type of signal has multiple values
Signal - the name of the signal
8© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
SDL Contents (Contd.)SDL trace file contents
Line Number | date + timestamp | node id | type of signal | signal name | state ofLine Number | date + timestamp | node id | type of signal | signal name | state of receiving process | receiving process | sending process | tag field | signal queue depth | additional info
The state of the receiving process is the state before the signal is processedThe state of the receiving process is the state before the signal is processed
The receiving process is the PID of the process recieving the SDL signal
The sending process is the PID of the process sending the SDL signalThe sending process is the PID of the process sending the SDL signal
The tag field is miscellaneous information that is not commonly used in troubleshooting
The signal queue depth outlines the number of signals waiting to be processed in each queue
The additional info field is typically the data associated with the signal being sent
9© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
The additional info field is typically the data associated with the signal being sent to the receiving process. For an error it will contain details about the error. It is valid for this field to be empty
T f SDI TTypes of SDI Traces
10© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
Callmanager : Types of Traces and their meaningg
Cisco CallManager Trace Fields Enable H245 Message Trace : This is used for H 245 signaling forEnable H245 Message Trace : This is used for H.245 signaling for
H.323 calls, including the media processing messages. Enable PRI Trace : This activates a trace of Primary Rate Interface
(PRI) devices. Enable ISDN Translation Trace : This is used for a Layer 3 trace of
Q.931 (ISDN messages). Enable H225 & Gatekeeper Trace : This is used to trace the H.225
signaling messaging for H 323 callssignaling messaging for H.323 calls. Enable Miscellaneous Trace : Activates a trace of miscellaneous
devices.E bl C f B id T U d t f t f th Enable Conference Bridge Trace : Used to perform trace of the conference bridges.
Enable CM Real-Time Information Server Trace : This is used for CallManager RIS traces - the real-time information server.
11© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CallManager RIS traces the real time information server. Enable CDR Trace : This enables tracing of call detail record (CDR)
processing.
Callmanager : Types of Traces and their meaning (Contd.)g ( ) Enable Analog Trunk Trace : This is used for a trace of all MGCP-based
devices using an analog interface.
Enable All Phone Device Trace : To get a trace of phone devices, including Cisco IP SoftPhones, and shows events such as on-hook, off-hook, key presses, and so on.
Enable MTP Trace : This trace if for media termination point devices and transcoders.
Enable All Gateway Trace : This activates a trace of all analog and digitalEnable All Gateway Trace : This activates a trace of all analog and digital gateways.
Enable Forward and Miscellaneous Trace : Used for trace of call forwarding and all subsystems not covered by another checkboxand all subsystems not covered by another checkbox.
Enable MGCP Trace : For MGCP messages for MGCP-based devices.
Enable Media Resource Manager Trace : This is trace for media resource
12© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
gmanager.
Other Traces There are other service traces as well in
addition to CUCM traces CTI traces to debug and troubleshoot CTI CTI traces – to debug and troubleshoot CTI related issues CDR – To troubleshoot CDR related issues DB and Admin To troubleshoot DB sync DB and Admin – To troubleshoot DB sync issues, MLA logging etc Performance and Monitoring – Same as windows Perfmon Security services – CAPF related traces Directory services – DirSync (LDAP trace) traces Backup and Restore – Related to DRS traces System Services – CCMWeb service, common interface etc traces
13© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
SOAP Services – AXL SOAP related traces Platform services – OS Admin interface traces
CUCM Trace
Configuration
14© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CUCM Trace Configurationg
CUCM Trace – Configuration ->
Ci U ifi d i bilit T T fi tiCisco Unified serviceability -> Trace -> Trace configuration
15© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CUCM Trace Configuration (Contd.)
16© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
Trace Collection
17© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CUCM Trace Collection
CUCM Traces from CUCM 5.x onwards are collected using RTMT
RTMT is a plugin available in CUCM plugins
Download RTMT for windows and install it on your pc
Login with CUCM admin credentials with Pub’s IP
Login with desired profile (default is CM_Default)
G t > T l > t d l t l Go to -> Tools -> trace and log central
18© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CUCM Trace Collection Procedure (Step by Step)by Step) Step 1: Turn on Traces in CallManager
Open up CallManager Admin page and navigate to Serviceability Then go toOpen up CallManager Admin page and navigate to Serviceability. Then go to Trace Configuration, select publisher and choose the CallManager Service. Make sure Trace is on, and that debug trace level is set to Detailed. We will also see a lot of options with checkboxes. Make sure that the “apply to all
d ” i l h k d d li knodes” is also checked and click save
Step 2: Download and Install RTMT (Real Time Monitoring Tool)( g )
This tool is accessed from the normal CallManager Administration page. It can be found under Application Plugins. Download and install this tool. Make sure we can run the RTMT tool and log in using the admin credentials you
t l i t th b d i Ch T l T T & Luse to log into the web admin page. Choose Tools ->Trace ->Trace & Log Central (ensure that is there’s a firewall between management PC and CUCM, the ports for RTMT are opened – 1090 and 1099 TCP )
19© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CUCM Trace Collection Procedure (Step by Step) Contd.by Step) Contd. Step 3. Place Your Test Call
Once we have RTMT loaded up we need to place your test call or create theOnce we have RTMT loaded up, we need to place your test call or create the condition we are trying to capture. Try to repeat the problem re creation twice so that, we can be sure that we can find it
Step 4. Collect Trace Files
Now go to RTMT window that we opened in step 2, and double click “Collect Files” Find CallManager in the list and click the checkbox for “All Servers”Files . Find CallManager in the list, and click the checkbox for All Servers since we may have a situation where a phone is registered to one server and the gateway is registered to another. This is good info to have beforehand so the search for the trace is a bit easier. Now for the file options, choose relative
d b k 5 10 i t M k h t h irange and go back 5 or 10 minutes. Make sure whatever we choose is sufficient to capture the test call or error condition
20© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CUCM Trace Collection
Click Collect files
21© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CUCM Trace Collection (Contd.) Choose CUCM traces (and any other traces desired)
22© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CUCM Trace Collection (Contd.) Accept the defaults for save (except delete collected log files from
server and also select Zip Files)
S l t th d t d ti f hi h t t b l d Select the date and time from which traces are to be analyzed
23© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CUCM Trace Collection (Contd.) The RTMT tool will collect the files for the selected time period
The files will be saved in selected location (default is C:\Program Fil \Ci \C llM S i bilit \JRt t\Files\Cisco\CallManager Serviceability\JRtmt\<CUCMname_CiscoCallManager_sdi_2009-06-15_12-46-57.zip>
24© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
Common Trace locations and collection
25© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
CUCM frequently used traces and file locationlocation SDI traces - /var/log/active/cm/trace/ccm/sdi
SDL traces - /var/log/active/cm/trace/ccm/sdl
Perfmon logs - /var/log/active/cm/log/ris/csv and /var/log/active/cm/log/amc/PerfMonLog
Callmanager app log /var/log/active/syslog/CallManager* * Callmanager app log - /var/log/active/syslog/CallManager*.*
Note – to get the files use the syntax - file get activelog cm/trace/ccm/sd*/*
Equivalent to CLI trace collection, in RTMT this can be done as follows –
For UCM sdl and sdi traces, check the "Cisco CallManager" box for each server
For RIS Perfon logs, check the box "Cisco RIS Data Collector PerfMonLog" for each server you are interested in
For the System Event Logs and Application logs, check the box for "Event Viewer -
26© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
For the System Event Logs and Application logs, check the box for Event Viewer System Log and Application log" for each server you are interested in
Reading CUCM Traces
27© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
Reading CallManager SDI Tracesg gHow to Read a Trace ?
Stationinit is a message from the end point to the CallManager.
StationD is the message from CallManager to endpoint.
Start finding dd=“xxx, where dd is Dialed Digits and xxx is number dialled
When you get the Dialed String, make sure to note the TCP Handle because it doesn’t changes for a Phone unless its reset.
28© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
Reading CallManager Traces (Contd.)Reading CallManager Traces (Contd.)
Now follow the tcp handle to go with flow of callNow, follow the tcp handle to go with flow of call
Alternatively (less preferred method) you can follow the Device ID y ( p ) y(Mac address) or the IP of the endpoint
If f i i l d t h dl i t d S If a conference is involved, a new tcp handle is generated. So, one now has to follow this new tcp handle till call lasts or conference lasts
The dialed string space also gives a clear picture of CSS, PSS, translation pattern hit, Route Pattern hit etc
29© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
Example Trace Filep
Trace File
30© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
An Example of CFB TraceAn Example of CFB Trace
In this trace we can see how the CFB Resources are configured on the CallManager Also we can see the error of resource unavailablethe CallManager. Also we can see the error of resource unavailable.
CallManager|UnicastBridgeControl::StationRegister MaxStream=0, ActiveStream=0|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25><CT::1,100,95,1.7687829><IP::10.7.0.25><DEV::CFB CALLMANAGER>_04/25/2002 16:41:58.948 Cisco CallManager|UnicastBridgeControl::StationRegister CurrentAvail= 0 MaxAvail=0|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25><CT::1,100,95,1.7687829><IP::10.7.0.25><DEV::CFB_CALLMANAGER>
04/25/2002 16:41:58.838 Cisco IP Voice Media Streaming App|kRequestedCFBStreamsFailed - CFB requested streams failure. Requestedfailure. Requestedstreams:48 Allocated streams:0 App ID:Cisco IP Voice Media Streaming App Cluster ID:CALLMANAGER01-Cluster NodeID:10.7.0.25|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25><CT::Alarm>
04/25/2002 16:41:34.306 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB DeviceName =CFB_CALLMANAGER|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25>04/25/2002 16:41:34.322 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB ServerName[0] =04/25/2002 16:41:34.322 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB ServerName[0] 10.7.0.25, Port =2000|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25>04/25/2002 16:41:34.338 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB Number of Parties =48|<CLID::CALLMANAGER01-Cluster><NID::10.7.0.25>04/25/2002 16:41:34.338 Cisco IP Voice Media Streaming App|CIpVMSMgr::ReadConfiguration CFB Run Flag =1|<CLID::CALLMANAGER01-Cluster><NID::10 7 0 25>
31© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
1|<CLID::CALLMANAGER01 Cluster><NID::10.7.0.25>
An Example of AC Client tracep
Call control initialization failed error message comes on ac.
1 09/07 07:55:25.953 DEBUG New trace level: DEBUG
2 09/07 07:55:25.953 INFO
3 09/07 07:55:25.953 INFO **************************************
4 09/07 07:55:25.953 INFO ****** Attendant Console Client ******
5 09/07 07:55:25.953 INFO * Version : 1.2(1 ES7)
6 09/07 07:55:25.953 INFO * Built on : Thu Nov 6 13:17:48 PST 2003
7 09/07 07:55:25.953 INFO **************************************
8 09/07 07:55:25.953 INFO
9 09/07 07:55:25.953 DEBUG Command Argument 0: C:\Program Files\Cisco\Call Manager Attendant Console\bin\ACClient
12 09/07 07:55:26.140 WARN Line state socket creation failed on /199.42.66.144:1105: java.net.BindException: Cannot assign requested address: Cannot bind
The issue was related to the XP ghost and on reinstalling afresh, th ll t l
32© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
the call control came up
An Example of TFTP Tracep
TFTP trace showing an issue with file not being found in TFTP
The phone is unable to register and keeps on rebootingThe phone is unable to register and keeps on rebooting
02/20/2008 17:34:15.144
TFTP|CTftpConnect::checkAndSetIsStatic()[0268F1F8][3976:132 143 64 20:25000]TFTP|CTftpConnect::checkAndSetIsStatic()[0268F1F8][3976:132.143.64.20:25000],
[sep00506003606c.cnf.xml] is [not a] Static File|<CLID::StandAloneCluster><NID::CM-1>
02/20/2008 17:34:15.144
TFTP|-->CTftpConnect::tftp[3976:132.143.64.20:25000] |<CLID::StandAloneCluster><NID::CM-1>
02/20/2008 17:34:15.144
TFTP|CTftpConnect::checkAndHandleDiagReq[0268F1F8],[3976:132.143.64.20:25000],
File[SEP00506003606C.cnf.xml]|<CLID::StandAloneCluster><NID::CM-1>
02/20/2008 17:34:15.144 TFTP|
CTftpConnect::tftp[3976:132.143.64.20:25000] [SEP00506003606C.cnf.xml] not found in bin/config cache Looking at disk|<CLID::StandAloneCluster><NID::CM-1>
33© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
This issue was resolved by rebuilding the CNF files from TFTP service parameters
An Example of PRI Tracep
PRI Trace showing issue with an incoming call on MGCP gateway
03/09/2010 11:26:46.940 CCM|MGCPpn9d - S0/SU1/DS1-1@FFX-BW-7-VoIP-GW - Collision Detected in PriNi2 Interface, callphase= 9|<CLID::StandAloneCluster><NID::162.70.58.2><CT::2,100,135,1.2928815> <IP::1 62.70.58.4><DEV::><LVL::State Transition><MASK::2000>
03/09/2010 11:26:46.940 CCM|Entering default,restarting | g , gchannel|<CLID::StandAloneCluster><NID::162.70.58.2><CT::2,100,135,1.2928815 > <IP::162.70.58.4><DEV::><LVL::State Transition><MASK::2000> 03/09/2010 11:26:46.941 CCM|MGCPpn9d::conflictingChannelHandler - Don't Restart this channel since it is marked OOS_NE under service parameter:S0/SU1/DS1-1@FFX-BW-7-VoIP-GW= or due to MGCP gateway hardware failure|<CLID::StandAloneCluster><NID::162.70.58.2><CT::2,100,135,1.2928815 >
This was a Hardware issue as all channels were in service
This issue was resolved by upgrading DSP firmware
34© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
Best Practices for est act ces oTrace reading
35© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
Best Practices for Trace Readingg
Always, have an Idea about the time period when issue occurs. It becomes very very difficult to go through hundreds of Trace filesbecomes very very difficult to go through hundreds of Trace files
Always Understand the Call Flow before troubleshootingy g
Information about network setup and topology always helps d t d th fl f i li d diunderstand the flow of signaling and media
Always try to save older trace files and begin afresh if debugging aAlways try to save older trace files and begin afresh if debugging a new issue
36© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
Q & A ?
37© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
Th k YThank You
38© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
39© 2010 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialAKHIL BEHL CUCM Traces
top related