Table Of Contents
VXSM Trouble Shooting
Collecting Trouble Shooting Data
Trouble Shooting Procedures
Getting Information on Current Voice Calls (H.248)
Examples
Getting Information on Current Voice Calls (xGCP)
Bearer Tracing Troubleshooting Tool
Introduction
Bearer Trace Operation
Bearer Trace Types
PCM Traces
Echo Canceller (Sout) Traces
Packet Traces
VPU Traces
T.38 Traces
Bearer Tracing Connectivity
Bearer Trace CLI Requirements
Other Considerations and Limitations
Configuring the Bearer Tracing Feature
Configuration Summary
Detailed Configuration
Examples of a Bearer Trace Configuration
Trace Files
Server File Requirements
FileNames
Troubleshooting a Bearer Trace Operation
Summary of Bearer Trace Commands
Other Useful Trouble Shooting Commands
VXSM Trouble Shooting
This chapter describes how to trouble shoot and resolve known types of problems on the Cisco MGX 8850 VXSM card. The following topics are covered in this chapter:
•
Collecting Trouble Shooting Data
•
Trouble Shooting Procedures
•
Getting Information on Current Voice Calls (H.248)
•
Getting Information on Current Voice Calls (xGCP)
•
Bearer Tracing Troubleshooting Tool
•
Other Useful Trouble Shooting Commands
Collecting Trouble Shooting Data
This section provides procedures for collecting trouble shooting data for when a VXSM card fails. These procedures use PXM45, VXSM, RPM-XF and AXSM commands. See Chapter 5, "VXSM CLI Commands" in this document for detailed descriptions of the VXSM CLI commands. Refer to the following documents for descriptions of the PXM45, RPM-XF and AXSM CLI commands:
•
Cisco MGX 8850 (PXM1E/PXM45), MGX 8950, and MGX 8830 Command Reference, Release 5.2.
•
Cisco ATM Services (AXSM) Software Configuration Guide and Command Reference for MGX Switches, Release 5.2
•
Cisco MGX Route Processor Module (RPM-XF) Installation and Configuration Guide, Release 4
Table 7-1 shows the initial steps you can take using CLI commands to trouble shoot any problem on the VXSM. These steps enable you to collect data and failure reports.
Table 7-1 Initial Trouble Shooting Steps Using CLI Commands
Run these CLI commands on the PXM45:
|
1. dspcds
|
2. dspcd
|
3. dsplog
|
4. dsperr
|
5. dspversion
|
6. dsprev
|
7. dspclksrc
|
| |
Run these CLI commands on the VXSM:
|
1. dspcd
|
2. dspversion
|
Trouble Shooting Procedures
Table 7-2 provides a list of known types of problems, their possible causes, and their possible solutions. The problems are listed alphabetically by topic.
Table 7-2 VXSM Trouble Shooting Procedures
Problem
|
Possible Cause
|
Possible Solutions
|
Access to Card— Cannot use the cc command to access VXSM card.
|
VSXM card is not in the active or standby state. If the VSXM card is not in the active or standby state, you cannot use the cc command to access the card.
|
Verify that the VXSM card is in either the active or standby state, by issuing the PXM45 CLI command, dspcds.
See Possible Solutions for "Active Card—VXSM card did not become active".
|
Active Card— VXSM card did not become active.
|
VXSM card was inserted into the wrong slot. Slot 7, 8, 15 and 16 are reserved for other service modules.
|
Remove the VXSM card. Verify that the pins on the backplane are not bent and the power key is intact. Attempt to insert the VXSM card into a slot that is not reserved.
|
PXM45 and VXSM run time images are not compatible
|
Check the latest Cisco MGX8850 Release Notes and make sure that the PXM45 and VXSM images are current and are compatible. If they are not, update the VXSM boot image and runtime image by issuing the following PXM45 CLI commands:
1. burnboot
2. setrev
|
VXSM runtime image is not available from the PXM45 hard drive or is corrupted.
|
Download the correct VXSM image to PXM45-HD. Use the PXM45 dsprevs CLI command in order to verify that the image file name and size are correct. Then, issue the clrsmcnf command and the setrev command to clear and reload the correct image.
|
The setrev command was not executed for the VXSM card.
|
Issue the PXM45 CLI command, setrev.
|
The VXSM card is inserted in a slot that was previously configured for a different card type and the slot is still in the Reserved state.
|
Insert the VXSM card into a different slot or issue the clrsmcnf CLI command.
|
Alarm— PVC is in alarm.
|
The OC3 cable is bad, does not exist, or is not connected properly.
|
1. Check for alarms on the AXSM line associated with the VXSM PVC using the dsplns and dsplnalm CLI commands.
2. On the AXSM, issue the addlnloop command with the option for local loopback set.
3. Check whether the alarm is cleared using the dspcons command.
4. If the alarm is cleared, check the TX/RX connections between the AXSM and the router's ATM interface.
5. If the TX/RX connections are correct, replace the OC3 cable.
|
The router's ATM interface is not configured.
|
On the router, issue the IOS command, show run, to verify whether the VPI/VCI and traffic parameters are correct. If necessary, reconfigure the VPI/VCI and traffic parameters.
|
The PVC has been added on the VXSM, but not on the AXSM, or vice versa.
|
Verify that the PVC has been added on both cards by issuing the dspcons command on both the VXSM and the AXSM. If necessary, add the connection, using the addcon command.
|
CA-Initiated Commands Ignored
|
An association does not exist between the CA and the VXSM.
|
Display the H.248 association state using the dsph248state CLI command (or SNMP equivalent).
|
VXSM is Rejecting Call Attempts
|
|
|
TDM/SCN termination does not exist.
|
Display the VIFs (Voice Interfaces) using the dspvifs CLI command (or SNMP equivalent). If the VIF corresponding to the TDM/SCN termination does not exist, it must be added using the addvif CLI command (or SNMP equivalent).
|
H.248 packages not associated with VIF.
|
Display the VIFs (Voice Interfaces) using the dspvifs CLI command (or SNMP equivalent). Verify that the appropriate H.248 packages have been added corresponding to the VIFs. H.248 packages can be added using cnfvifterm CLI command (or SNMP equivalent).
|
| |
|
Inadequate resources (CPU, memory, message queues, ...).
|
Display the system resources using the following CLI commands,
• dspconcacs
• dspcrr
• dspcpursc
• dspmemrsc
• dsprpcrsc
• dspmsgqrsc
|
A H.248 Add TDM termination command from the CA may be rejected if the TDM termination is already associated with a (non-null) context.
|
The CA should attempt to clean-up the orphaned TDM termination.
In the event that the CA is not able to subtract the orphaned termination, the cnfh248oos CLI command (or SNMP equivalent) can be used to forcefully subtract ALL of the terminations (and calls) on the VXSM. This should be used as a second-to-last resort.
In the event that the cnfh248oos CLI command won't subtract the termination, the card will need to be reset.
|
The CA port has not been provisioned on the VXSM.
|
The CA port can be provisioned using the addmgcgrpmgc CLI command (or SNMP equivalent).
|
Echo— Echo is present in voice call.
|
Echo cancellation (ECAN) is either not enabled or is not configured properly.
|
Ensure that the ECAN feature is enabled.
|
Fail— dspcds command output on the PXM shows VXSM card as failed.
|
VXSM card is in the initial boot process.
|
Use the PXM45 CLI dspcds command in order to monitor the boot process and card bring-up. Allow time for the boot process to complete.
|
VXSM boot flash or configuration file is corrupted.
|
Capture the output of the PXM45 dsplog -sl CLI command.Capture the console output. Issue the PXM45 CLI command clrsmcnf with the option all to clear the configuration for the slot. If the problem persists, contact Cisco Systems, Inc.
|
VXSM slot is configured for another service module
|
Physically reset the VXSM card or move it to another slot. Issue the PXM45 CLI command, clrsmcnf with the option all to clear configuration for the slot.
|
Firmware— Firmware does not download to VXSM card.
|
VXSM card is not present or is not seated properly.
|
Make sure that the VXSM card is seated properly in the slot so that the top and bottom portions of card are making electrical contact with back plane.
|
VSXM card is not in the active state or the standby state.
|
Verify that the VXSM card is in either the active or standby state, by issuing the PXM45 dspcds CLI command. Check for a card state of Boot/Init or Failed.
Perform the PXM45 dsprevs CLI command to verify that the proper firmware image resides on the PXM.
See Possible Solutions for "Active Card—VXSM card did not become active".
|
VXSM card or MGX slot is defective.
|
Attempt to insert the VXSM card into another slot. If the problem persists, contact Cisco Systems, Inc.
|
LEDs— All VXSM front panel LEDs are off.
|
Card is not seated properly in the slot.
|
Make sure that the VXSM card is seated properly in the slot so that the top and bottom portions of card are making electrical contact with back plane.
|
Mismatch— Front card, back card mismatch.
|
Configuration mismatch
|
Use the dspcds, dspcd and dsplog CLI commands to identify the configuration mismatch.
|
Mismatch— T1/E1 mismatch.
|
Configuration mismatch
|
Issue the PXM45 dspcds and dspsmcnf CLI commands to identify a configuration mismatch.
After the slot is identified, issue the PXM45 dsplog CLI command to show the card mismatch log entry.
|
Ping— Card does not respond to ping from router.
|
PVC connection is not established between VXSM and AXSM/RPM-XF.
|
Verifying that the physical connection between the AXSM and the router's ATM interface is properly connected.
Verify that the LEDs for the ATM interface and the VXSM interface are green.
Verify that the TX/RX cables are not crossed.
|
PVC connection is in fail or alarm status.
|
Verify the alarm status by issuing the PXM45 dspcons or dspalms commands.
|
The router's ATM interface is not configured.
|
From the router, issue the IOS show atm pvc and show run commands to verify that the ATM PVC status is up and that the VPI/VCI, traffic parameters and the IP address are configured properly.
|
The IP address is incorrect.
|
From VXSM and AXSM, issue the PXM45 CLI command dspcons to verify that the VPI/VCI for the PVC between the VXSM and AXSM is correct.
Issue the PXM45 dspconips CLI command and the IOS route show command to verify that the IP address and netmask for the PVC are correct.
|
An IP address has not been provisioned corresponding to the AAL5 bearer PVC.
|
Issue the VXSM addipcon CLI command.
|
Resets— Card constantly reboots-becomes active and then resets again.
|
The MGX switch is fully loaded and the power is reaching its shut off threshold.
|
Issue the PXM45 dspenvalms and dspndalms CLI commands to see if the power and temperature for the shelf and the card are within acceptable limits. If the problem persists, contact Cisco Systems, Inc.
|
VXSM card is getting too hot due to poor air circulation
|
The MGX fans are not functioning.
|
Resets— Card resets occasionally.
|
Too many calls are made while a debugging command is enabled.
|
Minimize the number of calls when debugging is enabled, reduce the trace level in order to minimize the number of trace messages or turn off debugging altogether.
|
Slot— VXSM card is in place, but the dspcds command on the PXM shows that the slot is empty.
|
The pins on the back plane are bent or a power key is missing.
|
Remove the VXSM from the slot and observe the back panel to make sure that the pins on that slot are not bent and that the power key (the orange or yellow plastic cap in the center of the slot) exists.
Insert the VXSM card into another slot.
If the problem persists, contact Cisco Systems.
|
Getting Information on Current Voice Calls (H.248)
With H.248, calls are modeled using a "context". A context contains one or more "terminations". In Release 5, a context can contain at most two terminations.In this release, there are two types of terminations: TDM (also referred to as SCN or Switched Circuit Network); and RTP.
A basic VoIP call "leg" is modeled as a context containing one TDM termination and one RTP termination.
Figure 7-1 VoIP Call Leg Model
.
A TDM hairpin call is modeled as a context containing two TDM terminations.
Figure 7-2 TDM Hairpin Model
Table 7-3 shows the commands used to get information about the current voice calls on the VXSM.
Table 7-3 How to Display Current Voice Calls on the VXSM
Step
|
CLI Commands
|
1. Display all of the H.248 terminations on the VXSM card.
|
dsph248calls
|
2. Display the status of all the H.248 contexts on the VXSM card.
|
dsph248status -cntxs
|
3. Display the status of a particular H.248 context on the VXSM card.
|
dsph248status -cntx <context ID>
|
4. Display the status of a particular H.248 TDM/SCN termination on the VXSM card.
|
dsph248status -term 1 1 <term_id>
dsph248status -term 1 2 <term_name>
dsph248status -term 1 3 <oc3> <ds1> <ds0>
|
5. Display the status of a particular H.248 RTP termination on the VXSM card.
|
dsph248status -term 3 1 <term_id>
dsph248status -term 3 2 <term_name>
|
6. Display information pertaining to all of the H.248 associations on the VXSM card.
|
dsph248status -assoc
|
7. Display information pertaining to a particular H.248 termination on the VXSM card.
|
dsph248call -term
|
8. Display counters corresponding to all of the H.248 physical (TDM/SCN) terminations on the VXSM card.
|
dsph248cnts -phyterm
|
9. Display H.248 command counters.
|
dsph248cnt -cmd <association>
|
10. Display counters pertaining to a particular H.248 physical (TDM/SCN) termination on the VXSM card.
|
dsph248cnt -phyterm 1 <bay.line.path.vtg/ds3.vt/ds1:ds0>
|
11. Display information regarding all of the H.248 contexts.
|
dsph248cnt -cntx <association>
|
12. Display counters pertaining to all H.248 ephemeral (RTP, ...) terminations on the VXSM card.
|
dsph248cnt -ephterm
|
13. Display counters pertaining to a particular H.248 association.
|
dsph248cnt -assoc <association>
|
Examples
The dsph248calls CLI command displays all of the H.248 terminations,
shelf.1.VXSM.a > dsph248calls
TermId TermType Media Codec Vad Ecan
------ -------- -------- ----- ----- ----
1 SCN 'Voice' 'G.711 U' ----- 'Enabled'
8067 RTP 'Voice' 'G.711 U' 'Enabled' -----
2 SCN 'Voice' 'G.711 U' ----- 'Enabled'
8068 RTP 'Voice' 'G.711 U' 'Enabled' -----
The dsph248status -cntxs CLI command displays the status of all H.248 contexts,
shelf.1.VXSM.a > dsph248status -cntxs
=======================================
Status for all contexts
=======================================
Number of active contexts:2
List of contexts
Status of context 4:
Creation date and time:02/03/2004, 17:31:21
Number of terminations in context:2
List of termination IDs:
8068 2
Status of context 3:
Creation date and time:02/03/2004, 17:31:21
Number of terminations in context:2
List of termination IDs:
8067 1
The dsph248status -cntx CLI command displays the status of a particular H.248 context,
shelf.1.VXSM.a > dsph248status -cntx 1
=======================================
Status of context 1
=======================================
Creation date and time: 07/04/2003, 13:55:45
Number of terminations in context: 2
List of terminations:
term_id: 8065, term_type: MG_TERM_TYPE_PDN_IP, term_name: RTP8065
term_id: 3, term_type: MG_TERM_TYPE_SCN, term_name: ds/1/1/3
The dsph248status -term CLI command displays the status of a particular H.248 termination,
shelf.1.VXSM.a > dsph248status -term 1 1 1
==========================================
Status of termination with ID 1
==========================================
Termination type:MG_TERM_TYPE_SCN
Termination name:ds/1/1/1
Termination state:MG_TERM_STATE_INSERVICE
Termination test flag:False
Termination context id:3
Id of profile on termination:0
Number of streams on termination:1
Termination about to be deleted:False
Next state of termination:MG_NEXT_STATE_INSERVICE
Last modified/updated:02/03/2004, 17:31:21
NOTE:This timestamp is not preserved across switchovers.
It is independently generated on each card.
Supported packages:
H248_GENERIC_PKG H248_TDMC_PKG
The dsph248call -term CLI command displays information pertaining to a particular H.248 termination,
shelf.1.VXSM.a > dsph248call -term 1
Term Id :1
Term Type :SCN
Term Name :ds/1/1/1
connMode :4 ('Send-recv')
loopBackType :1 ('None')
RTP Encapsulation Parameters ...
ssrc :424285698 (0x194a1602)
RTCP Enable :TRUE
dspCodec :2 ('G.711 U')
packetPeriod :'20 ms'
vadMode :2 ('Enabled')
vadThreshold :-38
vadHangoverTime :250 ms
ecan enabled :'Enabled'
ecanTail :128 ms
ecanFlags :0x0
Type <CR> to continue, Q<CR> to stop:
upspeedCodec :2 ('G.711 U')
jitterMode :2 ('Adaptive')
jitterMinDelay :5 ms
jitterMaxDelay :100 ms
jitterNomDelay :30 ms
jitterFaxNomDelay :0 ms
txGain :0
rxGain :0
nsePayloadType :100 (0x64)
nteTxPayloadType :0 (0x0)
nteRxPayloadType :0 (0x0)
profileType :0
profileNum :0
hsRedCount :0
lsRedCount :0
toneDetect :0x3c32
toneDetectBitmap :0x0
Type <CR> to continue, Q<CR> to stop:
digitDetect :1 ('DTMF')
digitRelayMethod :1 ('Send as Voice')
icsEnable :0 (FALSE)
nx64FramePattern :0 (CRML_PATTERN_NONE)
trunkMode :0 (CRML_MODE_NONE)
nx64FrameFlagCnt :0
dtmfTransport :0 (FALSE)
trunkCond :0 (FALSE)
The dsph248cnts -phyterm CLI command displays counters corresponding to all of the H.248 physical (TDM/SCN) terminations on the VXSM card,
shelf.1.VXSM.a > dsph248cnts -phyterm
=================================================================
All Gateway Physical Terminations Statistic
DS1:DS0 Termination Termination Num of Num of Num of OOS Num of OOS
line Id Name Add Failure from MGC from OAM
============ =========== ============ ====== ======= ==========
1.1.1.2.3:1 361 DS/1/16/1 1 0 0 0
1.1.1.2.3:2 362 DS/1/16/2 0 0 0 0
......
1.1.1.2.3:23 383 DS/1/16/23 0 0 0 0
1.1.1.2.3:24 384 DS/1/16/24 1 0 0 0
The dsph248cnt -ephterm CLI command displays counters corresponding to all of the H.248 ephemeral (RTP, ...) terminations on the VXSM card,
shelf.1.VXSM.a > dsph248cnt -ephterm
=========================================================
Gateway Ephemeral Terminations Statistic
=========================================================
Num of Add Commands : 2
Num of Commands Failure : 0
Getting Information on Current Voice Calls (xGCP)
The xGCP call model is based upon endpoints and connections. Connections have modes. Supported modes include Receive-Only, Send/Receive, Inactive, Loopback, and Continuity Test. The connection model is shown in Figure 7-3.
Figure 7-3 xGCP Connection Model
To obtain information regarding the status of TGCP activities. use the PXM and VXSM commands as follows/
The following PXM CLI commands will display the list of events and errors logged by the active and standby VXSM cards, starting with the most recent event or error.
Table 7-4 Useful PXM CLI Commands for Trouble Shooting the VXSM
Command
|
Description
|
dsperr -sl <primary slot>
|
Displays the error for a primary VXSM.
|
dsperr -sl <secondary slot>
|
Displays the error for a secondary VXSM.
|
dsplog -sl <primary slot>
|
Displays the log for a primary VXSM.
|
dsplog -sl <secondary slot>
|
Displays the log for a secondary VXSM.
|
The following VXSM CLI commands are useful to monitor the basic operation of the currently active VXSM card and to check if calls are being processed after switchover:
Table 7-5 Useful VXSM CLI Commands for Trouble Shooting the VXSM
Command
|
Description
|
dspxgcpcnts
|
Displays xgcp command counts.
|
dspxgcpdetailcnts
|
Displays xgcp details.
|
dspxgcpendpts
|
Displays xgcp endpoints.
|
dspxgcpendptcons
|
Dispaly xgcp endpoint connections
|
dspxgcpcnts - displays a static summary of the number of TGCP commands that failed/succeeded on the ACTV card at that moment; executing this command a few times will give the user an idea whether commands are being sent to the VXSM and whether VXSM is processing those commands
dspxgcpdetailcnts - displays number of times a particular TGCP command is received/sent/retransmitted
dspxgcpendpts <endpt-name> - displays all the endpt state, if connection is present on that endpoint.
`dspxgcpendpts *' can be used to display all
-dspxgcpendptcons <endpt-name> - displays the connection id, call id of the connection on that endpoint
Bearer Tracing Troubleshooting Tool
Introduction
The VXSM card includes a useful troubleshooting tool that has the ability to collect fundamental information about the content of bearer streams and to transmit the information in real-time to an external file server for offline analysis. This feature, known as Bearer Tracing, involves the VXSM card, the Media Gateway Controller for VoIP applications, and a file server to receive the bearer traces (see Figure 7-4).
Figure 7-4 Bearer Tracing Major Elements
Within the VXSM card, the Bearer Tracing feature is performed in the Voice Module as shown in Figure 7-5.
Figure 7-5 VXSM Bearer Tracing High Level Functional Blocks
The Voice Module contains a number of probes on the card that are situated at various points in the path of the bearer stream through the card.
There are eight such probes situated as shown in Figure 7-6.
Figure 7-6 Bearer Trace Probe Locations
Note
The numbering used to identify the probes in the diagram above is the same numbering that the VXSM Bearer Tracing command use to identify the probes
The probes are:
1 = PCM Input from TDM network
2 = PCM Output to the TDM network.
3 = Input from the packet network (not operational in VXSM Release 5.2)
4 = Output to the packet network (not operational in VXSM Release 5.2)
5 = Output from echo canceller (Ecan/Sout)
6 = Voice Playout Unit - Jitter buffer
7 = T.38 information at the Decoder - level 1
Bearer Trace Operation
Step 1
To be able to collect bearer traces on a endpoint, VXSM needs to:
a.
Have an active bearer channel
In VOIP: A call needs to be setup
or
In AAL2, an AAL2 nailed up connection needs to be added by adding a CID
And
b.
Be connected to a computer where the trace files are to be stored, referred to herein as a "file server, server or network"
Note
In the case where a call agent is used, the same machine can be a call agent and a file server at the same time
Step 2
To start collecting bearer traces
a.
If bearer tracing is setup and activated on an endpoint that has an active bearer channel, tracing starts immediately, otherwise, it will start as soon as a call (VOIP) or a connection (AAL2) is activated on the endpoint.
b.
If bearer tracing is setup and non-activated on an endpoint, tracing will have to be initiated manually even if there is an active call/connection on the endpoint
Step 3
To stop collecting bearer traces:
a.
End the active call/connection on the endpoint, or,
b.
Manually stop the trace
Note
As long as the trace is not stopped, VXSM will continue to send trace samples to the file server
Step 4
VXSM generates a trace filename each time a trace is started and active on an endpoint. The filename is based on the probe type, the endpoint index and on specific user input that constitutes the file signature.
Note
The file signature could be set to remain the same from one trace session to another or, in some cases; it could be set to be specific to each session, (see Trace Files).
If the connection to the file server can't be established, bearer tracing will be deactivated and the VXSM card may not function correctly. To reactivate bearer tracing, fix the network issues that are preventing the network connection to be established.
Step 5
All trace probes can be activated on the same endpoint. Each probe initiates its own trace stream to the network and will write its data to its own file except for VPU traces where VPU event traces and VPU segment traces are written to the same file.
Step 6
The level 1 and level 2 T.38 probes (7 and 8) can't be activated at the same time. Also these probes also need PCM trace to be on.
Step 7
The total number of probes activated at the same time is limited but the amount of data transferred per probe isn't.
Note
a. There needs to be enough free space on the file server to store the trace files generated by VXSM. For and estimation of the required space per trace, see Per Trace Bandwidth and File Size Requirements
b. If the network connection experiences problems while bearer tracing is writing trace data into the trace file, bearer tracing may be deactivated and the VXSM card may not function correctly. To reactivate bearer tracing, fix the network issues that are preventing the network connection to be established
Bearer Trace Types
Bearer Traces are transmitted to the server as files with file names that indicate the probe type, the endpoint being probed and, optionally, a timestamp. Files are assigned names by the Bearer Tracing feature that indicate these items. For more details on file naming see Trace Files.
PCM Traces
There are 2 PCM traces, pcmin and pcmout, to and from the voice network, respectively. In CLI, pcmin Probe 1 collects pcmin and probe 2 collects pcmout. The pcmin trace files generated by VXSM have the pcmin prefix and.pcm suffix, pcmout trace file have the pcmout prefix and the .pcm suffix.
The format for a PCM trace is either 8-bit PCM A-law or 8-bit PCM mu-law, depending on the physical interface being used. VXSM currently ties the PCM encoding to the physical interface.
Echo Canceller (Sout) Traces
The Sout trace is collected from the "output" of the echo canceller and is collected by probe 5.
The format of the Sout stream is a big-endian signed 16-bit linear format. The value 32757 represents full scale. The sout trace file generated by VXSM has the sout prefix and .pcm suffix
The Sout stream is synchronized to the PCM In and PCM Out signals.
The Sout tracing cannot be enabled if the Ecan is not enabled on the given connection
Packet Traces
The packet traces are collected at the input and output of the Packet IO interface.
The current implementation focuses on RTP traces. The packet trace is in raw IP pcap format that is readable by packet sniffers such as Ethereal.
VPU Traces
The VPU trace is on Probe 6. When a VPU segment trace is started, an event trace is started automatically.
The Voice Playout Unit (Jitter Buffer) Segment trace (vpseg) reports jitter-related data every segment, that is every 5ms.
The VPU Event trace (vpevt) reports jitter-related events when they happen.
Each type of VPU trace produces it own trace file. Segment files have the suffix .seg and Event files have the suffix .evt.
T.38 Traces
When a fax call is active using T.38 fax relay, traces can be gathered from the Fax module within the DSP. The user can select to gather 2 levels of traces:
Level 1 - basic events to decipher T.30 and other signaling progress
Level 2 - proprietary format
When enabling the tracing functionality for T.38 it is possible to enable tracing for an entire DS1 or a specific DS0. When monitoring a DS1 the first fax relay session activated on the monitored span will have tracing enabled. Enabling a trace for an entire DS1 is specified by a special probe in the addbearertraceendpt command.
A total of 10 simultaneous DS1 spans can be monitored at any one time. In addition to this a maximum number of 10 trace sessions can be obtained simultaneously.
Bearer Tracing Connectivity
External Interface
VXSM can be connected to the file server either through the Control PVC or through the VXSM Ethernet port.
If the control PVC is used, the provisioning of the control PCV needs to include the bandwidth required for the transfer of trace data to the network.
Per Trace Bandwidth and File Size Requirements
Bearer tracing requires that enough bandwidth be available to accommodate the bearer trace traffic data and the transport protocol packet and control overhead. Table 7-6 and Table 7-7 below show the estimate per-trace bandwidth requirements in the case of ftp and tftp. The file size of the accumulated bearer trace data on the server is also provided so that enough space is set aside for the trace files to be stored in the ftp server. The file size of the trace files is limited only by the free space on the file server. To make sure that trace files have no loss of data, provide enough space on the file server for the trace files.
Table 7-6 Bandwidth and File Sizes for FTP
Probe Type
|
AAL5 Bandwidth Per Trace in cells/s
|
Ethernet Bandwith in Bytes/s
|
File Size for 5 min of tracing in Bytes
|
PCM In, PCM Out, Sout
|
200
|
9000
|
2400000
|
Pkt in , Pkt out
|
250
|
11000
|
3600000
|
VPU Segment
|
100
|
3000
|
920000
|
Sout
|
400
|
18000
|
4800000
|
Table 7-7 Bandwidth and File Sizes for TFTP
Probe Type
|
AAL5 Bandwidth Per Trace in cells/s
|
Ethernet Bandwith in Bytes/s
|
File Size for 5 min of tracing in Bytes
|
PCM In, PCM Out, Sout
|
210
|
9500
|
2400000
|
VPU Segment
|
90
|
4300
|
920000
|
Sout
|
420
|
19000
|
4800000
|
Bearer Tracing for AAL2 connections
FTP and TFTP protocols run on an IP network. For Voice over IP, VXSM inherently uses an AAL5 control PVC to communicate with the MGC. The same control PVC can be used to communicate with the external FTP/TFTP server.
Control PVC isn't inherent in an AAL2 configuration (example: AAL2 trunking). To access the IP cloud in such a situation, either a control PVC or the Ethernet console need to be configured. Unless pinging the file server is possible, tracing will not operate.
IPSec and Bearer Tracing
The Bearer Tracing mechanism uses FTP/TFTP connections to transfer the traces to the server. The trace data are carried over the control PVC, which is also used to communicate with MGC. For many solutions, IPSec is used to secure the communication with the MGC over control PVC. In such cases, the FTP/TFTP traffic is bypassed by IPSec in order to transfer trace data to the bearer tracing server.
Bearer Trace CLI Requirements
Persistency
The operational state of bearer tracing is not saved in disk database. Hence, all the information is lost if the card resets. All the bearer tracing commands are non-MIB commands.
Backward Compatibility
Commands introduced in the PCM tracing functionality in the earlier VXSM releases are no longer functional. They are replaced by the new bearer tracing commands in VXSM Release 5.3.
Bearer Trace Configuration
Once a bearer trace or a server profile is added, the only way to change its configuration is by deleting the trace or the server profile and adding it again with the new configuration.
Note
NOTE 1: For bearer tracing to occur, there needs to be a connection between VXSM and the file server, an ftp/tftp server need to be running on the file server and access permission to the directory (and in the case of tftp, to the files) need been granted.
Note
Prior to deleting either configuration, make sure that the trace is stopped
Other Considerations and Limitations
1.
A VXSM card supports up to 10 simultaneous trace sessions.
2.
Only one trace per DSP core is possible, consequently, adding tracing on an endpoint will fail if the dsp core is found to have tracing enabled for a different endpoint
3.
Network latencies beyond 20 seconds have an impact on the quality of traces. To avoid any loss of data during the trace collection, it is preferable to have the file server on the same LAN segment as VXSM
4.
Bearer tracing requires about 1% of CPU utilization so, in order for tracing to operate properly, only enable the number of traces that the system's cpu utilization can actually handle. Also, under a heavy call-rate, there might not be enough CPU bandwidth to accommodate the maximum number of traces.
5.
All traces have to be setup at the same time as there is no provision to setup each one individually.
Configuring the Bearer Tracing Feature
Configuration Summary
The following steps provide a quick summary of the procedure for configuring the Bearer Tracing feature.
Step 1
In a typical network scenario, identify the ds0 that needs to be monitored for bearer trace collection. Use addbearertraceendpt to define the endpoint and the trace probes that need to be enabled.
Step 2
Use addbearertracesrvprof to add TFTP or FTP server profile.
Step 3
Use addbearertrace to tie the endpoint to the defined server profile.
Step 4
Make sure that the necessary files are created on the server with correct write permissions if TFTP transfer mode is to be used.
Step 5
Use bearertracestart to enable the tracing on the given endpoint if it is not already enabled using addbearertrace or simply start a call on the endpoint
Step 6
Use bearertracestop to stop the tracing on the given endpoint, or simply stop the call in a VOIP configuration or delete the connection in an AAL2 configuration
Detailed Configuration
To setup bearer tracing, use the following procedure.
Step 1
Setup a connection to an external server to receive the trace information. See Bearer Tracing Connectivity for connectivity details
Step 2
Use the addbearertraceendpt command create the TDM segment of a bearer trace session.
This command is used to create a session with a specified DS0 or DS1 endpoint for bearer tracing. See T.38 Traces for T.38 endpoints.
The format of this command is:
addbearertraceendpt <endptIndex> <LineNum> <ds0> -trace <probe>
For <endptIndex> enter a number in the range 1 to 10. This parameter uniquely identifies the bearer trace session.
For <LineNum>, enter line number of the endpoint in one of the following formats.
Interface Type Line Number Format Values
OC3/SDH bay.line.path.vtg.vt bay {1 - upper}
or bay.line.path.ds1 line (range=1..4)
path (range=1..3)
vtg (range=1..7)
vt (range=1..4)(ds1)
(range=1..3)(e1)
ds1 (range=1..28)
T1/E1 bay.line bay {1 - upper}
line (range=1..24)
T3 bay.line.ds1 bay {1 - upper}
line (range=1..6)
ds1 (range=1..28)
For <ds0>, enter:
A number in the range 1 to 24 or "all" for a T1 interface
A number in the range 1 to 32 or "all" for an E1 interface
For -trace<probe>, enter the probe number(s) to be used in the session. Enter either a single number or multiple numbers, for example, 1,3,5,7)
1 = PCM Input
2 = PCM Output
4 = Network Input
5 = Network Output
3 = Sout (echo canceller output)
6 = VPU - Events
7 = T38 Level 2
8 = T38 Level 1
The line number "all" option can be used only when one of the probes 7 and 8 is chosen.
The creation of the session can be verified by using the dspbearertraceendpt command. The format of this command is:
dspbearertraceendpt <endptIndex>
Step 3
Use the addbearertracesrvprof command to create the network segment of a bearer trace session.
The syntax of this command is:
addbearertracesrvprof <profIndex> <srvIP> <xferMode> <fileNameBase>[<uploadPath> <portNumber/login><password>]
For <profIndex> enter a number in the range 1 to 10. This parameter uniquely identifies the bearer trace server profile.
For <srvIP>, enter the IP address of the server to receive the bearer trace reports. The IP address can be provided either using the regular "IP dot-notation" (nnn.nnn.nnn.nnn) or a name (alpha.beta.com) with a proper DNS support
For <xferMode>, specify the transfer mode for communicating with the server. Enter 0 for FTP, or 1 for TFTP
For <fileNameBase>, specify a fileNameBase. For FTP, the user can also enter "#" to indicate the default fileNameBase is to be used (default is not allowed for TFTP).
VXSM will generate filenames as follows:
Table 7-8 Generated Bearer Trace Filenames
xferMode
|
fileNameBase
|
Filenames
|
1 = TFTP
|
FILENAMEBASE specified by user
|
PCM In - pcminFILENAMEBASE_EndPtID.pcm PCM Out - pcmoutFILENAMEBASE_ EndPtID.pcm Sout - soutFILENAMEBASE_ EndPtID.pcm VPU - vpuFILENAMEBASE_ EndPtID.seg (or evt) Network In - netwrxFILENAMEBASE_ EndPtID.pkt Network Out - netwtxFILENAMEBASE_ EndPtID.pkt T.38 level 1 - t38lvl1FILENAMEBASE_ EndPtID.t38 T.38 level 2 - t38lvl2FILENAMEBASE_ EndPtID.t38 MSG - msgFILENAMEBASE_EndPtID.msg (to and from DSP)
|
1 = TFTP
|
# for default
|
Not allowed
|
2 = FTP
|
FILENAMEBASE specified by user
|
PCM In - pcminFILENAMEBASE_EndPtID.pcm PCM Out - pcmoutFILENAMEBASE_ EndPtID.pcm Sout - soutFILENAMEBASE_ EndPtID.pcm VPU - vpuFILENAMEBASE_ EndPtID.seg (or evt) Network In - netwrxFILENAMEBASE_ EndPtID.pkt Network Out - netwtxFILENAMEBASE_ EndPtID.pkt T.38 level 1 - t38lvl1FILENAMEBASE_ EndPtID.t38 T.38 level 2 - t38lvl2FILENAMEBASE_ EndPtID.t38 MSG - msgFILENAMEBASE_EndPtID.msg (to and from DSP)
|
2 = FTP
|
# for default
|
PCM In - pcminMMDDYYHHMMSS_EndPtID.pcm PCM Out - pcmoutMMDDYYHHMMSS_ EndPtID.pcm Sout - soutMMDDYYHHMMSS_ EndPtID.pcm VPU - vpuMMDDYYHHMMSS_ EndPtID.seg (or evt) Network In - netwrxMMDDYYHHMMSS_ EndPtID.pkt Network Out - netwtxMMDDYYHHMMSS_ EndPtID.pkt T.38 level 1 - t38lvl1MMDDYYHHMMSS_ EndPtID.t38 T.38 level 2 - t38lvl2MMDDYYHHMMSS_ EndPtID.t38 MSG - msgMMDDYYHHMMSS_EndPtID.msg (to and from DSP)
|
For <uploadPath>, specify the path of the file in the server. For example. /pcm/tes
Note
The Server root directory will be used if the path is not specified)
For <portNumber/login>, enter a TFTP port number if xferMode = 1 or the FTP login name If xferMode = 0.
The TFTP port number is a number in the range 1 to 255 with a default of 69.
For login, use the FTP login name, default is "guest".
For <password>, use the FTP password, default is "guest".
The creation of the profile can be verified by using the dspbearertracesrvprof command. The format of this command is:
dspbearertracesrvprof <profIndex>
Step 4
Use the addbearertrace command to link the TDM segment and the network segment of the bearer trace session. Optionally, this command can also start the bearer trace function.
The format of the addbearertrace command is:
addbearertrace <endptIndex> <profIndex> <active>
For <endptIndex>, enter the endpoint index number of the bearer trace session. A number in the range of 1 to 10.
For <profIndex>, enter the profile number of the bearer trace profile. A number in the range of 1 to10.
For <active>, specify 1 for enable or 0 for disable. If this parameter is omitted, the default is enable.
Note
If the bearer tracing is enabled, tracing will start automatically whenever a call is added on the given bearer endpoint. If the bearer tracing is not enabled, then it needs to be explicitly started using bearertracestart command (there is also a corresponding bearertracestop command).
The creation of the association can be verified by using the dspbearertrace command. The format of this command is:
dspbearertrace<endptIndex>
Examples of a Bearer Trace Configuration
Example 1.
This example procedure sets up the bearer tracing endpoint, the ftp/tftp server, and a method of mapping them to each other before making and tracing a call. The call is on ds0=1 on ds1=1.
Configuring the bearer tracing endpoint and the servers:
Step 1
First configure the bearer trace endpoint with appropriate trace probes.
martler.2.VXSM.a > addbearertraceendpt 1 1.1.1.1.1 1 -trace 1,2,5
Step 2
Then configure the bearer trace server with appropriate parameters.
martler.2.VXSM.a > addbearertracesrvprof 1 172.17.38.159 0 sarang / root password
Step 3
Map the bearer trace endpoint with the bearer trace server
martler.2.VXSM.a > addbearertrace 1 1 1
Step 4
Verify that the mapping is successful
martler.2.VXSM.a > dspbearertraceendpts
=============================================
BEARER TRACE ENDPT PROFILES
=============================================
bearerTraceSrvProf : 0x1 <---- Server Profile is added
Step 5
Verify that the parameters for bearer trace server are correct
martler.2.VXSM.a > dspbearertracesrvprofs
=============================================
BEARER TRACE SERVER PROFILES
=============================================
Bearer Trace Server IP : 172.17.38.159
Server upload directory : /
FTP server password : password
---------------------------------------------
Step 6
Make a call on ds0=1 and ds1=1, and then delete it after some time. On the FTP server the following files are created.
-rw-r--r-- 1 root other 690480 Oct 16 12:17 pcminsarang_1.pcm
-rw-r--r-- 1 root other 690480 Oct 16 12:17 pcmoutsarang_1.pcm
-rw-r--r-- 1 root other 1380960 Oct 16 12:17 soutsarang_1.pcm
Example 2
The following example collects PCM traces for a call with both
endpoints on the same card:
Step 1
Configure the bearer tracing endpoint for PCMin and PCMout traces
addbearertraceendpt 1 1.1.1.1.1 1 -trace 1,2
addbearertraceendpt 2 1.1.1.1.1 2 -trace 1,2
Step 2
Add a Bearer Trace File Server
addbearertracesrvprof 1 172.17.38.159 0 rxie /tftpboot root password
Step 3
Map the bearer endpt to the server profile
addbearertrace 1 1 1
addbearertrace 2 1 1
When you make a call on endpoints 1 & 2 the PCM tracing will start automatically because of the last parameter in addbearertrace commsnd above.
However, If you use a different last parameter as in the following commands,
addbearertrace 1 1 0
addbearertrace 2 1 0
Tracing does not start automatically and the following commands are needed to start the tracing after the call is up:
bearertracestart 1
bearertracestart 2
Step 4
The following CLIs are needed to stop bearer tracing at any time :
bearertracestop 1
bearertracestop 2
Trace Files
Server File Requirements
1.
Files corresponding to the trace files that will be generated by VXSM need to be created on the file server prior to enabling bearer tracing.
2.
Write access to both the created traces files and to the directory where they reside need to be set.
3.
If a directory is specified when adding a server profile on VXSM, this directory is relative to the default tftp directory specified in the tftp config file (in unix, the default is set to /tftpboot in /etc/inet.conf .
FileNames
Bearer trace information is saved to the server as data chunks that will be stored in a file of which the name is generated by VXSM. The file naming scheme is slightly different depending upon whether TFTP or FTP are used as the transport protocol.
TFTP Filenames
Filenames used with TFTP consist of the following parts:
•
A probe type
•
A FILENAMEBASE
•
The endpoint index being probed
•
A file extension
For example, a trace for PCM In on a bearertrace endpoint of index 1 and with a FilenameBase Mytrace would yield a filename of:
pcminMytrace_1.pcm
File names corresponding to trace files must be created on the server prior to enabling the bearer trace function. Further, VXSM must have write access to the trace files and the directory where they reside. The path to the file is relative to the default tftp directory (usually set as /tftpboot)
Note
When using the FILENAMEBASE, subsequent traces for the same probe and same endpoint will overwrite the previous file in the server. It is the responsibility of the server to take action to prevent this occurrence.
FTP Filenames
For systems using FTP, the use of a FILENAMEBASE is optional. If a FILENAMEBASE is specified, the same file naming scheme used for TFTP is used (see previous paragraphs). If a FILENAMEBASE is not specified, VXSM will generate a filename having the following parts:
•
A probe type
•
A timestamp MMDDYYHHMMSS based upon a 24 hour clock)
•
The endpoint index being probed
•
A file extension
For example, a trace for PCM In on a bearertrace endpoint of index 1 would yield a filename of:
pcmin111205104520_1.pcm
Note
The filename method using timestamps automatically prevents the overwriting of previous trace files.
Note
Make sure there is enough space on the server to store all the different trace files for the same probe. The files can get very large if tracing is on for a long time.
Troubleshooting a Bearer Trace Operation
A number of issues can cause the trace to lose samples. In the bearer tracing infrastructure, the user needs to define the bearer tracing endpoints (ie: ds0) on which bearer traces are to be collected. The user needs to define bearer trace server where the bearer traces will be stored using FTP or TFTP. Any given bearer trace endpoint needs to be mapped to a bearer trace server, so that the bearer traces for that endpoint are stored at the mapped bearer trace server. The user can specify which directory the bearer traces can be stored and the base for the file names in which the traces are to be stored.
The user MUST make sure the following:
1.
The bearer trace server can be "ping"ed from the VXSM card
2.
The VXSM control PVC IP address can be "ping"ed from the bearer trace server
3.
If there is no control PVC (example: AAL2 trunking), the Ethernet console connection should be used. In any case, step 1 should pass
4.
The bearer trace server supports FTP and/or TFTP protocol
5.
In case of TFTP protocol, appropriate files are created with correct "rwx" permissions
6.
If the bearer trace transfer takes place through control PVC, the control PVC should have sufficient bandwidth allocated
Table 7-9 Bearer Trace Troubleshooting
Problem
|
Possible Cause
|
Potential Solution
|
Can't add tracing on an endpoint
|
Tracing already enabled on the same DSP
|
Remove existing trace or enable tracing on a different endpoint
|
Can't enabled a fax trace
|
Max number of fax session exceeded
|
Remove an existing trace before adding a new one
|
Trace files don't increase in size
|
Tracing might have been turn off because of network issues
|
Check your network connection and restart the tracing
|
Summary of Bearer Trace Commands
The commands for the Bearer Trace feature are listed below. For detailed descriptions of these commands, please refer to the "Cisco Voice Switch Services (VXSM) Configuration Guide for MGX Switches and Media Gateways, Release 5.2"
addbearertraceendpt <endptIndex> <LineNum