Table Of Contents
Troubleshooting Tools and Methodology
Command-Line Interface Troubleshooting Commands
Using the show processes CLI Command
Using the show processes cpu CLI Command
Using the show system resource CLI Command
Enabling Logging for Telnet or SSH
Troubleshooting Tools and Methodology
This appendix describes the troubleshooting tools and methodology available for the Cisco NX-OS and includes the following sections:
•
Command-Line Interface Troubleshooting Commands
•
•
•
Command-Line Interface Troubleshooting Commands
The command-line interface (CLI) allows you to configure and monitor Cisco NX-OS using a local console or remotely using a Telnet or SSH session. The CLI provides a command structure similar to Cisco IOS software, with context-sensitive help, show commands, multiuser support, and roles-based access control.
Each feature has show commands that provide information about the feature configuration, status, and performance. Additionally, you can use the following commands for more information:
•
switch# copy running-config startup-config[########################################] 100%2008 Jan 16 09:59:29 zoom %$ VDC-1 %$ %BOOTVAR-2-AUTOCOPY_FAILED: Autocopy of file /bootflash/n7000-s1-dk9.4.0.0.837.bin.S8 to standby failed, error=0x401e0008switch# show system error-id 0x401e0008Error Facility: sysmgrError Description: request was aborted, standby disk may be full•
Configuration Files
Configuration files contain the Cisco NX-OS commands used to configure the features on a Cisco NX-OS device. Cisco NX-OS has two types of configuration files, running configuration and startup configuration. The device uses the startup configuration (startup-config) during device startup to configure the software features. The running configuration (running-config) contains the current changes that you make to the startup-configuration file. You should create a backup version of your configuration files before modifying that configuration. You can either back up the configuration files to a remote server (see configuration file information in the Cisco NX-OS Fundamentals Configuration Guide, Release 4.0 ) or create a checkpoint copy of the configuration file that you can roll back to if problems occur (see the Rollback feature in the Cisco NX-OS System Management Configuration Guide, Release 4.0).
Cisco NX-OS features can create internal locks on the startup configuration file. In rare cases, these locks may not be removed by the features. Use the show system internal sysmgr startup-config locks command to determine if any locks remain on the startup configuration file. Use the system startup-config unlock command to remove these locks.
CLI Debug
Cisco NX-OS supports an extensive debugging feature set for actively troubleshooting a network. Using the CLI, you can enable debugging modes for each feature and view a real-time updated activity log of the control protocol exchanges. Each log entry has a timestamp and is listed chronologically. You can limit access to the debug feature through the CLI roles mechanism to partition access on a per-role basis. While the debug commands show real-time information, you can use the show commands to list historical and real-time information.
Caution
Note
By using the ? option, you can see the options that are available for any feature. A log entry is created for each entered command in addition to the actual debug output. The debug output shows a time-stamped account of the activity that occurred between the local device and other adjacent devices.
You can use the debug facility to track events, internal messages, and protocol errors. However, you should be careful when using the debug utility in a production environment, because some options may prevent access to the device by generating too many messages to the console or creating CPU-intensive events that could seriously affect performance.
Note
Note
Ping and Traceroute
Note
Ping and traceroute are two of the most useful tools for troubleshooting TCP/IP networking problems. The ping utility generates a series of echo packets to a destination across a TCP/IP internetwork. When the echo packets arrive at the destination, they are rerouted and sent back to the source.
The traceroute utility operates in a similar fashion but can also determine the specific path that a frame takes to its destination on a hop-by-hop basis.
This section includes the following topics:
Using Ping
Use the ping command to verify connectivity and latency to a particular destination across an IPv4 routed network.
Use the ping6 command to verify connectivity and latency to a particular destination across an IPv6 routed network.
The ping utility allows you to send a short message to a port or end device. By specifying the IPv4 or IPv6 address, you can send a series of frames to a target destination. Once these frames reach the target, they are looped back to the source and a time-stamp is taken.
Using Traceroute
Use traceroute to:
•
•
The traceroute utility identifies the path taken on a hop-by-hop basis and includes a timestamp at each hop in both directions. You can use traceroute to test the connectivity of ports along the path between the generating switch and the switch closest to the destination.
Use the traceroute command for IPv4 networks. Use the traceroute6 command for IPv6 networks.
If the destination cannot be reached, the path discovery starts, which traces the path up to the point of failure.
Monitoring Processes and CPUs
This section includes the following topics:
•
•
•
Using the show processes CLI Command
Use the show processes command to identify the processes that are running and the status of each process (see Example B-1). The command output includes the following:
•
•
•
•
•
•
Process states are as follows:
•
•
•
•
•
•
•
Note
Example B-1 show processes Command
switch# show processes ?cpu Show processes CPU Infolog Show information about process logsmemory Show processes Memory Infoswitch# show processesPID State PC Start_cnt TTY Process----- ----- -------- ----------- ---- -------------1 S b7f9e468 1 - init2 S 0 1 - migration/03 S 0 1 - ksoftirqd/04 S 0 1 - desched/05 S 0 1 - migration/16 S 0 1 - ksoftirqd/17 S 0 1 - desched/18 S 0 1 - events/09 S 0 1 - events/110 S 0 1 - khelper15 S 0 1 - kthread24 S 0 1 - kacpid101 S 0 1 - kblockd/0102 S 0 1 - kblockd/1115 S 0 1 - khubd191 S 0 1 - pdflush192 S 0 1 - pdflushn...Using the show processes cpu CLI Command
Use the show processes cpu command to display CPU utilization (see Example B-2). The command output includes the following:
•
•
•
•
Example B-2 show processes cpu Command
switch# show processes cpuPID Runtime(ms) Invoked uSecs 1Sec Process----- ----------- -------- ----- ----- -----------1 922 4294967295 0 0 init2 580 377810 1 0 migration/03 889 3156260 0 0 ksoftirqd/04 1648 532020 3 0 desched/05 400 150060 2 0 migration/16 1929 2882820 0 0 ksoftirqd/17 1269 183010 6 0 desched/18 2520 47589180 0 0 events/09 1730 2874470 0 0 events/110 64 158960 0 0 khelper15 0 106970 0 0 kthread24 0 12870 0 0 kacpid101 62 3737520 0 0 kblockd/0102 82 3806840 0 0 kblockd/1115 0 67290 0 0 khubd191 0 5810 0 0 pdflush192 983 4141020 0 0 pdflush194 0 5700 0 0 aio/0193 0 8890 0 0 kswapd0195 0 5750 0 0 aio/1...Using the show system resource CLI Command
Use the show system resources command to display system-related CPU and memory statistics (see Example B-3). The output includes the following:
•
•
•
•
Example B-3 show system resources Command
switch# show system resourcesLoad average: 1 minute: 0.30 5 minutes: 0.34 15 minutes: 0.28Processes : 606 total, 2 runningCPU states : 0.0% user, 0.0% kernel, 100.0% idleMemory usage: 2063268K total, 1725944K used, 337324K free2420K buffers, 857644K cacheUsing Onboard Failure Logging
Cisco NX-OS provides the facility to log failure data to the persistent storage, which can be retrieved and displayed for analysis. This onboard failure logging (OBFL) feature stores failure and environmental information in nonvolatile memory on the module. This information will help analysis failed modules.
The data stored by the OBFL facility includes the following:
•
•
•
•
•
•
•
•
•
•
•
•
•
•
For more information about configuring OBFL, see the Cisco NX-OS System Management Configuration Guide, Release 4.0.
Using GOLD Diagnostics
Generic Online Diagnostics (GOLD) defines a common framework for diagnostic operations across Cisco platforms. The GOLD implementation checks the health of hardware components and verifies proper operation of the system data and control planes. Some tests take effect when the system is booting up, other tests take effect when the system is operational.
A booting module goes through a series of checks before coming online to allow the system to detect faults in the hardware components at boot-up and to ensure that a failing module is not introduced in a live network.
Defects are also diagnosed during system operation or runtime. You can configure a series of diagnostic checks to determine the condition of an online system. You must distinguish between disruptive and nondisruptive diagnostics tests. Although nondisruptive tests occur in the background and do not affect the system data or control planes, disruptive tests do affect live packet flows. You should schedule disruptive tests during special maintenance windows. The show diagnostic content module CLI output displays test attributes such as disruptive or nondisruptive tests.
You can configure runtime diagnostics checks to run at a specific time or to run continually in the background.
Health-monitoring diagnostic tests are nondisruptive, and they run in the background while the system is in operation. The role of online diagnostic health monitoring is to proactively detect hardware failures in the live network environment and inform you of a failure.
GOLD collects diagnostic results and detailed statistics for all tests including the last execution time, the first and last test pass time, the first and last test failure time, the total run count, the total failure count, the consecutive failure count, and the error code. These test results help administrators determine the condition of a system and understand the reason for a system failure. Use the show diagnostic result command to view diagnostics results.
For more information about configuring GOLD, see the Cisco NX-OS System Management Configuration Guide, Release 4.0.
Using Embedded Event Manager
Embedded Event Manager (EEM) is a policy-based framework that allows you to monitor key system events and then act on those events through a set policy. The policy is a preprogrammed script you can loaded that defines actions that the switch should invoke based on set events occurring. The script can generate actions, including, but not limited to, generating custom syslog or SNMP traps, invoking CLI commands, forcing a failover, and much more.
For more information about configuring EEM, see to the Cisco NX-OS System Management Configuration Guide, Release 4.0.
Using Ethanalyzer
Ethanalyzer is a Cisco NX-OS protocol analyzer tool based on the Wireshark (formerly Ethereal) open source code. Ethanalyzer is a command-line version of Wireshark that captures and decodes packets. You can use Ethanalyzer to troubleshoot your network and analyze the control-plane traffic.
To configure Ethanalyzer, use the following commands:
Ethanalyzer does not capture data traffic that Cisco NX-OS forwards in the hardware.
Ethanalyzer uses the same capture filter syntax as tcpdump. For more information, see the following URL:
http://www.tcpdump.org/tcpdump_man.html
For information on the syntax of the display filter, see the following URL:
http://wiki.wireshark.org/DisplayFilters
This example shows captured data (limited to four packets) on the management interface:
switch(config)# ethanalyzer local interface mgmt brief limit-captured-frames 4Capturing on eth12008-02-18 13:21:21.841182 172.28.230.2 -> 224.0.0.2 HSRP Hello (state Standy)2008-02-18 13:21:21.842190 10.86.249.17 -> 172.28.231.193 TCP 4261 > telnet [AC] Seq=0 Ack=0 Win=64475 Len=02008-02-18 13:21:21.843039 172.28.231.193 -> 10.86.249.17 TELNET Telnet Data ..2008-02-18 13:21:21.850463 00:13:5f:1c:ee:80 -> ab:00:00:02:00:00 0x6002 DEC DNRemote Console4 packets capturedThis example shows detailed captured data for one HSRP packet:
switch(config)# ethanalyzer local interface mgmt capture-filter "udp port 1985"limit-captured-frames 1Capturing on eth1Frame 1 (62 bytes on wire, 62 bytes captured)Arrival Time: Feb 18, 2008 13:29:19.961280000[Time delta from previous captured frame: 1203341359.961280000 seconds][Time delta from previous displayed frame: 1203341359.961280000 seconds][Time since reference or first frame: 1203341359.961280000 seconds]Frame Number: 1Frame Length: 62 bytesCapture Length: 62 bytes[Frame is marked: False][Protocols in frame: eth:ip:udp:hsrp]Ethernet II, Src: 00:00:0c:07:ac:01 (00:00:0c:07:ac:01), Dst: 01:00:5e:00:00:02(01:00:5e:00:00:02)Destination: 01:00:5e:00:00:02 (01:00:5e:00:00:02)Address: 01:00:5e:00:00:02 (01:00:5e:00:00:02).... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast).... ..0. .... .... .... .... = LG bit: Globally unique address (factorydefault)Source: 00:00:0c:07:ac:01 (00:00:0c:07:ac:01)Address: 00:00:0c:07:ac:01 (00:00:0c:07:ac:01).... ...0 .... .... .... .... = IG bit: Individual address (unicast).... ..0. .... .... .... .... = LG bit: Globally unique address (factorydefault)Type: IP (0x0800)Internet Protocol, Src: 172.28.230.3 (172.28.230.3), Dst: 224.0.0.2 (224.0.0.2)Version: 4Header length: 20 bytesDifferentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00)1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30).... ..0. = ECN-Capable Transport (ECT): 0.... ...0 = ECN-CE: 0Total Length: 48Identification: 0x0000 (0)Flags: 0x000... = Reserved bit: Not set.0.. = Don't fragment: Not set..0. = More fragments: Not setFragment offset: 0Time to live: 1Protocol: UDP (0x11)Header checksum: 0x46db [correct][Good: True][Bad : False]Source: 172.28.230.3 (172.28.230.3)Destination: 224.0.0.2 (224.0.0.2)User Datagram Protocol, Src Port: 1985 (1985), Dst Port: 1985 (1985)Source port: 1985 (1985)Destination port: 1985 (1985)Length: 28Checksum: 0x8ab9 [correct][Good Checksum: True][Bad Checksum: False]Cisco Hot Standby Router ProtocolVersion: 0Op Code: Hello (0)State: Active (16)Hellotime: Default (3)Holdtime: Default (10)Priority: 105Group: 1Reserved: 0Authentication Data: Default (cisco)Virtual IP Address: 172.28.230.1 (172.28.230.1)1 packets capturedThis example uses a display filter to show only those HSRP packets that have an active HSRP state:
switch(config)# ethanalyzer local interface mgmt brief display-filter "hsrp.state==Active" limit-captured-frames 2Capturing on eth12008-02-18 14:35:41.443118 172.28.230.3 -> 224.0.0.2 HSRP Hello (state Active)2008-02-18 14:35:44.326892 172.28.230.3 -> 224.0.0.2 HSRP Hello (state Active)2 packets capturedFor more information on Wireshark, see the following URL:
http://www.wireshark.org/docs/
DCNM Tools
Cisco DCNM gathers events and statistics for each supported feature.
For more information about DCNM, see the Cisco DCNM Fundamentals Configuration Guide, Release 4.0.
SNMP and RMON Support
Cisco NX-OS provides extensive SNMPv1, v2, and v3 support, including Management Information Bases (MIBs) and notifications (traps and informs).
The SNMP standard allows any third-party applications that support the different MIBs to manage and monitor Cisco NX-OS.
SNMPv3 provides extended security. Each device can be selectively enabled or disabled for SNMP service. In addition, each device can be configured with a method of handling SNMPv1 and v2 requests.
Cisco NX-OS also supports Remote Monitoring (RMON) alarms and events. RMON alarms and events provide a mechanism for setting thresholds and sending notifications based on changes in network behavior.
The AlarmGroup allows you to set alarms. Alarms can be set on one or multiple parameters within a device. For example, you can set an RMON alarm for a specific level of CPU utilization on a device. The EventGroup allows you to configure events that are actions to be taken based on an alarm condition. The types of events that are supported include logging, SNMP traps, and log-and-trap.
For more information about configuring SNMP and RMON, see the Cisco NX-OS System Management Configuration Guide, Release 4.0.
Using RADIUS
The RADIUS protocol is used for to exchange attributes or credentials between a head-end RADIUS server and a client device. These attributes relate to three classes of services:
•
•
•
Authentication refers to the authentication of users for access to a specific device. You can use RADIUS to manage user accounts for access to an Cisco NX-OS device. When you try to log into a device, Cisco NX-OS validates you with information from a central RADIUS server.
Authorization refers to the scope of access that you have once you have been authenticated. Assigned roles for users can be stored in a RADIUS server with a list of actual devices that the user should have access to. Once the user has been authenticated, the device can then refer to the RADIUS server to determine the access that the user will have.
Accounting refers to the log information that is kept for each management session in a switch. You can use this information to generate reports for troubleshooting purposes and user accountability. You can implement accounting locally or remotely (using RADIUS).
The following example show a accounting log entries:
switch# show accounting logSun Dec 15 04:02:27 2007:start:/dev/pts/0_1039924947:adminSun Dec 15 04:02:28 2007:stop:/dev/pts/0_1039924947:admin:vsh exited normallySun Dec 15 04:02:33 2007:start:/dev/pts/0_1039924953:adminSun Dec 15 04:02:34 2007:stop:/dev/pts/0_1039924953:admin:vsh exited normallySun Dec 15 05:02:08 2007:start:snmp_1039928528_172.22.95.167:publicSun Dec 15 05:02:08 2007:update:snmp_1039928528_172.22.95.167:public:Switchname
Note
Using syslog
The system message logging software saves messages in a log file or directs the messages to other devices. This feature provides the following capabilities:
•
•
•
You can use syslog to store a chronological log of system messages locally or to sent this information to a central syslog server. The syslog messages can also be sent to the console for immediate use. These messages can vary in detail depending on the configuration that you choose.
The syslog messages are categorized into seven severity levels from debug to critical events. You can limit the severity levels that are reported for specific services within the device. For example, you may wish only to report debug events for the OSPF service but record all severity level events for the BGP service.
Log messages are not saved across system reboots. However, a maximum of 100 log messages with a severity level of critical and below (levels 0, 1, and 2) are saved in NVRAM. You can view this log at any time with the show logging nvram command.
This section includes the following topics:
•
Logging Levels
Cisco NX-OS supports the following logging levels:
•
•
•
•
•
•
•
•
By default, the device logs normal but significant system messages to a log file and sends these messages to the system console. Users can specify which system messages should be saved based on the type of facility and the severity level. Messages have a timestamp to enhance real-time debugging and management.
Enabling Logging for Telnet or SSH
System logging messages are sent to the console based on the default or configured logging facility and severity values.
Users can disable logging to the console or enable logging to a given Telnet or secure shell (SSH) session.
•
•
Note
The no logging console command (shown in Example B-4) disables console logging and is enabled by default.
Example B-4 no logging console Command
switch(config)# no logging consoleThe terminal monitor command (shown in Example B-5) enables logging for telnet or SSH and is disabled by default.
Example B-5 terminal monitor Command
switch# terminal monitorFor more information about configuring syslog, see the Cisco NX-OS System Management Configuration Guide, Release 4.0.
Using SPAN
You can use the Switched Port Analyzer (SPAN) utility to perform detailed troubleshooting or to take a sample of traffic from a particular application host for proactive monitoring and analysis.
When you have a problem in your network that you cannot solve by fixing the device configuration, you typically need to take a look at the protocol level. You can use debug commands to look at the control traffic between an end node and a switch. However, when you need to focus on all the traffic that originates from or is destined to a particular end node such as a host or a disk, you can use a protocol analyzer to capture protocol traces.
To use a protocol analyzer, you must insert the analyzer inline with the device under analysis, which disrupts input and output (I/O) to and from the device.
In Ethernet networks, you can solve this problem by using the SPAN utility. SPAN allows you to take a copy of all traffic and direct it to another port within the device. The process is nondisruptive to any connected devices and is facilitated in the hardware, which prevents any unnecessary CPU load.
SPAN allows you to create up to 16 independent SPAN sessions within the device. Each session can have up to four unique sources and one destination port. In addition, you can apply a filter to capture only the traffic received or the traffic transmitted. You can even capture traffic from a particular VLAN.
To start the SPAN utility, use span session session_num command, where session_num identifies a specific SPAN session. When you enter this command, the system displays a submenu, which allows you to configure the destination interface and the source VLAN or interfaces.
switch2# config terminalswitch2(config)# span session 1 <<=== Create a span sessionswitch2(config-span)# source interface e1/8 <<=== Specify the port to be spannedswitch2(config-span)# destination interface e1/3 <<==== Specify the span destination portswitch2(config-span)# endswitch2# show span session 1Session 1 (active)Destination is e1/3No session filters configuredIngress (rx) sources aree1/8,Egress (tx) sources arefe1/8,For more information about configuring SPAN, see the Cisco NX-OS System Management Configuration Guide, Release 4.0.
Using the Blue Beacon Feature
On some platforms, you can cause the platform LEDs to blink. This is a useful way to mark a piece of hardware so that a local administrator can quickly identify the hardware for troubleshooting or replacement.
To flash the LEDs on a hardware entity, use the following commands:
To flash a single port LEDs on a module, use the following command in interface configuration mode:
