Cisco ICS 7750 Troubleshooting Guide, 2.5.0 - Solving Hardware Problems

Table Of Contents

Solving Hardware Problems

Performing Diagnostics

POST Sequence of Events

Cisco ICS 7750 Card LEDs

System Card LED and Shutdown Buttons

LED Behavior During System Bootup

LED Behavior After System Bootup

LED Behavior After Shutdown

Diagnostic Commands

Using the ping Command

Using the traceroute Command

Power Subsystem

System Alarm Processor

Troubleshooting the SAP

Power Supply Modules

Troubleshooting Power Supply Modules

Fans

Checking Fan Speed and Operation

Troubleshooting Fans

System Processing Engine

Troubleshooting SPEs

Routing and Switching Subsystem

Troubleshooting the Routing and Switching Subsystem

How to Reset Chassis Slot Cards Through The SSP

How to Remotely Reset Chassis Slot Cards Through The SAP

Troubleshooting ASIs, MRPs, and WICs

Troubleshooting the SSP

Troubleshooting Cisco ICS 7750 Booting Problems

Monitoring the Cisco ICS 7750 Boot Process

MRP300, MRP3-8FXS, and MRP3-16FXS Boot Sequence

Initial Bootup and Discovery

ASI and MRP200 Boot Sequence

General ASI or MRP200 Boot Problems

Troubleshooting the SPE Running System Manager

Best Practices for Using the IOS CLI

Specific System and MRP-Related Bootup Problems

Troubleshooting MRP200, ASI81, and ASI160 Bootup Problems

Troubleshooting MRP300, MRP3-8FXS, and MRP3-16FXS Bootup Problems

Best Practices for Reseating Cards in the Chassis

Replacing an MRP200 or ASI Card with a Flash-Based MRP Card

SSP Card Boot Problems

Useful MRP Troubleshooting Commands

Support for VLAN Functionality

Enabling VLAN Functionality

Configuring VLAN Support on the Flash-Based MRP Card

Configuring VLAN Support on the SSP

Backing Up and Restoring the VLAN Configuration on the SSP

Troubleshooting VLANs

Solving Hardware Problems

This chapter explains how to solve hardware problems with the Cisco Integrated Communications System (ICS). The chapter is organized as follows:

•Performing Diagnostics

•Power Subsystem

•Fans

•System Processing Engine

•Routing and Switching Subsystem

•Troubleshooting Cisco ICS 7750 Booting Problems

•Useful MRP Troubleshooting Commands

•Support for VLAN Functionality

Note For a description of the features, modifications, and caveats for the
Cisco Integrated Communications System 7750 (Cisco ICS 7750) release 2.5.0, refer to the Release Notes for System Software Release 2.5.0 on the
Cisco ICS 7750.

Performing Diagnostics

You can use a card's power-on self-test (POST) or other diagnostics to locate hardware faults. In some cases, you can run diagnostics on an individual card while the rest of the system continues to operate because only the card being tested is out of service. In other cases, you must take the system off line to run diagnostics; for instance, testing the system processing engine (SPE) running System Manager may take the system off line if there is no secondary SPE310 running Cisco CallManager.

You can run diagnostics remotely through a Telnet or modem connection, locally from a console connected to the console port on the system alarm processor (SAP), or directly through a monitor, mouse, and keyboard attached to the SPE310 card.

POST diagnostics are available for the following cards:

•SPEs, including disk drives

•Analog station interface (ASI) and multiservice route processor (MRP) cards, including installed WAN interface cards (WICs), voice interface cards (VICs), and voice WAN interface cards (VWICS)

•SAP

•System switch processor (SSP)

POST diagnostics can be initiated either by unseating and reseating a card or by using a hardware reset command from the SSP. For instance, from enable mode on the SSP, enter the following command:
SSP#hw-module chassis slot 5 restart hold 5
Entering this command causes the card in slot 5 to restart after holding its power off for 5 seconds.

The set request command used in enable mode (after invoking the SLPENABLEPASSWORD command) on the SAP can also initiate POST. Viewing POST information through the console port on the SAP provides significantly more information than any other method, such as Telnet. From the SAP menu, select the card that you want to check immediately after its POST test has been initiated.

Caution There are only one SAP card and one SSP card in a chassis. If you take the SAP off line, the system's ability to detect alarms associated with the operating environment, fans, and power supply modules will be degraded until the SAP is returned to an on line status. If you take the SSP off line, the system loses LAN connectivity, and calls being made to or from Cisco IP Phones that are routed through the Cisco ICS 7750 will be disconnected until the SSP is brought back on line.

Note Depending on your system configuration, taking SPEs, ASIs, or MRPs off line can adversely affect users connected to the system. For example, if Cisco CallManager is running on only one SPE, taking that SPE off line disconnects calls to or from the PSTN and prevents the system from processing further PSTN traffic until that SPE is on line. Similarly, taking an ASI or MRP off line that is in the process of routing voice or WAN traffic will prevent that traffic from reaching its destination.

POST Sequence of Events

During POST, some or all of the following events can occur:

•A diagnostic image is downloaded to the card to be tested. The downloading process erases the IOS software from the card memory and installs the diagnostic software image.

•Normal card operation is suspended; the card being tested no longer sends or receives network traffic.

•As testing proceeds, the console displays diagnostic messages.

•A card pass or fail message is displayed on the console when testing ends.

•If the card passes the diagnostic tests, the IOS software is reloaded into the card's memory, and the card resumes normal operations.

Note If a card fails diagnostics, see the troubleshooting help in this chapter.

Cisco ICS 7750 Card LEDs

The Cisco ICS 7750 is a chassis-based system that consolidates multiple cards with different functions (SPEs, MRPs, SSP, and SAP) and power supply modules into a single system unit. Each system card has LEDs that provide visual indications about the status of the card.

The following sections describe the LEDs and LED behavior on the system cards:

•System Card LED and Shutdown Buttons

•LED Behavior During System Bootup

•LED Behavior After System Bootup

•LED Behavior After Shutdown

System Card LED and Shutdown Buttons

The LEDs that are common to each of the Cisco ICS 7750 cards are the alarm and status LEDs. Some cards have additional special-purpose LED(s), depending on the card's function. Each card also has one shutdown button.

Table 3-1 summarizes the LEDs that are available on each system card.

Table 3-1 Cisco ICS 7750 Card LEDs

LED

LED Color

SPE

MRP

SSP¹

SAP²

Alarm

Yellow, amber, red

X

X

X

X

Status

Green

X

X

X

X

Slot

Green

X³

Power⁴

Green

X

Fan

Yellow

X

Temperature

Yellow

X

¹The SSP also contains two port status LEDs. These LEDs are controlled by the IOS running on the SSP. Refer to the
Catalyst 2900 Series XL Installation and Configuration Guide for a description of the port status LEDs.

²The SAP has two power LEDs, one for each power supply module.

³The MRP has two slot LEDs, one for each VIC, WIC, or VWIC slot. Exceptions are the MRP3-16FXS and ASI160 cards, which do not contain any slot LED.

⁴The system power supply module has three LEDs—AC OK, DC OK, and Over Temp.

LED Behavior During System Bootup

During system bootup, the LEDs on each system card are controlled by the card on which they reside.

Table 3-2 shows the color and status of each LED on the MRP and SSP cards during the cards' various states from system power up to system bootup.

Table 3-2 LED States on MRP and SSP Cards

LED

Rommon Mode

During IOS Bootup

After IOS Bootup

Alarm LED (yellow)

Steady

Steady

Off, controlled by IOS¹

Status LED (green)

Blinking

Blinking

Steady, controlled by IOS

Slot LEDs² (green)

Steady if the slot is populated

Steady if the slot is populated

Steady if the slot is populated

¹This is a new behavior beginning with the IOS software packaged with ICS System Software release 2.4.0 (MRP IOS version 12.2(4)YH and SSP IOS version 12.0(5)WC5). With the IOS versions packaged before ICS System Software release 2.4.0, the LED is in a steady state after IOS bootup until ICS System Manager turns it off.

²The SSP does not have slot LEDs.

Table 3-3 shows the color and status of each LED on the SPE card during its various states from system power up to system bootup.

Table 3-3 LED States on SPE Card

LED

In BIOS

During System Bootup

After System Bootup

Alarm LED (yellow)

Steady

Steady

Off, controlled by FMM

Status LED (green)

Blinking

Blinking

Steady, controlled by FMM

Table 3-4 shows the color and status of each LED on the SAP card during its various states from system power up to system bootup.

Table 3-4 LED States on SAP Card

LED

During Firmware Bootup

After Firmware Bootup

Alarm LED (yellow)

Steady

Off, controlled by SAP firmware

Status LED (green)

Blinking

Steady

Power LEDs

Steady

Steady if it is normal

Blinking if it fails; controlled by SAP firmware

Fan LEDs

Off

Off if it is normal

Blinking if it fails; controlled by SAP firmware

Temperature LED

Off

Off if it is normal

Blinking if it fails; controlled by SAP firmware

LED Behavior After System Bootup

Once the Cisco ICS 7750 boots up, each card in the chassis transfers the alarm LED control functionality to ICS System Manager and FMM (on the SPE running System Manager). ICS System Manager and FMM determine the appropriate behavior of the alarm LED on each card, based on the traps or events sent by the cards in the chassis. The only colors used by ICS System Manager and FMM are yellow and amber, depending on the severity of the failure.

After system bootup, the status LED remains in its bootup state, as described in Table 3-2, Table 3-3, and Table 3-4, unless the shutdown button is pressed.

Slot LEDs on the MRP also remain in their bootup state, as described in Table 3-2.

Power, fan, and temperature LEDs on the SAP are controlled by the SAP firmware. Status changes are communicated to ICS System Manager and FMM.

LED Behavior After Shutdown

When you press the shutdown button on a system card, the card completely shuts down. The shutdown sequence is as follows:

•The status LED begins to blink immediately and keeps blinking during the shutdown preparation process; on the MRP, SSP, and SAP, the status LED blinks for 15 seconds.

•On the SPE, the operating system shuts down first. Then the card powers down.

•On the MRP, SSP, and SAP, card functionality completely stops.

•The software or firmware on a system card turns off the status LED on the card after the shutdown preparation period.

Note Refer to the Cisco Interface Cards Installation Guide for information on LED behavior for VICs, WICs, or VWICs.

Diagnostic Commands

Many of the cards in the Cisco ICS 7750 support commands that can help you better understand what is happening in your internetwork. This section provides information on the following topics:

•Using the ping Command

•Using the traceroute Command

Using the ping Command

The ping command is a test that is available on Cisco internetworking devices and on many host systems. In TCP/IP, this diagnostic tool is also known as an Internet Control Message Protocol (ICMP) echo request.

To check host reachability and network connectivity, use the IOS ping command. This command enables you to confirm basic network connectivity on many kinds of networks.

For IP connections, in EXEC mode, the ping command sends ICMP echo messages. If a station receives an ICMP echo message, it sends an ICMP echo reply message to the source.

In privileged EXEC mode, the ping command permits you to specify the supported IP header options. With these options specified, the target device can perform a more extensive range of test options.

It is a good idea to use the ping command when the network is functioning properly to see how the command works under normal conditions, so that you have something to compare to when troubleshooting.

Using the traceroute Command

In EXEC mode, the IOS traceroute command discovers the routes that packets follow to their destinations. In privileged EXEC mode, traceroute (also referred to as an extended trace) enables you to specify the supported IP header options, allowing the target device to perform a more extensive range of testing.

The traceroute command uses the error message generated by devices (routers or cards) when a datagram exceeds its time-to-live (TTL) value. Probe datagrams are sent initially with a TTL value of 1, which causes the first device to discard the probe datagrams and send back "time exceeded" error messages.

The traceroute command then sends several probes and displays the round-trip time for each. After every third probe, the TTL is increased by 1.

Each outgoing packet can result in one of two error messages. A "time exceeded" error message indicates that an intermediate device has seen and discarded the probe. A "port unreachable" error message indicates that the destination node has received the probe and discarded it because it could not deliver the packet to an application.

If the timer goes off before a response comes in, an asterisk (*) is displayed.

The trace operation terminates when the destination responds, when the maximum TTL is exceeded, or when you interrupt the trace with the escape sequence.

As with ping, it is a good idea to use the traceroute command when the network is functioning properly to see how the command works under normal conditions, so that you have something to compare to when troubleshooting.

For more information on ways to use the traceroute EXEC and privileged EXEC commands, refer to the "Troubleshooting and Fault Management Commands" chapter in the Cisco IOS Configuration Fundamentals Command Reference.

Power Subsystem

This section explains how to determine and resolve problems associated with the power subsystem, which includes the following components:

•System Alarm Processor

•Power Supply Modules

Note The SAP monitors the physical state of the system, including the availability of system power and cooling air.

System Alarm Processor

This section explains how to isolate problems associated with the SAP. Figure 3-1 shows the front panel of the SAP card.

Figure 3-1 SAP Card

Troubleshooting the SAP

Complete the following steps to isolate SAP problems:

Step 1 Check the system LEDs. If no SAP LEDs are on, ensure that the SAP is properly seated in the chassis and that the power supply modules and fans are functioning properly. (See the "Power Supply Modules" section or the "Fans" section for additional information.)

Step 2 Check all the SAP LEDs:

•If the ALARM LED is on (amber or yellow), the system has most likely found a problem outside the SAP. Check system status for additional indications that can help you isolate the problem.

•Verify that the STATUS LED is on (green), indicating that the system software has initialized successfully and that the system is operational.

•If the STATUS LED is off, the SAP might have pulled away from the backplane. If the SAP is not seated properly, it may cause the system to hang.

•If one or both of the PWR LEDs are off when they should be on (green), see the "Power Supply Modules" section.

•If the FAN LED is off or the TEMP LED is on (yellow), see the "Fans" section.

Power Supply Modules

This section explains how to isolate power supply module problems. Figure 3-2 shows the front panel of the power supply module.

Figure 3-2 Power Supply Module

Troubleshooting Power Supply Modules

Follow these steps to isolate power supply module problems; refer to Table 3-5 if necessary:

Step 1 Verify that the fans are operating and that the STATUS LED of at least one SPE is on (green). Look at the SAP card: If the FAN and STATUS LEDs are on (green) but the power supply AC OK LED is off, there is probably a faulty power supply module LED.

Step 2 Verify that the system power switch is set to the on position.

Step 3 Verify that the power source, power cable, and at least one power supply module are functioning correctly. Swap parts to determine whether one of the components is faulty. (Refer to the Cisco ICS 7750 Installation and Configuration Guide.)

Table 3-5 lists symptoms of and possible solutions for power problems.

Table 3-5 Power Supply Module Problems and Solutions

Symptom

Power Supply Module LED Status¹ Change

Possible
Cause

Solutions

The system shuts down after being on for a short time.

AC OK: on

OVERTEMP: off

DC OK: off

Power or cooling problem

•Verify that the area in which the system is installed meets the environmental requirements in the Cisco ICS 7750 Installation and Configuration Guide and in the "Site Requirements" section in the Regulatory Compliance and Safety document that came with your system.

•Verify that nothing is blocking the air intake or exhaust. (See the "Fans" section.)

•The power supply module has failed. Install a new power supply module.

The system attempts to boot, but all LEDs remain off.

AC OK: on

OVERTEMP: off

DC OK: off

Power problem

•Ensure that the power on/off switch is in the on position.

•The power supply module has failed. Install a new power supply module.

The power supply module is not operating within its normal operating tolerances (in a single power supply module configuration).

AC OK: on

OVERTEMP: off

DC OK: off

Defective power supply module

•If the system is still operating, insert a new power supply module into the unused power supply slot, and turn it on. If the newly installed power supply module is functioning properly, remove the defective power supply module in the other slot.

•If the system is not operating, replace the defective power supply module.

The power supply module has exceeded its maximum operating temperature and is about to shut down (in a single power supply module configuration).

AC OK: on

OVERTEMP: on

DC OK: off

Cooling problem

•Correct any problems associated with the fans. (See the "Fans" section.) The power supply module will restart if the high-temperature condition is caused by a problem with the fans.

•Replace the power supply module.

A power supply module is producing out-of-tolerance power (in a dual power supply module configuration).

AC OK: on

OVERTEMP: off

DC OK: off

Defective power supply module

Replace the defective power supply.

A power supply module has exceeded its maximum operating temperature and is about to shut down (in a dual power supply module configuration).

AC OK: on

OVERTEMP: on

DC OK: off

Cooling or power supply module problem

•Correct any problems associated with the fans. (See the "Fans" section.) The power supply module will restart if the high-temperature condition is caused by a problem with the fans.

•Replace the defective power supply module.

¹In dual power supply module configuration, the LED Status column shows the condition of the power supply module that is reporting an error.

Fans

This section explains how to isolate problems associated with the fans, which are located in the fan tray at the bottom of the system chassis.

Checking Fan Speed and Operation

The SAP on the Cisco ICS 7750 monitors the operational status of each component in the system.

Working with the Fault Management Module (FMM) and ICS System Manager, the SAP can remotely alert you of environmental, functional, or operational problems detected within the system. Access is available through the console port on the front of the card.

One element of the Cisco ICS 7750 that can affect performance is the speed of its fans. The fans run at different speeds, depending on the system's operation:

•At power up, the fans should run at high speed (high noise level) for up to 10 seconds. After that time, they should go into normal operation mode.

•At normal operation, the fans should be spinning at a low speed (low noise level).

•During a software upgrade, the fans should run quietly.

•Following a software upgrade, the fans should run as they do during the power-up process, spinning fast for about 10 seconds and then slowing down to normal operation.

Follow these steps to check the fans for proper operation:

Step 1 On a PC, open a HyperTerminal session with the SAP card.

Step 2 At the prompt, enter enable mode by typing slpenablepassword, followed by the password for the card.

Step 3 Enter the following command:
get fan-speed :<fan#> 
where fan# is the number of the fan that you wish to check.

The following is an example of output that shows fans running at normal operation:
AlarmCard>get fan-speed :1
INF FAN-SPEED : 1, 2250
AlarmCard>get fan-speed :2
INF FAN-SPEED : 2, 2250
AlarmCard>get fan-speed :3
INF FAN-SPEED : 3, 2250
AlarmCard>get fan-speed :4
INF FAN-SPEED : 4, 2280
Troubleshooting Fans

Follow these steps to isolate fan problems. Refer to Table 3-6 if necessary.

Step 1 To verify that the fans are operating, place your hand near the air exhaust on the back of the chassis, and check for a warm flow of air. If the fans are not operating, there might be a problem with the fan tray or with the power source. If the fans are not operating and if the FAN LED on the SAP is off, ensure that the fan tray is seated properly. If the LEDs on the SAP are off, ensure that the SAP is also seated properly. If there are no LEDs illuminated on the SAP, disengage the ejector levers to unseat the card, and then reseat it to ensure that it is seated properly in the chassis.

Step 2 If the system and the fans start up but shut down after about 2 minutes, one or more fans might have failed.

If one or more fans fail to operate or if one or more fans have tachometer speeds that are not within the acceptable range, you might need to replace the fan tray.

Step 3 If you see the following message at startup, the system has detected an overtemperature condition or out-of-tolerance power inside the chassis:
The system is overheated 
This message might indicate an environmental problem; ensure that there is adequate ventilation and unrestricted air flow in the location where the Cisco ICS 7750 is installed. This message might also indicate a malfunctioning component or a faulty temperature sensor within the Cisco ICS 7750 chassis.

Step 4 Ensure that heated exhaust air from other equipment is not entering the inlet vents and that there is sufficient clearance around the chassis to allow cooling air to flow.

Step 5 Ensure that the SAP is properly installed. If the SAP is not properly installed, the fans will not operate.

Table 3-6 lists symptoms of and possible solutions for fan problems.
Table 3-6 Fan Problems and Solutions
Syslog Message

Symptom

SAP LED
Status Change

Possible
Cause

Solutions
The  
system is 
overheated
The system is overheated and will shut down 10 seconds after sending a warning message.

ALARM: on (amber)

TEMP: on

High ambient air temperature or air intake or exhaust blockage

•Verify that the room temperature in which the system is located is less than 104°F (40°C).

•Verify that there are no air intake blockages at the front of the chassis.

•Verify that there are no air exhaust blockages at the back of the chassis.

•Verify that there are no internal system blockages.
Card x is 
overheated
A system card (other than the SAP) is overheated.

ALARM: on (amber)

TEMP: on

High ambient air temperature, air intake or
exhaust blockage, or card failure

•Verify that the room temperature in which the system is located is less than 104°F (40°C).

•Verify that there are no air intake blockages at the front of the chassis.

•Verify that there are no air exhaust blockages at the back of the chassis.

•Verify that there are no internal system blockages.

•Replace the card.
The ambient 
temp x 
exceeded 
recommended 
value
The temperature inside the chassis is over 104°F (40°C).

ALARM: on (amber)

TEMP: on

High ambient air temperature or air intake or exhaust blockages

•Verify that the room temperature in which the system is located is less than 104°F (40°C).

•Verify that there are no air intake blockages at the front of the chassis.

•Verify that there are no air exhaust blockages at the back of the chassis.

•Verify that there are no internal system blockages.
Fan x 
failure
A fan tachometer reading drops below the designated threshold, or a fan stops completely.

ALARM: on (amber)

FAN: on

TEMP: on

Improperly seated or defective fan tray

•Remove, inspect, and reinsert the fan tray.

•If the fans are still not operating, install a new fan tray.

•Refer to the
Cisco ICS 7750 Installation and Configuration Guide.
Fan tray 
absent
All fan tachometer readings fall to zero.

ALARM: on (amber)

FAN: on

TEMP: on

Missing or improperly seated fan tray

•Remove, inspect, and reinsert the fan tray.

•If the fans are still not operating, install a new fan tray.

•If the fans are still not operating, install a new SAP.

•Refer to the
Cisco ICS 7750 Installation and Configuration Guide.
System Processing Engine

This section explains how to isolate problems associated with the SPE. Figure 3-3 shows the front panel of the SPE310.

Figure 3-3 SPE Card

Note Refer to the Installing Memory, PVDM, and VPN Modules in ASI Cards, MRP Cards, and SPE Cards in the Cisco ICS 7750 documentation for a detailed view of the SPE310 card.

Note The Cisco ICS 7750 supports a maximum of 5 SPEs in a single chassis.

Troubleshooting SPEs

Follow these steps to isolate problems with the SPE:

Step 1 Check the system LEDs. If no SPE LEDs are on, ensure that the SAP, power supply modules and fans are functioning properly. (See the "Power Supply Modules" section and the "Fans" section for additional information.)

Step 2 Check all the SPE LEDs.

a. If any ALARM LED is on (amber or yellow), except during system boot or POST, the system has detected a hardware failure in that SPE. Contact a technical support representative for instructions.

b. Verify that the STATUS LED on all SPEs is on (green), indicating that the system software has initialized successfully and that the system is operational.

c. If the STATUS LED on an SPE is off, that SPE might have become disengaged from the backplane or might be in a shutdown state. If an SPE is not seated properly, it may hang the system.

Table 3-7 lists symptoms of and possible solutions for hardware-related problems on the SPE.

Table 3-7 SPE Problems and Solutions

Symptom

SAP LED Status Change

SPE LED Status Change

Possible
Cause

Solutions

The system cannot read or write to an SPE hard disk.

ALARM: on (amber)

None

Inadequate security privileges (read/write permission) or hard disk errors

•The SPE may be out of disk space. Use Windows Explorer to determine how much space is left on the hard disk and, if necessary, move data to another server.

•You may not have the necessary authorization. If you are attempting to write to the disk, use ICSConfig to determine whether you can access the system as an administrator.

•The hard disk may have errors. Close all applications running on the disk, and use the Windows 2000 Check Disk utility to scan for and attempt recovery of bad sectors.

Note The disk is not available to run other tasks while Check Disk is running.

•Replace the SPE.

SPE memory parity errors have occurred.

ALARM: on (amber)

ALARM: on (amber)

Defective memory or hard disk

Replace the SPE.

SPE failed to boot.

ALARM: on (yellow)

ALARM: on (yellow)

STATUS: off

Defective memory module and defective hard disk in SPE

Replace the SPE.

Routing and Switching Subsystem

This section explains how to isolate problems associated with the routing and switching subsystem, which includes the following components:

•One or more ASI cards or MRP cards. Each ASI81 card and MRP card can include different combinations of WICs, VICs, and VWICs.

•SSP.

Note For problems associated with VICs and trunks or for other voice-related issues, see "Solving Voice Problems." For instructions on how to solve problems concerning Catalyst 3524-PWR XL switches, refer to the documentation that came with the Catalyst 3524-PWR XL switches.

Figure 3-4 shows the front panels of an MRP card (on the left) and an SSP card (on the right).

Note Refer to the Installing Memory, PVDM, and VPN Modules in ASI Cards, MRP Cards, and SPE Cards in the Cisco ICS 7750 documentation for a detailed view of the MRP200, ASI81, ASI160, MRP300, MRP3-8FXS, and MRP3-16FXS cards.

Figure 3-4 Routing and Switching Subsystem (MRP and SSP)

Troubleshooting the Routing and Switching Subsystem

The problems listed below could require replacement of an ASI, MRP, or SSP:

•An ASI or MRP fails, even when moved to another slot. (If the card fails in one slot but operates properly in another, there might be a problem with the backplane or SSP.)

•Diagnostics that loop data through the SSP fail on two or more Catalyst 3524-PWR XL switches that are connected to that SSP.

•Traffic is not passing through the system, but the ASIs, the MRPs and the SSP all appear to be functioning normally.

•The system has data transmission problems that do not go away when you replace the card that appears to be failing, or the system has problems that occur in several cards simultaneously. (Problems of this type can also indicate a faulty backplane.)

•You cannot fully insert an ASI, MRP, or SSP into its slot. This problem most likely indicates damage to the connectors on either the card or the backplane. Inspect all the connectors. If you find damage, replace the card or the chassis.

How to Reset Chassis Slot Cards Through The SSP

This procedure can be useful when you are remotely troubleshooting a Cisco ICS 7750, or when you mistakenly shut down the SPE instead of restarting it. Generally, when restarts of an SPE fail, you need to unseat the SPE and then reseat it in order to get it to boot. To perform these actions, you need to have hands-on access to the Cisco ICS 7750. If the Cisco ICS 7750 is remotely located, you will need to obtain on-site support in order to proceed.

To restart Cisco ICS 7750 cards using the SSP, start at the command-line interface (CLI) for the SSP. This can be through Telnet or through the console of the Cisco ICS 7750, using a 2511 access server if you are not locally connected to the system.

At the CLI prompt, issue the following command:
Switch#hw-module chassis slot n restart
where n represents the slot number in which the line card to restart is installed.

Options for this command include a delay before the line card is restarted, as well as a hold in the unpowered state between the bring-down and bring-up phases. Examples follow:
Switch#hw-module chassis slot 1 restart ?
  delay  Seconds to delay before asserting reset
  hold   Seconds to assert reset
  <cr>
Switch#hw-module chassis slot 1 restart delay ?
  <0-2147483647>  Delay in seconds
  <cr>
Switch#hw-module chassis slot 8 restart hold ?
  <0-2147483647>  Seconds to assert reset
How to Remotely Reset Chassis Slot Cards Through The SAP

You can use the SSP for remote troubleshooting purposes, as described in the "How to Reset Chassis Slot Cards Through The SSP" section. You can also access the SAP menu without being directly connected to the Cisco ICS 7750 chassis. Either method may be used, depending on user preference, the state of the system, and the task to be performed.

Follow these steps to set up remote troubleshooting capability, using the SAP console:

Step 1 At the Cisco ICS 7750 chassis, plug the blue RJ-45 console cable into the console port on the SAP.

Step 2 Plug the other end of the console cable (DB-9 connector) into Com port 1 or Com port 2 (if you are using voice mail, plug this end into Com port 2).

Step 3 Access the SPE running System Manager, using a Terminal Services Client connection.

Step 4 Open the HyperTerminal application on the SPE, using Com 1 or Com 2 (make sure to set up your HyperTerminal connection to recognize this port).

You will be presented with the SAP prompt as if you were directly connected to the Cisco ICS 7750.

Step 5 From the SAP menu, enter the SLPENABLEPASSWORD command to access the administrator functions on the SAP.

Step 6 Enter the administrator password.

Step 7 At the prompt, enter the set reset command, as follows:
AlarmCard>set reset 5,1,1
In this example, the SAP will reset (re-enable) the card in slot 5 after holding for 1 second and waiting for 1 second

where:

•5 represents the slot number of the card to be reset.

•1 represents the time_to_hold_reset (the time in seconds for the SAP to hold during the reset operation).

•1 represents the time_to_wait_reset (the time in seconds for the SAP to wait before performing the reset operation); this is an optional parameter.

Troubleshooting ASIs, MRPs, and WICs

Table 3-8 lists symptoms of and possible solutions for problems with ASIs, MRPs, and WICs.

Table 3-8 ASI, MRP, and WIC Problems and Solutions

Symptom

ASI or MRP LED Status Change

Possible Cause

Solutions

All ASIs and MRPs do not display status as expected.

STATUS: off
(all ASIs and MRPs)

Power problem

Check your system power connections and power supply modules. (See the "Power Subsystem" section.)

A single ASI or MRP does not display status as expected.

STATUS: off (single ASI or MRP)

Improperly seated card or chassis slot problem

•If the STATUS LED remains off, verify that the card is seated properly.

•Assuming that the power LED of other ASIs or MRPs is on, try inserting the card in a different slot.

A WAN interface is down.

ALARM: on (amber)

Improperly configured or defective WIC

•If the state change is unexpected, verify that the WIC that is reporting the problem is properly configured.

•Contact your technical support representative for assistance.

There is loss of signal on a WAN serial link.

ALARM: on (amber)

Cable or WIC problem

•Replace the cable attached to the WIC that is reporting the error.

•Have your telephone service provider perform a loopback test to verify the integrity of the WAN link.

•Replace the faulty WIC.

•Contact your technical support representative for assistance.

Note See Chapter 6, "Solving Serial Connection Problems," for additional information.

The WIC is not supported.

ALARM: on (amber)

WIC installed in ASI81 or MRP not supported by currently loaded IOS software

•Determine whether the WIC is supported by the IOS software release running on the ASI or MRP.

•Contact your technical support representative for assistance.

An ASI or MRP initialization error has occurred.

ALARM: on (amber)

Checksum error detected while downloading WIC firmware

Copy the error message exactly as it appears, and contact your technical support representative.

The ASI or MRP has failed to boot.

ALARM: on (yellow)

STATUS: off

Improperly seated or defective ASI or MRP, IOS image not downloaded from SPE (see Table 3-10 for additional information)

•Remove and reinsert the ASI or MRP.

•Replace the ASI or MRP.

•Using System Manager, check the System Manager Software Upgrade table on the SPE running System Manager to see if an appropriate IOS image has been delivered to the ASI or MRP card.

A WIC minor alarm occurs.

ALARM: on (yellow)

WIC link integrity failure or incorrect WIC configuration

•Verify that the WIC configuration is correct.

•Have your telephone service provider perform a loopback test.

There are WIC transmit or receive errors.

ALARM: on (yellow)

•Consecutive frame bits on receive line

•Errors on receive or transmit lines

•Remote signal or frame errors on receive and transmit lines

•Check the status of the carrier line.

•Contact your technical support representative for assistance.

Note WIC, VIC, and VWIC connectivity that is lost for an extended period of time, such as 30 minutes, may not be regained without resetting the far end of the connection.

Troubleshooting the SSP

Table 3-9 lists symptoms of and possible solutions for problems with an SSP.

Table 3-9 SSP Problems and Solutions

Symptom

SSP LED Status Change

Possible Cause

Solutions

SSP does not display status as expected.

STATUS: off

Power problem or improperly seated card

•Check system power connections and power supply modules. (See the "Power Subsystem" section.)

•If the STATUS LED remains off, verify that the SSP is seated properly.

An SSP interface is down.

ALARM: on (yellow)

Interface intentionally disabled or errors on network

•If the state change is unexpected, verify that the interface has not been explicitly disabled by an administrator.

•Look for an address violation, such as an address mismatch or duplication. If there is any address duplication, disable address violation detection.

•Verify that there are no network connection errors, such as a loss of link beat or jabber.

•Contact your technical support representative for assistance.

SSP fails to boot.

STATUS: blinking (green)

Switch software has become corrupted, or there is no bootable file in Flash

This situation can cause the SSP to drop into ROMMON mode with no bootable file in Flash.

•Reseat the SSP in the chassis.

•If the problem persists, use XMODEM to copy the correct image from the PC to the SSP (ensure the baud rate on the SSP is configured with the default value of 9600). Refer to Recovery From Corrupt or Missing Software Image - Cisco Catalyst 2900XL, 3500XL, and 2950 Series Switches for additional information.

•Contact your technical support representative for assistance.

Troubleshooting Cisco ICS 7750 Booting Problems

This section describes various ways to troubleshoot problems that may prevent or interfere with the ability of the Cisco ICS 7750 to boot. This section is organized as follows:

•Monitoring the Cisco ICS 7750 Boot Process

•MRP300, MRP3-8FXS, and MRP3-16FXS Boot Sequence

•ASI and MRP200 Boot Sequence

•General ASI or MRP200 Boot Problems

•Specific System and MRP-Related Bootup Problems

•SSP Card Boot Problems

Monitoring the Cisco ICS 7750 Boot Process

You can monitor the boot process on the Cisco ICS 7750 to verify that it comes up correctly, as follows:

Step 1 Using a HyperTerminal connection, connect a PC to the console port of the SAP card.

The HyperTerminal connection will display information that is flowing across the serial connection.

Step 2 Power on the Cisco ICS 7750 chassis.

The SAP card should display fan, temperature, power supply, and hardware sensor information.

Step 3 Once the display stops, press CTRL-backslash (\), and choose the SSP switch card (option C).

The SSP will boot in the regular Catalyst switch sequence. However, although it appears to be done booting, it is not done.

Several minutes will elapse with little apparent activity. During this time, the SPE running System Manager is launching various essential services.

•If this is a new Cisco ICS 7750 system or if it is the first time the SPE has been booted up, 8 to 10 minutes may elapse. The amount of time will also depend on the software package installed on the SPE.

•If this is just a reboot of a configured system, the time will be less—3 to 5 minutes. There is little point in trying to monitor that directly. It is recommended that you do not attempt to access the SPE during this period.

After a few minutes, the SPE running System Manager will download a configuration to the SSP switch card. You will know that the download is completed when you see (on the SSP) the following:
*Mar  1 00:00:23.938 UTC: %SYS-5-CONFIG_I: Configured from console by 
vty0 (10.0.0.1)
Note The IP address, 10.0.0.1, is the default IP address for the SPE running System Manager. This IP address may be different after you have configured the system using ICSConfig.

If you then display the configuration of the SSP, you will find the following:

•It has a password. (If this is the initial boot of the system or of a replacement SPE running System Manager, the password will be changeme.)

•It has an IP address on Interface VLAN 1. (If this is the initial boot of the system or of a replacement SPE running System Manager, the IP address will be in the 10.0.0.0 network, with a subnet mask of 255.255.255.0.)

Note If any of these settings is incorrect, the system will not function, even if you attempt to change settings or otherwise intervene manually. Contact the Cisco TAC for assistance.

If the SSP has the correct configuration (IP address and password), press CTRL-backslash (\), and enter the slot number of an MRP card.

The MRP may still be broadcasting a BOOTP request. If so, let it continue because it may take several minutes for the MRPs to receive their boot information—their IP address, IOS image, and configuration.

The reason why MRP BOOTP requests go unanswered before the MRP boots up is that the MRPs are the last cards to boot in the chassis. Before the SPE running System Manager can answer the MRPs' boot requests, that SPE must first boot itself and start ICS System Manager. It then answers the BOOTP requests from other cards, such as MRP200 and ASI cards, that may be present. Also, before the Cisco Network Registrar (CNR) responds to a BOOTP request, it must ensure that the request has originated from within the same chassis. Typically, the requests from an MRP or ASI, that fall within the first 30 seconds after the SPE running System Manager has fully started, will not receive a response.

When the MRP starts booting from 10.0.0.1 (the SPE running System Manager), you can assume that the boot process is likely to succeed.

When all the MRPs boot from the SPE running System Manager, the Cisco ICS 7750 is ready to operate.

You can use the same monitoring process to verify that the system boots correctly after running the initial configuration program (ICSConfig). Of course, the IP address and password on the SSP must match the ones you assigned in ICSConfig.

If the boot process is not successful, see the "Specific System and MRP-Related Bootup Problems" section for additional information.

MRP300, MRP3-8FXS, and MRP3-16FXS Boot Sequence

The MRP200 and MRP300 cards are both voice-and-data-capable routers that support digital and analog voice trunks and WAN routing interfaces to link remote Ethernet LANs to the Public Switched Telephone Network (PSTN), to existing private branch exchanges (PBXs), and to most common analog devices, such as fax machines and teleconferencing stations.

The MRP300, MRP3-8FXS, and MRP3-16FXS cards are Flash-based cards. They have additional functionality provided by 16 MB of onboard Flash memory, with 64 MB of add-on Flash memory available as an option.

Beginning with ICS System Software release 2.5.0, the Flash-based MRP cards also have the ability to be configured as virtual local area network (VLAN) routers. See the "Support for VLAN Functionality" section.

Configuration files for the Flash-based cards (MRP300, MRP3-8FXS, and MRP3-16FXS) are loaded and saved on these cards in nonvolatile RAM (NVRAM). The performance impact is an improvement in boot time. When there is an IOS image in Flash memory, the Flash-based MRP card boot time is reduced because it does not have to TFTP the image from the SPE.

Note The MRP Flash-based cards should obtain their IP address and initial configuration from the ICS System Manager that is running in the same chassis, and not from any other external Dynamic Host Control Protocol (DHCP) server.

Initial Bootup and Discovery

The MRP Flash-based cards are shipped with a default IOS image in Flash memory, and with no configuration in their NVRAM. The first time the Flash-based MRP boots up in a Cisco ICS 7750 chassis, the card will send DHCP requests (see number 2).

If the boot field of the configuration register, which controls the way the system boots, is set to "Boot to ROMMON" (0x0), then the system will remain at the ROM Monitor (ROMMON) prompt. Setting the configuration register to 0x2102 will not initiate BOOTP requests, and the MRP will try to boot from Flash memory.

If the boot field of the configuration register is set to anything other than "Boot to ROMMON," then either of the following will occur:

•The system will stay in ROMMON mode if the Flash memory is corrupt or if there is no image in Flash. No attempt is made to obtain an IOS image.

•ROMMON boots from the image in Flash memory if there is an IOS image in Flash memory.

After IOS has booted, the following should occur:

1. IOS enters the standard IOS prompt if the Fast Ethernet interface and all its subinterfaces (if there are any), are configured with IP addresses by the startup-configuration in NVRAM.

2. If the Fast Ethernet interface or any of its subinterfaces are not configured with IP addresses because the startup-configuration in NVRAM did not complete, IOS will attempt to discover the IP addresses by sending DHCP requests. IOS will not perform DHCP auto-installation. DHCP responses are accepted from any DHCP server.

3. If there is no startup-configuration in NVRAM, then IOS will attempt to discover the IP addresses and download the startup-configuration through the standard DHCP auto-installation mechanism, from ICS System Manager in the same chassis only.

–The Vendor class identifier field in DHCP request messages sent during DHCP auto-installation will be filled with any one of the following strings, as appropriate:

•ICS7750-MRP300—for MRP300

•ICS7750-MRP3-8FXS—for MRP3-8FXS

•ICS7750-MRP3-16FXS—for MRP3-16FXS

4. The DHCP response message from ICS System Manager will contain the IP address, the IP address of the TFTP server, and the configuration filename. During DHCP auto-installation, the Flash-based MRP card looks for the pattern ICS7750-SPE in sub-option 66 of Option 43 (Vendor-specific information) in the DHCP response. This entry ensures that the Flash-based MRP card obtains its IP address and initial configuration from the ICS System Manager, and not from any external DHCP server.

5. If the DHCP response contains the expected pattern (in sub-option 66 of Option 43) and a configuration filename, the Flash-based MRP card downloads the configuration file from the TFTP server and merges it with the running-configuration file.

6. The Flash-based MRP card is now configured with the passwords and IP address delivered by the ICS System Manager. The configuration is saved locally in the Flash-based card's NVRAM. When a configuration change is made to the Flash-based card by using ICSConfig, the ICS System Manager Telnets to the Flash-based card and issues the copy running-config startup-config command, which saves the configuration in that card's NVRAM. When manual changes are made to the configuration by using the CLI from the MRP's console, the running-config will be written to