-
Cisco ICS 7750 Troubleshooting Guide, 2.5.0
-
Preface
-
Monitoring the System
-
System Troubleshooting Guidelines
-
Solving Hardware Problems
-
Installing and Upgrading Cisco ICS 7750 System Software
-
Troubleshooting Cisco ICS 7750 Software
-
Solving Serial Connection Problems
-
Solving Ethernet Problems
-
Solving IP Problems
-
Solving Voice Problems
-
Error Message Summary
-
Reimaging Cisco ICS 7750 SPEs
-
Using Peripherals with Cisco ICS 7750 SPEs
-
Index
-
Table Of Contents
Handling Log Messages with ICS System Manager
Saving Log Messages to a Syslog Server
How to Change the Log Configuration
Configuring the Syslog Daemon on UNIX Syslog Servers
Changing Syslog Server Logging
Enabling or Disabling Availability Notification
Setting E-Mail Notification Defaults
Setting and Testing Page Notification Defaults
Additional Verification of Correct E-Mail Settings
SNMP Management Information Base
Understanding Community Strings
Modifying the Base Community String
Monitoring with ICS System Manager
Monitoring with CiscoWorks2000
Using CiscoWorks2000 for Network Management
SNMP and the CiscoWorks2000 Interface
Monitoring with Cisco IOS Software
Evaluating Reachability and Response Times
In-Band Versus Out-of-Band Monitoring
Searching and Filtering Output of show Commands
Monitoring an Uninterruptible Power Supply (UPS)
Connecting and Powering Up the UPS Components
Configuring the System to Monitor the UPS Through a Serial Connection
Configuring the System to Monitor a UPS Through an Ethernet Connection
Verifying That the Cisco ICS 7750 Can Communicate with the UPS
Monitoring the System
This chapter explains how to monitor the Cisco ICS 7750. The chapter is organized as follows:
•
Monitoring with ICS System Manager
•
•
•
Note
Cisco Integrated Communications System 7750 (Cisco ICS 7750) release 2.6.0, refer to the Release Notes for System Software Release 2.5.0 on the
Cisco ICS 7750.
Alarms
This section describes alarms, which indicate problems on the Cisco ICS 7750 or on systems with which it is communicating.
Alarms are associated with the following faults and event generators:
•
•
Applications running on the SPE, such as Cisco CallManager and Cisco Unity Voice Messaging, are capable of sending syslog messages, but these applications must first be configured to send the messages and must be configured with a destination (the SPE running System Manager).
•
MRPs also generate syslog messages when specific conditions occur, such as a link-up/link-down state. These syslog messages are received by the CSyslog service on the SPE running System Manager and logged in the syslog database. Each syslog message is analyzed against the rules in FMM. If there is an action specified in FMM for that syslog message, the action is taken (such as setting the alarm LED).
•
•
Note
Alarm Notification
The following system components are responsible for issuing notifications of alarms in the following ways:
•
•
•
•
Alarm Levels
The cards in the Cisco ICS 7750 each have an alarm LED that indicates the status of the hardware and software of that card. The SAP has an alarm LED that indicates the status of the overall system.
Using these alarm LEDs, the system can report the following alarm levels:
•
–
–
–
–
–
•
–
–
Logging
This section provides the following information about logging:
•
How to Access Log Messages
You can access log messages in any of the following ways:
•
•
Handling Log Messages with ICS System Manager
ICS System Manager provides several options for handling the log messages directed to it. By default, the system sends log messages to the SPE, where they are stored on disk.
The FMM module on each SPE running core software is also configured to send FMM syslog messages to ICS System Manager. Syslog messages generated by other applications on the SPE running core software will not be forwarded to ICS System Manager, however, unless you configure the application. If the application is configured to send syslog messages to ICS System Manager, then the messages will be logged in the ICS System Manager syslog database.
Other components in the Cisco ICS 7750 chassis, such as the MRP and SSP, are configured with a syslog destination in the IOS configuration to send syslog messages to ICS System Manager. This configuration is done through ICSConfig.
Note
Saving Log Messages to a Syslog Server
The system saves syslog messages to an internal buffer. You can configure the system to read messages from the buffer and send them to a specified syslog server.
Syslog messages are reported based on severity. The first number following the percent sign (%) in a syslog message indicates the severity of the message. For details, see the "Severity Levels" section.
Note
How to Read Log Messages
When viewed on a log server, the mandatory portion of a log message begins with a percent sign (%) and can contain up to 80 characters. The message fields that precede the percent sign (received and sent dates and times) are optional.
Table 1-1 describes the elements of log messages that can be viewed in Event Manager (see the "Event Manager" section).
Table 1-1 Log Message Elements
Received date and time
yyyy mmm dd
hh:mm:ssThe date and time when the message was received.
Sent date and time
yyyy mmm dd
hh:mm:ssThe date and time when the message was sent.
FACILITY
STRING
Two or more uppercase letters that indicate the facility to which the message refers (see Table 1-2).
From
n.n.n.n
The IP address of the device sending the message.
Message
string
A description of the event.
CISCO FACILITY (optional)
STRING
Two or more uppercase letters that indicate the facility to which the message refers. Facilities include hardware devices, protocols, and system software modules.
CISCO SUBFACILITY (optional)
STRING
Two or more uppercase letters that indicate the subfacility for Channel Interface Processor (CIP) messages. CIP messages have a Cisco subfacility code of CIP.
Cisco Severity (optional)
0-7
A single-digit code from 0 to 7 that indicates the severity of the message (see Table 1-3). The lower the number, the more serious the situation.
CISCO MNEMONIC (optional)
STRING
A code that uniquely identifies the message.
Facilities
Table 1-2 describes the facility types supported by log messages.
Severity Levels
Table 1-3 describes log message severity levels.
Note
How to Change the Log Configuration
The system sends log messages to ICS System Manager by default. You should not change this default configuration; changing the default configuration may impact the functionality of ICS System Manager. You can, however, add another syslog destination, such as buffers and UNIX hosts that are running a syslog server, to direct these messages.
This section provides the following information about log configurations:
•
•
Default Log Configuration
System IOS components (analog station interface [ASI] cards, MRP cards, and the SSP card) ship with the default logging configuration described in Table 1-4.
Tip
Configuring the Syslog Daemon on UNIX Syslog Servers
Before you can send log messages to a UNIX syslog server, you must configure the syslog daemon on the UNIX server. To configure the syslog daemon, log in as root and include a line such as the following in the file syslog.conf:
facility.level /syslog path/myfile.logwhere
•
•
•
•
The syslog daemon (syslogd) sends messages at the level specified in syslog.conf, provided that the file exists and provided that syslogd has permission to write to it.
Changing Syslog Server Logging
Table 1-5 shows IOS commands that can be used in global configuration mode to change syslog server logging behavior. However, the use of CLI commands to change the syslog logging host on the Cisco ICS 7750 is prohibited; use ICSConfig instead of CLI commands.
Table 1-5 Syslog Server Logging Behavior Commands
Configure an IOS device to log messages to a syslog server, where host is the name or IP address of the target syslog server.
logging host
Remove a host from the list of syslog servers.
no logging host
Configure an IOS device to limit the log messages that it sends to the syslog server(s), based on the severity level, where level is one of the log message severity keywords listed in Table 1-3.
logging trap level
Disable logging to the syslog server(s).
no logging trap
Note
Note
Event Manager
ICS System Manager includes an Event Manager that monitors system events. The Event Manager can be configured to invoke action when specific types of events are received. These events are defined by Event Manager rules; the events include conditions, such that when Event Manager receives a system message, that message is compared to the conditions defined in the rules. If the message fulfills the conditions of a rule, then Event Manager will take appropriate action, as defined.
Event Manager enables you to view system events (messages) and to specify how you want the system to respond to a particular type of message. For example, for certain types of log messages, you might want to configure the system to automatically generate an e-mail message or send a page.
To facilitate performance improvements and to provide enhanced support to the Cisco ICS 7500, there is an alert feature in Event Manager which, if enabled, forwards certain performance and availability monitoring event information to the designated system user and to Cisco Systems. SSP, FMM, chassis, and MRP events cause system information to be sent when this feature is enabled.
Note
Enabling or Disabling Availability Notification
The availability notification feature can be enabled or disabled at any time by accessing the Event Manager tab labeled "Availability and Preferences."
Step 1
Step 2
Step 3
Step 4
To set up e-mail alerts, you must consent to the availability agreement displayed on the second page of the ICSConfig user interface during the initial configuration process. The first screen displays the general license information, and the second page shows the e-mail processing agreement.
Through the Event Manager Preferences page, the following values can be defined:
•
•
•
•
A description of the fields, links, and controls available through Preferences can be found in the online help section of the Event Manager tab.
Setting E-Mail Notification Defaults
To set e-mail notification, follow these steps:
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Setting and Testing Page Notification Defaults
In addition to e-mail notifications, paging parameters may also be configured to alert you of specific events as they occur on the Cisco ICS 7750. When the paging operation is configured, a numeric page is sent from the Cisco ICS 7750 when condition(s) of a custom rule are met. You must define and create the custom rule that will send a page upon a specific condition. Multiple rules must be created for separate conditions, sending separate pages.
When you create an Event Manager rule that specifies a page operation, you specify a rule condition such as "severity greater than critical." The page operation requires the page phone number to be specified in the Page To field and the numeric message to be specified in the Numeric Message field in the Page Operation configuration page. (This page phone number will be automatically populated if the Page Setting is entered in the Preferences page.) The numeric message is unique to the condition(s) specified in the rule.
See the ICS System Manager online help for additional information about defining custom Event Manager rules.
To use the paging operation, you must also connect a modem to the console port on the SAP using a serial modem cable (also known as a Data Communications Equipment, or DCE, cable). The US Robotics 56K Fax Modem is recommended, but the paging functionality can be provided by using any modem that can save and restore its settings when the modem is powered on, as long as similar modem settings (as those used on the US Robotics modem) are set.
Note
Follow these steps to configure the paging parameters and to test the paging operation on the Cisco ICS 7750:
Step 1
Step 2
Step 3
For example, you can define the page setting to include a pager phone number and a PIN or numeric password, such as:
555-1234,,,,,,5678where 555-1234 is the phone number to which the pages should be sent, the ,,,,,, represents pauses, and 5678 indicates the PIN or numeric password.
If no PIN or numeric password is required, enter the phone number only (the pauses are not required).
Step 4
Test the paging operation to determine successful completion.
Step 5
A popup window appears.
Step 6
For example, if you enter 7750 in the popup window, you should receive a page showing 7750.
If the page does not complete successfully, check the Windows 2000 Event Viewer to determine the error messages that were received.
Step 7
Step 8
The following error messages originating from the FMMServer and CSyslogd components provide an example of the errors that might be indicated if a modem is not attached to the SAP, or if the modem is incorrectly configured:
Event Type: InformationEvent Source: FMMServerEvent Category: NoneEvent ID: 0Date: 10/30/2002Time: 11:20:15 AMUser: N/AComputer: VNT4-SPE-CM1Description:Modem State changed to DIAL_TIMEOUTEvent Type: ErrorEvent Source: CSyslogdEvent Category: NoneEvent ID: 75Date: 10/30/2002Time: 11:20:29 AMUser: N/AComputer: VNT4-SPE-CM1Description:11:20:29.625(4488): SAPageThreadInstance: DialOut() Failed,hResult = -2147467259 (0x80004005)Event Type: InformationEvent Source: FMMServerEvent Category: NoneEvent ID: 0Date: 10/30/2002Time: 11:20:29 AMUser: N/AComputer: VNT4-SPE-CM1Description:AlarmCardControl::executeCommand: failed (err=20)
Note
Validating E-Mail Settings
To validate and test the e-mail settings on the ICS System Manager Event Manager:
Step 1
where xxx.xxx.xxx.xxx is the IP address of the SPE running System Manager.
Step 2
Step 3
Step 4
•
•
•
•
Testing E-Mail Settings
To test Event Manager e-mail settings, do the following:
Step 1
where xxx.xxx.xxx.xxx is the IP address of the SPE running System Manager.
Step 2
Step 3
Step 4
If you do not receive an alert notification from the Cisco ICS 7750, it is possible that the following conditions occurred:
•
•
•
•
•
•
Additional Verification of Correct E-Mail Settings
If you completed the steps in the "Testing E-Mail Settings" section but did not receive a test e-mail within 10 minutes, follow these steps to log into the Event Viewer and verify the accuracy of the e-mail settings:
Step 1
Step 2
Step 3
Step 4
If the SMTP e-mail client software is having difficulty connecting to the e-mail server specified in the Event Manager e-mail settings, CSyslogd will log an error to the Event Viewer log.
If a change has been made to the location to which alerts are to be sent, but the notifications are still going to the previously defined location, do the following:
•
•
If you can see the correct information in the System Manager Event Manager Preferences GUI, the correct information is in the SQL table. This would indicate that the CSyslogd service was not correctly notified that the Preferences information had changed or that an error occurred when CSyslogd attempted to reread the information from the SQL table. The remedy described in this section should resolve the problem.
SNMP Basics
SNMP facilitates the exchange of management information among network devices. SNMP is part of the Transmission Control Protocol/Internet Protocol (TCP/IP) protocol suite. SNMP enables you to manage network performance, find and solve network problems, and plan for network growth.
SNMP Components
An SNMP-managed network consists of three key components: managed devices, agents, and network management systems (NMSs).
•
–
–
–
–
•
In the Cisco ICS 7750 chassis, each system card runs its own SNMP agent. The SPEs run the Microsoft SNMP agent; the SPE running System Manager also runs its own proxy agent in addition to the Microsoft SNMP agent. The proxy agent runs on the default port (port 161), while the Microsoft SNMP agent runs on a different port.
The individual agents on each system card can be queried directly by the SNMP agent, or they can be queried through the ICS System Manger proxy agent by appending the slot number to the community string.
For example, to access the Management Information Bases (MIBs) from the card in slot 5, you can direct the request to the SPE running System Manager with the following community string:
public@Slot5See the "SNMP Management Information Base" section for additional information about MIBs. See the "Understanding Community Strings" section for additional information about community strings.
•
–
–
–
Figure 1-1 shows the relationships among the managed devices and agents and the NMS.
Figure 1-1 Major Components of SNMP-Managed Networks
The following system components, though not SNMP-managed devices, receive SNMP support through ICS System Manager:
•
•
•
•
SNMP Management Information Base
A Management Information Base (MIB) is a collection of information that is organized hierarchically. MIBs are accessed by using a network management protocol such as SNMP. They comprise managed objects, which are identified by object identifiers.
A managed object (sometimes called a MIB object or an object) is one of any number of specific characteristics of a managed device. Managed objects comprise one or more object instances, which are essentially variables.
Using SNMP with MIB Variables
System MIB variables are accessible through SNMP as follows:
•
•
Instead of defining a large set of commands, SNMP places all operations in get-request, get-next-request, and set-request format. For example, an SNMP manager can get a value from an SNMP agent or store a value in that SNMP agent.
On the Cisco ICS 7750, the ICS System Manager software (the NMS) typically sends SNMP requests to a single IP address to access the SNMP MIBs of any system component. The SNMP agent can then respond to MIB-related queries sent by the NMS. Similarly, if CiscoWorks2000 is the NMS, it uses the MIB variables to set device variables and to poll devices on the network. You can then display the data that CiscoWorks2000 collects as a graph and analyze it to enhance network performance, to monitor traffic loads, or to troubleshoot problems. (See "Monitoring with CiscoWorks2000" section.)
As Figure 1-2 shows, the SNMP agent gathers data from the MIB, which is the repository for information about device parameters and network data. The agent can send traps (see "Understanding Traps" section) to the manager.
Figure 1-2 SNMP Network
The SNMP manager uses information in the MIB to perform the operations described in Table 1-6.
Supported MIBs
The Cisco ICS 7750 supports the following MIBs:
•
•
•
•
•
•
•
The supported MIBs, by individual system card, are as follows:
•
–
–
–
•
–
–
–
•
–
–
–
–
–
–
–
–
–
–
–
–
–
•
–
–
–
–
Understanding Traps
An agent can send traps to the manager that identify important system events. The following are examples of situations in which an agent might send an SNMP trap message to an NMS specified as a trap receiver:
•
•
•
When an agent detects an alarm condition, it reacts by logging information about the time, type, and severity of the condition and generates a trap—or notification message—that is sent to certain IP addresses.
Cisco ICS 7750 Traps
Cisco ICS 7750 ASI cards, MRP cards, and the SSP card can generate traps such as the following:
•
•
•
•
•
•
•
Understanding Community Strings
SNMP community strings authenticate access to MIB objects and function as embedded passwords.
The Cisco ICS 7750 uses a base community string, to which the string @SLOTnumber can be appended to form a composite community string:
base community string@SLOTnumberwhere
•
•
Note
For example, if a particular SNMP request needs to reach an MRP card in slot 3, the following composite community string could be used:
ICS7750@SLOT3where ICS7750 represents the standard system get (public) or set (changeme) community string.
Modifying the Base Community String
You can use ICS System Manager to modify the default read-only and read-write community strings. (Refer to the ICS System Manager online help.)
Monitoring with ICS System Manager
ICS System Manager monitors the Cisco ICS 7750 device information through the Monitor applet. Monitor provides information such as the following:
•
•
From the SPE running System Manager, use the Monitor tab to access the Monitor page. The Monitor page lets you view current data about the Cisco ICS 7750, by system card, such as the CPU usage and disk space available on your SPE, or the line protocol status and number of inbound and outbound packets transferred over a Fast Ethernet port on the SSP.
The Monitor page consists of three frames: the Inventory Tree, the Monitorable Fields List, and the Monitored Fields List. These frames can be resized by clicking and dragging the borders between the frames.
You can configure the Monitor page to monitor specific Cisco ICS 7750 system cards, which are organized into a hierarchical tree called the Inventory Tree. The Inventory Tree shows each slot in your Cisco ICS 7750 and the system cards that populate each slot. The Monitor applet uses SNMP MIB queries to retrieve the requested information.
Each system card has various associated inventory items. These inventory items are specific pieces of hardware, such as an Ethernet port, or specific file systems, such as the Windows 2000 system.
Each system card and inventory item has various associated fields. Each field represents one monitorable piece of data.
Note
Inventory Tree
The Inventory Tree depicts the Cisco ICS 7750 as a hierarchical tree.
The top, or root, level indicates the active SPE that is connected to the browser. This is represented by the system host name at the top of the tree. The next level of the tree depicts the system cards, and the third level of the tree depicts the inventory items associated with each system card.
To display the inventory items associated with a system card, click the triangle next to the system card. The triangle points down, indicating that the items associated with that system card are shown.
Note
To select a system card or inventory item, click it. The fields associated with the selected system card or inventory item appear in the Monitorable Fields List.
The Inventory Tree has a popup menu that you can activate by right-clicking anywhere within the frame. The popup menu has the following functions:
•
•
Monitorable Fields List
The Monitorable Fields List displays all the data fields associated with a system card or inventory item selected in the Inventory Tree. Each field represents one monitorable piece of data related to the selected inventory item. The name of the system card or inventory item to which these fields are associated is indicated by the item selected in the Inventory Tree.
To monitor a field, double-click it. Selected fields appear in bold in the Monitored Fields List, which displays the value of the field. You can select and monitor any number of fields.
Note
The Fields List contains the following two buttons:
•
•
The Fields List has a popup menu that you can activate by right-clicking anywhere within the frame. The popup menu has the following items:
•
•
•
•
•
•
Monitored Fields List
The Monitored Fields List displays the values of the fields that have been selected for monitoring.
Each row in the list represents one field. There are two columns:
•
•
The Monitored Fields List includes the following buttons:
•
•
•
•
The Monitored Fields List has a popup menu that you can activate by right-clicking one of the fields listed. The popup menu has the following options:
•
•
•
•
•
•
•
Setting Monitor Polling
Monitor fields can be automatically updated at various intervals or can be set to update only when an update is manually specified. When a field is initially added to the Monitored Fields List, it defaults to the automatic update rate specific to the field. The default automatic date rate varies for each field and can range from 10 to 60 seconds.
Follow these steps to change the update status for a field:
Step 1
Step 2
Follow these steps to change the frequency that a field is automatically updated:
Step 1
Step 2
The Field Properties Dialog Box appears.
Step 3
Step 4
Viewing Current System Data
All viewable data about your Cisco ICS 7750 is arranged logically into lists of fields, with each field representing one piece of data. For example, to view the number of octets sent out over a Fast Ethernet interface on the SSP, you would view the Outbound Octets field that is associated with that particular interface.
To select fields and view current system data:
Step 1
Step 2
Step 3
Step 4
Note
The selected fields and the current values of those fields are displayed in the Monitored Fields List.
Note
