Last updated: January 2008
The Service Diagnostics feature provides a bundled set of Tool Command Language (Tcl) scripts and Embedded Event Manager (EEM) policies written and tested by
subject matter experts to facilitate diagnosing common networking issues in the areas
of Border Gateway Protocol (BGP), Open Shortest Path First (OSPF), and Quality of Service (QoS).
Definitions
Introduction to Service Diagnostics
Deploying Service Diagnostic Scenarios via tclsh Parser Mode
Installing Scripts
1. Download the diagnostic scripts from Cisco Beyond Website: http://www.cisco.com/go/ciscobeyond
Scripts will be organized into diagnostic scenarios, i.e., sdiag_bgp_1.0.zip
2. Unzip the scenario archive.
3. On the router's disk create a directory to store all the scripts.
4. Example: Router# mkdir svc-diag
5. The files to be copied to the dir svc-diag (per scenario) are:
BGP
• sdiag_bgp.tcl
• bgp_neighbor_loss.tcl
• bgp_neighbor_formation.tcl
• bgp_neighbor_route.tcl
• sdiag_bgp_display.tcl
• sdiag_bgp_undeploy.tcl
• collectEmailParameters.tcl
OSPF
• ospf_adj_duplicate_rid.tcl
• ospf_intf_down_detached.tcl
• ospf_miss_area_id.tcl
• ospf_stuck_cases.tcl
• ospf_timer_expired.tcl
• sdiag_ospf.tcl
• sdiag_ospf_display.tcl
• sdiag_ospf_undeploy.tcl
QOS
• sdiag_qos.tcl
• sdiag_qos_display.tcl
• sdiag_qos_undeploy.tcl
• qos_drop_packet.tcl
6. Copy tftp disk#:/svc-diag
7. The router will prompt for :-
• Address or name of remote host [ ] ?
• Source filename [ ]? Give the full path where the script has to be copied from
• Destination filename [ ]? Give the path as disk#:/svc-diag/filename.tcl
8. Create a directory to store the user libraries and the template to send email.
Example: Router* mkdir user_library
BGP
• email_template
• userlib.tcl
• diag_lib.tcl
OSPF
• email_template_cmd
• email_setup.tcl
• diag_lib.tcl
• tclIndex
• lib.tcl
• ospf_lib.tcl
• acl_lib.tcl
QOS
• email_template_cmd
• email_setup.tcl
• diag_lib.tcl
• tclIndex
• lib.tcl
Note: The Diagnostic Tcl scripts posted on Cisco Beyond will be digitally signed. If your Cisco IOS router image supports Tcl script signing, you may use this feature to verify the integrity of the downloaded scripts. For more information, please see the topic "Signed Tcl Scripts" here:
Deploying Scenarios
• The value for notification can be "email or syslog or all"
• The value for configuration history option, event history option can be "TRUE or FALSE"
• The value for user policy and user library directories is the respective full path where the scripts and library files are stored
BGP Neighbor Loss
• sdiag_bgp.tcl is the registration script
• bgp_neighbor_loss.tcl is the EEM policy script
• The notification is email
• The configuration history option is set to TRUE
• The event history option is set to TRUE
• The user policy directory is disk2:
• The user library directory is disk2:/user_lib
BGP Neighbor Formation
• sdiag_bgp.tcl is the registration script
• bgp_neighbor_formation.tcl is the EEM policy script
• The notification is email
• The configuration history option is set to TRUE
• The event history option is set to TRUE
• The user policy directory is disk2:
• The user library directory is disk2:/user_lib
• The timer value is 120
BGP Neighbor Route
Router#tclsh disk#:sdiag_bgp.tcl bgp_neighbor_route.tcl email TRUE TRUE disk#: disk#:/user_lib 120
• sdiag_bgp.tcl is the registration script
• bgp_neighbor_route.tcl is the EEM policy script
• The notification is email
• The configuration history option is set to TRUE
• The event history option is set to TRUE
• The user policy directory is disk2:
• The user library directory is disk2:/user_lib
• The timer value is 120
QoS
OSPF Stuck Cases
OSPF Timer Expired
Displaying Inputs
BGP
• sdiag_bgp_display.tcl is the script that displays the BGP inputs on the router related to the BGP- EEM policy script passed as the argument
• bgp_neighbor_loss.tcl is the BGP neighbor loss policy script
• sdiag_bgp_display.tcl is the script that displays the BGP inputs on the router related to the BGP- EEM policy script passed as the argument
• bgp_neighbor_formation.tcl is the BGP neighbor formation policy script
• sdiag_bgp_display.tcl is the script that displays the BGP inputs on the router related to the BGP- EEM policy script passed as the argument
• bgp_neighbor_route.tcl is the BGP route policy script
Removing Scenarios
BGP
• sdiag_bgp_undeploy.tcl is the script that removes the BGP inputs on the router related to the BGP- EEM policy script passed as the argument
• bgp_neighbor_loss.tcl is the BGP neighbor loss policy script
• sdiag_bgp_undeploy.tcl is the script that removes the BGP inputs on the router related to the BGP- EEM policy script passed as the argument
• bgp_neighbor_formation.tcl is the BGP neighbor formation policy script
• sdiag_bgp_undeploy.tcl is the script that removes the BGP inputs on the router related to the BGP- EEM policy script passed as the argument
• bgp_neighbor_route.tcl is the BGP neighbor formation policy script
Router#tclsh disk#:/svc-diag/sdiag_qos_undeploy.tcl qos_drop_packet.tcl
Router#tclsh disk#:/svc-diag/sdiag_ospf_undeploy.tcl ospf_stuck_cases.tcl
BGP Diagnostics
BGP Neighbor Loss
Triggers/Symptoms
1. The interface on the neighbor or the local router is shut.
2. Access-list is configured on the local or the neighbor router that is explicitly or implicitly denying the tcp, ip or udp packets from the other.
3. Any BGP configuration on the neighbor router is removed.
Actions/Outputs
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR LOSS IS:
Neighbor x.x.x.x has gone down and does not seem to be reachable through ping. Check network connectivity.
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR LOSS IS:
Neighbor x.x.x.x has gone down. However, configured neighbor is reachable through ping. Check BGP configuration at the peer and any access-list restrictions between the peers.
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR LOSS IS:
Neighbor x.x.x.x has gone down and does not seem to be reachable through ping. Check network connectivity, BGP configuration at peer and any access-list restrictions between the peers.
BGP Neighbor Formation Problem
Triggers/Symptoms
1. The interface on the local or neighbor router is down.
2. Access-list is configured on the local or the neighbor router that is explicitly or implicitly denying the tcp, ip or udp packets from the other.
3. For iBGP neighbor "update-source Loopback" is not configured OR is misconfigured on the local or neighbor router.
4. For eBGP neighbor "update-source Loopback" or "ebgp-multihop is not configured OR is misconfigured on the local or neighbor router.
5. For indirectly connected eBGP or iBGP neighbors the static route or ip routing between the peers may be missing.
Actions/Outputs
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR FORMATION IS:
Neighbor x.x.x.x does not seem to be reachable through ping. Check network connectivity, BGP configuration at the peer and any access list restrictions between the peers.
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR FORMATION IS:
Neighbor x.x.x.x is reachable through ping. Check BGP configuration at the peer and any access list restrictions between the peers.
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR FORMATION IS:
Neighbor x.x.x.x does not seem to be reachable through ping. Check network connectivity and routing between peers.
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR FORMATION IS:
Check BGP configuration and routing at the peer and any Access list restrictions between the peers.
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR FORMATION IS:
The configuration 'neighbor x.x.x.x update-source Loopback' may be required on the local router. The configuration 'neighbor x.x.x.x ebgp-multihop' may be required on the local router.
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR FORMATION IS:
The configuration 'neighbor x.x.x.x update-source Loopback' may be required on the local router. The configuration 'neighbor x.x.x.x ebgp-multihop' may be required on the local router.
BGP Route Problem
Triggers/Symptoms
1. In the BGP, configuration synchronization is turned on (local router).
2. The configuration "next-hop-self" command is missing on the local router.
3. There is a mismatch in the prefix mask between the BGP configuration and the routing table.
4. All the routes in the bgp table are not valid and are not best path routes.
Actions/Outputs
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP ROUTE PROBLEM IS:
There is a mismatch between prefix mask in the IP routing table and BGP configuration network x.x.x.x mask x.x.x.x.
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP ROUTE PROBLEM IS:
Synchronization is turned on in the BGP configuration. Prefixes might not be installed on the Routing table if they are not known through IGP.
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP ROUTE PROBLEM IS:
BGP Next-hop x.x.x.x of bgp prefix ix.x.x.x/y is not reachable as ping fails; next-hop-self need to be configured in the bgp configuration of the router.
THE SERVICE DIAGNOSTIC MESSAGE FOR BGP ROUTE PROBLEM IS:
THE ROUTES: x.x.x.x which is not marked as best path prefixes in the BGP table are not installed in the routing table and so will not be advertised to peer.
QoS Diagnostics
Triggers/Symptoms
Actions/Outputs
THE SERVICE DIAGNOSTIC MESSAGE FOR QOS DROP COUNTER is:
Interface gi0/0, service policy direction "input" on policymap "qospol1":
New drop during these 15 minutes for classmap qosc1 is 732087551, its match statement is
"protocol ip"
--------------------------------------------------------------
Email result will not be sent, please look at console or buffer on service diagnostic messages.
--------------------------------------------------------------
Interface gi0/0, service policy direction "output" on policymap "qospol10":
• This first classmap qosc1 is a classmap in interface giga0/0 on input direction.
• The second classmap qosc1 is a classmap in interface giga0/0 on output direction.
OSPF Diagnostics
OSPF Neighbor Loss
Triggers/Symptoms
1. ospf_intf_down_detached.tcl
This policy checks the syslog message for interface down and detached.
%OSPF-5-ADJCHG:.*DOWN, Neighbor Down: Interface down or detached"
The possible reason can be:
• The interface shut in its own router is configured (admin down)
• An interface shut in remote end is configured and near end has a Down Down
• Passive-interface is configured.
2. ospf_miss_area_id.tcl
The policy checks the syslog error message for miss area id:
%OSPF-4-ERRRCV: Received invalid packet: mismatch area ID"
It gives addition information to explain additionally on the message and give what the user has configured in the router.
3. ospf_timer_expired.tcl
The policy checks syslog message for timer_expired:
%OSPF-5-ADJCHG:.*DOWN, Neighbor Down: Dead timer expired
The possible reason can be:
• Remote end shut in Ethernet interface.
Please note, when remote end in Fastethernet interface, the syslog message is not Dead timer expired. Instead it is
"\%OSPF-5-ADJCHG:.*DOWN, Neighbor Down: Interface down or detached"
• ip address list is configured.
• ospf is removed from the remote end.
Actions/Outputs
OSPF Neighbor Formation Problem
Triggers/Symptoms
1. ospf_adj_duplicate_rid.tcl
The script detects the syslog message:
and gives addition information for which kind of case can cause the duplicate router id.
2. ospf_stuck_cases.tcl
The script is a timer based script to check if the router is stuck at one of the following states:
• Stuck at attempt state, which can be caused by wrong neighbor configuration.
• Stuck at init state, can be caused by the access list in the remote end side. The script can provide a suggestion for this case.
• Stuck at exstart/exchange state, can be caused by an mtu problem.
Actions/Outputs
For duplicate router rid is:
THE SERVICE DIAG MESSAGE FOR ADJ DUPLICATE RID is:
The message happens when two routers are configured with the same router id. Check the router ids on them to make sure they have individual router ids and restart the ospf protocol by doing "no router ospf <ospf_id>" and "router ospf <ospf_id>" in configuration mode or "clear ip ospf <process_id> process" in super user mode.
--------------------------------------------------------------
*Nov 15 09:29:30.635: %HA_EM-6-LOG: tmpsys:/eem_policy/ospf_adj_duplicate_rid.tcl:
Email result will not be sent, please look at console or buffer on service diagnostic messages.
--------------------------------------------------------------
For stuck at INIT:
THE SERVICE DIAG MESSAGE FOR STUCK CASES is:
OSPF neighbor 131.107.1.2 is stuck at INIT might be due to access list on remote end blocking OSPF hellos or authentication config is present on one side, Please check the access-list or enable authentication on both sides.
--------------------------------------------------------------
*Nov 27 10:55:07.067: %HA_EM-6-LOG: tmpsys:/eem_policy/ospf_stuck_cases.tcl:
Email result will not be sent, please look at console or buffer on service diagnostic messages.
--------------------------------------------------------------
For stuck at EXCHANGE:
THE SERVICE DIAG MESSAGE FOR STUCK CASES is:
OSPF neighbor 131.107.1.2 is stuck at EXCHANGE might be due to unmatched mtu, the stuck interface GigabitEthernet0/0 has mtu value 3456. Please check mtu value in remote side to make sure they are synchronized.
--------------------------------------------------------------
*Nov 14 15:29:27.559: %HA_EM-6-LOG: tmpsys:/eem_policy/ospf_stuck_cases.tcl:
Email result will not be sent, please look at console or buffer on service diagnostic messages.
--------------------------------------------------------------
For stuck at ATTEMPT:
THE SERVICE DIAG MESSAGE FOR STUCK CASES is:
Stuck at ATTEMPT might be due to wrong neighbor configuration. The neighbor address 131.107.5.11 might not exist. Please check the correct one and reconfigure it.
--------------------------------------------------------------
*Nov 27 10:43:18.083: %HA_EM-6-LOG: tmpsys:/eem_policy/ospf_stuck_cases.tcl:
Email result will not be sent, please look at console or buffer on service diagnostic messages.
1. Sometimes there is an EEM Tcl Error which is due to email_template not being restaged. Since this issue does not have definite steps to be reproduced the workaround for this issue is to do the following on the router:
2. All the BGP scripts will work only with IPv4.