Cisco IOS XR Getting Started Guide, Release 3.4 - Chapter 3 - Bringing Up the Cisco IOS XR Software on a Multishelf System [Cisco IOS XR Software]

Table Of Contents

Bringing Up the Cisco IOS XR Software on a Multishelf System

Contents

Prerequisites

Software Requirements

Hardware Requirements

Restrictions

Information About Bringing Up a Multishelf System

Bringup Overview

Preparing a Rack Number Plan

Configuring the External Cisco Catalyst 6509 Switches

Prerequisites

Software Requirements

Hardware Requirements

Restrictions

Before You Begin

Information About the Catalyst Switch Configuration

Configuring the Catalyst Switches

Example: Single-FCC Multishelf System Configuration

Example: Four-FCC Multishelf System Configuration

Verifying the Catalyst Switch

Verify the Interface Status

Verify Communication Between the Catalyst Switch and an LCC or FCC

Verify that the Links are Not Unidirectional

Integrated Switch System

Prerequisites for an Integrated Switch System

Software Requirements

Hardware Requirements

Restrictions for an Integrated Switch System

Before You Begin

Information About the Integrated Switch Implementation

Integrated Switch Overview

Integrated Switch Functions

Integrated Switch Control Network Topology

LED Definitions for the Integrated Switch System

Implementing the Integrated Switch System

Implementing the Integrated Switch Through ROMMON

Implementing the Integrated Switch in Cisco IOS XR

Booting Up the Integrated Switch Network

Reenabling the Ports

Verifying the Connections of the Integrated Switch Control Network

Verifying the Control Ethernet Connection

Verifying the Port Statistics

Verifying Bidirectionality

Verifying Unidirectional Link Detection (UDLD) Protocol Information

Verifying Spanning Tree Protocol Information

Bringing Up and Configuring Rack 0

Examples

Bringing Up and Verifying FCCs

Examples

Bringing Up and Verifying the Non-DSC LCC

Verifying the Spanning Tree

Examples

Verifying Fabric Cabling Connections

Where to Go Next

Bringing Up the Cisco IOS XR Software on a Multishelf System

This chapter describes how to bring up the Cisco IOS XR software on a Cisco CRS-1 Carrier Routing System Multishelf System for the first time.

Contents

This chapter contains the following sections:

•Prerequisites

•Restrictions

•Information About Bringing Up a Multishelf System

•Configuring the External Cisco Catalyst 6509 Switches

•Integrated Switch System

•Bringing Up and Configuring Rack 0

•Bringing Up and Verifying FCCs

•Bringing Up and Verifying the Non-DSC LCC

•Verifying the Spanning Tree

•Verifying Fabric Cabling Connections

•Where to Go Next

Prerequisites

The following sections describe the software and hardware requirements for bringing up a multishelf system.

Software Requirements

The multishelf system requires the following software:

•Cisco IOS XR Software Release 3.4

•ROMMON 1.40 or higher on each RP in the system

Caution The ROM Monitor software must be upgraded to version 1.42 or a later version on all RPs before a Cisco CRS-1 system is upgraded to Cisco IOS XR Software Release 3.4.0 or a later release. If the router is brought up with an incompatible version of the ROM Monitor software, then the standby RP may fail to boot. For instructions to overcome a boot block in the standby RP in a single-chassis system, see Cisco IOS XR ROM Monitor Guide. If a boot block occurs in a multishelf system, contact your Cisco Systems support representative for assistance. See Obtaining Technical Assistance.

In addition, Cisco CRS-1 multishelf systems should be upgraded to ROMMON release 1.40 before being upgraded to Cisco IOS XR Release 3.4.0 to ensure that RPs are assigned the correct rack numbers during system boot.

For more information, see Cisco IOS XR ROM Monitor Guide.

Hardware Requirements

Before you can bring up a multishelf system, the system components must be physically installed and tested. Three multishelf system configurations are supported, and they require the following components:

•Two 16-slot line card chassis containing eight FC/M (S13) fabric cards

•Two external Gigabit Ethernet Cisco Catalyst 6509 switches

Single-FCC systems require one FCC containing eight SFC (S2) fabric cards. Two-FCC systems require two FCCs, and four-FCC systems require four FCCs. In two- and four-FCC configurations, the eight SFC (S2) fabric cards are distributed equally in the FCCs.

For instructions to install, cable, and verify a multishelf system, see the documents listed on the Cisco CRS-1 documentation web page listed in the "Related Documents" section.

Restrictions

The following restrictions apply to multishelf systems in Cisco IOS XR Software Release 3.4.0.

•The multishelf system supports:

–Two 16-slot line card chassis.

–One, two, or four FCCs.

–Two external Catalyst switches to form a control Ethernet plane used for administrative management and monitoring of the system.

•The 4-slot and 8-slot LCCs are not supported.

•Although Cisco IOS XR Software Release 3.4.0 supports the addition of a second line card chassis, the removal of a line card chassis is restricted. Consult your Cisco Systems support representative for more information (see the "Obtaining Technical Assistance" section).

Information About Bringing Up a Multishelf System

The following sections provide information that is good to know before you bring up a multishelf system:

•Bringup Overview

•Preparing a Rack Number Plan

Bringup Overview

The bringup procedure for a multishelf system starts after the hardware installation is complete. The bringup procedure tasks configure the system components to work together and verify the operation and configuration of system components. To bring up the multishelf system, complete the following procedures in the sequence shown:

1. Configuring the External Cisco Catalyst 6509 Switches

2. Bringing Up and Configuring Rack 0

3. Bringing Up and Verifying FCCs

4. Bringing Up and Verifying the Non-DSC LCC

5. Verifying the Spanning Tree

During the bringup procedure, you need the information presented in the following section.

Preparing a Rack Number Plan

In a multishelf system, each chassis must be assigned a unique rack number, as shown in Figure 3-1. This rack number is used to identify a chassis in the system, and maintain the software and configurations for the chassis.

Caution Failure to assign a unique rack number to each chassis in the system can result in serious system error and potential downtime. Unique rack numbers must be assigned and committed on Rack 0 before the additional chassis are powered on and brought on line.

Figure 3-1 DSC in a CRS-1/M-F1 Multishelf System

Note Chassis, shelf, and rack are used interchangeably. Each term refers to the physical tower that contains the installed cards, power, and cooling equipment. In general, chassis describes the system components. Rack is used in software to assign a rack number to each chassis.

A rack number plan lists each chassis in a system with the correct chassis serial ID and an assigned rack number. The serial ID is the chassis serial number, which can be accessed by the software and uniquely identifies the chassis. The rack number for an LCC is a number in the range of 0 to 255, which is easier to remember and read than serial numbers in display messages.

The rack number plan is used during the startup and configuration of Rack 0. The LCC that hosts the DSC must be configured as Rack 0. The non-DSC LCC must be configured to use a rack number in the range of 1 to 255. FCC rack numbers range from F0 to F3, as shown in Table 3-1, Table 3-2, and Table 3-3.

Table 3-1 shows a sample rack number plan for a single-FCC system.

Table 3-1 Sample Rack Number Plan for a Single-FCC Multishelf System

Chassis

Serial ID

Rack Number

LCC containing the active DSC

0

Non-DSC LCC

1

Fabric chassis

F0

Table 3-2 shows a sample rack number plan for a two-FCC system.

Table 3-2 Sample Rack Number Plan for a Two-FCC Multishelf System

Chassis

Serial ID

Rack Number

LCC containing the active DSC

0

Non-DSC LCC

1

Fabric chassis 0

F0

Fabric chassis 1

F1

Table 3-3 shows a sample rack number plan for a four-FCC system.

Table 3-3 Sample Rack Number Plan for a Four-FCC Multishelf System

Chassis

Serial ID

Rack Number

LCC containing the active DSC

0

Non-DSC LCC

1

Fabric chassis 0

F0

Fabric chassis 1

F1

Fabric chassis 2

F2

Fabric chassis 3

F3

To complete the rack number plan, change the rack number for the non-DSC LCC if you want, and record the serial number for each chassis. The chassis serial number is attached to the back of the chassis, as shown in Figure 3-2 and Figure 3-3.

Figure 3-2 Location of the Serial Number on a Fabric Card Chassis

Figure 3-3 Location of the Serial Number on a Line Card Chassis

Caution Always assign a rack number to each chassis in the system before the chassis is booted. If a chassis is not assigned a rack number, or if the rack number conflicts with an existing chassis, it may not be recognized by the system or cause other operational difficulties.

If you cannot locate or read the chassis serial number on a chassis, you can view the serial number stored in software as described in the following documents:

•To display the chassis serial numbers in administration EXEC mode, see Cisco IOS XR System Management Configuration Guide, Release 3.4.

•To display the configured chassis serial numbers in administration EXEC mode, see Cisco IOS XR System Management Configuration Guide, Release 3.4.

•To display the chassis serial numbers in ROM Monitor, see Cisco IOS XR ROM Monitor Guide.

See the "Bringing Up and Configuring Rack 0" section for complete instructions to bring up a new router and configure the rack numbers.

Configuring the External Cisco Catalyst 6509 Switches

The control Ethernet network is formed by interconnecting each RP and shelf controller Gigabit Ethernet (SCGE) card in the system through two external Catalyst switches. (The SCGE card is the control card in an FCC.) The Catalyst switches are also directly connected using one or more Gigabit Ethernet links (see Figure 3-4).

These Catalyst switches must also be configured for operation with the Cisco CRS-1 multishelf router. This section includes instructions to configure and verify the Catalyst switches using the Cisco IOS software. For instructions to install and cable the Catalyst switches, see CRS-1 Multishelf System Interconnection and Cabling Guide.

This section includes the following topics:

•Prerequisites

•Restrictions

•Before You Begin

•Information About the Catalyst Switch Configuration

•Configuring the Catalyst Switches

•Verifying the Catalyst Switch

Figure 3-4 Control Ethernet Network Connections in a Single-FCC System

Prerequisites

The following sections describe the software and hardware requirements for bringing up Catalyst 6509 switches in a multishelf system.

Software Requirements

Each Cisco Catalyst 6509 switch requires the same software:

•Cisco IOS Release 12.2(14r)S9 with SUP720 Supervisor Engine controller module

•System Bootstrap (ROMMON), Version 1.3 or later

•BOOTLDR: s72033_rp Software (s72033_rp-PSV-M), Version 12.2(17d)SXB7

Both switches should use the same software. The filename of the software is 72033-psv-mz.122-17d.SXB8.bin, and is available on CCO at:

http://www.cisco.com/public/sw-center/lan/cat6000.shtml

Hardware Requirements

•Two external Cisco Catalyst 6509 switches correctly cabled to the Cisco CRS-1 multishelf router.

•The recommended hardware configuration for an AC-powered Cisco Catalyst 6509 system is shown in the following table:

Quantity

Description

Part

1

Catalyst 6509 Chassis, 9slot, 15RU, No Pow Supply, No Fan Tray

WS-C6509

1

Cisco CAT6000-SUP720 IOS IP
(see Software Requirements for complete details).

S733Z-12217SXB

1

Catalyst 6500/Cisco 7600 Supervisor 720 Fabric MSFC3 PFC3B

WS-SUP720-3B

1

Catalyst 6500 Sup720 Compact Flash Mem 256MB

MEM-C6K-CPTFL256M

1

Catalyst 6000 16-port Gig-Ethernet Mod. (Req. GBICs)

WS-X6416-GBIC

8

1000BASE-LX/LH long haul GBIC (singlemode or multimode)

WS-G5486

1

Catalyst 6509 High Speed Fan Tray

WS-C6K-9SLOT-FAN2

2

Catalyst 6000 2500W AC Power Supply

WS-CAC-2500W

2

Power Cord, 250Vac 16A, straight blade NEMA 6-20 plug, US

CAB-AC-2500W-US1

Restrictions

The following restrictions apply to Cisco Catalyst 6509 switches that are installed in a multishelf system:

•Both Catalyst switches must operate with the same Cisco IOS software release.

•The spanning tree implementation of Cisco CRS-1 control Ethernet assumes that all Catalyst switch ports connected to the multishelf system are kept in VLAN 1.

•The Gigabit Interface Converter (GBIC) transceiver module used on the Catalyst switches must match the SFP optic used on each RP and SCGE card in the system. The GBIC can be either LX/LH or SX, but the same type must be used on both ends.

Note Both Catalyst switches must be dedicated for use with the multishelf system. The Catalyst switches should not be used for any other purpose.

Before You Begin

Before you begin to bring up the Catalyst 6509 switches, consider the following:

•The Catalyst switches must be installed, including all cables properly connected between the switches and the Cisco CRS-1 router.

•See the "Related Documents" section for a hyperlink to documents on installing and connecting the Catalyst switches.

•For additional information regarding Cisco IOS commands and usage, see the "Cisco IOS Software Configuration" page at the following URL:

http://www.cisco.com/univercd/cc/td/doc/product/software/index.htm

Information About the Catalyst Switch Configuration

The configuration described in the following sections places all Catalyst ports in VLAN 1. The configuration on the Catalyst switches is the same as the configuration on the Cisco CRS-1 router internal Broadcom switches—they all participate in a Multiple Spanning Tree (MST) region with one MST instance.

The Catalyst switches are made the root of the network by assigning them the highest priority. Because there are two Catalyst switches, one is selected as the root-bridge device. Configure the primary Catalyst switch with priority 0 to make the switch the root of the network. Configure the second Catalyst switch with a number greater than 0 and less than 32768. If the primary Catalyst switch (priority 0) fails, the second switch becomes the root of the network.

Configuring the Catalyst Switches

The Cisco IOS software configuration must be applied to both external Catalyst switches.

Note Configure the primary Catalyst switch with priority 0 to make the switch the root of the network. Configure the second Catalyst switch with a number greater than 0 and less than 32768. If the primary Catalyst switch (priority 0) fails, the second switch becomes the root of the network.

To configure the Catalyst 6509 switches, use the following procedure:

SUMMARY STEPS

1. configure

2. spanning-tree portfast default

3. spanning-tree mode mst

4. spanning-tree mst configuration

5. name mst_region

6. revision number

7. instance instance_id vlan range

8. end

9. spanning-tree mst hello-time seconds

10. spanning-tree mst forward-time seconds

11. spanning-tree mst max-age seconds

12. spanning-tree mst max-hops hops

13. spanning-tree mst instance_id priority priority

14. udld aggressive

15. udld message time interval

16. interface gigabitethernet slot/port

17. switchport

18. switchport mode access

19. switchport access vlan 1

20. end

21. Repeat Step 16 through Step 20 for all interfaces.

22. Repeat all steps for the second switch.

DETAILED STEPS

Command or Action

Purpose

Step 1

configure
Example:

router# configure

Places the switch in global configuration mode.

Step 2

spanning-tree portfast default
Example:

router(config)# spanning-tree portfast default

Enables PortFast by default on all access ports.

Step 3

spanning-tree mode mst
Example:

router(config)# spanning-tree mode mst

Selects the MST mode for the Spanning Tree Protocol.

Step 4

spanning-tree mst configuration
Example:

router(config)# spanning-tree mst configuration

Places the router in spanning tree MST configuration mode.

Step 5

name mst_region
Example:

router(config-mst)# name STP_1

Defines a name for an MST region.

Step 6

revision number
Example:

router(config-mst)# revision 1

Sets a revision number for the MST configuration.

•This number must be identical on both switches.

Step 7

instance instance_id vlan range
Example:

router(config-mst)# instance 1 vlan 1

Maps the MST instance to a range of VLANs.

Step 8

end
Example:

router(config-mst)# end

Exits spanning tree MST configuration mode.

Step 9

spanning-tree mst hello-time seconds
Example:

router(config)# spanning-tree mst hello-time 1

Sets the hello-time delay timer for all instances on the switch.

•We recommend 1 second.

Step 10

spanning-tree mst forward-time seconds
Example:

router(config)# spanning-tree mst forward-time 6

Sets the forward-delay timer for all MST instances on the switch.

•We recommend 6 seconds.

Step 11

spanning-tree mst max-age seconds
Example:

router(config)# spanning-tree mst max-age 8

Sets the maximum-age timer for all MST instances on the switch.

•We recommend 8 seconds.

Step 12

spanning-tree mst max-hops hops
Example:

router(config)# spanning-tree mst max-hops 4

Specifies the number of possible hops in the region before a BPDU is discarded.

•We recommend 4 hops.

Step 13

spanning-tree mst instance_id priority priority
Example:

router(config)# spanning-tree mst 0-1 priority 28672

Sets the spanning tree priority for the switch.

•The primary Catalyst switch should be configured with priority 0. Priority 0 makes the switch the root of the network.

•The second Catalyst switch should be configured with a number greater than 0 and less than 32768. If the primary Catalyst switch (priority 0) fails, the second switch becomes the root of the network.

Step 14

udld aggressive
Example:

router(config)# udld aggressive

Enables the Unidirectional Link Detection (UDLD) protocol aggressive mode.

Step 15

udld message time interval
Example:

router(config)# udld message time 7

Configures the time between UDLD probe messages on ports that are in advertisement mode and are currently determined to be bidirectional.

•Valid values are from 7 to 90 seconds.

•We recommend 7 seconds.

Step 16

interface gigabitethernet slot/port
Example:

router(config)# interface GigabitEthernet3/1

Enters interface configuration mode for the specified interface.

Step 17

switchport
Example:

router(config-if)# switchport

Configures a LAN interface as a Layer 2 interface in preparation for additional switchport commands.

Step 18

switchport mode access
Example:

router(config-if)# switchport mode access

Specifies a nontrunking, nontagged single-VLAN Layer-2 interface.

Step 19

switchport access vlan 1
Example:

router(config-if)# switchport access vlan 1

(Optional) Assigns ports to VLAN 1, which is the default selection.

Step 20

end
Example:

router(config-if)# end

Exits interface configuration mode and returns to global configuration mode.

Step 21

Repeat Step 16 through Step 20 for all interfaces.

Configures remaining interfaces.

•Repeat this configuration for each port, including ports that are not currently being used (for example, interface gigabitethernet 0/1).

Step 22

Repeat all steps for the second switch.

Configures a second switch for redundancy.

Example: Single-FCC Multishelf System Configuration

Note When configuring the Catalyst 6509 switches, the difference between configuring single-, two-, and four-FCC multishelf systems is the number of interfaces that require configuration. When additional FCCs are present, additional interfaces must be configured for the connections to those FCCs.

First Catalyst Switch Configuration:
CAT6k-1# configure
CAT6k-1(config)# spanning-tree portfast default
CAT6k-1(config)# spanning-tree mode mst
CAT6k-1(config)# no spanning-tree optimize bpdu transmission
CAT6k-1(config)# spanning-tree mst configuration
CAT6k-1(config-mst)# name STP_1
CAT6k-1(config-mst)# revision 1
CAT6k-1(config-mst)# instance 1 vlan 1
CAT6k-1(config-mst)# end
CAT6k-1(config)# spanning-tree mst hello-time 1
CAT6k-1(config)# spanning-tree mst forward-time 6
CAT6k-1(config)# spanning-tree mst max-age 8
CAT6k-1(config)# spanning-tree mst max-hops 4
CAT6k-1(config)# spanning-tree mst 0-1 priority 0
CAT6k-1(config)# udld aggressive
CAT6k-1(config)# udld message time 7
CAT6k-1(config)# interface gigabitethernet 0/1
CAT6k-1(config-if)# switchport
CAT6k-1(config-if)# switchport mode access
CAT6k-1(config-if)# switchport access vlan 1 
CAT6k-1(config-if)# end
Second Catalyst Switch Configuration:
CAT6k-2# configure
CAT6k-2(config)# spanning-tree portfast default
CAT6k-2(config)# spanning-tree mode mst
CAT6k-2(config)# no spanning-tree optimize bpdu transmission
CAT6k-2(config)# spanning-tree mst configuration
CAT6k-2(config-mst)# name STP_1
CAT6k-2(config-mst)# revision 1
CAT6k-2(config-mst)# instance 1 vlan 1
CAT6k-2(config-mst)# end
CAT6k-2(config)# spanning-tree mst hello-time 1
CAT6k-2(config)# spanning-tree mst forward-time 6
CAT6k-2(config)# spanning-tree mst max-age 8
CAT6k-2(config)# spanning-tree mst max-hops 4
CAT6k-2(config)# spanning-tree mst 0-1 priority 28672
CAT6k-2(config)# udld aggressive
CAT6k-2(config)# udld message time 7
CAT6k-2(config)# interface gigabitethernet 0/1
CAT6k-2(config-if)# switchport
CAT6k-2(config-if)# switchport mode access
CAT6k-2(config-if)# switchport access vlan 1 
CAT6k-2(config-if)# end
Example: Four-FCC Multishelf System Configuration

Note When configuring the Catalyst 6509 switches, the difference between configuring single-, two-, and four-FCC multishelf systems is the number of interfaces that require configuration. When additional FCCs are present, additional interfaces must be configured for the connections to those FCCs.

The following configuration display shows an example configuration for one of the Catalyst 6509 Switches in a four-FCC multishelf system:
Router# show running-config 
Building configuration...
Current configuration : 2873 bytes
!
version 12.2
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
service counters max age 10
!
hostname Router
!
logging snmp-authfail
!
ip subnet-zero
!
!
!
mpls ldp logging neighbor-changes
no mls flow ip
no mls flow ipv6
mls cef error action freeze
!
power redundancy-mode combined
!         
spanning-tree mode mst
spanning-tree portfast default
no spanning-tree optimize bpdu transmission
spanning-tree extend system-id
!
spanning-tree mst configuration
 name STP_1
 revision 1
 instance 1 vlan 1
!
spanning-tree mst hello-time 1
spanning-tree mst forward-time 6
spanning-tree mst max-age 8
spanning-tree mst 0-1 priority 28672
diagnostic cns publish cisco.cns.device.diag_results
diagnostic cns subscribe cisco.cns.device.diag_commands
!
redundancy
 mode sso
 main-cpu
  auto-sync running-config
  auto-sync standard
!         
vlan internal allocation policy ascending
vlan access-log ratelimit 2000
!
!
interface GigabitEthernet1/1
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/2
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/3
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/4
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/5
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/6
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/7
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!         
interface GigabitEthernet1/8
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/9
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/10
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/11
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/12
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/13
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/14
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/15
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet1/16
 no ip address
 switchport
 switchport mode access
 spanning-tree portfast
!
interface GigabitEthernet5/1
 no ip address
 shutdown
!
interface GigabitEthernet5/2
 no ip address
 shutdown
!
interface Vlan1
 no ip address
 shutdown
!
ip classless
no ip http server
!         
!
!
!
!
line con 0
line vty 0 4
 login
!
End
Verifying the Catalyst Switch

As each rack in the multishelf system is brought up, verify that the Catalyst switch links are operating correctly by completing the tasks in the following sections:

•Verify the Interface Status

•Verify Communication Between the Catalyst Switch and an LCC or FCC

•Verify that the Links are Not Unidirectional

Verify the Interface Status

To verify that the interfaces are connected, enter the command show interfaces status. Enter the command on a terminal connected to each Catalyst switch.
CAT6k-1# show interfaces status 
Port Name Status Vlan Duplex Speed Type
Gi1/1 connected 1 full 1000 1000BaseLH
Gi1/2 connected 1 full 1000 1000BaseLH
Gi1/3 connected 1 full 1000 1000BaseLH
Gi1/4 connected 1 full 1000 1000BaseLH
Gi1/5 connected 1 full 1000 1000BaseLH
Gi1/6 connected 1 full 1000 1000BaseLH
Gi1/7 disabled 1 full 1000 1000BaseSX
Gi1/8 disabled 1 full 1000 1000BaseSX
Gi1/9 disabled 1 full 1000 1000BaseSX
Gi1/10 disabled 1 full 1000 1000BaseSX
Gi1/11 disabled 1 full 1000 1000BaseSX
Gi1/12 disabled 1 full 1000 1000BaseSX
Gi1/13 disabled 1 full 1000 1000BaseSX
Gi1/14 disabled 1 full 1000 1000BaseSX
Gi1/15 disabled 1 full 1000 1000BaseSX
Gi1/16 disabled 1 full 1000 1000BaseSX
Gi5/1 disabled 1 full 1000 1000BaseSX
Gi5/2 connected routed a-full a-100 10/100/1000BaseT
CAT6k-1# 
Verify Communication Between the Catalyst Switch and an LCC or FCC

To verify that the Catalyst switch is communicating with an LCC or FCC in forwarding mode, enter the command show spanning tree.

This command displays the states of the spanning tree ports. Verify that the ports used to connect the DSC, remote LCC RP, and FCC SCGE are in the FWDG state.

The listed interfaces should include the port to which you have connected. If the port is not listed, contact Cisco Technical Support. For contact information, see the "Obtaining Technical Assistance" section.
CAT6k-1# show spanning-tree 
MST00
Spanning tree enabled protocol mstp
Root ID Priority 0
Address 0013.1a4f.75c0
This bridge is the root
Hello Time 1 sec Max Age 8 sec Forward Delay 6 sec
Bridge ID Priority 0 (priority 0 sys-id-ext 0)
Address 0013.1a4f.75c0
Hello Time 1 sec Max Age 8 sec Forward Delay 6 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Gi1/1 Desg FWD 20000 128.1 P2p 
Gi1/2 Desg FWD 20000 128.2 P2p 
Gi1/3 Desg FWD 20000 128.3 P2p 
Gi1/4 Desg FWD 20000 128.4 P2p 
Gi1/5 Desg FWD 20000 128.5 P2p 
Gi1/6 Desg FWD 20000 128.6 P2p 
MST01
Spanning tree enabled protocol mstp
Root ID Priority 1
Address 0013.1a4f.75c0
This bridge is the root
Hello Time 1 sec Max Age 8 sec Forward Delay 6 sec
Bridge ID Priority 1 (priority 0 sys-id-ext 1)
Address 0013.1a4f.75c0
Hello Time 1 sec Max Age 8 sec Forward Delay 6 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Gi1/1 Desg FWD 20000 128.1 P2p 
Gi1/2 Desg FWD 20000 128.2 P2p 
Gi1/3 Desg FWD 20000 128.3 P2p 
Gi1/4 Desg FWD 20000 128.4 P2p 
Gi1/5 Desg FWD 20000 128.5 P2p 
Gi1/6 Desg FWD 20000 128.6 P2p 
CAT6k-1#
Verify that the Links are Not Unidirectional

After an LCC or FCC is brought up, verify that the Catalyst links are operating correctly. If a link has a partial fiber cut or a bad optic, the control Ethernet network can become unidirectional and cause a loop.

To verify the links in a Catalyst switch using the Cisco IOS software, enter the command show interface in EXEC mode.

In the following example, the command is entered for a specific port. The keywords | inc Gig narrows the output to Gigabit Ethernet ports.
Router# show interface gi 6/1 | inc Gig
GigabitEthernet6/1 is up, line protocol is up (connected)
Router#
The output of this command should display "connected." If it does not, then the connector may have a partial fiber cut or a bad optic. You may need to jiggle the GBIC wire to ensure that it is firmly inserted. Re-enter the command show interface until the port displays a status of "connected" or "disabled" for every port that displays a connector type.

Caution If this problem is not resolved and the Cisco CRS-1 router enters the forwarding state, a loop occurs.

Integrated Switch System

Integrated switches are two Gigabit Ethernet switches placed on system controller cards in the fabric chassis. The system controller card is called 22-port shelf controller Gigabit Ethernet (22-port SCGE), because it contains 22 ports on the front panel.

Each 22-port SCGE card provides 22 Gigabit Ethernet (GE) links, which are used to interconnect control network connections of the different Cisco CRS-1 chassis.

Note When the 22-port SCGE cards are installed, there is no requirement for an external switch device (Catalyst 6509) to provide control Ethernet connectivity.

For information about the cabling schemes for a single-FCC multishelf system, two-FCC multishelf system, and four-FCC multishelf system, see Cisco CRS-1 Carrier Routing System Multishelf System Interconnection and Cabling Guide.

This section includes the following topics:

•Prerequisites for an Integrated Switch System

•Restrictions for an Integrated Switch System

•Before You Begin

•Information About the Integrated Switch Implementation

•Implementing the Integrated Switch System

Prerequisites for an Integrated Switch System

The following prerequisites are presented for the integrated switch system:

•Software Requirements

•Hardware Requirements

Software Requirements

•Requires ROMMON 1.43 or higher on all RP and 22-port SCGE nodes. The 22-port SCGE card comes with ROMMON 1.43 or later version.

Note ROMMON 1.43 is the first ROMMON version to support 22-port SCGE cards and other cards.

•Requires Cisco IOS XR Software Release 3.4.1 to support 22-port SCGE cards.

Hardware Requirements

Route processors (RPs) should be revision 8 or higher. SMF cables are required and LX optics is recommended.

Restrictions for an Integrated Switch System

A mixture of 22-port SCGE and Cisco Catalyst 6K is not a supported configutation (except during migration procedures). For information about migration procedures, see Cisco CRS-1 Carrier Routing System Multishelf System Upgrade and Conversion Guide.

Before You Begin

Before you begin to bring up the integrated switch control network, consider the following items:

•See the "Related Documents" section for a hyperlink to documents about installing and connecting Catalyst switches.

•For additional information regarding Cisco IOS commands and usage, see the "Cisco IOS Software Configuration" page at the following URL:

http://www.cisco.com/univercd/cc/td/doc/product/software/index.htm

Information About the Integrated Switch Implementation

To implement the integrated switch, you must understand the following concepts:

•Integrated Switch Overview

•Integrated Switch Functions

•Integrated Switch Control Network Topology

•LED Definitions for the Integrated Switch System

Integrated Switch Overview

Four switches are present on the 22-port SCGE card. Two switches provide connectivity to all cards inside the chassis. Two more Gigabit Ethernet (GE) switches on the board allow for all the external connections.

Table 3-4 lists the differences between intra-rack switch and inter-rack switch for the 22-port SCGE card.

Table 3-4 Differences Between Intra-Rack and Inter-Rack

Rack Type

Description

Intra-Rack Switch

Provides connectivity inside the rack through FE ports. These switches are similar to the RPs.

Note The GE1 link on intra-rack switches on the 22-port SCGE card is not connected.

Inter-Rack Switch

Provides connectivity between the racks.

Integrated Switch Functions

The 22-port SCGE performs the following functions:

•Arbitrate for shelf ownership (active mode or standby mode) with the other 22-port SCGE card that is installed on the rack.

•Provide 22 GE ports for external system management communication across racks.

•Performs the same type of control plane functions as the current SCGE for the multichassis system.

•Performs the same type of control plane management functions as the Catalyst 6509. For example, 22-port SCGE card provides physical GE ports that allow control plane traffic to be distributed throughout the fabric and line card chassis.

•Supports 22 GE ports and has the capability to support up to 72 line card chassis.

•Allows you to perform a hitless migration from an external Catalyst 6509 switch based network to an internal 22-port SCGE based network with no loss of user or control traffic. For more information, see Cisco CRS-1 Carrier Routing System Multishelf System Upgrade and Conversion Guide.

•Validates the link state before running the Spanning Tree Protocol (STP) that prevents a link from being unidirectional and causes a spanning tree loop.

•Checks the link for any unidirectional mode when a cable is plugged into the 22-port SCGE. If the 22-port SCGE software detects this condition, the port cannot be allowed to participate in the spanning tree algorithm.

In active mode, you can perform the following functions:

•Download the Ethernet MAC addresses from the backplane EPROM and assign them to all boards in the rack.

•Start up and monitor power supplies, rack fans, and thermal sensors within the fabric rack upon request from the system management network.

•Start up board power supplies, download software images to the fabric cards in the rack upon request from the system management network. Start and reset board processors.

•Send alarms, reset, and shut down portions of the rack hardware in case of abnormal or dangerous conditions in the rack.

•Keep a log of the 22-port SCGE cards and rack activity on nonvolatile memory. Take core dumps onto the hard disk.

•Initiate a self-reset and rearbitration for shelf ownership in case of a watchdog timeout.

•Control and monitor the fan speed.

In standby mode, you can perform the following functions:

•Test the FE links to all the rack hardware periodically.

•Keep the local state information synchronized to the rack master.

•Rearbitrate the shelf ownership if the primary router releases ownership.

Integrated Switch Control Network Topology

Once the 22-port SCGE cards are installed, the control network topology ceases to be a simple hub-and-spoke set of connections.

A control network topology provides the following functions:

•Each RP in a line card chassis is connected to two different 22-port SCGE cards in a fabric chassis.

•The 22-port SCGE cards are interconnected in a full mesh to provide an available control network with multiple redundant Ethernet connections.

•The 22-port SCGE cards appear to be a backbone in which different RPs are connected from the outside.

•Both the 22-port SCGE cards and RPs run the rapid spanning tree protocol (RSTP) to provide a loop-free topology.

LED Definitions for the Integrated Switch System

The 22-port SCGE displays the LEDs on the front panel foe every port. Table 3-5 lists the LEDs that are used to obtain information about the link.

Table 3-5 LEDs for the Integrated Switch

LED

Description

Green

Link Up

Blinking Green

Activity

Amber

Port error disabled Unidirectional Link Detection (UDLD)

Off

Link Down

Implementing the Integrated Switch System

This section presents topics on how to implement the integrated switch system:

•Implementing the Integrated Switch Through ROMMON

•Implementing the Integrated Switch in Cisco IOS XR

•Booting Up the Integrated Switch Network

•Reenabling the Ports

Implementing the Integrated Switch Through ROMMON

When the 22-port SCGE comes out of reset, ROMMON must initialize the switches so that no loops get formed and the processor can communicate with the rest of the system. ROMMON configures the switches.

Table 3-6 lists the ROMMON switch configuration.

Table 3-6 ROMMON Switch Configuration

Type

Description

Switch connections

The BCM5618s create the intra-rack control network. The BCM5690s create the inter-rack control network. There are two BCM5690 switches that are connected through a 10 Gbps stacking link. Port 11 on BCM5690-sw1 is connected to BCM5618-sw0-GE, which is also referred to as BCM5618-GE0. This is the link that connects the intra-rack control network to the inter-rack control network. Port 0 on BCM5618-sw0 is connected to the control Ethernet port for the CPU.

Port configuration

Both BCM5690 switches are configured to forward traffic only to port 11 of BCM5690-sw1 (for example, the CPU bound port). Because forwarding is not enabled between any other ports on those switches, the switches can never participate in a loop. The BCM5618s are configured in a very similar manner by enabling forwarding to only port 0 (CPU port) of BCM5618-sw0 to or from any other port.

Implementing the Integrated Switch in Cisco IOS XR

When the RP and SCGE node boots to Cisco IOS XR software, rapid spanning tree protocol (RSTP) starts to run on that node. On the RP, the RSTP configures the state of the 2-GE and inter-RP (backplane) FE port. RSTP runs on all ports of inter-rack switches in addition to the intra-rack switch ports.

Assigning a Bridge Priority

The switches on the 22-port SCGE, which are connected to each other, form the core of the network. The RP connections form the edge (regardless of whether the 22-port SCGEs are connected in a full or partial mesh). In ste