Document ID: 6933
Contents
Introduction
Before You Begin
Conventions
Prerequisites
Components Used
Background
Task Detail
Stage 1: Planning
Stage 2: Network Preparation
Stage 3: The Upgrade
Appendix A - Network Health Check
Related Information
Introduction
This document describes the Cisco recommended 23 point process for a successful MGX 8250 Edge Concentrator or MGX 8850 Edge Switch ungraceful upgrade using the PXM-1. The use of MGX 8850 throughout the document also applies to the MGX 8250. This document does not address the MGX 8850 PXM-45 and should not be used as an upgrade reference.
Before You Begin
Conventions
For more information on document conventions, see the Cisco Technical Tips Conventions.
Prerequisites
Ungraceful upgrades cause service disruption and are required when upgrading:
-
To an incompatible firmware version
-
A non-redundant MGX 8850, single Processor Switch Module (PXM).
The tasks listed in this document are recommended for non redundant MGX 8850 firmware upgrades. The tasks were verified in the order shown in a laboratory test of a non redundant MGX 8850 upgrade from release 1.0.00 to release 1.1.23. This document lists the minimum required steps, and then addresses each step in some detail. The service disrupting tasks are highlighted and may result in a service outage in excess of 1 hour. The MGX 8850 and MGX 8250 is based on the same platform as the MGX 8220 and it is recommended that the MGX 8220 Upgrade and Downgrade Matrices, Concepts and Definitions be reviewed to familiarize the reader with general upgrade concepts. The screen displays used to illustrate the tasks were taken from laboratory equipment and are in no way intended to specify IP addressing or naming schemes.
This document is intended to be used as an aid for conducting successful firmware upgrades, but is not a substitute for proper planning with your Cisco Sales Engineer, Systems Engineer, or Account Manager.
Note: This document is updated with information for an ungraceful upgrade for an MGX 8250 with a single PXM-1 from version 1.1.23 to1.1.25. The updated tasks are:
-
13
-
16
-
17
Tasks 14 and 15 are not used for an ungraceful upgrade from 1.1.23 to1.1.25 as those functions are performed in tasks 13 and 17. All other tasks do not change for an ungraceful upgrade from 1.1.23 to 1.1.25.
Caution:
-
Only one image must be loaded onto the PXM per TFTP session.
-
Multiple TFTP sessions are required to load boot code and firmware images onto a PXM.
-
If multiple firmware images are loaded in one TFTP session, all files copied after the initial image will be corrupted.
Components Used
This document is not restricted to specific software and hardware versions.
Background
This section explains IP addressing on the MGX 8850 shelf in general. There are three separate IP addresses for on MGX 8850 shelf with two PXMs.
-
one cnifip IP address
-
two bootChange IP addresses
The cnifip address is the live IP Address of the Active PXM Ethernet port on the MGX 8850. It is the IP address used to manage the MGX 8850 shelf. If a switchcc occurs, the new MAC address of the Standby PXM card is automatically broadcast out and takes over the cnfifip IP address. To verify the existing IP address issue the dspifip command. The dspifip output also displays the ATM and SLIP addresses assigned to the MGX 8850 shelf. The ATM address is used for inband IP routing (NWIP) management of the MGX 8850 shelf. The SLIP address is a legacy assigned to the MGX 8850. The SLIP interface does not support statistics collection. The cnfifip and bootChange IP addresses are retained after the clrallcnf command is issued.
bootChange is a Service-level command that is used as necessary for MGX 8850 bring up when the PXMs have no run time firmware. The bootChange IP address should be different than the cnfifip IP address. The bootChange IP address of the Active PXM should also be different than the bootChange IP address of the Standby PXM. The bootChange IP address is active only when the PXM is in the boot mode and is used to FTP firmware and the bootcode directly into itn PXM. Refer to Bringing up the PXM with no runtime firmware for more information. Once the PXM is booted up, the cnfifip IP address is active. The bootChange gateway address specifies the next hop that allows the shelf to communicate with a laptop (PC) or Cisco WAN Manager (SV+) station on a different LAN segment while the MGX 8850 is in the boot mode. To view the bootChange IP address of the Active PXM when the MGX 8850 shelf is using runtime firmware, issue the version command.
sj_core.1.7.PXM.a > bootChange
'.' = clear field; '-' = go to previous field; ^D = quit
boot device : lnPci
processor number : 0
host name : solwandbg1
file name :
inet on ethernet (e) : 10.1.2.15:ffffff00
inet on backplane (b):
host inet (h) :
gateway inet (g) : 10.1.1.1
user (u) : autoprog
ftp password (pw) (blank = use rsh):
flags (f) : 0x0
target name (tn) : pxm-7
startup script (s) :
other (o) :
sj_core.1.7.PXM.a > dspifip
Interface Flag IP Address Subnetmask Broadcast Addr
--------------- ---- --------------- --------------- ---------------
Ethernet/lnPci0 UP 10.1.2.44 255.255.255.0 10.1.1.1
SLIP/sl0 DOWN 0.0.0.0 255.255.255.0 (N/A)
ATM/atm0 DOWN 0.0.0.0 255.255.255.0 0.0.0.0
sj_core.1.7.PXM.a >
Task Detail
Stage 1: Planning
The following summarizes the planning steps that are necessary for a successful upgrade. All steps should be completed irrespective of network size.
-
Evaluate known anomalies in the selected release.
Some anomalies may require additional preparation in order to ensure a smooth upgrade. This may mean:
-
Additional upgrade steps
-
Parameter changes
-
Workarounds
-
-
Review release notes for upgrade steps specific to this release.
As in Task 1, this task may result in:
-
Additional upgrade steps
-
Parameter changes
-
Workarounds
-
-
Write scripts, which is an optional task to aid the parameter changes required in certain sections of Stage 3.
Writing and testing scripts will:
-
Make the parameter change process easier to execute
-
Highlight any commands that have changed in the new firmware release.
There are various products that can be used to aid in setting parameters in preparation for a network upgrade.
-
Stage 2: Network Preparation
The following summarizes the network preparation steps that are necessary for a successful upgrade. All steps should be completed irrespective of network size.
Note: This stage needs to be completed one week before firmware upgrade.
-
Network health check.
See Appendix A.
-
Monitor network closely until time of upgrade.
Step 1 should highlight any existing network issues, but it is prudent to monitor the network for new firmware errors and card errors right up to the time of the upgrade. Report recurring errors to Cisco TAC.
See Appendix A for details on checking for firmware errors and card errors.
-
Verify network management connectivity to network nodes.
Ensure that every network MGX 8850 shelf can be connected to using Out of Band access. Using TELNET, connect to each MGX 8850 in the network.
-
Verify bootChange address configuration.
Use the Service level bootChange command to assign a unique IP address to each PXM in the MGX 8850 shelf. The bootChange IP address is used to load runtime firmware onto a PXM. The bootChange IP address must also be different from the IP address assigned to the MGX 8850 shelf using the cnfifip command.
jet.1.7.PXM.a > bootChange '.' = clear field; '-' = go to previous field; ^D = quit boot device : lnPci processor number : 0 host name : solwandbg1 file name : inet on ethernet (e) : 192.168.1.65:ffffff00 inet on backplane (b): host inet (h) : gateway inet (g) : 192.168.1.1 user (u) : autoprog ftp password (pw) (blank = use rsh): flags (f) : 0x0 target name (tn) : pxm-7 startup script (s) : other (o) :
To verify the bootChange IP address of the Active PXM issue the version command.
jet.1.7.PXM.a > version VxWorks (for POPEYE) version 5.3.1. Kernel: WIND version 2.5 Made on Mar 30 1999, 12:20:01. Boot line: lnPci(0,0)solwandbg1: e=192.168.1.65 g=192.168.1.1 u=autoprog tn=pxm-7 PXM firmware version : 1.0.00 Boot Image version : 1.0.00Dc1
Issue the cnfifip command to assign the IP address used to connect to the MGX 8850 shelf. The IP address assigned by the cnfifip command is the IP address used to connect to the MGX 8850 when the shelf is in a normal operating state.
jet.1.7.PXM.a > cnfifip 26 192.168.1.23 255.255.255.0 192.168.1.255
To verify the shelf IP address issue the dspifip command.
jet.1.7.PXM.a > dspifip Interface Flag IP Address Subnetmask Broadcast Addr --------------- ---- --------------- --------------- --------------- Ethernet/lnPci0 UP 192.168.1.23 255.255.255.0 192.168.1.255 SLIP/sl0 DOWN 0.0.0.0 255.255.255.0 (N/A) ATM/atm0 DOWN 0.0.0.0 255.255.255.0 0.0.0.0The ATM address is used for in-band management of the MGX 8850 shelf over the feeder trunk to the Cisco BPX 8600 Series Switch.
Stage 3: The Upgrade
The following summarizes the steps that are necessary for a successful upgrade. All steps should be completed irrespective of network size.
-
Provisioning freeze starts.
Halt provisioning of new services until completion of upgrade.
-
If required, save MGX 8850 PXM and Service Module (SM) configuration.
Save a snapshot of the MGX 8850 configuration on a CWM (SV+) workstation. If the MGX 8850 configuration is not saved, the entire configuration must be manually re-entered.
jet.1.7.PXM.a > saveallcnf jet.1.7.PXM.a > ll C:/CNF size date time name -------- ------ ------ -------- 512 MAY-21-1999 17:46:12 . < DIR> 512 MAY-21-1999 17:46:12 .. <DIR> 182762 MAY-06-2000 03:02:44 jet_0306000302.zip 182762 MAY-06-2000 03:02:46 jet.zip In the file system : total space : 819200 K bytes free space : 712933 K bytesFrom the TFTP server issue the following commands to save the configuration file to the server. The TFTP server can be a Unix workstation or CWM workstation.
unix-prompt>tftp 192.168.1.23 tftp>bin tftp>get CNF/jet_0306000302.zip Received 182762 bytes in 2.4 seconds tftp>quit
-
View and record card errors and clear all error log files.
On all nodes to be upgraded record the card errors and clear the card errors using the following commands on the respective cards:
dspcderrs on the PXM, FRSM, AUSM, VISM, CESM. clrcderrs on the FRSM, AUSM. clrerr on the PXM. clrlog on the PXM. -
Load new revision into CWM (SV+) stations.
Load new firmware version into CWM (SV+) stations. Verify that the images have loaded successfully by comparing file sizes to those listed in the Firmware Release Notes.
-
Remove the cause of all MAJOR alarms and if possible all MINOR alarms.
Ideally, the network should be alarm free at the time of the firmware upgrade. If this is not possible, at least the reason for all major alarms should be identified and noted, and then suitable reconfiguration should be made in order to remove the alarm. Verify connection totals by issuing the dsptotals command as described in Appendix A.
Any minor alarms should be noted so that, after the upgrade, a comparison can be made.
-
Load new boot code revision into PXM.
Upload the new PXM boot code to the MGX 8850 using the TFTP process and verify the checksum.
unix-prompt>tftp 192.168.1.23 tftp>bin tftp>put pxm_bkup_1.1.23.fw POPEYE@PXM.BT Sent 798532 bytes in 7.2 seconds tftp>quit jet.1.7.PXM.a > Program length = 798532 Calculated checksum = 0x2a5a41f2 stored checksum = 0x2a5a41f2 Fw checksum passed
The PXM executes boot code sequentially, so if there is an older image loaded, the PXM will execute the oldest image. To avoid this problem, either delete the existing boot code image or rename the filename with a .old extension. If the existing boot code image is renamed, the FW directory contents will have two boot code files, one with a .old extension. A sample FW directory is provided below. To view the contents of the FW directory; from the C: drive issue the cd FW command and then the ll command. The current boot code file and two old boot code files are highlighted.
jet.1.7.PXM.a > ll size date time name -------- ------ ------ -------- 512 MAY-21-1999 17:46:12 . <DIR> 512 MAY-21-1999 17:46:12 .. <DIR> 2006756 MAY-21-1999 11:30:46 pxm_1.1.00.fw 886820 OCT-21-1999 09:18:54 sm130.fw 1133500 MAY-21-1999 12:03:20 sm50.fw 617584 MAY-21-1999 12:06:08 sm90.fw 794408 MAY-21-1999 12:06:46 sm35.fw 1253388 NOV-16-1999 06:42:38 pxm_bkup_1.1.13.fw 1246872 OCT-20-1999 11:07:28 pxm_bkup_1.1.12.old 2105328 OCT-20-1999 11:58:34 pxm_1.1.11.fw 644624 OCT-20-1999 12:07:38 pxm_bkup_1.1.01.old 2006664 OCT-20-1999 12:02:16 pxm_1.1.01.fw 2117676 NOV-16-1999 06:45:22 pxm_1.1.12.fw 377836 FEB-02-2000 21:04:44 sm50_14.bt 2168276 FEB-29-2000 03:49:44 pxm_1.1.22.fw 21 FEB-29-2000 03:55:30 ComMat.dat In the file system : total space : 819200 K bytes free space : 712621 K bytes
Note: The firmware files displayed using the ll command are a superset of the firmware files displayed by the dspfwrev command.
jet.1.7.PXM.a > dspfwrev Card Type Date Time Size Version File Name ----------- ------------------- -------- ------------------- ------------------ PXM1 05/21/1999 11:30:46 2006756 1.1.00 pxm_1.1.00.fw FRSM-VHS 10/21/1999 09:18:54 886820 10.0.05 sm130.fw AUSM-8T1E1 05/21/1999 12:03:20 1133500 10.0.01 sm50.fw CESM-8T1E1 05/21/1999 12:06:08 617584 10.0.01 sm90.fw FRSM-8T1E1 05/21/1999 12:06:46 794408 10.0.01 sm35.fw PXM1 11/16/1999 06:42:38 1253388 1.1.13 pxm_bkup_1.1.13.fw PXM1 10/20/1999 11:58:34 2105328 1.1.11 pxm_1.1.11.fw PXM1 10/20/1999 12:02:16 2006664 1.1.01 pxm_1.1.01.fw PXM1 11/16/1999 06:45:22 2117676 1.1.12 pxm_1.1.12.fw PXM1 02/29/2000 03:49:44 2168276 1.1.22 pxm_1.1.22.fw
PXM bootcode and firmware are loaded during the same task for an ungraceful upgrade from version 1.1.23 to 1.1.25. Use the TFTP process and verify the checksum.
unix prompt>tftp 172.16.185.10 tftp>bin tftp>put pxm_bkup_1.1.25.fw POPEYE@PXM.BT sp8250-1.1.8.PXM.a > Program length = 1274260 Calculated checksum = 0xd59c091c stored checksum = 0xd59c091c Fw checksum passed
-
If network has been stable for 30 minutes after the successful boot code download, install the boot code into the PXM flash.
Install the uploaded boot code file into the PXM flash and reset the MGX 8850 shelf. Note that the downloadflash command is required for firmware releases prior to 1.1.12. For firmware releases 1.1.21 and later, use the install bt command.
jet.1.7.PXM.a > downloadflash Board recognised as a PXM1A board ... Checksum size is 798532 ... Erasing the flash .... FLASH erase complete Downloading C/FW/pxm_bkup_1.1.23.fw into the flash ... verifying flash contents .... Flash ok .... Flash download completed ... jet.1.7.PXM.a > resetsys Do you want to proceed (Yes/No)? Yes Syncing ...... The shelf boots up and displays the VxWorks prompt ->
-
Install PXM runtime firmware.
Upload the PXM firmware to the bootChange IP address and verify the firmware transfer. Note that for firmware upgrades from versions 1.0.00Dc1 or lower a drive re-format is required after loading the new boot code, to ensure that the databases between the firmware versions are upgradeable.
-> ataFormat IDE format in progress. This takes a while ......... ............................................. Disk format complete. Reboot the system .... value = 0 = 0x0 (This takes approx 25 minutes) Disk format complete. Reboot the system ..... -> reboot unix-prompt>tftp 192.168.1.65 tftp>bin tftp>put pxm_1.1.23.fw POPEYE@PXM.FW Sent 2168768 bytes in 26.3 seconds tftp>quit -> Program length = 2168768 Calculated checksum = 0x2a5a61b2 stored checksum = 0x2a5a61b2 Fw checksum passed
-
Install the ComMat.dat file
Upload the ComMat.dat file to the bootChange IP address. The ComMat.dat file contains compatibility matrix data that specifies the firmware version ranges that support graceful upgrades. Copy the ComMat.dat file to the C:/FW directory of the PXM.
unix-prompt>tftp 192.168.1.65 tftp>bin tftp>put ComMat.dat Sent 21 bytes in 0.3 seconds tftp>quit ->copy "ComMat.dat","FW/ComMat.dat"
For an ungraceful upgrade from version 1.1.23 to 1.1.25, upload the ComMat.dat file and copy to the PXM firmware directory.
unix-prompt>tftp 172.16.185.10 tftp>bin tftp>put ComMat.dat sp8250-1.1.8.PXM.a >copy C:/ComMat.dat C:/FW/ComMat.dat
-
Bring up the MGX 8850 PXM with the target firmware.
Execute the target PXM firmware version and reboot the shelf.
->setPXMPrimary "1.1.23" ->reboot
After the reboot command is successful, the PXM will run the new firmware and present the standard Login: prompt.
For an ungraceful upgrade from version 1.1.23 to 1.1.25, install the target PXM boot code.
sp8250-1.1.8.PXM.a >install bt 1.1.25 writing pxm_bkup_1.1.25.fw to flash... Board recognised as a PXM1B board ... Checksum size is 1274260 ... Erasing the flash .... FLASH erase complete Downloading C:/FW/pxm_bkup_1.1.25.fw into the flash ... verifying flash contents .... Flash ok .... Flash download completed ... command completed ok on the active pxm.. The new boot code will be used after the next reset
Then, execute the target bootcode and firmware.
sp8250-1.1.8.PXM.a >install 1.1.25 redundancy is not available the other card is not available you are not in redundant mode, do you want to try an ungraceful upgrade (yes or no)? yes Syncing ...... Warning: firmware reset on active PXM car
-
Load saved configuration from Task 9 onto PXM.
Assign the IP Address for the MGX 8850 shelf using the cnfifip command as described in step 4 of Stage 2. Restore the configuration if saved during step 6 of Stage 2. The MGX 8850 shelf name automatically returns after the configuration is restored. To manually configure the nodename issue the cnfname command.
NODENAME.1.7.PXM.a > cnfname jet jet.1.7.PXM.a >
From the TFTP server, put the saved configuration file onto the PXM using the following commands:
unix-prompt>tftp 192.168.1.23 tftp>bin tftp>put jet_0306000302.zip CNF/jet_0306000302.zip sent 182762 bytes in 2.3 seconds tftp>quit
Once the saved configuration file has been loaded onto the PXM restore the configuration using the following command:
jet.1.7.PXM.a >restoreallcnf -fjet_0306000302.zip The saved FW version (1.0.00) is not same as the current FW version (1.1.23) Do you want to proceed with the saved FW version (1.0.00)? no Do you want to keep the current FW version (1.1.23) (WARNING, the databases in these FW versions must be compatible or upgradeable for this to work)? Yes All current config will be replaced with the specified restored config and the shelf will be reset. Do you want to proceed (Yes/No)? yes Syncing ...... Flash download completed. -
Load new Service Module boot code and firmware versions into the PXM.
If the new firmware version does not require a Service Module boot code upgrade, omit the boot code step. Upload the new firmware and boot code for each Service Module to the shelf. Note that the checksum results are only displayed for firmware uploads.
Service Module firmware is copied onto the MGX 8850 PXM harddrive in the /FW directory. If no slot is specified when loading Service Module firmware by using the 0, any Service Module can be inserted into a valid slot and retrieve necessary firmware from the PXM. Loading Service Module firmware without specifying a slot will overwrite the old version of firmware if it exists on the hard drive.
To upload the new boot code :
unix-prompt>tftp 192.168.1.23 tftp> bin tftp>put frsm_8t1e1_FR8_BT_1.0.02.fw POPEYE@SM_1_3.BOOT Sent 297988 bytes in 14.2 seconds tftp>quit
Note: Syntax of the put command is put <backup boot> popeye@SM_1_<slot#>.BOOT
To upload the new firmware:
unix-prompt>tftp 192.168.1.23 tftp> bin tftp>put frsm_8t1e1_10.0.11.fw POPEYE@SM_1_0.FW Sent 799440 bytes in 18.3 seconds tftp>quit jet.1.7.PXM.a > Program length = 799440 Calculated checksum = 0x2a5a61b2 stored checksum = 0x2a5a61b2 Fw checksum passed
After all of the Service Module firmware has been loaded onto the MGX 8850 shelf, the resetcd command must be issued for each card to execute the new version of boot code and firmware. This will cause service interruption to connections built on each Service Module for approximately five minutes if there is no redundant Service Module.
Note: Syntax of the put command to apply to all Service Modules of the same model and not be slot-specific is put <firmware file name> POPEYE@SM_1_0.FW
-
Let network settle and run customer specific validation tests.
After 10 minutes, login to the target node and verify health using the following commands:
-
dsplog
-
dsperr -en
-
dsptotals
This period provides an ideal time to run tests to check that the new firmware is functioning correctly.
Interrogate all external management systems that are used to manage any routers that are connected to the MGX 8850 network. This interrogation is done to ensure that all devices are reachable.
If possible, end users should be contacted and asked to check that all network connections are in proper working order.
Note: In the unlikely event that a decision is taken to revert back to the previous firmware revision, Cisco TAC should be contacted prior to switching to the old revision. Important information as to why the new firmware is not functioning correctly will be lost after switching back to the old revision.
-
-
Network health check.
See Appendix A
-
Save MGX 8850 PXM and Service Module (SM) configuration.
See step 2 of Stage 3.
-
Provisioning freeze ends.
Appendix A - Network Health Check
Follow these steps to check the network health:
-
Audit the parameters within the following commands.
Settings should be consistent across all nodes of the same type within the network. Document differences and any variations from the default values.
dsptotals dsplog dspalms dspshelfalm -
Audit network for recent errors (active and standby controller cards), card errors, load model inconsistencies and alarms.
Use the following commands to accomplish these tasks:
dsperr -en dsplog s dsplog printlog dspcderrs or the dspcderrs <slot #> dspalms -
Investigate the following:
-
Recent firmware errors: Any nodes that continually log errors or have logged recent errors should be reported to Cisco TAC.
-
Card errors: Cards that are logging failures or have a history of hardware errors should be investigated by Cisco TAC.
-
Any trunks that are logging errors: Should be fixed for the duration of the upgrade.
-
All alarms should be accounted for. The real purpose of this check is to make sure that there are no alarms that will require special intervention before the upgrade.
-
-
Ensure that any necessary corrections are made before the start of the upgrade.
Related Information
- MGX 8220 Upgrade and Downgrade Matrices, Concepts and Definitions
- Configuring PXM Cards
- 1.1.00 Version Software Release Notes Cisco WAN MGX 8850 Software
- MGX 8850 Boot Code and Firmware Graceful Upgrade Script
- Cisco WAN Switching Solutions – Cisco Documentation
- Guide to New Names and Colors for WAN Switching Products
- Software Center – WAN Switching Software
- Technical Support – Cisco Systems
| Updated: Oct 04, 2005 | Document ID: 6933 |