Downloads |
Table Of Contents
Adjacency Types That Require Special Handling
Hardware CEF Resiliency on Cisco 12000 Series E2, ISE, and E5 Line Cards
Enabling CEF Consistency Checkers
Displaying CEF Table Inconsistencies
Clearing CEF Table Inconsistencies
Configuring Default Packet Handling for E2 or ISE Memory Failure
Configuring the E2 or ISE Out-of-Resource Threshold For Alarms
Verifying Hardware CEF Resiliency
ip cef table consistency-check
ip cef table hardware resource-failure action
ip cef table hardware resource-failure alarm threshold yellow
Cisco Express Forwarding
First Published: October 15, 2003Last Updated: June 26, 2006Cisco Express Forwarding (CEF) is advanced, Layer 3 IP switching technology. CEF optimizes network performance and scalability for networks with large and dynamic traffic patterns, such as the Internet, on networks characterized by intensive Web-based applications, or interactive sessions.
Feature History for Cisco Express Forwarding
Finding Support Information for Platforms and Cisco IOS Software Images
Use Cisco Feature Navigator to find information about platform support and Cisco IOS software image support. Access Cisco Feature Navigator at http://www.cisco.com/go/fn. You must have an account on Cisco.com. If you do not have an account or have forgotten your username or password, click Cancel at the login dialog box and follow the instructions that appear.
Feature Overview
This feature module describes CEF. It contains the following sections:
•
Hardware CEF Resiliency on Cisco 12000 Series E2, ISE, and E5 Line Cards
Benefits
CEF offers the following benefits:
•
•
•
Although you can use CEF in any part of a network, it is designed for high-performance, highly resilient Layer 3 IP backbone switching. For example, Figure 1 shows CEF being run on Cisco 12000 Series Internet Routers at aggregation points at the core of a network where traffic levels are dense and performance is critical.
Figure 1 Cisco Express Forwarding
In a typical high-capacity Internet service provider (ISP) environment, Cisco 12000 Internet Routers as aggregation devices at the core of the network support links to Cisco 7500 series routers or other feeder devices. CEF in these platforms at the network core provides the performance and scalability needed to respond to continued growth and steadily increasing network traffic. CEF is a distributed switching mechanism that scales linearly with the number of interface cards and the bandwidth installed in the router.
Restrictions
•
•
•
•
CEF Components
Information conventionally stored in a route cache is stored in several data structures for CEF switching. The data structures provide optimized lookup for efficient packet forwarding. The two main components of CEF operation are described in the following sections:
Forwarding Information Base
CEF uses a FIB to make IP destination prefix-based switching decisions. The FIB is conceptually similar to a routing table or information base. It maintains a mirror image of the forwarding information contained in the IP routing table. When routing or topology changes occur in the network, the IP routing table is updated, and those changes are reflected in the FIB. The FIB maintains next hop address information based on the information in the IP routing table.
Because there is a one-to-one correlation between FIB entries and routing table entries, the FIB contains all known routes and eliminates the need for route cache maintenance that is associated with switching paths such as fast switching and optimum switching.
Adjacency Tables
Nodes in the network are said to be adjacent if they can reach each other with a single hop across a link layer. In addition to the FIB, CEF uses adjacency tables to prepend Layer 2 addressing information. The adjacency table maintains Layer 2 next-hop addresses for all FIB entries.
Adjacency Discovery
The adjacency table is populated as adjacencies are discovered. Each time an adjacency entry is created (such as through ARP), a link-layer header for that adjacent node is precomputed and stored in the adjacency table. Once a route is determined, it points to a next hop and corresponding adjacency entry. It is subsequently used for encapsulation during CEF switching of packets.
Adjacency Resolution
A route might have several paths to a destination prefix, such as when a router is configured for simultaneous load balancing and redundancy. For each resolved path, a pointer is added for the adjacency corresponding to the next hop interface for that path. This mechanism is used for load balancing across several paths.
Adjacency Types That Require Special Handling
In addition to adjacencies associated with next hop interfaces (host-route adjacencies), other types of adjacencies are used to expedite switching when certain exception conditions exist. When the prefix is defined, prefixes requiring exception processing are cached with one of the special adjacencies listed in Table 1.
Unresolved Adjacency
When a link-layer header is prepended to packets, the FIB requires the prepend to point to an adjacency corresponding to the next hop. If an adjacency was created by the FIB and not discovered through a mechanism, such as ARP, the Layer 2 addressing information is not known and the adjacency is considered incomplete. Once the Layer 2 information is known, the packet is forwarded to the Route Processor (RP), and the adjacency is determined through ARP.
Supported Media
CEF currently supports ATM/AAL5snap, ATM/AAL5mux, ATM/AAL5nlpid, Frame Relay, Ethernet, FDDI, PPP, HDLC, and tunnels.
CEF Operation Modes
CEF can be enabled in one of two modes described in the following sections:
Central CEF Mode
When CEF mode is enabled, the CEF FIB and adjacency tables reside on the RP, and the RP performs the express forwarding. You can use CEF mode when line cards are not available for CEF switching or when you need to use features not compatible with dCEF switching.
Figure 2 shows the relationship between the routing table, FIB, and adjacency table during CEF mode. The Catalyst switches forward traffic from workgroup LANs to a Cisco 7500 series router on the enterprise backbone running CEF. The RP performs the express forwarding.
Figure 2 CEF Mode
Distributed CEF Mode
When dCEF is enabled, line cards, such as VIP line cards or Cisco 12000 Series line cards, maintain an identical copy of the FIB and adjacency tables. The line cards perform the express forwarding between port adapters, relieving the RSP of involvement in the switching operation.
dCEF uses an Inter Process Communication (IPC) mechanism to ensure synchronization of FIB tables and adjacency tables on the RP and line cards.
Figure 3 shows the relationship between the RP and line cards when dCEF mode is active.
Figure 3 dCEF Mode
In this Cisco 12000 Series Internet Router, the line cards perform the switching. In other routers where you can mix various types of cards in the same router, all of the cards you are using may not support CEF. When a line card that does not support CEF receives a packet, the line card forwards the packet to the next higher switching layer (the RP) or forwards the packet to the next hop for processing. This structure allows legacy interface processors to exist in the router with newer interface processors.
Note
Hardware CEF Resiliency on Cisco 12000 Series E2, ISE, and E5 Line Cards
Starting in IOS Release 12.0(28)S, the Hardware CEF Resiliency feature is supported on Cisco 12000 Series Engine 2 (E2) and IP Services Engine (ISE) line cards. Hardware CEF resiliency is a protection mechanism for CEF hardware memory and ASIC-forwarding resources.
Starting in IOS Release 12.0(32)SY, the Hardware CEF Resiliency feature is supported on Cisco 12000 Series Engine 5 (E5) line cards.
Hardware CEF resiliency prevents CEF from being disabled and packet forwarding from being impacted in case of resource exhaustion or an error condition, such as such as low memory or IPC failure. The line card device driver handles resource failures internally without involving upper layers.
In the event of resource exhaustion, Hardware CEF resiliency provides early warnings and graceful degradation of CEF services. CEF behavior is consistent across all Cisco 12000 Series E2 and ISE line cards. Enhanced failure detection provides a simple, proactive monitoring and notification system for reporting critical events across the entire network. A caution or warning alarm is printed on the system console and logged to alert operators of the situation. Additional information about possible corrective actions to take is also logged. These messages allow sufficient time for network operators to resolve the resource problem.
Hardware CEF resiliency builds on the CEF Self-healing feature that provides an automatic restart capability. As soon as an error condition is corrected, an E2 or ISE line card automatically recovers from resource exhaustion provided that the network steady-state operation does not exceed line card capacities.
Without the Hardware CEF Resiliency feature, CEF is not automatically re-activated on Cisco 12000 Series E2 and ISE line cards after an error condition has been corrected. Instead, manual intervention is necessary to re-enable dCEF using the ip cef distributed command. While this behavior was implemented to prevent CEF instability and ensure network stability, the lack of resilience can result in prolonged network outages following network events, such as default route flaps, prolonged routing protocol instability, and errant redistribution events that disable forwarding.
For information about how to configure the resource monitoring functions in the Hardware CEF Resiliency feature, see Configuring Default Packet Handling for E2 or ISE Memory Failure and Configuring the E2 or ISE Out-of-Resource Threshold For Alarms.
Configuration Tasks
To configure CEF, perform the tasks described in the following sections. The task in the first section is required; the tasks in the remaining sections are optional.
•
•
•
•
•
•
Enabling CEF or dCEF
Enable CEF when your router has interface processors that do not support dCEF.
To enable CEF, use the following command in global configuration mode:
Enable dCEF when you want your line cards to perform express forwarding so that the route processor (RP) can handle routing protocols or switch packets from legacy interface processors.
Note
To enable or disable dCEF operation, use one of the following commands in global configuration mode as needed:
Router(config)# ip cef distributed
Enables dCEF operation.
Router(config)# no ip cef distributed
Disables dCEF operation.
When you enable CEF or dCEF globally, all interfaces that support CEF are enabled by default. If you want to turn off CEF or dCEF on a particular interface, you can do so.
To disable CEF or dCEF on an interface, use the following command in interface configuration mode:
When you disable CEF or dCEF, Cisco IOS software switches packets received on the interface using the next fastest switching path. In the case of dCEF, the next fastest switching path is CEF on the RP.
If you have disabled CEF or dCEF operation on an interface and want to re-enable it, you can do so by using the ip route-cache cef command in interface configuration mode.
Note
Enabling CEF Consistency Checkers
CEF uses routing information that is retrieved from the Routing Information Base (RIB), Route Processor (RP), and the line card (LC) databases to perform express forwarding. As updates occur to these databases, inconsistencies may result due to the asynchronous nature of the distribution mechanism for these databases.
If you find a database inconsistency, such as an IP prefix missing from a line card or an RP; you can investigate and resolve these instances by referencing the CEF system error messages that occur and by issuing CEF debug and show commands.
To enable CEF consistency checkers, use the following command in global configuration mode:
Enables CEF table consistency checker types and parameters.
You can enable the following CEF consistency checker types:
•
•
•
•
Displaying CEF Table Inconsistencies
To display CEF table inconsistency records found by the lc-detect, scan-rp, scan-rib, and scan-lc detection mechanisms, use the following command in privileged EXEC mode:
Clearing CEF Table Inconsistencies
To clear CEF table inconsistencies, use the following commands in privileged EXEC mode:
Configuring Default Packet Handling for E2 or ISE Memory Failure
In the event that hardware-forwarding memory (pointer look-up (PLU) or table look-up (TLU)) runs low or fails on a Cisco 12000 Series E2 or ISE line card, a pre-defined default behavior determines whether incoming packets are dropped or punted to an adjacent memory cache for future fast-path switching.
The driver software on an E2 or ISE line card handles the resource failure without involving higher switching levels, such as CEF and IP Routing protocols. No error is returned to CEF. Instead, when a memory allocation request fails, a caution or warning alarm is sent to the system console and logged in the syslog file for troubleshooting.
During the memory failure, the E2 or ISE device driver assigns a drop or punt adjacency to service memory allocation requests. (For more information on adjacency types, see Adjacency Types That Require Special Handling.) You specify the default action for handling packets during a memory failure by using the ip cef table hardware resource-failure action command.
When a memory allocation failure re-starts, a timer-based resource monitoring process is activated in the background. When the timer expires, the process checks to see if sufficient adjacency memory is available to resume normal CEF switching on the line card. If enough memory is available, an automatic recovery takes place.
To configure the default behavior for handling packets during a memory failure on an Cisco 12000 Series E2 or ISE line card, use the following command in global configuration mode:
Configuring the E2 or ISE Out-of-Resource Threshold For Alarms
When hardware-forwarding memory (PLU or TLU) runs low or fails on a Cisco 12000 Series E2 or ISE line card, the resource monitoring function prints an alarm (error message or warning) on the system console and logs the alarm (with suggested troubleshooting actions) in the syslog file for operator intervention.
Table 2 describes the thresholds (percentage of PLU or TLU hardware-forwarding memory used) that determine when a warning or error message is issued.
When a memory allocation failure starts, a timer-based resource monitoring process is activated in the background. The process checks the percentage of PLU and TLU hardware-forwarding memory used at one-minute intervals. When the percentages of hardware memory exhaustion described in Table 2 are exceeded, an alarm is generated.
A sample warning (issued when the yellow threshold is exceeded) and error message (issued when the red threshold is exceeded) are shown below. Note that, in each sample alarm, resource refers to one of the following E2 or ISE resources:
•
•
•
•
Sample Out-of-Resource Warning
SLOT 2 is running low on resource. CEF will begin resource constrained forwarding operation if problem persists. For additional details please see "show ip cef resource" and "show ip cef summary".Sample Out-of-Resource Error Message
Out of hardware forwarding resource. CEF will now begin resource constrained forwarding operation which may result in performance degradation. System will automatically resume normal forwarding once network issue is corrected.To configure the (yellow) threshold used to generate warnings for PLU and TLU memory exhaustion on a Cisco 12000 Series E2 or ISE line card, use the following command in global configuration mode:
Verifying CEF
To verify CEF-related information, use the following commands in privileged EXEC mode:
Verifying Hardware CEF Resiliency
To verify information about hardware-forwarding resources used for Hardware CEF resiliency on Cisco 12000 Series E2 and ISE line cards, use the following commands in privileged EXEC mode:
Additional References
The following sections provide references related to CEF and Hardware CEF resiliency.
Related Documents
Description of CEF
"Cisco Express Forwarding Overview" chapter in the Cisco IOS Switching Services Configuration Guide, Release 12.1
Procedures for configuring CEF or distributed CEF (dCEF)
"Configuring Cisco Express Forwarding" chapter in the Cisco IOS Switching Services Configuration Guide, Release 12.1
Cisco Express Forwarding and how it is implemented on Cisco 12000 Series Internet Routers
Cisco Express Forwarding and CEF Resilience
Procedures for verifying CEF switching
Show commands used to display system information
Standards
No new or modified standards are supported by this feature, and support for existing standards has not been modified by this feature.
—
MIBs
RFCs
Technical Assistance
Command Reference
This section documents new commands. All other commands used with this feature are documented in the Cisco IOS Release 12.1 command reference publications.
•
•
•
clear ip cef event-log
To clear the Cisco Express Forwarding (CEF) event-log buffer, use the clear ip cef event-log command in EXEC mode.
clear ip cef event-log
Syntax Description
This command has no arguments or keywords.
Command Modes
EXEC
Command History
Usage Guidelines
This command clears the entire CEF table event log that holds forwarding information base (FIB) and adjacency events.
Examples
The following example clears the CEF event-log buffer:
Router# clear ip cef event-logRelated Commands
Enables CEF table consistency checker types and parameters.
Controls CEF table event-log characteristics.
Displays all recorded CEF FIB and adjacency events.
clear ip cef inconsistency
To clear the Cisco Express Forwarding (CEF) inconsistency statistics and records found by the CEF consistency checkers, use the clear ip cef inconsistency command in EXEC mode.
clear ip cef inconsistency
Syntax Description
This command has no arguments or keywords.
Command Modes
EXEC
Command History
Usage Guidelines
This command clears the CEF inconsistency checker statistics and records that accumulate when the ip cef table consistency-check command is enabled.
Examples
The following example clears all CEF inconsistency checker statistics and records:
Router# clear ip cef inconsistencyRelated Commands
Enables CEF table consistency checker types and parameters.
Displays CEF IP prefix inconsistencies.
debug ip cef fragmentation
To report fragmented IP packets when Cisco Express Forwarding (CEF) is enabled, use the debug ip cef fragmentation command in privileged EXEC mode. To disable debugging, use the no form of this command:
debug ip cef fragmentation
no debug ip cef fragmentation
Syntax Description
This command has no arguments or keywords.
Defaults
This command is disabled by default.
Command Modes
Privileged EXEC
Command History
Usage Guidelines
This command is used to troubleshoot fragmentation problems when CEF switching is enabled.
Examples
The following is sample output from the debug ip cef fragmentation command:
Router# debug ip cef fragmentation00:59:45:CEF-FRAG:no_fixup path:network_start 0x5397CF8E datagramstart 0x5397CF80 data_start 0x397CF80 data_block 0x397CF40 mtu 1000 datagramsize 1414 data_bytes 141400:59:45:CEF-FRAG:send frag:datagramstart 0x397CF80 datagramsize 442 data_bytes 44200:59:45:CEF-FRAG:send frag:datagramstart 0x38BC266 datagramsize 1006 data_bytes 100600:59:45:CEF-FRAG:no_fixup path:network_start 0x5397C60E datagramstart 0x5397C600 data_start 0x397C600 data_block 0x397C5C0 mtu 1000 datagramsize 1414 data_bytes 141400:59:45:CEF-FRAG:send frag:datagramstart 0x397C600 datagramsize 442 data_bytes 44200:59:45:CEF-FRAG:send frag:datagramstart 0x38BC266 datagramsize 1006 data_bytes 1006Table 3 describes the significant fields shown in the display.
debug ip cef table
To enable the collection of events that affect entries in the Cisco Express Forwarding (CEF) tables, use the debug ip cef table command in privileged EXEC mode. To disable debugging, use the no form of this command.
debug ip cef table [access-list | consistency-checkers]
no debug ip cef table [access-list | consistency-checkers]
Syntax Description
access-list
(Optional) Controls collection of consistency checker parameters from specified lists.
consistency-checkers
(Optional) Sets consistency checking characteristics.
Defaults
This command is disabled by default.
Command Modes
Privileged EXEC
Command History
11.2 GS
This command was introduced.
11.1 CC
Multiple platform support was added.
12.0(15)S
The consistency-checkers keyword was added.
Usage Guidelines
This command is used to record CEF table events related to the forwarding information base (FIB) table. Possible types of events include the following:
•
•
•
•
Examples
The following is sample output from the debug ip cef table command:
Router# debug ip cef table01:25:46:CEF-Table:Event up, 1.1.1.1/32 (rdbs:1, flags:1000000)01:25:46:CEF-IP:Checking dependencies of 0.0.0.0/001:25:47:CEF-Table:attempting to resolve 1.1.1.1/3201:25:47:CEF-IP:resolved 1.1.1.1/32 via 9.1.104.1 to 9.1.104.1 Ethernet2/0/001:26:02:CEF-Table:Event up, default, 0.0.0.0/0 (rdbs:1, flags:400001)01:26:02:CEF-IP:Prefix exists - no-op changeTable 4 describes the significant fields shown in the display.
ip cef table consistency-check
To enable Cisco Express Forwarding (CEF) table consistency checker types and parameters, use the ip cef table consistency-check command in global configuration mode. To disable consistency checkers, use the no form of this command.
ip cef table consistency-check [type {lc-detect | scan-lc | scan-rib | scan-rp}] [count count_number] [period seconds]
no ip cef table consistency-check [type {lc-detect | scan-lc | scan-rib | scan-rp}] [count count_number] [period seconds]
Specific to Suppress Errors During Route Updates
ip cef table consistency-check [settle-time seconds]
no ip cef table consistency-check [settle-time seconds]
Syntax Description
Defaults
All consistency checkers are disabled by default.
Command Modes
Global configuration
Command History
Usage Guidelines
This command configures CEF consistency checkers and parameters for the following detection mechanism types:
Examples
The following example enables the CEF consistency checkers:
ip cef table consistency-checkRelated Commands
ip cef table event-log
To control Cisco Express Forwarding (CEF) table event-log characteristics, use the ip cef table event-log command in global configuration mode.
ip cef table event-log [size event-number] [match ip-prefix mask]
no ip cef table event-log [size event-number] [match ip-prefix mask]
Specific to Virtual Private Network (VPN) Event Log
ip cef table event-log [size event-number] [vrf vrf-name] [match ip-prefix mask]
no ip cef table event-log [size event-number] [vrf vrf-name] [match ip-prefix mask]
Syntax Description
Defaults
Default size for event log is 10000 entries.
Command Modes
Global configuration
Command History
Usage Guidelines
This command is used to troubleshoot inconsistencies that occur in the CEF event log between the routes in the Routing Information Base (RIB), Route Processor (RP) CEF tables and line card CEF tables.
The CEF event log collects CEF events as they occur without debugging enabled. This allows the tracing of an event immediately after it occurs. Cisco technical personnel may ask for information from this event log to aid in resolving problems with the CEF feature.
When the CEF table event log has reached its capacity, the oldest event is written over by the newest event until the event log size is reset using this command or cleared using the clear ip cef event-log command.
Examples
The following example sets the CEF table event log size to 5000 entries:
ip cef table event-log size 5000Related Commands
Enables CEF table consistency checker types and parameters.
Displays all recorded CEF FIB and adjacency events.
Clears the CEF event-log buffer.
ip cef table hardware resource-failure action
To configure the default behavior for handling fast-path packet switching during low hardware memory or a memory failure on a Cisco 12000 Series E2 or ISE line card, use the ip cef table hardware resource-failure action command in global configuration mode:
ip cef table hardware resource-failure action {drop | punt}
no ip cef table hardware resource-failure action {drop | punt}
Syntax Description
Defaults
Packets forwarded during low hardware memory or a memory failure are punted to a software-switching path.
Command Modes
Global configuration
Command History
12.0(28)S
This command was introduced on Cisco 12000 Series E2 and ISE line cards.
Usage Guidelines
Use the ip cef table hardware resource-failure action command to set the default behavior on a Cisco 12000 Series E2 or ISE line card for handling packets in CEF switching requests received during periods of low PLU or TLU memory or memory failure.
Drop specifies that the packets in a failed memory allocation request are dropped from adjacency memory; punt specifies that the packets are sent to the switching software on the line card instead of being forwarded by the hardware.
When sufficient adjacency memory is available to resume CEF switching on the line card, an automatic recovery takes place and resource requests assigned a punt adjacency are serviced in fast-path switching.
The driver software on an E2 or ISE line card handles a low memory condition or memory exhaustion without involving higher switching levels, such as CEF and IP Routing protocols. No error is returned to CEF. Instead, when a memory allocation request fails, an error message or warning is sent to the system console and logged in the syslog file for troubleshooting.
Examples
The following example shows how to reset the default packet handling behavior during periods of low E2 or ISE hardware memory or memory exhaustion so that packets in failed memory allocation requests are punted to the software driver on an E2 or ISE line card for processing:
ip cef table hardware resource-failure action puntRelated Commands
ip cef table hardware resource-failure alarm threshold yellow
To configure the (yellow) threshold used to generate a warning for PLU and TLU memory exhaustion on a Cisco 12000 Series E2 or ISE line card, use the ip cef table hardware resource-failure alarm threshold yellow command in global configuration mode:
ip cef table hardware resource-failure alarm threshold yellow number
no ip cef table hardware resource-failure alarm threshold yellow number
Syntax Description
Defaults
The default yellow threshold is set at 90% of hardware-forwarding (PLU or TLU) memory.
Command Modes
Global configuration
Command History
12.0(28)S
This command was introduced on Cisco 12000 Series E2 and ISE line cards.
Usage Guidelines
When PLU or TLU memory runs low or fails on a Cisco 12000 Series E2 or ISE line card, the resource monitoring function prints an alarm (warning or caution message) on the system console and logs the alarm (with suggested troubleshooting actions) in the syslog file for operator intervention.
When PLU or TLU memory is 95% or more exhausted, an error message is sent to the system console and syslog file. This (red) threshold is not configurable
When the percentage of PLU or TLU memory that is exhausted exceeds the current yellow threshold value, a warning is sent to the system console and syslog file. You configure this (yellow) threshold using the ip cef table hardware resource-failure alarm threshold yellow command.
For an example, of the warning and error messages used for Hardware CEF resiliency, see Configuring the E2 or ISE Out-of-Resource Threshold For Alarms.
During a memory allocation failure, a timer-based resource monitoring process is activated in the background. The process checks the percentage of PLU and TLU hardware-forwarding memory used at one-minute intervals. If the yellow or red threshold of memory is exceeded, a new error message or warning is generated. To avoid having repeated warnings sent to the system console every minute, configure the yellow threshold setting to a higher number.
The following warning is printed on the system console when the yellow threshold for hardware memory usage is exceeded:
SLOT 2 is running low on resource. CEF will begin resource constrained forwarding operation if problem persists. For additional details please see "show ip cef resource" and "show ip cef summary".Where resource refers to one of the following E2 or ISE hardware-forwarding memory resources:
•
•
•
•
Note
Examples
