The Cisco IOS® NetFlow feature set allows for the tracking of individual IP flows as they are received at a Cisco router or switching device. Network administrators can use the NetFlow flow records for a variety of purposes, including accounting, billing, network planning, traffic engineering, and user or application monitoring.
NetFlow services are available on Cisco IOS Software-based routers including Cisco 800 thru 7500 Series Routers, as well as the Cisco Catalyst® 6500 Series Switch, Cisco 7600, 10000, 12000 Series Routers and CRS-1 devices.
Although many Cisco customers want to deploy NetFlow services, they are naturally cautious about introducing new technology into their network without completely understanding the potential performance impact. This paper examines the CPU impact of enabling NetFlow services in various scenarios on several different Cisco hardware platforms.
This information is valuable for planning potential NetFlow implementations, but these results are not a replacement for proper customer lab testing, pilot deployments, and other types of solution validation.
Testing Methodology
For the following selection of routers, tests were performed using Cisco IOS Software Release 12.0S. The platforms and configurations tested include:
• Cisco 2600 Router
• Cisco 2851 Router
• Cisco 3640 Router
• Cisco 3745 Router
• Cisco 7200 Router with Network Processing Engine NPE-300
• Cisco 7200 Router with Network Services Engine NSE-1
• Cisco 7500 Router with Route Switch Processor 8 using Cisco Express Forwarding
• Cisco 7500 Router with Route Switch Processor 8 using Distributed Cisco Express Forwarding
• Cisco 12000 Internet Router running Distributed Cisco Express Forwarding, with 1:100 sampling enabled
For the following selection of routers an enhanced selection of tests were run using Cisco IOS Software Release 12.4T:
• Cisco 1841 Router
• Cisco 2811 Router
• Cisco 3845 Router
• Cisco 7200 Router with Network Processing Engine NPE-400
• Cisco 7200 Router with Network Processing Engine NPE-G1
• Cisco 7200 Router with Network Processing Engine NPE-G2
• Cisco 7301 Router
Seventeen test cases were defined and not all are run on all the platforms listed. The test cases are documented below. A mnemonic has been assigned to each test case to make it easier to understand the charts and graphs in the "Test Results" and "Test Analysis" sections of this document. Table 1 describes the test cases.
Table 1. NetFlow Test Cases
Mnemonic
Test Description
Baseline
Baseline test without NetFlow enabled; provides a context for the tests that follow
NF-load
Statistics generated immediately after NetFlow services are enabled on the router; tests any unusual initialization requirements of NetFlow. No traffic is running.
NF-enable
NetFlow version 5 enabled but no export destination defined; documents the effects of NetFlow on the router itself
NF-NDE
NetFlow version 5 enabled and NetFlow Data Export (NDE) destination also defined; tests the effects of NDE on the router
NF-NDE-2
NetFlow version 5 enabled and two different NDE destinations also defined; tests the effects of NDE with multiple destinations on the router
v9-NDE1
NetFlow version 9 enabled and NetFlow Data Export (NDE) destination also defined; tests the effects of NDE on the router
v9-NDE2
NetFlow version 9 enabled and two different NDE destinations also defined; tests the effects of NDE with multiple destinations on the router
NF-NDE-AS
NetFlow version 5 enabled and recording autonomous-system origin of packets; NDE destination also defined; used to test the effects of maintaining information about autonomous systems with NetFlow
V9-NDE-AS-NH
NetFlow version 9 enabled and recording autonomous-system peer of packets and BGP next-hop; NDE destination also defined; used to test the effects of maintaining information about autonomous systems with NetFlow
NF-Prefix-V8
NetFlow configured with a version 8 prefix aggregation scheme (but no NDE); compare results with NetFlow version 5
NF-AS-V8-NDE
NetFlow configured with a version 8 autonomous system aggregation scheme and NDE; compare results with NetFlow v5
NF-AS-TOS
NetFlow configured with a version 8 autonomous system- Type of Service (ToS) aggregation scheme and NDE; compare results with NetFlow version 5
FNF-load
Statistics generated immediately after Flexible NetFlow services are enabled on the router; tests any unusual initialization requirements of Flexible NetFlow
FNF-enable
NetFlow-original on all interfaces, no export; documents the effects of NetFlow on the router itself
FNF-NDE-AS
NetFlow version 9 enabled and recording autonomous-system origin of packets; NDE destination also defined; used to test the effects of maintaining information about autonomous systems with Flexible NetFlow
FNF-NDE-AS-NH
NetFlow version 9 enabled and recording autonomous-system peer of packets and BGP next-hop; NDE destination also defined; used to test the effects of maintaining information about autonomous systems with Flexible NetFlow
Each test case was performed with three different IP flow sets: 10,000, 45,000, and 65,000 flows (this represents the number of unique IP flows that were seen by the router for each test). For some platforms 70,000 flows were tested instead of 65,000 flows. The flows were sent in a loop, so that the NetFlow cache was populated by the first iteration of the traffic stream and used for switching the packets on subsequent iterations.
For the enhanced selection of tests, multicast traffic was always flowing and for certain platforms, an additional set of tests with 2,000 flows were also run. The enhanced selection of tests include Flexible NetFlow.
To ensure accuracy of the results and to eliminate any anomalies, output of the tests was not collected until NetFlow had been running for ten minutes (the NF-load test case was the exception to this rule). Six samples of output were taken at one-minute intervals following that. The numbers presented in the "Test Results" portion of this document represent the average of those results.
As the timeouts for each flow set are constant, the packet per second (pps) rates presented to the router are different for each flow set.
Table 2. Flow Counts and PPS
Flow Count
PPS
2,000
274
10,000
1394
45,000
6326
65,000
8903
Note that the testing provided a "worst-case scenario" in terms of the traffic flows seen by the routers, and the results must be viewed in that context.
All packet sizes were 64 bytes, a scenario that tends to be more stressful on a router than a mixed traffic stream of various sizes.
At each flow rate, there was no duplication of flows until the test began its second iteration through the testing loop.
Test Results
This section presents the results of the test cases described in the previous section. Interpretations and conclusions to be drawn from the data are discussed in the "Test Analysis and Conclusions" section of this document.
The results are presented in raw, tabular format, so that readers will have all available information, and can utilize the numbers to extrapolate the results into their own environment.
CPU Load Results
Tables 2 through 18 give results from the CPU utilization tests.
