Executive Summary
Cisco and Intel are collaborating in a number of important areas, including validating and optimizing interoperability and performance for the next generation of 802.11n WLAN networks, allowing Cisco and Intel customers to enjoy a greater than five times throughput improvement over traditional 802.11abg networks along with improved reliability and predictability in RF coverage.
Cisco and Intel are jointly testing 802.11n performance in Intel's Over-the-Air test facility. This effort provided Cisco and Intel an opportunity to tweak their implementations to optimize both client and infrastructure performance. The resulting test data demonstrates the improved throughput, reliability and predictability of 802.11n. For joint Cisco and Intel customers, this collaboration ensures that the 802.11n system is interoperable and delivers a high confidence level in a deploying the joint 802.11n solution.
Demonstrating market leadership in performance, the joint testing yielded impressive results with throughput numbers up to 195 Mbps in a real-world testing scenario. In sustained testing, throughput rates averaged 182 Mbps, showing a 900% throughput improvement over 802.11abg. The results also showcased the benefits of MIMO for improved coverage, and the backwards compatibility between 802.11n and 802.11abg.
More details on the Cisco and Intel alliance can be found at the Cisco Intel alliance homepage: http://www.ciscointelalliance.com.
Enterprise Class 802.11n for the Real World
Cisco and Intel are collaborating in a number of important areas. Whether it's improving voice over wireless, combining forces to defend against security threats, or building the next-generation supply chain, Cisco and Intel are helping improve the way companies and consumers work, interact, and stay connected. The Cisco and Intel alliance is focused on driving the convergence of networking communications and computing through innovative, standards-based solutions. More details on the Cisco and Intel alliance can be found at the Cisco and Intel alliance home page: http://www.ciscointelalliance.com
One of the critical areas where Intel and Cisco are collaborating is in the wireless space, where 802.11n represents the next evolutionary step in standards-based wireless networking and a new level of mobility in enterprise networks. Now with the advent of 802.11n and supported data rates up to 300 Mbps, bandwidth is increased at least 5 times over today's 802.11ag WLAN offerings (54 Mbps), creating breathing room for more demanding-use cases. In addition to throughput increases, 802.11n will bring enterprise-class predictability and reliability to Wi-Fi connections thanks to multiple-input multiple-output (MIMO) technology, which brings multiple antennas and advanced signal processing capabilities.
Cisco and Intel: A Cardinal Collaboration in Wireless and Mobility
Even though the 802.11n portion of the WLAN industry is just now coming to fruition, Cisco and Intel are already established market leaders. On the infrastructure side, Cisco currently holds a majority market share (63 percent) of enterprise WLAN products and has expanded on this lead in the 802.11n-specific segment to nearly 73 percent (Dell'Oro Q1CY08 Report). On the client side, Intel has shipped well over 15 million 802.11n devices, signaling clear market leadership.
Cisco and Intel recognize the importance to our customers of engaging in collaborative testing procedures aimed at harmonizing and optimizing the interaction between their respective WLAN infrastructure and client components. To the Cisco and Intel customer, the benefits of this alliance abound. First, Joint Cisco and Intel WLAN system is tested to deliver maximum level of performance with resolution for failure points before reaching customer premises. Second, this ongoing collaboration facilitates the optimization of performance and reliability for both current and future mobility solutions from both vendors. Lastly, given Cisco and Intel's market leadership, customers can be confident that their wireless deployment will be successful, with the backing of well-established providers of enterprise-grade mobile solutions.
802.11n Testing for Real-World Deployments
There are two common approaches to Wi-Fi testing: conductive testing and over-the-air testing. Although both approaches have their use cases, their fundamental difference lies between the manner in which RF energy is transferred between infrastructure and client. On one end, conductive testing relies on a directly cabled connection between wireless client and access point, which allows the Wi-Fi signals to flow free and uninterrupted along a guided wired medium. While this flavor of testing is ideal for assessing a product for quality assurance and simulated environments, it does very little to capture the behavior exhibited in the customer's deployment scenario.
The Cisco/Intel 802.11n testing adheres to real-world, end-to-end solution verification. First, testing occurs in an over-the-air (OTA), dedicated-office environment modeled after a typical enterprise environment. The test environment features actual RF obstacles that would exist in a customer's premises. Second, multiple access points and client devices are utilized to replicate the inherent levels of contention, co-channel interference, and complexity found in an enterprise grade wireless network. Third, the entire testing process includes environmental control and reproducibility while facilitating changes in singular variables to obtain empirical data. Lastly, all throughput, roaming, and high client density assessments occur with actual benchmarks running at the application layer, stressing the cumulative end-to-end solution's performance and viability.
Real-World 802.11n Testing
In order to appreciate the importance of OTA testing, it's helps to understand the limitations of conductive testing which is often used by other vendors and testing publications as a benchmark. While directly connecting the access point to the client with an RF cable is useful for quality assurance and lower layer performance verification-it does not capture the true performance challenges of 802.11 contention, RF multipath and roaming behavior of the wireless LAN client.
Conductive testing relies on using coaxial RF cable to directly connect the antenna leads on the access point to those found inside the WLAN client device. Because this methodology removes the antenna subsystem from the equation, it assesses only a subset of what actually matters to the customer. In an actual deployment, antenna and radio performance play a large role in WLAN coverage characteristics, especially important with the multiple antenna capabilities found in 802.11n. In addition, conductive testing does not accurately test the contention-based MAC layer interactions between clients and access points in a shared medium like 802.11. Lastly, the client to access point connectivity does not suffer from any multipath, or obstructions, that would exist in any real world environment.
The Over-the-Air Test Facility
The best way to accurately assess the customer experience is with over-the-air testing utilizing infrastructure and clients in their final form. To this end, Intel has constructed a dedicated, 27,000-square-foot, OTA environment test facility in Oregon that is a replica of an enterprise-class office. Obstacles typically present in the enterprise-class office, ranging from desks, cubes, filing cabinets and load-bearing walls help ensure that the testing site is outfitted with the same conditions that would be found in actual customer deployments. This facility carefully mirrors a typical deployment where wireless devices manage and deal with multiple signals, all with varying fade rates and signal strengths.
All test scenarios are fully automated and repeatable. This facility uses programmable robots for performing roaming tests in a consistent and repeatable manner, taking into consideration the real world multipath and dynamically ocurring RF attenuation characteristics that are difficult to simulate.
The main infrastructure components in the OTA test facility are:
• Cisco 4404 WLAN controllers
• Cisco Aironet 1240 Series (802.11abg) and 1250 Series (802.11abgn) Access Points
• Cisco Wireless Control System (WCS)
The ambient RF characteristics of the OTA facility have been and are continually characterized with Cisco Spectrum Expert. The test client devices are all Intel® Centrino® processor technology with Wi-Fi clients, including:
• Intel® PRO/Wireless 2915ABG Network Connection (Tri-mode 802.11a/b/g)
• Intel® Wireless WiFi Link 4965AGN 802.11a/g/n draft v2.0
• Intel® WiFi Link 5300 (3x3 MIMO) 802.11a/g/n draft v2.0
The Collaborative Tests
Cisco and Intel have worked together to define a suite of tests for measuring OTA performance. These tests include:
• Line of sight (LoS) and nonline of sight (NLoS)
• High client density testing
• Multiple 802.11n and 802.11abg capacity testing
• Enterprise roaming
These tests are designed to ensure maximum performance and interoperability between Cisco and Intel devices. Specifically targeted to optimize and characterize the gains in predictability and reliability from 802.11n, and to identify and remedy faults before they reach customers in real-world OTA scenarios. We present some highlights of the data that illustrate the power of this collaboration.
LoS and NLoS Test
Testing LoS and NLoS performance in a real-world OTA environment demonstrates the benefits and reliability of the Intel WLAN client on a Cisco WLAN infrastructure. Testing conductively simply does not reproduce accurate rate versus range data, and removes the huge value-add of properly engineered antennae. OTA multipath and reflection within a real enterprise office building environment allows Cisco and Intel to evaluate the behavior of the combined solution in a repeatable and scientific method. This test gives Cisco and Intel engineering teams the ability to design, develop, and fine-tune the combined solution for optimal performance.
The Testing Methodology
In this test, a test laptop is mounted on a turntable on a robot. The robot then moves through the office environment on a programmed path, while the laptop is slowly rotated on the turntable at 1 RPM. The client's antennas and position as it is sitting on the automated vehicle are precisely set to allow for repeatability. Figure 1 illustrates the testing methodology.
Figure 1. Testing Methodology
Throughout the entire path, throughput scripts are run while the client's associated data rate, packet retransmits, and other metrics are collected. Ultimately, this data is recorded for review and repeated several times and averaged for consistency. As the client moves in and around the test range that has the same construction and layout of the typical enterprise environment, the behavior is observed and analyzed. Both the clients and infrastructure are challenged with the changing reflected signal to maintain the spatial separation and signal strength necessary for achieving the desired behavior our customers expect.
Figure 2 shows a map of the main data collection points in the test facility. Note that each data collection point represents a common, real-world scenario. For example, sampling location 50 on the right-hand side of the figure is 68 meters from the access point on the left-hand side, represented by the green circle. RF signals between the access point and a client at location 50 must pass through eight walls and four standard office cubicle walls.
Figure 2. Data Collection Points
Testing Results
Ultimately, results from these tests scenarios reflect the improvements to throughput, predictability, and reliability of 802.11n. The graph in Figure 3 is a powerful example of the benefits of MIMO and the performance optimizations that come from the partnership between Cisco and Intel.
Figure 3. LoS and NLoS Throughput Results
There are three data sets represented in this graph. The first data set, in light blue, represents test results from a trial using a Cisco Aironet 1242 Series Access Point (802.11a) with Intel 3945ABG client. The middle data set, in red, represents the same Cisco Aironet 1242 Series Access Point (802.11a) with an Intel 5300, which is an 802.11n, 3x3 MIMO client device. The third data set, represented in dark blue, is from a trial using the Cisco Aironet 1252 Series Access Point (802.11n) with the Intel 4965AGN client card.
First, consider the data sets acquired using the Cisco Aironet 1242 802.11a AP at station 40 which is a line of sight location six meters from the access point. This location has no signal obstructions and experiences no multipath, so it is a good baseline location to compare against. At this location, both the 802.11a Intel 3945 and Intel 5300 MIMO client achieve slightly more than 20 Mbps of throughput. These results meet expectations since both devices are connected at 802.11a data rates and there is no multipath to show the benefits of MIMO. However, as the sample points move through the range at progressively more obstructed locations, the benefits of the Intel 5300 MIMO client are apparent. Note station 50, where the 802.11a client barely delivers any throughput, whereas the MIMO client continues to deliver about 10 Mbps of throughput. The multiple transmit and receive antennae on the client side clearly benefit the client, even in an environment with only 802.11a infrastructure. Using an 802.11n AP with an 802.11abg client will yield similar results.
The data additionally shows the benefits of 802.11n on both the infrastructure and client side. At station 40, in a pure line of sight location six meters from the AP, the throughput is a staggering 187 Mbps! The sample locations throughout the test range show clearly the benefits of 802.11n MIMO. At stations 48 and 50, where the signals are most obstructed and multipath the worst, the system continues to deliver performance in excess of the best legacy 802.11a data. These data points clearly demonstrate how 802.11n MIMO technology leverages multiple transmit antennae, multiple spatial streams, and maximal ratio combining to deliver superior performance, predictability, and reliability.
High Client Density Testing
The high client density test suite is designed to help Cisco and Intel understand the impact of client density and cell size, leading to better development and optimization in these environments. High client density environments are common in hot spots, education, and healthcare environments. Also, in the enterprise, it is not uncommon to have high densities of clients in conference rooms and shared areas. Cisco and Intel customers will enjoy superior throughput, reliability and predictability in these environments.
The Test Methodology
A large enterprise room contains many 802.11n capable laptops, each connected on the wireless network and running test automation scripts. Starting with a single laptop, actual throughput is measured and characterized using industry-standard performance tools. The test then adds an additional client to the mix with each subsequent test run. The tests run with both upstream and downstream traffic.
Since the RF medium is shared, this test characterizes the ability of both Intel clients and the Cisco infrastructure to manage contention.
The Test Results
The high capacity test suite measures and characterizes the effect of contention on total throughput. The graph in Figure 4 represents data from the high client density tests with up to 10 client devices, with bidirectional traffic.
Figure 4. High Client Density Testing Results
This test scenario shows how the system is able to manage contention in a network and does not lower aggregate throughput. The test results show consistent high performance, with an average of 182 Mbps across the test runs, with peak throughput at 195 Mbps. The client devices enjoy fair access to the medium and the high performance is consistent across the different test runs. The test results demonstrate clearly that 802.11n technology enables bandwidth hungry applications with adequate performance. Furthermore, the tests provide Intel and Cisco engineers with data that can be used to further improve and optimize total performance.
Multiple 802.11n and 802.11abg Capacity Testing
No real-world deployments will have just a single client, nor will most be only 802.11n clients, commonly referred to as a "Greenfield" deployment. Cisco and Intel asked some basic questions: What can our customers expect to see when they deploy 802.11n access points with an installed base of 802.11abg clients? And, how does a mix of 802.11abg and 802.11n clients affect the performance of aggregate systems?
The Testing Methodology
Intel and Cisco crafted several test scenarios to observe and characterize how clients running at both 802.11n and 802.11abg data rates impact performance. For these test scenarios, 8 laptops are logically placed and setup to run a suite of performance tests with upstream, downstream, and bidirectional traffic using industry standard tools.
The tests start with a single 802.11n client and seven clients running at 802.11a data rates. The test is then iterated; with successive iterations, the number of 802.11n clients is incremented by one while the number of 802.11a clients is decremented by one. The iterations continue until only one 802.11a client is present, with the rest 802.11n clients.
The Test Results
The test is designed to characterize and help us understand the behavior of a system in the presence of both 802.11n clients and 802.11a clients and to demonstrate the performance improvements of 802.11n even in environments with high numbers of 802.11a clients.
Due to 802.11 rules for managing contention, 802.11n clients are expected to have less throughput with 802.11a data rate clients. This also means aggregate system performance will be lower in the presence of more 802.11a clients. Naturally, more 802.11n clients in comparison to the number of 802.11a clients will logically result in superior aggregate system performance.
As the graph in Figure 5 shows, test data from the multiple 802.11n and 802.11a clients capacity tests confirmed these expectations:
Figure 5. 802.11a to 802.11n Client Migration
In the graph, the data sets on the far left are from a test run with seven 802.11a clients and a single 802.11n client, as indicated by the label "7A1N" on the X axis. Each successive series of data moving to the right represents test runs with one fewer 802.11a and one more 802.11n client. The blue data represents aggregate 802.11a throughput; the red data represents aggregate 802.11n throughput; the tan area in the background represents combined 802.11a and 802.11n throughput.
The data demonstrates that even in mixed mode environments with a high number of 802.11a devices, which occurs as 802.11n devices start to penetrate the enterprise, 802.11n infrastructure improves aggregate system performance. So even in the presence of 802.11a clients, Cisco and Intel customers will still enjoy improved system performance with an 802.11n infrastructure. More importantly as applications and clients advance and the 802.11a clients are upgraded to 802.11n, the total system performance improves substantially. In this test, the system performance with one 802.11a client and seven 802.11n clients is over 340% better than with a single 802.11n client and seven 802.11a clients. Furthermore, the data shows that per client performance is consistent in a mixed mode environment. Across all test runs, the 802.11a clients achieved about 2 Mbps of throughput while individual 802.11n clients enjoyed about 25 Mbps of throughput.
Enterprise Roaming
True mobility means that the workspace is wherever the user happens to be, rather than where a wired connection exists. More importantly, it means the user can "move" the workspace to meeting locations, to a balcony, or to a private location, all without losing connectivity to business-critical applications and data.
The lack of a roaming algorithm in the 802.11 standard has forced all client vendors to implement their own algorithm. Early 802.11 devices relied on algorithms, which were sufficient for most consumer deployments which typically have a single access point. However, as 802.11 based WLAN entered the enterprise, more sophisticated algorithms were needed to enable clients to seamlessly roam across multiple access points across enterprise facilities.
Furthermore, because 802.11n MIMO technology changes the way client devices see signals from infrastructure access points, client roaming algorithms designed for 802.11abg need to be revisited with 802.11n.
The enterprise roaming test suites are designed to characterize the roaming behavior of Intel clients with a Cisco infrastructure. This has allowed Cisco and Intel engineers to focus on optimizing roaming algorithms for enterprise use.
The Testing Methodology
In the exhaustive roaming test, a test laptop is placed on the programmable robot, which follows a defined roaming path while running industry-standard performance test suites. Figure 6 shows the roaming path for the robot, represented by the orange line.
Figure 6. Floor Plan for Roaming Test
Data is collected as the robot traverses the programmable path. The green circles represent access point locations. The robot can traverse the path multiple times, with different combinations of access points turned on, and at different output powers. The tests are completely automated; a full test run takes over 17 hours.
The Test Results
Figure 7 demonstrates the initial enterprise roaming test.
Figure 7. Ineffective Client Roaming
Each series of colored circles represents the client's roaming locations for a particular iteration of the robot's programmed path. Since the test is crafted to be repeatable and predictable, a well-behaved client should roam in the same location with each iteration. The figure shows test results from a pre-optimized client test. As can be seen from the figure, the roaming locations are highly irregular and unpredictable. Ultimately, this kind of inconsistent and unpredictable roaming leads to poor end user experience for enterprise applications.
Figure 8 represents data from a test run using optimized client roaming algorithms.
Figure 8. Optimized Client Roaming
As can be seen in the figure, the roaming locations are now clustered, indicating that the client device is now roaming in a predictable and consistent manner. Optimized, consistent and predictable roaming is ultimately good for enterprise applications.
Summary
Cisco and Intel are collaborating in a number of important areas, including validating and optimizing interoperability and performance for the next-generation, 802.11n WLAN networks. Enterprises deploying a joint Cisco and Intel WLAN system are ensured that the solution has been assessed to deliver the maximum level of throughput, reliability and predictability. This testing allows the companies to address any faults in their products before they reach the customer premises. This ongoing alliance optimizes performance and reliability for both current and future mobility solutions from both industry leaders. Lastly, Cisco and Intel customers can be confident that their wireless deployments will be successful with the joint effort of two industry leaders in delivering enterprise-class mobility solutions.
The Cisco and Intel alliance is focused on driving the convergence of networking communications and computing through innovative, standards-based solutions. More details on the Cisco and Intel alliance can be found at the Cisco and Intel alliance homepage: http://www.ciscointelalliance.com.
Appendix
Test Configurations
802.11n test data derived from:
• Dell Latitude D830, Vista32 Ultimate (SP1), Intel(R) Core(TM)2 Duo 2.50GHz (T9300), RAM: 4 GB
• Intel® WiFi Link 5300 (3x3 MIMO)
• 11F EEPROM
• Intel® PROSet/Wireless Network Connection Software version 12.0.4.0
• Dell Service Tag: FMNT0G1
• Cisco Aironet 1250 Series Access Point; FW 124-10b.JA1
• Traffic Generator: Console IxChariot 6.0; Endpoint Ixia 6.1
802.11a test data derived from:
• Dell Precision M65, XP32 Pro (SP2), Intel(R) T2300 @ 1.66GHz, RAM: 1 GB
• Intel® PRO/Wireless 3945ABG Network Connection (Tri-mode 802.11a/b/g)
• Intel® PROSet/Wireless Network Connection Software version: 11.5.0.32
• Dell Service Tag: HM8GTB1
• Cisco Aironet 1242 Series Access Point; FW 12.3(7)JA3
• Traffic Generator: Console IxChariot 6.0; Endpoint Ixia 6.1