Cisco on Cisco

A presidential visit is not the type of event that most companies need to consider in their network planning. But a panel discussion on the American Competitiveness Initiative that brought U.S. President George W. Bush, California Governor Arnold Schwarzenegger, and other dignitaries to Cisco® headquarters in April 2006 created a significant impact on the Cisco corporate network.

To avoid the road closures around the Cisco San Jose campus, coupled with the extra security measures for the presidential visit, an unusually large number of local Cisco employees telecommuted from home that day. This large peak in remote access to the corporate network, especially to watch the live video broadcast of the event, overloaded Cisco’s local VPN circuits.

Video Broadcast Saturates VPN Circuits

The event took place on a Friday afternoon, which typically is a time of low demand on the Cisco corporate network as employees worldwide conclude their work for the week. However, because the event was announced only one week in advance, Cisco network engineers had little time to prepare for the anticipated surge in traffic on local VPN circuits. “The service providers would not have enough time to physically upgrade our circuits,” says Mike Osako, a Cisco network engineer. “Although these circuits had a bursting capability to handle unexpected increases in traffic levels, the network traffic generated by this event saturated even that capacity.”

Normally, the VPN circuits at the Cisco San Jose campus experience peak traffic levels of approximately 7700 simultaneous connections during the morning hours, Monday through Thursday. During President Bush’s visit, traffic levels increased dramatically, reaching nearly 9000 connections as employees watched the event broadcast online. This high level of VPN connections and the bandwidth saturation by the video streams meant that some employees may have experienced difficulty in connecting to the VPN or slower network performance.

Cisco network engineers were able to make a few advance changes in network elements to better handle the anticipated traffic demand on the VPN. For example, one VPN concentrator was reconfigured to handle both hardware and software VPN clients, which allowed more flexibility in serving employee access. In addition, the live video broadcast was converted from a multicast stream to unicast transmissions that would allow employees using software VPN clients to watch the event on their PCs. (Converting multicast streams to unicast for large meeting broadcasts is a standard practice for Cisco IT.)

A Valuable Learning Experience for Network Preparedness

At the time of the American Competitiveness event, Cisco used several OC-3 circuits to provide local VPN access to the corporate network. “Based in part on the lessons that we learned from this event, we determined these circuits would not be able to handle another high-demand traffic surge,” says David Iacobacci, a member of the Cisco technical staff who supports remote access. Osako says, “We decided to upgrade our local VPN links to OC-12 circuits with a bursting capability and gigabit-capacity Metro Ethernet services that offer flexible bandwidth. These links help us to better prepare for future high-demand events and disaster recovery in San Jose. It also became clear that we need burstable VPN access services in other world regions, especially for disaster recovery.”

To serve more remote access connections, Cisco network engineers have deployed the Cisco VPN 3000 Series Concentrators worldwide and have started migrating to Cisco ASA 5500 Series Adaptive Security Appliances for improved VPN performance.

Osako says, “VPN access traffic is becoming the primary user of bandwidth in the Cisco corporate network as more employees telecommute full-time or extend the workday by accessing collaboration tools such as Web conferences, voice over IP (VoIP), and other applications. Our experience with this event reinforced the importance of remote network access to the way Cisco conducts its internal business.”

For More Information

Cisco on Cisco
Enterprise Class Teleworker Service
Enterprise Class Teleworker Case Study
Software VPN Case Study