The Transition to
IP Telephony
at Cisco Systems

Executive Summary

As a principal inventor and driver of integrated data, voice, and video networking, Cisco Systems was among the first companies to deploy integrated IP networks in locations worldwide, including its campus in San Jose, California. Beginning with its acquisition of Selsius Systems, Inc. in 1998, Cisco planned to incorporate IP telephony into its converged, quality-of-service (QoS) enabled network. As a pioneer in this technology, Cisco learned valuable lessons through its own migration from 1998 to 2001. These lessons directly contributed to the enhancement of the Cisco IP Telephony product line and voice over IP (VoIP) technologies. Cisco developed best practices for achieving a successful transition from traditional private branch exchange (PBX) telephony to integrated IP telephony based on Cisco AVVID (Architecture for Voice, Video and Integrated Data). This discussion outlines how Cisco customers can learn and benefit from that experience.

Introduction

Cisco has long made a practice of using its own technology. Where it lacks capability within its own product line and solutions, it often acquires companies that have those capabilities. One objective of this strategy is to establish Cisco as the leader in end-to-end internetworking, especially IP networking. In 1998, Cisco set a goal to develop solutions for converged networking: giving the world the ability to use a single, QoS-enabled, IP network infrastructure to carry data, voice, and video traffic simultaneously. To accomplish this, Cisco had to overcome several technical hurdles as outlined below.

A converged network needs the ability to separate each traffic type and handle it according to its unique requirements. For example, data traffic is not time-sensitive; it travels in bursts and requires accurate delivery. The TCP standard was designed to ensure accurate data transport. However, voice traffic—traditionally carried by some type of time-division multiplexing (TDM) mechanism—such as a PBX—is very time sensitive. Adding voice packets to a "bursty" IP environment required the development of many new voice features and QoS mechanisms, such as traffic classification and marking, advanced queuing mechanisms, and data packet fragmentation and interleaving techniques.

There is now a proven portfolio of such voice-enabling features within Cisco IOS^® Software. Because there is extensive documentation available on these features, they are designated outside the scope of this discussion. However, no one can successfully integrate voice and data into an IP environment without them; therefore, it is essential that anyone considering such an undertaking ensure that the network infrastructure is voice-enabled with an appropriate suite of QoS mechanisms and VoIP features. An end-to-end Cisco network allows organizations to guarantee the QoS required to attain high-quality voice services.

Selsius was a pioneer in IP telephony solutions. Soon after the Selsius acquisition, Cisco began a trial deployment of Selsius equipment in its San Jose campus network. At the time, Cisco AVVID was under development. The IP telephony paradigm represents a radical shift in how voice networks are constructed. The traditional PBX is cabinets in a locked environment with some kind of uninterruptible power supply (UPS) such as a battery backup. An IP telephony network is distributed in elements throughout the network. During its testing, Cisco learned valuable lessons during technology trials, and these lessons translated into best practices to assist Cisco customers with their own IP telephony deployments.

• Developing best practices—Cisco IT engineers learned valuable lessons during technology trials, and these lessons translated into best practices to assist Cisco customers with their own IP telephony deployments.
  
  

The current Cisco campus in San Jose has over 80 buildings and more than 20,000 employees connected to an IP Ethernet LAN/MAN. During 2001, Cisco is migrating its San Jose campus and many field offices to a production network based on Cisco AVVID and IP Telephony. The size of the deployment proves the scalability of the solution, ensuring that similar large organizations can safely deploy Cisco IP telephony solutions.

First Telephone Network

In 1998, the Cisco voice network in its San Jose campus had two Lucent Definity 3Gr PBXs and three Octel Aria 350 voicemail systems (Figure 1). The PBX systems were interconnected with several Lucent Definity Distributed Communication System (DCS) links for feature transparency and number portability. The network provided telephony services to all Cisco employees and call center services to the Cisco Technical Assistance Center (TAC).

Figure 1: Original, Dedicated Voice Network

At the time, Cisco owned number blocks in three prefixes: 526, 527, and 525. Each Octel system was assigned its own prefix for ease of management. Each Octel system was integrated with the Definity systems via Lucent digital phone-set emulation Per-port Integration Card—AT&T (PIC-A). OctelNet, proprietary voicemail-networking protocol, interconnected the three on-campus systems to other Octel systems located in approximately 80 Cisco remote sites, while the PBXs were connected to the public switched telephone network (PSTN).

First Trial Deployment

After acquiring Selsius, Cisco rolled out a pilot deployment to 200 employee volunteers late in 1998. Each employee had two phones: a Lucent PBX phone and a Cisco IP telephone. A Primary-Rate Interface (PRI) link between the CallManager 2.2 and the Definity PBX provided five-digit desk-to-desk dialing and access to and from the PSTN for IP telephony users (Figure 2).

Figure 2: Initial Deployment Model

The 200 test Selsius IP phones had voicemail access, but not full voicemail capabilities. The Octel system is designed to communicate with one PBX, not multiple PBXs. Cisco IT staff set up phantom numbers on the Definity switch so that unanswered or busy calls would forward to the Octel voicemail box. However, the system could not deliver a message-waiting indication (MWI) light to the IP telephone.

Each Selsius IP telephone had its own direct-inward dial (DID) phone number and a non-private, routable, registered IP address. This raised another issue; that is, if an organization chooses to migrate thousands of employees to IP telephones, assigning each person a new telephone number is neither scalable nor viable. It could also cause potential issues with the local telephone provider. Also, while the original Selsius product allowed the use of private (RFC 1918) IP addresses, the test deployment used real IP addresses.

Test users with two phones on their desk found it difficult to participate in the test without confusion. They did not know which phone to use or which phone number to give out. The easiest way to avoid confusion was to use the existing and familiar Lucent telephone and forget about the IP phone. This became the predominant practice. Another factor contributing to misuse was that the Selsius phone had a 10-megabit Ethernet hub. It was connected to the Ethernet network between the user PC and the wall jack. When users sent large files across the LAN while on the telephone, it disrupted voice quality.

A last technical issue was power. Selsius phones require local power, so they have their own power transformer and electrical cord. However, Cisco found that it was often difficult to place a Selsius phone into a conference room, break room, or lobby, because the Ethernet and power jacks were usually on opposite sides of the room.

So while there were several successful ways of deploying the existing system, Cisco engineers learned several lessons that were incorporated into the current generation of solutions that allow customers to break down some of the barriers to deployment.

Production Deployment of IP Telephony

The initial trials continued as Cisco took IP telephony to the next level. By November 1999, Cisco had grown considerably, and needed a new prefix from its local provider, Pacific Bell. Pacific Bell assigned the 853 DID range to Cisco. Cisco also had CallManager version 2.3, which was more stable and feature rich. Cisco embarked on a production-level deployment of IP telephony. Beginning in November, 1999, all new hires at the Cisco San Jose campus were provisioned with a Cisco IP phone. This was the only phone they had to conduct their daily business.

Cisco IT implemented the IP telephony network by assigning 853-prefix numbers to the Cisco CallManager 2.3 and leasing a fourth Octel Aria 350 voicemail system, which was dedicated to 853 numbers and directly connected to the CallManager server via a simplified message desk interface (SMDI) link. The CallManager had direct PSTN access through Cisco DT24+ gateways for both inbound and outbound calls. Cisco assigned IP telephones from the 853 range to newly hired employees, compelling them to have a single IP phone on their desk (Figure 3). Cisco maintained primary-rate interface (PRI) trunks to the Definity systems for desk-to-desk calls between Definity and Cisco IP Phone users.

This approach proved much cleaner than the previous one because no one had two telephone numbers. It also solved the MWI light problem from the initial deployment because the Octel voicemail system was directly connected to the CallManager. Before long, the single CallManager was upgraded to version 2.4 and had 2500 registered users on a system specified for a maximum of 200 users (Figure 3).

Figure 3: Revised Deployment Model

During this period, most failures in the phone system were traced to the network infrastructure rather than the IP telephony components. As a networking company, Cisco employees sometimes test new products on the production network that should be reserved for test laboratories. These "extras" add complexity to the network and can unwittingly cause problems for existing systems such as Cisco IP phones. As these problems became apparent, Cisco IT responded by developing and implementing a stricter change-management system and securing traffic from laboratories.

Improved Product Line

In tandem with the Selsius IP phone rollout, Cisco was developing a new breed of IP telephone. Most of the new products were introduced in late 1999 and shipped in 2000 along with key Cisco AVVID infrastructure components such as inline powered cards for the Cisco Catalyst^® 6500 products. Among the improvements and additions to the product line were:

• Improved handset design—To give it a contemporary appeal, Cisco added a more ergonomic key layout and a large display screen for Web-based services such as directory lookups using Lightweight Directory Access Protocol (LDAP) version 3, daily and financial news, and other XML-based services
  
  
• Inline power—By providing the option to power sets from the wiring closet switch rather than from local sources, users do not have to hunt for power in order to use a phone
  
  
• Internal 10/100BaseT switch—Call quality improved when Cisco added a switch and more bandwidth, which combined to improve call quality; the internal switch also supports dedicated, 802.1 p/q virtual LANs (VLANs) for PCs and phones, enabling further separation, address management, and control; internal QoS prioritization automatically tags voice packets as higher priority for expedited transport
  
  
• Redundant CallManager—By enabling server "clusters," overall scalability, spatial redundancy, and availability of the phone system improved
  
  
• Full-featured CallManager software—Adds unique, new functionality such as integration with LDAP directories and Web servers
  
  
• Dedicated analog voice gateways such as the Cisco VG200 round out the requirements for a robust, all-IP phone system
  
  
• Advanced software applications for AVVID networks such as unified messaging, IP Contact Center, Cisco IP SoftPhone, Cisco Web Attendant, and Cisco IP Interactive Voice Response (IVR) products
  
  

Campus Migration

In April 2000, Cisco cut over its existing CallManager 2.4 server to a CallManager 3.0 cluster. This added capacity and reliability. The cluster includes eight servers: four primary servers, two backup servers, and dedicated Publisher and Trivial File Transfer Protocol (TFTP) servers. The four primary servers are separated into redundant pairs, each of which has a single backup server. The Publisher server communicates with the other six members of the cluster, and the TFTP server is solely responsible for delivering configuration files to IP Phones. The Publisher manages adds, moves, and changes to phones and gateways on the CallManager network. It is dedicated because changes consume significant processing power, and Cisco did not want its primary servers to experience any performance degradation. The Publisher server replicates its changes to the other servers in the cluster with minimal processing overhead. The cluster currently supports the 853-prefix CallManager and about 5000 users, but can support up to 10,000 users in the future.

At the same time, Cisco IT migrated its "853 prefix" users to new Cisco 7960 IP Phones. During a weeklong exchange period, Selsius phone users could bring their old IP phones to the cafeteria to exchange for a new phone, which they installed themselves. IT staff pre-programmed new telephones with existing telephone numbers. The Media Access Control (MAC) address of the new phone was scanned into a comma-separated value (CSV) file. Using the Batch Administration Tool (BAT), the CSV file was "batched" into the CallManager database and assigned the user's existing 853 phone number.

Once plugged into the network, the new phones self-configured within approximately 30 seconds. No technicians had to hard-wire cross-connects, as they would have with a PBX system. All training documentation for the new phone was placed on the internal Web for users to access at their convenience. This transition took about 5 business days.

Figure 4: Phase 1 Migration: 853 Prefix Users

The upgraded phone network uses private (RFC 1918), network-10 IP addresses to preclude impact to Cisco's several class-B addresses. It also adds a layer of security because the phones are not accessible from the Internet. Call quality is improved largely because desktop bandwidth increased from 10 to 100 Mbps; also, the three-way switch in every Cisco IP Phone allows Cisco IT staff to implement a trust boundary at the wiring closet to ensure priority handling of voice packets. The internal switch classifies packets with IP Precedence values as follows:

The Cisco LAN at this time did not yet have end-to-end QoS queuing mechanisms activated, because much of the Layer-2 switched infrastructure has older Catalyst switches that do not support advanced QoS features. Also, users initially had to use local power for the new telephones. At the time, most wiring closets had Catalyst 5500 and smaller switches, which do not support inline power cards.

By the end of 2001, Cisco plans to complete a campus-wide migration to IP telephony. It has a major infrastructure upgrade in progress to support this effort. It is enabling its entire campus LAN with the latest Cisco AVVID design and components. The project meets the following requirements:

• No added traditional PBX infrastructure expenditure
  
  
• Users must keep existing DID numbers
  
  
• Users must receive the equivalent voicemail service before and after migration
  
  
• Cisco IT must continue to supply new employees with telephone service throughout the migration period
  
  

The upgraded campus LAN features Cisco Catalyst 6500 Layer-3 switches in the distribution layer, and Catalyst 6500 Layer-2 switches in the wiring closets, complete with 10/100 Ethernet cards with inline power for Cisco IP phones. The migration is taking place building by building. As of September 2000, all newly occupied buildings are fully deployed with a Cisco AVVID infrastructure and IP Phones. An alternative migration strategy would have been to move entire blocks of numbers at a time. This is easier from a PBX management perspective, but in practicality works best for smaller campuses with only a few buildings. At Cisco, users within a designated number block are distributed throughout the 80-building campus. Instead, Cisco opted to migrate users building by building despite the added complexities of call routing between dual telephone networks throughout the transition period. As each building is upgraded, its employees swap their Lucent PBX telephones for Cisco 7960 IP Phones. Because phones in each building have a mix of prefixes, Cisco must keep its entire PBX infrastructure in place until upgrades in the last building are completed.

To support this migration, Cisco installed a total of four CallManager clusters, one for each prefix, and all identical to the first cluster in terms of configuration, minus the TFTP server. A fifth CallManager cluster and Octel voicemail system supports a new, all-IP prefix of 902 to support future growth. For availability, spatial redundancy, and security reasons, clusters and their backup servers are located in different data center buildings on the Cisco campus. A single TFTP server coordinates IP phone registration among all five clusters to reduce processing drain on Publisher servers, and act as a centralized configuration file repository.

The biggest challenge to successful migration is maintaining transparent voicemail services for the 526, 527, and 525 prefixes. During its first pilot of the Selsius system in 1998, Cisco learned that if users with IP phones were connected to voicemail via phantom numbers on the Definity PBX, they lost services such as message-waiting indicators. Most voicemail systems, including the Octel, cannot handle a dual-dissimilar PBX integration; therefore, an intermediary appliance is required to deliver the message-waiting indicator signal to the IP phones. Cisco asked Lucent how much it would cost to change the digital ports on the Octel system to analog ports and Lucent quoted in excess of US$100,000 per system. Clearly this was not a cost-effective alternative!

Cisco acquired a company called Calista in late 1999. At the time, Calista was developing what is now known as the Digital PBX Adapter (DPA) 7630, which offers 24 ports of digital set-emulation on one side and an IP port on the other. With additional work, the IP port now looks like an IP Phone port to a CallManager. It easily inserts between the Octel voicemail system and Cisco CallManager 3.0 to direct-connect message-waiting ports from the Octel voicemail system. It adapts to the steadily migrating network with "hybrid" connections to both the Definity PBX and the CallManager. The side that has the extension activates and deactivates the message-waiting indicator light on the correct phone. The other side sends an "invalid" signal back to the DPA, which it ignores. This flexible—and relatively inexpensive—solution allows Cisco IT to incrementally transfer numbers from the PBX to the CallManager with transparent voicemail service (Figure 5).

Figure 5: Transition Network During Migration

Most Cisco field offices are equipped with a Nortel Option 11 PBX and Octel voicemail combination. These offices are typically no larger than 200 employees; therefore, "flash-cutting" these offices from traditional PBX to IP telephony systems over a weekend is a viable option that is already underway with the help of system integrators. The Octel voicemail systems in field offices must either convert from digital set-emulation to SMDI with analog ports for integration with the local CallManager server, or they must install a Cisco DPA device at a much lower cost. The Cisco DPA uses Skinny Client Control Protocol (SCCP) to communicate with CallManager. SCCP is an open, unlicensed protocol based on the H.323 standard.

An All-IP Telephony Network

When Cisco completes its migration to an integrated, voice-data Cisco AVVID LAN at its San Jose campus, it will have an infrastructure in place to implement end-to-end QoS and take advantage of pre-configured IP Precedence marking in the IP phones. Further, the IT staff can separate all voice traffic into a separate VLAN to facilitate management and guarantee QoS. The Lucent PBXs will be removed, and there will be H.323 communication between CallManager clusters. The Cisco DPA 7630 will remain to provide simple connections between the Octel voicemail systems and CallManager clusters. There will be backup links to the PSTN from each CallManager cluster (Figure 6) in case of a LAN link or component failure.

Figure 6: Final AVVID Network with All-IP Telephony

Cisco already provides Web-based directory services, stock quotes, and other information to users via the display screens of IP phones. With all its campus users on the IP phone network, Cisco can then add new IP services, such as unified messaging, and replace its Definity G3 Call Centers with IP Contact Center. Cisco is also experimenting with IP telephony for telecommuters.

At present, there are no plans to extend IP telephony across the IP WAN to regional offices, because Cisco enjoys a very inexpensive domestic long-distance rate that would not justify the cost of increasing interoffice WAN bandwidth, and because there is relatively little inter-office calling. Most calls from the regional offices go via the PSTN to Cisco customers and local system integrator partners.

Lessons Learned: Best Practices

Throughout its 3-year transition to IP telephony, Cisco learned many lessons, some of them the hard way. While this is expected in a very new technology, it does not have to be the normal operating procedure. Accordingly, Cisco imparts its best practices to its customers to alleviate or prevent such pain.

Cisco ensured high quality voice calls by accommodating QoS features on its entire network.

Planning a QoS strategy prior to deployment saves time and money, and eliminates unhappy users.

A simple dial plan can ease administration and ongoing maintenance. Avoid potential routing loops, and develop a plan for dealing with unauthorized phones. One strategy allows unauthorized phones to register but then places restrictions on outgoing calls.

Use the RFC1918 network-10 addressing option and separate VLAN features to separate traffic and manage IP addresses. This allows administrators to use one switch port per desk (phone plus PC), rather than two switch ports.

When an IP network carries voice, reliability is essential. In case of an emergency, people need to summon assistance by dialing 911. When using inline power to switches and routers, make sure they are connected to an uninterruptible power supply to guarantee dial tone if the power should go out.

If bandwidth permits, use a single codec throughout the campus to minimize the need for transcoding resources, which can add complexity to network design.

If you plan to keep your existing voicemail system provide transparent functionality to IP phone users during migration. Cisco discovered the need for transparent voicemail capability during its first pilot test, and met that need during its campus-wide transition by using the Cisco DPA 7630.

Do not give users an option. They will accept a new IP phone more readily if it is the only phone on their desk. This also reduces the number of handsets to track.

It is true that telecom and data teams do not always cooperate on joint projects. For true success, engage both teams early in the migration process. This way, telecom staff will embrace the switch-like configuration and operation of CallManager. The data staff may come to appreciate voice-management practices and adopt policies such as frequently changing passwords on switches and routers to restrict access. Above all, everyone needs advance notification of pending maintenance activities.

For More Information

For detailed information about migrating to IP telephony, deploying quality of service features, and deploying AVVID networks, Cisco offers many resources on its Web site. Among these are the following:

http://www.cisco.com/univercd/cc/td/doc/product/voice/index.htm

http://www.cisco.com/univercd/cc/td/doc/product/voice/ip_tele/index.htm