White Paper

MPLS and IP Quality of Service In Service Provider ATM Networks

Introduction

Equipment that combines switching and routing is becoming increasingly important. Carriers are discovering that they need to offer IP services on their ATM or Frame Relay networks in order to meet their customer needs. ISPs are realizing they need to add traditional "Layer 2" capabilities and services, such as traffic engineering and virtual private networks (VPNS), to their IP networks. Cisco introduced Tag Switching, now standardized as Multiprotocol Label Switching (MPLS), to meet these service provider requirements.

One of the most important applications of MPLS is IP+ATM networks: networks that offer traditional ATM and Frame Relay services along with providing optimized IP support using ATM MPLS. Equally important are the new services such as IP VPNs, which MPLS brings to both IP+ATM networks and router networks.This white paper concentrates on MPLS in ATM networks, and covers the following topics:

• Drivers behind MPLS
  
  
• Integrating IP with ATM
  
  
• Routing on ATM switches
  
  
• Building Internets on ATM
  
  
• MPLS in the Layer 3 packet world
  
  
• • Control component
    
    
  • Forwarding component
    
    
• MPLS in the ATM WAN
  
  
• • MPLS elements
    
    
  • Forwarding component
    
    
  • Control component
    
    
  • Cell interleave problem
    
    
  • Virtual circuit (VC) merge-capable switches
    
    
  • Labeling VC connections and cross-connects
    
    
  • Label switch controllers
    
    
  • IP+ATM capability
    
    
• An ATM MPLS point of presence (POP)
  
  
• Dual backbones: Traditional ATM and ATM MPLS or Packet over SONET
  
  
• Virtual private networks
  
  
• • Forwarding in a Cisco MPLS VPN service
    
    
  • Control in a Cisco MPLS VPN service
    
    
  • Attributes of Cisco MPLS VPNs
    
    
• Migrating MPLS over a traditional ATM cloud
  
  
• Quality of service (QoS) in an MPLS network
  
  
• • Best-effort traffic and IP QoS requirements
    
    
  • The differential services approach to QoS
    
    
  • MPLS traffic engineering
    
    
  • Harder QoS in IP+ATM networks
    
    
  • QoS for MPLS VPNs
    
    
  • Other QoS topics
    
    

Drivers Behind MPLS

MPLS is an innovative method for forwarding packets through a network. Ordinary IP routing is based on the exchange of network reachability information. This information is used at each hop in the network to determine how to handle the packet. Data in IP packets such as IP Precedence information, as well as information on VPN membership, is usually not considered when forwarding packets; in order to get maximum forwarding performance, only the destination address is considered.

MPLS is based on the assignment of labels to packets. The label summarizes all the information about how to direct the packet, including:

• Destination
  
  
• Precedence
  
  
• VPN membership
  
  
• QoS information from Resource Reservation Protocol (RSVP)
  
  
• The route for the packet, as chosen by traffic engineering (TE)
  
  

Because forwarding decisions based on some or all of these different sources of information can be achieved with a single table lookup from a fixed-length label, forwarding decisions based on more than just the destination address become feasible. Consequently, MPLS has many uses:

• MPLS can be used to offer IP VPN services that are highly scalable and very easy to manage.
  
  
• In wide-area networks (WANs) with ATM infrastructures, MPLS is an easy solution for integrating routed protocols with ATM. "Traditional" IP over ATM involves setting up a mesh of permanent virtual circuits (PVCs) between routers around an ATM cloud, and the Next Hop Resolution Protocol (NHRP) achieves a similar result with switched virtual circuits (SVCs). This approach has numerous problems, all arising from the way that the PVC links between routers are overlaid on the ATM network, making the ATM network structure invisible to the routers. A single ATM link failure, for example, may make several router-to-router links fail, creating problems with large amounts of routing update traffic and subsequent processing. Another problem is that, without extensive tuning of routing weights, all PVCs are seen by IP routing as single-hop paths with the same cost. This setup may lead to inefficient routing in the ATM network.
  
  
• Overlaying IP on ATM has other disadvantages, particularly in support of advanced IP services such as IP multicast and RSVP. Support of these services requires much time and work in the standards bodies and implementation, and the resulting mapping between IP features and ATM features is often approximate.
  
  
• When applied to ATM, MPLS solves these problems by truly integrating IP and ATM rather than overlaying IP on ATM, making the ATM infrastructure visible to IP routing and removing the need for approximate mappings between IP and ATM features. MPLS does not need ATM addressing and routing techniques such as Private Network-Network Interface (PNNI), although these can be used in parallel if required.
  
  
• When used with ATM hardware, MPLS makes use of the ATM queuing and buffering capabilities to provide different classes of service (CoSs). This scenario allows direct support of IP Precedence and CoSs on ATM switches, without complex translations to the ATM Forum service classes.
  
  
• MPLS can be used to provide VPNs. VPN services are important for providing enterprises with private IP networks needed to build their infrastructures. When an ISP carrier offers a VPN service, the carrier supports many individual VPNs on a single infrastructure. With an MPLS backbone, VPN information can be dealt with only at the ingress and exit points, with MPLS labels carrying packets across a shared backbone to their correct exit point. In addition to MPLS, the Multiprotocol Border Gateway Protocol (BGP) is used to deal with information about the VPNs. The combination of MPLS and Multiprotocol BGP makes MPLS-based VPN services fundamentally easy to manage, with straightforward operations to manage VPN sites and VPN membership. It also makes MPLS-based VPN services extremely scalable, with one network able to support tens or hundreds of thousands of VPNs.
  
  
• VPN services are an example of how MPLS can support a hierarchy of routing knowledge. Another important application of routing hierarchy is the isolation of Internet routing tables in service provider network cores. As with VPN information, MPLS allows the Internet routing table to be dealt with only at the ingress and exit points of a service provider network. With MPLS, transit traffic entering at the edge of the provider's autonomous system¹ (AS) can be given labels that are associated with specific exit points. As a result, internal transit routers and switches have to deal with connectivity among only the provider's edge routers, shielding the core devices from the overwhelming routing volume exchanged in the Internet. This separation of interior routes from full Internet routes also provides better fault isolation and improved stability.
  
  
• Other benefits of MPLS include its TE capabilities that are needed for the efficient use of network resources. Traffic engineering gives the user the ability to shift traffic load from overutilized portions to underutilized portions of the network. This shift can be done based on different predicates such as the traffic destination, traffic type, traffic load, time of day, and so on.
  
  

Cisco also supports an alternative IP-over-ATM scheme, Multiprotocol over ATM (MPOA), which uses NHRP. Unlike MPLS, MPOA overlays IP on ATM rather than fully integrating them. Although they do not share many of the advantages of MPLS in the WAN, MPOA and NHRP are cost-effective technologies for interconnecting nearby emulated LANs (ELANs) at high speeds.

Integrating IP with ATM

So far, IP routing protocols have always run on top of ATM or Frame Relay, with no real integration. ISPs, for example, have built ATM or Frame Relay cores inside their routed networks and have used them to build pipes between the routed edges. In other words, two IP routed networks are connected together using PVCs across an ATM or Frame Relay cloud. This setup creates an overlay model that is neither scalable nor manageable (Figure 1a), mainly because all routers on the cloud become IP neighbors. It also uses network resources inefficiently because the ATM links are invisible to IP routing. This means, for example, that a PVC using many hops is used by IP routing just as readily as a single-hop PVC, as both PVCs are each a single IP hop.

Another problem arises because routing protocols such as Open Shortest Path First (OSPF) do not perform very well over large fully meshed clouds because of the link-state update duplication and the large number of neighbor state machines that have to be maintained. The route oscillation caused by circuit failures sends router CPU utilization through the roof and can cause an indeterministic route convergence behavior. Experience has shown that this becomes a problem with a full mesh bigger than about 30-50 routers.

Figure 1: IP Over ATM

MPLS solves the meshing problem by eliminating the notion of an ATM "cloud;" the ATM links are treated as IP links and each ATM switch can become an IP routing peer (Figure 1b). Putting IP intelligence into the ATM switches resolves the IP scalability problem by eliminating the overlay of IP links on ATM, and making a one-to-one mapping between them. This true integration of the layers offers a distributed routing/switching model that takes advantage of the wealth of capabilities offered in each layer. The router component is needed to make use of the routing algorithms such as OSPF and BGP Version 4 (BGP4) for exchanging reachability information and calculating paths. The MPLS control component is needed to translate that reachability information into elements that can be understood by the switches. Finally, the switching component uses advanced hardware capabilities to switch data at wire speed.

Routing on ATM Switches

Routing on ATM switches can be achieved by either integrating the routing engine inside the switch or by using separate routing controllers (a router).

The integrated solution can be achieved by running routing and MPLS software on the switch control processor. This configuration is available on the Cisco LightStream^® 1010, Catalyst^® 5500, and Catalyst 8540 multiservice switch router (MSR) ATM switches. The controller model makes use of a separate router that controls the switch hardware. This separate router is called a label switch controller (LSC). The LSC can be either a routing card in the switch shelf or an external router. The LSC handles all the IP functionality and interacts with the switch via either the backplane (for a router card) or an external control interface. The first LSC offered by Cisco is an external controller for the BPX^® 8650. Also the IGX 8400 series supports an external LSC. The MGX^™ 8800 series will later use an LSC running on the route processor module (RPM) in the switch shelf.

From an outcome point of view, both the integrated and controller methods deliver the same result—provide an intelligent way of integrating routing and ATM. The differences lie in the new operational model of using separate controllers rather than integrated solutions. The controller model has the advantage of separating the services into separate logical entities, each having a roadmap that does not interfere with the other. In other words, if an external router is controlling a BPX 8650 or IGX 8400 series switch that is running PNNI and SVC services, an IP MPLS upgrade can be done without disturbing the operation of the PNNI and SVC services. In the WAN space, this functionality is attractive.

On the other hand, extra space occupied by external device, and the cost of an external controller, both in hardware and use of a port on the switch, may not be justified in all situations. Therefore, it was decided to use the integrated approach on the LightStream 1010.

Building Internets on ATM

The Internet is a collection of service providers offering IP services to their customers and is interconnected either directly or via high-speed network access points (NAPs). The NAPs are usually managed by a dedicated provider acting as a point of contact for coordination and connectivity purposes. Each ISP maintains multiple POPs that serve as concentration points for customer connectivity in multiple regions. POPs can be interconnected via an ATM infrastructure or via direct high-speed, leased-line connections. Currently, ISPs use BGP4 for the purposes of interdomain connectivity. BGP4 offers a wide range of capabilities in segmenting providers' networks and offering routing policies that define the providers' administrative and political boundaries.

Integrating ATM infrastructures into the Internet model is as simple as providing an IP continuity between the ATM network and the rest of the IP world. IP is integrated over ATM inside the AS using MPLS, and the AS is connected to the rest of the Internet via BGP, as illustrated in Figure 2.

This configuration is accomplished using IP+ATM, with the ATM switches continuing to operate according to the ATM Forum and ITU-T standards while running MPLS in parallel. In other words, other network applications such as PNNI, SVC, and automatic routing manager can still operate independently of the MPLS application offering routed services.

Figure 2 shows an IP+ATM network offering both Internet and ATM services, illustrated by the two shaded areas. The virtual path identifiers/virtual channel identifier (VPI/VCI) space on the multiservice switches (where the two shaded areas overlap in Figure 2) is divided between MPLS and the ATM Forum services. The MPLS network is connected to other ISPs via the NAPs. The ATM network is running PNNI internally and is connected to other service providers. Note that there is no interaction between PNNI and MPLS except that they both share the ATM switch resources and link bandwidths. The MPLS implementations on the LightStream 1010, Catalyst 8540 MSR, BPX 8650, IGX 8400 series and MGX 8850 switches allow control of the resource allocation between MPLS and PNNI.

The remainder of this paper provides more details on defining the signaling needed to achieve MPLS over ATM. But first, it helps to give a background on how MPLS operates on whole packets as opposed to ATM cells.

Figure 2: An IP+ATM Multiservice Network

MPLS in the Layer 3 Packet World

MPLS is composed of two elements—forwarding and control. This section covers these elements as they are used in ordinary MPLS packet forwarding. Advanced services, including TE, VPNs, and CoS, are considered later.

Control Component

The control component of MPLS consists of IP routing protocols (typically OSPF or Intermediate System-to-Intermediate System [IS-IS]) running in conjunction with MPLS label allocation and maintenance procedures. The control component is responsible for setting up label forwarding paths along IP routes. The control component also maintains accuracy for the paths, given topology changes that might occur.

The OSPF or IS-IS routing protocol runs in the normal way, automatically creating forwarding tables in each MPLS label switch router. The MPLS Label Distribution Protocol (LDP) works in a way that is linked to the routing protocols, and works in parallel with them. Based on the routing information provided by OSPF or IS-IS, LDP exchanges the labels needed by the forwarding function. In a packet environment, LDP is used in a downstream label-allocation scheme, which works in the following manner: For each route in its routing table, the label switch router (LSR) allocates a label and creates an entry in its Label Forwarding Information Base (LFIB) with the incoming label set to the allocated label. The LSR then advertises the binding between the label (incoming) it created and the route to other adjacent label switch routers. When a LSR receives label binding information for a route and that information was originated by the next hop for that route, the switch places the label into the outgoing label of the LFIB entry associated with the route. This scenario creates the binding between the outgoing label and the route; it is illustrated in Figure 3.

Figure 3: Downstream Label Allocation

Forwarding Component

The forwarding component is based on the notion of label swapping. When a LSR based on router hardware (for example, a Cisco 7200 or 12000 series router) receives a packet with a label, the label is used as an index in a LFIB. Each entry in the LFIB consists of an incoming label and one or more subentries of the form (outgoing label, outgoing interface, outgoing link-level information). For each subentry, the LSR replaces the incoming label with the outgoing label and sends the packet over the outgoing interface with the corresponding link-level information.

In Figure 4, an unlabeled IP packet with destination 128.89.25.4 arrives to router A (RTA). RTA looks into its LFIB and matches the destination with prefix 128.89.0.0/16. The packet gets labeled with an outgoing label of 4 before being sent toward its next-hop router B (RTB). RTB receives a packet with a label (incoming) of 4, which is be used as an index in the LFIB. As such, the incoming label of 4 is swapped with outgoing label 9, and the packet is sent over outgoing interface 0 with the appropriate Layer 2 information (such as Media Access Control [MAC] address) according to the LFIB. Note that RTB did not do any prefix IP lookup based on the IP destination such as in RTA. Instead, RTB has used the label information to do the label forwarding. When the packet arrives at router C (RTC), RTC removes the label from the packet and forwards it as an unlabeled IP packet.

Figure 4: LFIB in an IP Packet Environment

MPLS in the ATM WAN

Before getting into more details on MPLS components over ATM, it helps to identify the different terminology and elements that will be used in this document.

MPLS Elements

This section defines multiple MPLS elements. Figure 5 illustrates these elements in a network environment.

• Label switch router (LSR)—A device that implements the MPLS control and forwarding components as already described.
  
  
• Label-controlled ATM (LC-ATM) interface—An ATM interface controlled by the MPLS control component. Cells traversing such an interface carry labels in the VCI field of a user-selected range of VPIs. The control component could be integrated in the switch or on an outside controller.
  
  
• ATM LSR—A LSR based on an ATM switch. It has LC-ATM interfaces.
  
  
• Packet-based LSR—A LSR that forwards complete packets between its interfaces. A packet-based LSR can have zero or more LC-ATM interfaces. Packet-based LSRs typically consist of MPLS software running on ordinary router platforms, for example, the Cisco 3600, 4700, 7200, or 7500 series. Sometimes there are some hardware features specifically for MPLS, as on the Cisco 12000 series.
  
  
• ATM edge LSR—A packet-based LSR that is connected to the ATM-LSR cloud via LC-ATM interfaces. The function of the ATM edge LSR is to add labels to unlabeled packets and strip labels from labeled packets. Note that edge LSRs are part of the same service provider network as the ATM-LSRs. Edge LSRs are not intended to be customer premises equipment (CPE) or customer located equipment.
  
  

Similar to MPLS in a packet environment, MPLS in an ATM environment comprises a forwarding component and a control component. This section covers these elements as they are used in ordinary MPLS packet forwarding. Advanced services, including TE, VPNs, and CoS, are considered later.

Figure 5: MPLS Elements

Control Component

The control component for MPLS over ATM is similar to router-based MPLS. A standard IP routing protocol such as OSPF or IS-IS runs in the network, alongside the LDP. The LDP is needed to bind VCIs to IP routes. As discussed earlier, the control component can be integrated in the switch or on outside controllers.

ATM LSRs use the downstream-on-demand allocation mechanism. Each ATM LSR uses standard IP routing protocols to populate a Forwarding Information Base (FIB) that contains a list of all IP routes that the ATM LSR uses. This function is either embedded in the switch or runs on an outside controller. For each route in its FIB, the ATM edge LSR identifies the next hop for a route. It then issues a request via the LDP to the next hop for a label binding for that route.

When the next-hop ATM LSR receives the request, it allocates a label and creates an entry in its LFIB, with the incoming label set to the allocated label. The next action depends on whether the label allocation is in "independent" mode or "ordered" mode. In independent mode, it immediately returns the binding between the incoming label and the route to the LSR that sent the request. However, this return may mean that it is not able to forward labeled packets that arrive, because the ATM LSR may not yet have an outgoing label/VCI for the route. In ordered mode, it does not immediately return the binding, but waits until it has an outgoing label.

Forwarding Component

The ATM forwarding operation is based on switching cells by swapping VCIs and VPIs, and in an ATM environment the MPLS forwarding function is done by the normal switch operation. The label information needed for MPLS can be carried in the VCI field within one or a small number of virtual paths (VPs). Figure 6 shows the forwarding operation of an ATM label switch. Note that the labels are actually the VCIs.

In Figure 6, an unlabeled IP packet with destination 128.89.25.4 arrives to RTA edge label switch router A (RTA). RTA looks into its LFIB and matches the destination with prefix 128.89.0.0/16. RTA sends the ATM adaption layer 5 (AAL5) frame as a sequence of cells on VCI 40. RTB, which is an ATM LSR controlled by a LSC, performs a normal switching operation by switching cells coming on interface 2/VCI 40 to interface 0/VCI 50.

In ordered mode, the next-hop LSR sends a new binding request to its next hop, and the process will repeat until the destination ATM edge LSR is reached. This returns a label binding to the previous ATM-LSR, causing it to return a label binding, and so on until the label bindings along the path are established.

Figure 6: LFIB in an ATM Environment

Figure 7 illustrates ordered behavior. ATM edge LSR RTA is an IP routing peer to ATM-LSR RTB. In turn, ATM-LSR RTB is an IP routing peer to ATM-LSR RTC. RTA-RTB and RTB-RTC exchange IP routing updates over VPI/VCI 0/32.

RTA sends a label binding request toward RTB in order to bind prefix 128.89.0.0/16 to a specific VCI. RTB allocates VCI 40 and creates an entry in its LFIB with VCI 40 as the incoming label. RTB then sends a bind request toward RTC.

RTC issues VCI 50 as a label. RTB sets the outgoing label to VCI 50. This information is now used by RTB to switch cells coming on VCI 40 to VCI 50. RTB then sends a reply to RTA with the binding between prefix 129.89.0.0/16 and VCI 40. RTA creates an entry in its LFIB and sets the outgoing label to VCI 40.

Independent mode operation is similar to that shown in Figure 7, except that the steps labeled 7 and 8 may occur concurrently with Step 3.

In independent mode, the LSR that initiated the request receives the binding information, creates an entry in its LFIB, and sets the outgoing label in the entry to the value received from the next hop. The next-hop ATM LSR repeats the process, sending a binding request to its next hop, and the process continues until all label bindings along the path are allocated.

Figure 7: Downstream-on-Demand Label Allocation, Ordered Mode

Cell Interleave Problem

The problem of having multiple sources transmitting data to the same destination causes some challenges with label VC allocation over ATM. An ATM LSR that receives binding requests from different upstream neighbors toward the same prefix have to request multiple outbound labels from its downstream neighbor. If the ATM LSR allocates only one outgoing VCI, then cells from different AAL5 frames would potentially be interleaved and dropped at the receiving end. Allocating different outbound VCIs for the same destination ensures that cells are received in order. This setup is illustrated in Figure 8.

Figure 8a shows a hypothetical situation. RTB has received two different binding requests for prefix 128.89.0.0/16 from RTA and router D (RTD). Hence, RTB creates two entries in its LFIB and assigns incoming labels for each request. In this example, RTB has assigned VCI 40 for RTA and VCI 70 for RTD. In case RTB doesn't already have an outbound label for the prefix, RTB sends a binding request toward RTC and gets assigned VCI 50 as an outbound label. As a result, cells arriving from RTA and RTD on VCIs 40 and 70 would be sent over VCI 50 and would potentially get interleaved, causing AAL5 frames to be discarded.

Figure 8b shows the same scenario, with the difference that RTB has now requested two outgoing labels for prefix 128.89.0.0/16. RTB is assigned two VCIs, 50 and 60. Cells from RTA are switched using crossconnect (40, 50) and cells from RTD are switched using crossconnect (70, 60). As such, complete noninterleaved AAL5 frames are received at the destination. This example explains how MPLS supports switches that do not have "VC merge" capability.

Figure 8: Problem of Cell Interleave

VC Merge-Capable Switches

In order to better utilize the VC space, it is possible to implement VC merge. VC merge allows the switch to transmit cells coming from different VCIs over the same outgoing VCI toward the same destination. In other words, it allows multipoint-to-point connections to be implemented by queuing complete AAL5 frames in input buffers until the end of frame has been received. The cells from the same AAL5 frame will all be transmitted before cells from any other frame are sent. This setup requires more buffering capabilities inside the switch, but no more buffering than is required in IP networks. The small additional delay caused by VC merge is of little concern, because VC merge is designed for IP traffic and does not need to be used for delay-sensitive traffic. IP traffic has good delay tolerance compared to other traffic that might be carried on an ATM network. VC merge is illustrated in Figure 9.

Figure 9: VC Merge

In Figure 9, RTA and RTD are sending traffic toward prefix 128.89.0.0/16. RTB has a single outbound VCI 50 bound to that prefix. Cells coming over VCIs 40 and 70 are buffered in separate queues of RTB until complete AAL5 frames have been formed. In this example, an end of frame has been detected over VCI 70 and the complete frame has been transmitted over VCI 50. An end of frame has not been detected for cells coming over VCI 40, and these cells are held back in the input buffer, solving the cell interleave problem and minimizing VC usage.

Label VC Connections and Crossconnects

A link between two ATM LSRs, or between an ATM edge LSR and an ATM LSR, is an ordinary ATM link. Since ATM MPLS uses the VCI fields of a few separate VPIs to carry a label, each label on a link corresponds to a different virtual circuit (VC). These VCs are called label virtual circuits (LVCs). LVCs are neither switched virtual circuits (SVCs) nor permanent virtual circuits (PVCs), and are set up using LDP instead of ATM Forum signaling protocols. LVCs, PVCs, and SVCs may all be used on the same link; however, they use different parts of the VPI/VCI space.

As illustrated in Figure 10, at least two distinct types of LVCs are used on each link:

• Signaling LVC—This VC is used to carry IP packets that are reassembled and examined at each ATM LSR. It is used to carry routing information (BGP, OSPF, IS-IS, and so on) and LDP. It might also be used to carry management traffic, such as Simple Network Management Protocol (SNMP) traffic or Internet Control Message Protocol (ICMP) traffic. By default, this VC has VPI and VCI (0, 32), which can be reconfigured if desired.
  
  
• Ordinary LVCs—These LVCs carry label-switched data. Packets on ordinary LVCs are crossconnected by ATM LSRs without being reassembled. On each link, all ordinary LVCs are within the same VP or small set of VPs.
  
  

Figure 10: Interconnecting ATM Label Switch Routers

An ATM LSR differs from an ordinary ATM switch in the way connections are set up. Normally an ATM connection is set up by control software running a connection routing protocol such as PNNI or automatic routing management. In ATM MPLS, a piece of software called a label switch controller is used.

Label Switch Controllers

A label switch controller (LSC) is part of an ATM LSR. The LSC runs an IP routing protocol such as OSPF or IS-IS, in addition to MPLS software. The IP routing software maintains knowledge of the layout of the network. Using this information, LDP establishes labels (such as VCs) on links connected to the ATM LSR. When the LSC has established incoming and outgoing labels for the same route in its LFIB, it then instructs the switch fabric to set up a connection with the parameters (incoming interface, incoming label VCI, outgoing interface, outgoing label VCI). Figure 11 shows possible locations for the LSC.

There are three possible locations for the LSC:

• Switch control card—The LSC software might run on the ATM switch, on the main control card. In LightStream 1010 or Catalyst 5500 or 8500 ATM-LSR, the LSC software runs on the main control card, the ATM switch processor.
  
  
• Another card in the switch shelf—The software might run on the ATM switch on a card separate from the main switch control card. In the MGX 8800, a route processor module (RPM) card in the switch will be used as the LSC later; similarly LSC function will be supported on a universal router module (URM) in the IGX 8400 series.
  
  
• Separate hardware—The LSC may also be a separate piece of hardware. A Cisco BPX 8650 IP+ATM switch consists of a BPX 8600 ATM switch shelf and an LSC based on a Cisco 7200 series router. The LSC and switch are interconnected by a switch control link. For the BPX 8650, the switch control link is an ATM link. This link is used in a different way with the other ATM interfaces on the LSR—it is used to connect the signaling LVCs from all other interfaces on the switch to the LSC, but it does not often carry any data. Exactly the same architecture is supported in the IGX 8400 series.
  
  

A LSC sets up connections in the switch fabric by way of a switch control interface. In the case of the LightStream 1010 or Catalyst 5500 or 8500, this interface is an internal interface within Cisco IOS^® software. In the case of the BPX 8650, IGX 8400 series and MGX 8850, a switch control interface is used, which is either an external interface or an channel between two cards in the switch.

BPX 8650 or IGX 8400 Label Switch Router: Controlling a Switch with a LSC

Figure 11b shows how a LSC is connected in a BPX 8650 or IGX 8400 series switch. The physical connection between the LSC and the BPX or IGX-series ATM switch shelf is the virtual switch interface (VSI) control link, which is an ATM link. The VSI control link could be an STM-1 link, connected to one port of a four- or eight-port broadband switch module (BXM) or a two- or four-port universal switch module (UXM) Synchronous Transport (STM-1) card.

Figure 11: Label Switch Controller Locations

Enabling LSC control of a switch requires:

• Declaring that an ATM interface on the LSC is a LSC interface
  
  
• Enabling a port on the switch as a control interface
  
  

The data connections between the LSC and switch shelf consist of two sets of VCs:

• Signaling LVCs—The signaling label VCs from every interface of the ATM switch must be connected through to the LSC (Figure12 a). The signaling LVC on each interface is on VPI and VCI (0, 32) by default, but are generally crossconnected to a different VCI on the switch control link. This VCI is chosen by the LSC software, which requests the setup of the crossconnects as part of its initialization.
  
  
• Switch control VCs—The LSC uses an interface control protocol to discover the port configuration of the switch and make switch connections. This protocol operates using VCs connected to each port card, called the external control VCs (Figure 12b). In the BPX 8650, there may be up to 12 of these, one for each BXM port card in the BPX8650; in the IGX 8400 series there may be up to 30, one per UXM card. The external control VCs are set up automatically if external control is enabled.
  
  

Figure 12: Connecting a BPX 8650 and Label Switch Controller

Using the infrastructure of signaling LVCs and external control VCs, the LSC can establish label bindings with the neighboring ATM edge LSRs, and consequently request the set up of LVC crossconnects in the switch. Most data LVCs bypass the LSC², as shown in Figure 12c.

IP+ATM Capability

We have explained how a label switch controller can be added to an ATM switch to give it MPLS capability. We have also described how an ATM network can be used to simultaneously provide MPLS service and traditional ATM switching. The key to doing this configuration is the "IP+ATM" capability of Cisco ATM switches. Figure 13a shows an ATM switch with an MPLS label switch controller, which has already been shown in more detail in Figure 12. PNNI control software can be connected to an ATM switch in the same way. Furthermore, Cisco IP+ATM switches allow an LSC and a PNNI controller to be simultaneously connected to the same switch. In other words, the same switch can support both optimized IP services using MPLS and traditional ATM services using PNNI.

Figure 13: Cisco IP+ATM Switches

Because Cisco IP+ATM switches directly support both MPLS and PNNI services, Cisco IP+ATM networks offer both native IP services and native ATM services. A Cisco IP+ATM network physically consists of ordinary ATM switches and links. As part of the initial configuration of the network, the operator assigns resources of the ATM network to PNNI and MPLS (see Figure 14). The resources involved include:

• Bandwidth on links
  
  
• VPI/VCI space on links
  
  
• VC connection table spaces
  
  
• Traffic management
  
  

Figure 14: Cisco IP+ATM Switches

This "partitioning" of resources is quite flexible, with arbitrary divisions of resources between the different "control planes." Partitioning can involve giving fixed allocations of resources to the control planes, or can involve a "pool" of link bandwidth or connection table spaces shared between the control planes. Furthermore, the concept of having controllers independently control a switch extends to more than two controllers—Cisco IP+ATM switches have the ability to support four or more control planes.

An ATM MPLS Point of Presence

An example of an ATM MPLS POP is shown in Figure 15a. A POP consists of an ATM LSR and several edge routers connected to customer sites. Signaling LVCs connect every edge LSR in the POP to the LSC in the ATM LSR. In this example, each of the edge routers is connected to only one ATM LSR, but this need not be the case.

Figure 15: An ATM MPLS POP

Using an Access Switch in an ATM MPLS POP

In the previous POP example, the customer access lines are connected directly to the edge LSRs. Figure 16 shows another alternative, in which the customer lines are brought into the POP by way of an access switch such as the MGX 8220. Customer lines are connected to the access switch, and then "backhauled" to the edge LSRs by way of PVCs.

Figure 16: Using an Access Switch in an ATM MPLS POP

A Fully Integrated POP

One more alternative is to use a single device that integrates an ATM LSR, edge LSRs, and an access switch, as shown in Figure 17. The MGX 8800 IP+ATM switch does this. In the MGX 8850, router cards called route processor modules (RPMs) act as edge LSRs. One of the RPMs also acts as a label switch controller, giving ATM-LSR function to the MGX 8800. All these functions are combined in a single chassis, shown in Figure 17b. The Cisco 6400 access switch has similar capabilities.

Figure 17: MGX 8800 as an Integrated ATM MPLS POP

Dual Backbones: Traditional ATM and ATM MPLS or Packet over SONET

The networks shown so far have combined MPLS and traditional ATM services on the same switches. Another alternative, which may be preferred by some providers, is to have two separate backbones in the networks—one for traditional ATM services, and one for IP services. Cisco IP+ATM edge switches support this alternative in a very flexible way, as shown in Figure 18. A single IP+ATM edge switch supports full edge LSR and LSC functions, as well as traditional ATM soft PVC (SPVC) and SVC services.

Figure 18: Supporting IP+ATM Services Using Dual Backbones

From a functional perspective, ATM MPLS services and traditional ATM services are then carried separately. However, with a Cisco IP+ATM backbone, as previously described, the separate MPLS and traditional ATM switching functions are carried on the same switch backbone. This setup is illustrated in Figure 18b. Another alternative, shown in Figure 18c, is to use physically separate backbones for MPLS services and traditional ATM services. The MPLS backbone can use ATM LSRs, or router-based LSRs such as the Cisco 7500 series, or Cisco 12000 series gigabit switch routers (GSRs). Nearly any link types can be used in the MPLS backbone, including packet over Synchronous Optical Network/ Synchronous Digital Hierarchy (SONET/SDH), or packet over wave-division multiplexing (WDM), if desired.

Cisco IP+ATM edge switches readily connect to dual backbones. Even from a single link to an end-customer site, some data-link connection identifiers (DLCIs) or VCIs can connect into the MPLS-based IP services, while others connect into traditional ATM or Frame Relay SPVC services.

Virtual Private Networks

Figure 19 shows the concept of an IP VPN service. One service provider network supports many different IP VPNs. Each VPN appears to its users to be a private network, separate from all other networks. Within each VPN, there is any-to-any connectivity: each site can send IP packets directly to any other site in the VPN, without having to go through a central site. A simple use of a VPN is an intranet, forming the wide-area IP network of a corporation. Another use is an extranet, linking several corporations.

Figure 19: Many VPNs Provided by One Network

In a VPN service based on Cisco MPLS, each individual VPN is identified by a route distinguisher (RD)³. In Figure 19, three different VPNs are shown with RDs 12, 101, and 2332. Although Figure 19 shows only three VPNs, many more can be supported by a single network. Cisco MPLS VPN networks initially support thousands of VPNs, and later this support is extendable to hundreds of thousands, and even millions if required.

Figure 20 shows more detail on how a Cisco MPLS network supports VPNs. An ordinary MPLS network is surrounded by provider edge (PE) routers, which are edge LSRs that support VPN functions. In combination with the provider edge routers, the MPLS network forms a network supporting VPN services. Customer sites are connected to the provider edge routers in any of the ways supported by any MPLS network: Frame Relay, ATM, digital subscriber line (xDSL), Point-to-Point Protocol (PPP), and so on. Any Cisco device that can act as an edge LSR can act as a PE router, including the MGX 8800.

Figure 20: Providing VPN Services Using an MPLS Network

In a Cisco MPLS VPN service, the customer sites run ordinary IP. They do not need to run MPLS, or IPSecurity (IPSec), or other special VPN functions. At the PE router, a RD is associated with each link to a customer site. The links may be physical links, PPP links, individual Frame Relay or ATM virtual circuits, xDSL links, or other links. The route distinguisher is configured at the PE router as part of the setup of a VPN site. It is not configured on the customer equipment, and is not visible to the customer.

As with other MPLS networks, functions in MPLS VPN equipment can be divided into a forwarding component and a control component:

The core network uses ordinary MPLS forwarding and control, and operates as described earlier in this white paper. The provider edge routers have some additional functions to support VPN services. Critical to the function and scalability of VPN services is the use of the BGP4, which is described below.

Forwarding in a Cisco VPN Service

Figure 21 shows how packets are forwarded in a Cisco MPLS+BGP VPN service.

Figure 21: Forwarding Packets in a Cisco MPLS VPN Service

Note that the IP addresses 10.4.1.1 and 171.69.2.4 have different "scopes." The IP address 10.4.1.1 is part of the VPN with route distinguisher 2332. Only packets inside that VPN are able to reach this address. There may be other IP addresses 10.4.1.1 in other VPNs, but they are distinguished by having different RDs. Address 171.69.2.4 is part of the service provider's backbone, and is not part of any VPN. It is probably a registered IP address from a range issued by the Internet Assigned Numbers Authority.

Control in a Cisco MPLS+BGP VPN Service

Figure 22 shows the extra control functions in a Cisco MPLS+BGP VPN service. The PE routers communicate with each other using the BGP4, with multiprotocol extensions. BGP4 is the standard routing protocol for communicating Internet routing tables. Multiprotocol BGP4 is a standard extension to BGP4 (RFC 2283). The PE routers use multiprotocol BGP4 to send VPN routing information to each other, allowing each PE router to build a complete VPN routing table. Part of a VPN routing table is shown in Figure 22.

Figure 22: Control Functions in a Cisco MPLS VPN Service

PE router RTA knows that it has a site with RD = 2332, because it has been configured with this information. It learns that an IP address range (RD = 2332, Destination prefix = 10.1.0.0/16) is reachable there. RTA gets this information either by running an IP routing protocol (for example, Routing Information Protocol [RIP] or OSPF) out to the customer site, or by static configuration. RTA records the address range (RD = 2332, Destination prefix = 10.1.0.0/16), and also picks a label value, say 789, to correspond to this "route." RTA tells all the other PEs about this route and label using Multiprotocol BGP. RTA records its own IP address, 171.69.2.3, as the destination PE router for this address⁴.

Similarly, RTB learns that the route (RD = 2332, Destination prefix = 10.4.0.0/16) is on a link directly connected to RTB. It picks a label, such as 122, to correspond to this route. RTB then advertises this route to all other PE routers, which record the route and label in their tables, along with the RTB IP address (171.69.2.4). This scenario explains how, in Figure 21, RTA knows it should put label 122 on a packet that matched the route (RD = 2332, Destination prefix = 10.4.0.0/16). RTC and RTD also send routes for RD = 2332. In this way, each PE router gets access to complete routing information for all the RDs it supports. Note that the assignment of VPN labels does not need to be coordinated across different PE routers. For example, in Figure 22, label 1001 from RTC (171.69.2.5) is different from label 1001 from RTD (171.69.2.6), because the labels are applied to packets sent to different PE routers. The outer label shown in Figure 21 ensures that the each packet reaches the correct PE router.

Attributes of Cisco MPLS+BGP VPNs

Privacy and Security

• Cisco MPLS+BGP VPNs have the same level of privacy as Frame Relay networks. In a Frame Relay network, each packet carries a label, called a DLCI, which ensures correct delivery of the packet, provided the network is configured correctly. Because MPLS+BGP VPNs label each packet with destination information, they achieve the same level of privacy as Frame Relay networks.
  
  
• In Cisco MPLS+BGP VPN services, routing information is kept private. Route distribution is controlled by the route distinguisher⁵. This setup keeps each customer's routing information separate, and ensures that customers can reach only legal addresses for their VPN. If routing protocols (RIP, OSPF, and so on) run out to customers' sites, then each customer gets access only to routes with the correct route distinguisher.
  
  
• "Spoofing" is impossible. Because MPLS labels are applied by the provider's edge router, a customer cannot pretend to be a member of another VPN by applying different labels. This pretense would cause the PE routers to simply discard the packets. Only ordinary IP packets are accepted from customer sites, and the network forwards them as normal for the customer's VPN⁶.
  
  
• Denial-of-service attacks may be prevented using mechanisms that are described in more detail in the section "Quality of Service in an MPLS Network." Briefly, any packets sent in excess of the customer's traffic contract can be marked and carried at lower precedence than in-contract traffic. If this setup is adopted, then excess, out-of-contract traffic from any source cannot adversely affect service for any traffic compliant with its traffic contract.
  
  

In addition, authentication and encryption can be used to give even greater security. Note that most Frame Relay customers are satisfied with Frame Relay privacy and do not encrypt their traffic. The same can be expected of MPLS+BGP VPN customers.

Customer Independence

• Customers can use any IP addresses. Because the provider network keeps customers' IP address ranges separate with route distinguishers, customers may use any IP addresses in their network. Many customers may use identical IP addresses. Customers can use registered addresses, private addresses (for example, IP network 10.x), or even completely illegal unregistered addresses.
  
  
• Customer equipment does not run MPLS or any special features. Any IP-capable equipment can be used at customer sites. The customer sites do not need to run "VPN routers" with IPSec, or any other special equipment⁷. However, running Cisco routers at customer sites may offer some advantages. Some optional Frame Relay congestion-prevention features are available if the CPE uses Cisco routers; Cisco CPE routers have industry-leading capability for setting and marking the desired CoS for IP packets.
  
  
• Customers automatically get to send any-to-any traffic within their VPN. There is no need to backhaul traffic to a central site in the VPN.
  
  

Scalability and Stability

• Routing distribution is highly scalable. In Cisco MPLS+BGP VPN services, a single set of BGP peerings between PEs is used, irrespective of how many different VPNs are supported. Alternative, unscalable techniques run different routing protocols in the provider backbone for each VPN. In addition, existing BGP techniques such as route reflectors can be used to help VPN route distribution scale even further. Initially, Cisco MPLS+BGP VPN services will support tens of thousands of VPNs and over 100,000 routes. This capability will soon be extended to hundreds of thousands of VPNs and millions of routes, and there is no reason why it cannot be extended even further. In addition, BGP has techniques such as "route flap damping," which prevents badly behaving customer sites from affecting BGP stability. Note that BGP the only routing protocol that has the scalability and stability to support networks the size of the Internet. The current Internet would not exist without BGP.
  
  
• Packet forwarding is highly scalable and stable. Cisco MPLS+BGP VPN services use two-level labeling, and only the top-level label is seen by the core MPLS network. In other words, the number of VCs used in the network depends only on the number of PE routers used. MPLS LVCs are not created for individual customer flows, making packet forwarding highly scalable. The core LSRs have no knowledge of VPNs, and do not respond to any changes in VPNs, making packet forwarding highly stable.
  
  

Management

Cisco MPLS+BGP VPNs have significant management advantages. These advantages are important even for smaller carriers and service providers, for whom the scalability advantages may not be important:

• The provider network uses only one set of BGP peerings, no matter how many VPNs it supports, resulting in simple administration of routing in the network.
  
  
• Adding new VPNs and sites is straightforward. Figure 23 shows the steps necessary to add an existing site to a small VPN with full connectivity. With VPNs based on Frame Relay or ATM PVCs, or IPSec or similar Layer 3 tunnels, the number of steps is dependent on the number of sites already in the VPN. In the example in Figure 23(a), there are four existing sites. Consequently, configuration Steps 1-4 are the setup of PVCs or Layer 3 tunnels. Steps 5-8 involve configuring the endpoints of these links at the existing customer sites. Steps 9-12 are the configuration at the tunnel endpoints at the new site. There may be more configuration steps at all routers for configuring IP routing to the new site. Obviously, far more configuration steps are required if a VPN has many sites.
  
  

Figure 23: Management Operations: Adding a Site to a VPN

With Cisco MPLS+BGP VPN services, configuration is much simpler, irrespective of the number of sites in a VPN. There are three steps: establish the link to the new site; configure the link at the PE router (with RD, and enabling a routing protocol or static address); configure the link at the customer site (enabling a routing protocol or default route). BGP automatically distributes reachability information and labels for the new site, and full communication to the new VPN site is established. This distribution requires no manual configuration, and typically takes a minute or less.

In this way, Cisco MPLS+BGP VPN services have extreme simplicity of management built into the technology that runs the service itself.

Migrating MPLS into a Traditional ATM Network

MPLS can be deployed into a traditional ATM network gradually, starting with just a single pair of ATM-LSRs in an otherwise purely ATM network. MPLS can be deployed across non-MPLS-capable switches using VP connections through the traditional ATM switches. These VP connections are called VP tunnels, because they allow MPLS to "tunnel" through traditional ATM switches. VP tunnels provide for easy migration to a full MPLS integration, although they do have several disadvantages. A possible migration strategy for introducing MPLS in an existing ATM network is shown in Figure 24.

Figure 24: Migrating MPLS over a Traditional ATM Cloud

• Figure 24a shows a starting position with routers connected by permanent virtual paths (PVPs) through an ATM cloud. This setup has most of the disadvantages of traditional IP-over-ATM networks, notably poor router peering scaling and poor bandwidth efficiency. However, it can support MPLS VPN services.
  
  
• Deploying some ATM LSRs in the network as shown in Figures 24b and c improves scalability: the number of PVPs to each edge LSR may be reduced to one (two if dual homed), and in some cases to zero, if an edge LSR is adjacent to an ATM LSR. ATM LSRs can be interconnected with ordinary ATM switches in a variety of ways. Careful deployment of ATM LSRs and PVPs can be used to make the PVP mesh more closely match the ATM link topology, and hence improve the bandwidth efficiency.
  
  
• Since ATM LSRs can also support traditional ATM services, ordinary ATM switches may be phased out, leading to full deployment of ATM MPLS, as shown in Figure 24d. The use of PVPs is no longer required.
  
  

An alternative to using PVPs is to use PVCs, which work only with edge LSRs, and not with ATM LSRs. In other words, the network in Figure 24a could use PVCs instead of PVPs, but none of the other networks in Figure 24 could use PVCs.

Quality of Service in an MPLS Network

For much IP traffic, the requirements for QoS are fundamentally weaker and more flexible than requirements for traditional virtual-circuit-switched data traffic. In order to be competitive, providers' IP networks must provide service-level agreements (SLAs) of an appropriate form for IP traffic with relatively weak QoS requirements, at low cost. The networks must also provide stronger QoS for some traffic. Cisco MPLS networks have unique flexibility for meeting all requirements for IP QoS:

• They support classes of service. Service providers might offer SLAs for some CoS, giving agreed QoS levels, averaged over periods of minutes or hours. These features can be engineered and provided at low cost.
  
  
• Stronger QoS guarantees may be provided for traffic that is critically important, or otherwise intolerant of varying QoS.
  
  
• They provide scalable and easy-to-manage ways of providing QoS to VPNs, and individual users and sites within VPNs.
  
  

Best-Effort Traffic and IP QoS Requirements

A major reason for the success of the Internet is that IP treats connectivity as being the fundamental requirement of a communications network. The Internet has been successful in its goal of allowing any host to communicate with any other, without setting up virtual circuits, reserving bandwidths, or performing any other actions with high overheads or costs. In the Internet, other considerations such as QoS are not treated as being as important as allowing any-to-any connectivity. Because of this philosophy, IP traffic is, in general, extremely tolerant of varying QoS. The typical World Wide Web user does not care whether a Web page downloads at, say, 100 kbps or 5 kbps. Transmission Control Protocol (TCP), which is used as the transport protocol for over half of IP traffic, automatically adjusts to the available end-to-end bandwidth and loss. User Datagram Protocol (UDP) has generally been used only by applications that are tolerant of packet loss. Although some IP applications now do require stronger QoS, most IP traffic still requires only a very loose guarantee of "connectivity," meaning that the available bandwidth between a given pair of IP addresses should be at least a few hundred bits per second, and that the delay should be no more than a couple of seconds⁸. This traffic merely requires "best effort" from the provider network(s).

Certain IP traffic does require better QoS guarantees, particularly voice over IP and similar real-time applications. Furthermore, many users, especially corporate customers, are prepared to pay a premium for good QoS for a proportion of their traffic, to ensure good application performance. In addition, users generally may tend to migrate in the longer term toward the networks that give, at a reasonable cost, the best QoS for "best-effort" traffic. Despite this fact, it is likely that a large proportion of the traffic in any MPLS network will continue to be very tolerant of widely varying QoS in day-to-day operations.

Another important aspect of IP traffic is that it is connectionless. While this is obvious, its effects on traffic patterns are not as obvious. Because IP is connectionless, IP applications have extreme flexibility in each location. Companies are finding that their departments are setting up Web servers and file servers away from their traditional head-office centers. In other words, because IP applications treat networks as being connectionless, the traffic in WAN links in corporate IP networks tends to become more meshed and "any-to-any" in nature. This scenario does not fit well with traditional "hub-and-spoke" VC connectivity. (See Figure 25 for an example.) Hub-and-spoke architectures lead to traffic passing through intermediate CPE in order to get to its destination. This setup is undesirable for customers, because they must pay for traffic to go across VCs twice, wasting bandwidth at the "hub" sites, and requiring that routing be managed. In response, many customers are adding more and more meshing to their VC networks. In other words, the VC connectivity increasingly reflects the "any-to-any" connectionless nature of IP traffic. Carrying connectionless traffic on increasingly complex meshes of VCs is a most inadequate solution, for several reasons:

• The complexity of CPE management increases with increasing routing complexity.
  
  
• Even if the provider network is able to support all the extra VCs required for meshes, the operational overheads of managing VC meshes become extreme.
  
  
• Determination of reasonable QoS parameters for all the PVCs becomes difficult.
  
  

The underlying problem is that IP traffic does not naturally fit with a connection-oriented service from a provider. For maximum efficiency and lowest cost, providers must offer a connectionless service to customers. Market research estimates [CIMI Corporation, 1998] have suggested that over 50 percent of current demand for IP VPN services is unmet. It is reasonable to assume that the lack of true connectionless IP services offered by carriers is an important reason why this demand is unmet. This paper has already characterized how connectionless MPLS VPN services simplify management for connectionless IP services. Now consider how QoS should be specified and supplied for a connectionless service.

Figure 25: Effects of Connectionless Traffic

In traditional VPNs, such as Frame Relay networks, committed information rates (CIRs) must be specified for every link, as shown in Figure 26a. As networks become more meshed, this becomes more difficult. For example, a full-mesh network of 100 sites would require 9900 separate CIRs to be provisioned. It is obviously unreasonable to dimension any-to-any networks in this way⁹.

Figure 26: Specifying Bandwidths for an IP Service

Aside from being connectionless, there are other reasons why providing QoS for IP traffic is fundamentally different from providing QoS in connection-oriented networks. Connection-oriented QoS is based on the premise that most traffic has QoS requirements that must almost always be met in order to provide adequate performance. Most IP applications, on the other hand, are tolerant of widely varying bandwidth, can tolerate periods of seconds or more of high loss, and are usually extremely tolerant of delay and delay variance. Because of this fundamental difference, traditional connection-oriented QoS tools for connection admission control and per-VC QoS guarantees are an unnecessary overhead for most IP traffic. They are also difficult or impossible to use without a fully specified matrix of traffic requirements, and we have seen how this is an unnatural requirement for an IP service.

Internet services already use a different model of QoS specification. As shown in Figure 26b, Internet users subscribe to a service by specifying access bandwidths for each of their sites. They do not specify a full matrix of bandwidths or any connection-oriented information, the natural way of structuring QoS demands for any connectionless IP service, because the access bandwidth requirements are easily estimated in proportion to the number of hosts or servers at each site.

Providers who offer IP SLAs without using traditional connection-oriented QoS methods have an important advantage. They do not have to deal with the equipment costs and operational overhead for connection-oriented QoS, and they closely meet their customers' requirements for QoS agreements suitable for connectionless networks. Providing "connectionless" SLAs is an important requirement for meeting current and future demand for IP services. Technologies that assist in providing QoS for IP traffic are considered next.

The Differential Services Approach to Quality of Service

Contracts for Access Bandwidths

Even though a full matrix of point-to-point bandwidths are not normally specified for MPLS networks, usage parameter control or "policing" can still be applied to constrain users' use of network resources. This fact is important, as it provides a basis for service providers' traffic planning to meet SLAs. In Figure 26b, a customer contracts for a certain access bandwidth at each site. In a simple case, the customer could be restricted to this bandwidth by setting the data link of the access line. However, more complex access contracts are possible and desirable.

An IP-layer policing function called committed access rate (CAR) is available in Cisco routers. CAR acts independently on each customer access link, or on each VC on a channelized access link. This use of CAR is shown in Figure 27. When used on edge LSRs or other provider access routers, CAR both enforces traffic contracts and marks packets according to the traffic contract. For example, a simple CAR contract may specify that a user site gets 64 kbps of traffic carried as premium traffic, while the remaining traffic, up to a total site bandwidth of 256 kbps of traffic, is carried as best effort.¹⁰

Figure 27: Cisco Committed Access Rate Policers

Far more sophisticated contracts are possible. Another possible contract for a site follows:

• The first 56 kbps is carried in class Premium 2.
  
  
• The next 128 kbps is carried in class Premium 1.
  
  
• The remaining traffic, up to a maximum of 1 Mbps, is carried as best effort.
  
  
• Any Simple Mail Transfer Protocol (SMTP) and File Transfer Protocol (FTP) traffic is carried as best effort and not premium.
  
  

The last point illustrates an important advantage of policing at the IP layer. CAR is able account for IP header information. In this example, it is used to specify that certain types of IP traffic, which are very tolerant of varying QoS, are automatically carried in the best-effort class and not counted against the limits for premium traffic.

CAR enforces the bandwidth contracts using token bucket policers, which permit burstiness in a short time frame, while limiting rates in a longer tomfooleries. Traffic classes are marked on the IP packets that are admitted into the provider network. Currently, CAR sets IPv4 precedence bits on packets. The meaning of these bits is being changed in the forthcoming differentiated services (DiffServ) standards from the Internet Engineering Task Force (IETF), but DiffServ is backwards-compatible with the original IPv4 formats. CAR will soon be fully compliant with the new meanings of the DiffServ "DS" bits on IP packets¹¹. CAR is compliant with the DiffServ architecture, which requires technologies like CAR to mark precedence on IP packets. DiffServ standards are discussed in more detail later in this paper.

The core of the network supports different differentiated services classes. In MPLS networks, after CAR has been used to mark CoS using the precedence or DS bits on IP packets, the IP packets are sent on different LVCs according to their CoS—there is, in general, a different LVC for each CoS. In other words, CAR sets IP CoS, which is then supported by MPLS.

As described above, a service provider can use CAR to both police IP traffic and mark CoS on IP packets using precedence or DS bits. Alternatively, the customers can choose the CoS for their packets according to their own policies. CAR can be used on CPE to do this, and will soon be expanded to allow customers to coordinate CoS assignments using directory-enabled networking. If CAR is used on CPE to preset CoS, then CAR on the edge LSRs acts purely as a policer. The use of CAR on both CPE and edge LSRs is shown in Figure 28.

Figure 28: Using CAR on Customer Premises

Using "Best-Effort" Traffic to Assist in Guaranteeing Bandwidths

The prevalence of "best-effort" traffic in IP networks has already been discussed. The presence of best-effort traffic in a network produces important advantages:

• It is likely that a relatively small amount of traffic in an IP network requires guarantees of QoS, and that the rest of the traffic requires best effort.
  
  
• If most of the traffic is best effort, good QoS can be ensured simply by giving precedence to the "higher classes" of traffic. The traffic requiring good QoS can be given the ability to "push" best-effort traffic out of the way, in order to provide good QoS for the premium traffic.
  
  

The second bullet explains why best-effort traffic is an advantage in providing QoS: it can be used to "cushion" premium traffic, ensuring that the premium traffic gets premium service. This scenario is the key to the "Differentiated Services" approach to service quality.

Figure 29 shows how differentiated services can work to ensure that premium traffic gets good service. The network operator allocates bandwidth to two classes of service on a particular link. Best-effort traffic is allocated B0, and the premium traffic is allocated Bp. Note that Bp is greater than the estimated bandwidth of premium traffic, meaning that the premium traffic gains access to its required bandwidth even if the actual requirement is much greater than the estimated requirement. The best-effort traffic guarantees almost nothing—after all, it is best effort traffic, meaning that the best-effort traffic can have bandwidth denied in order to meet the requirements of premium traffic. This scenario occurs for most of the time in Figure 29: the best-effort traffic does not get all the bandwidth that it could use. On the other hand, all the premium traffic is carried. Using this method, differentiated services can give excellent QoS to premium traffic, under two conditions:

• A substantial amount of traffic is best-effort traffic.
  
  
• An estimate can be made of the premium traffic load on the link, so that a larger value Bp can be allocated in proportion¹².
  
  

Since almost all IP traffic today is best effort, and IP traffic loads are growing much faster than other traffic, for example, circuit-switched voice, the first condition is likely to hold in future networks. Unusual cases where this doesn't hold are discussed in the section "What if There Isn't Much Best-Effort Traffic in My Network?." The second condition is easy to meet, and this topic is discussed next.

The discussion above has used an example in which there is only one class of premium traffic, as well as best-effort traffic. Multiple classes of premium traffic can be supported in the same way, provided that the two conditions are met.

Figure 29: Ensuring Access to Bandwidth Using Differentiated Services

Modeling Network Traffic Flows in Order to Meet Service-Level Agreements

Now consider the engineering steps required for a DiffServ network to provide SLAs. We've seen how CAR or similar technologies can be used to enforce access rate contracts for sites, meaning that at each edge LSR, it is easy to calculate the sum of allowed premium-class access bandwidths. An example is shown in Figure 30a.

Figure 30: Estimating Network Loads

In order to engineer the core network to ensure delivery of premium-class packets, an estimate of the actual distribution of premium traffic is required, even though customers do not specify traffic matrices. Note that the traffic estimates are required only for aggregate flows between edge LSRs. For example, consider a provider network with 100 edge LSRs serving 200,000 customer sites. For this network, traffic estimates are required for the flows between the 100 edge LSRs, and not for the individual flows between the 200,000 customer sites. Also note that the estimates do not have to be exact, for reasons that have just been discussed.

It is trivial to calculate the maximum possible traffic flows for each origin destination pair. In Figure 30, for example, the sum of the premium-class access bandwidths at edge LSR A is 320 Mbps, so the maximum possible flow of premium-class traffic from A to any other edge LSR is 320 Mbps. This scenario is illustrated in Figure 30b. This form of estimating results in an extreme overestimate: it is impossible for a given source LSR to send its full rate of premium traffic to every other edge LSR simultaneously, but this scenario is what is assumed by this traffic estimate. Despite this assumption, the "maximum possible" traffic matrix may be useful in some circumstances:

• During trials and early production phases of introducing an MPLS network, it may be appropriate to dimension the network according to the "maximum possible" traffic, in order to reduce the risks associated with more efficient traffic models. The early phases would be used to gather statistics about actual traffic distributions.
  
  
• In networks with very small amounts of premium traffic compared to best-effort traffic, the inefficiencies of the "maximum possible" traffic model may be irrelevant, because the best-effort traffic is able to use all the bandwidth overallocated to premium traffic. In this case, there is no need to dimension the network according to a more realistic traffic model.
  
  

However, it is usually better to use a more realistic traffic model.

A more realistic traffic model assumes that traffic is distributed evenly among all possible destinations. In the example in Figure 30, there are four edge LSRs, and the total premium-class access bandwidth at edge LSR A is 320 Mbps. In this "evenly distributed" estimate, (320/4)Mbps is sent from A to all the edge LSRs¹³. Similarly, edge LSR B sends (1000/4)Mbps to every edge LSR, and so on. This scenario is shown in Figure 30c.

This method can be further refined by assuming that traffic is sent in proportion to the access bandwidths of the receiving sites. In other words, for example, a site with many large access lines is assumed to receive more traffic than a site with fewer, smaller access lines. In Figure 30d for example, the sums of the access bandwidths at nodes A, B, C, and D are in the proportion 320:1000:384:384. Consider the 1000 Mbps of premium-class traffic entering site B. According to these proportions, the 1000-Mbps traffic from node B is divided among nodes A, B, C, and D as 153, 479, 184, and 184 Mbps. The traffic from the other nodes is divided similarly. Dividing traffic by this method leads to balanced estimates of loads; that is, the traffic in one direction between a given pair of nodes is equal to the traffic in the other direction. This "proportional to POP size" estimated traffic is quite realistic and likely to be useful in many networks.

The setting of link or circuit parameters according to these traffic estimates covered in the following sections. Before moving on, this paper covers a recommended method for using traffic models in practice.

A Recommended Process for Estimating and Modeling Traffic

The traffic model used to dimension the network does not have to be exact. Recall from the earlier discussion that premium traffic is first estimated, and then actual allocations of bandwidth to premium traffic can be made in proportion to the estimates; these allocations, however, should be larger than the estimates to allow a margin of safety. A recommended process for dealing with traffic estimates and modeling as follows:

Step 1.   In trials and early production phases, use the "maximum possible traffic" model to dimension premium-class bandwidth allocations to ensure that all premium-class traffic is delivered under all conditions except equipment or link failures.

Step 2.   During the early production phases, measure the actual origin destination bandwidths. Cisco NetFlow is an excellent tool for collecting these statistics. Alternatively, packet or cell counts can be collected for all the backbone links. Compare this traffic to an estimate derived using the "proportional to POP bandwidth" estimate. It will usually be found that the proportional to POP bandwidth estimate is a good approximation to the real traffic. In unusual cases, some other model may need to be developed. In any case, choose a traffic model for the network.

Step 3.   Change the dimensioning of premium-class bandwidths to agree with the chosen model, but initially overallocate premium-class bandwidths by 100 percent or more to allow for unexpected variations. Recall that this overallocation is not usually a result of wasted bandwidth, because best-effort traffic typically can use spare bandwidth.

Step 4.   Continue to collect statistics and compare them to the traffic model, modifying the model if necessary. As confidence in the traffic model grows, reduce the amount of overallocation of premium-class bandwidths.

This process is familiar to many network engineers. Aside from the particular traffic models used, it is identical to the process that has been used in the introduction of oversubscribed Frame Relay and ATM networks.

Engineering DiffServ Per-Hop Behaviors

It was mentioned previously that DiffServ networks use queuing technologies such as Weighted Fair Queuing (WFQ) to provide differential service to the different CoSs. Link-by-link engineering of WFQ parameters is the approach suggested by the IETF DiffServ Working Group. The treatment of a particular CoS on a particular link (or "hop"), using technologies such as WFQ, is referred to as a per-hop behavior (PHB). Cisco supports engineering of per-hop behaviors on links in both ATM MPLS and packet-based MPLS networks, as well as ordinary IP networks. The principles are the same in all network types, although there are differences in the way CoS information is carried in packets for different networks.

As a prerequisite for setting the parameters for PHBs, the estimated demand for traffic of different PHBs must be derived for each hop. This estimate can be derived from the estimated traffic matrices described earlier. As an example, Figure 31a shows the traffic demand for a network where one class of premium traffic is carried, in addition to best-effort traffic. Figure 31b shows the physical network, including the links and core LSRs that carry the traffic. Figure 31b also shows the routes that have been chosen for IP routing in the network. IP routing protocols such as OSPF and Intermediate System-to-Intermediate System (IS-IS) normally choose the shortest possible route from a given origin to a given destination. For complex networks, a tool such as Cisco Netsys is helpful to review and analyze the routes used in a network.

Figure 31: Estimating Network Loads

The traffic matrix and routing information together specify the bandwidth used along various paths, as shown in Figure 31c. From this information, summing the total bandwidths on each link is straightforward. For example, there are three components of the premium-class traffic flow in the link from A to E: 153 Mbps A->B, 58 Mbps A->C, and 58 Mbps A->D. The sum of these numbers is 269 Mbps, the premium-class bandwidth requirement on the link A->E shown in Figure 31d. The other premium-class bandwidths are similarly calculated.

The end results of these calculations are values that should be allocated for premium-class bandwidth on each link. In this example, there has been one class of premium traffic, and the premium-class bandwidth reservation on link A->E should be at least 269 Mbps. In other words, referring back to Figure 29, the WFQ bandwidth Bp assigned on the link A->E should be at least 269 Mbps. This example has used one class of premium traffic. The same process can be used for multiple classes of premium traffic, with the bandwidth requirements for each class calculated as described here.

DiffServ Classes and Cisco IP+ATM Switches

The last discussion has shown how engineering of DiffServ networks leads to specifications of required bandwidths for various classes of service on various links of the network. This is quite different from traditional per-VC bandwidth management in ATM networks. The per-VC bandwidth concept used for constant bit rate (CBR), variable bit rate (VBR), and other ATM Forum traffic management types is illustrated in Figure 32a. Per-VC bandwidth management requires bandwidths to be specified for every origin destination flow. If per-VC bandwidth management is used in conjunction with approximate estimates of origin destination flows, bandwidth is distributed unfairly. In other words, per-VC bandwidth management is less useful in connectionless networks than the DiffServ mechanisms.

Per-VC bandwidth management is quite different from DiffServ concepts of CoS and per-hop behavior, and there is no simple, straightforward way to support DiffServ using ATM Forum traffic management types. Therefore, Cisco does not attempt to use ATM Forum traffic management types to support DiffServ. Instead, class-based queuing is used. As shown in Figure 32b, class-based queuing involves having a separate queue in the ATM switch for each CoS. Cells from all LVCs of each CoS are queued in a single queue for that CoS. The bandwidth parameters of a CoS on a link are set directly on the CoS queue. The only parameter that is signaled for each LVC is the CoS for the LVC, meaning that the ATM MPLS control component described in the section "MPLS in the ATM WAN" is used unchanged, except that multiple LVCs are set up for each destination: one LVC per destination per class of service.

Cisco IP+ATM switches support DiffServ for MPLS traffic, alongside ATM Forum traffic management types for PVCs and SVCs. Each DiffServ or ATM Forum traffic management type gets is own "CoS buffer." Per-VC queuing can be used in addition to the CoS buffers, and it is used for ATM Forum traffic management types. WFQ is used to assign bandwidths to the IP CoS buffers, meaning that the IP classes share bandwidth as illustrated in Figure 29.

Figure 32: Per-VC Service and Class of Service in ATM Switches

Using class-based queuing instead of per-VC queuing for the IP traffic has several advantages:

• The number of parameters programmed into the network is much smaller with class-based queuing; if a network has N nodes, the number of parameters required is proportional to N² with per-VC queuing, but proportional to N with class-based queuing.
  
  
• Class-based queuing is fairer, given approximate information. This fact is important because engineering of an IP network is based on approximate estimates and models of customer traffic. With class-based queuing, premium-class traffic from any origin to any destination gets preferential access to any premium-class bandwidth left spare from other origin destination pairs. This setup is much more difficult to achieve if bandwidths are assigned to individual origin destination LVCs.
  
  
• Class-based queuing can be used on any link types, including those that do not support virtual circuits: PPP over SDH and WDM. Use of class-based queuing helps make a network flexible, and open to future changes in technology without major changes in operations, administration, and management. Cisco already makes switch routers with ATM, PPP over SDH, and WDM interfaces.
  
  
• Class-based queuing works better than per-VC queuing with VC merge. In fact, per-VC queuing negates the advantages of VC merge in helping signaling scale. (See Figure 33). If per-LVC queuing were used, each LVC in the "tree" of LVCs merging to a given destination would need a bandwidth assigned to it according to the sum of bandwidth requirements merging in from other branches. If any addition or change was made to the bandwidths of the merging VCs, a ripple of signaling would go right through the network, negating one of the important advantages of VC merge: namely that VC merge removes the requirement for end-to-end signaling for most LVCs¹⁴.
  
  

Because of these reasons, Cisco strongly recommends that networks supporting IP services are engineered using class-based queuing.

Figure 33: Per-VC Service With VC Merge

Service-Level Agreements Using DiffServ

This section has covered:

• Enforcement of access contracts at the edge of a network using Cisco CAR
  
  
• Using the access contracts as a basis for modeling traffic
  
  
• Refinement of traffic models based on operation of a network
  
  
• Setting of the queuing parameters of the links according to the traffic models
  
  

These steps can lead to guarantees of packet delivery:

• Stronger guarantees of packet delivery can be given if "maximum possible" traffic models described above are used.
  
  
• Otherwise, SLAs are statistical, and are based on the accuracy of the traffic modeling process.
  
  

Unless "maximum possible" traffic models are used, the meeting of SLAs is reliant on monitoring network performance over time, reacting to unexpected traffic events, and modifying traffic models. An important set of statistics to monitor is the amount of premium-class traffic per link. If premium-class traffic is nearing its allocated bandwidth, then there is a danger that SLAs may not be met. However, since this process is reactive, it is important that SLAs are structured to permit the provider to have time to react; in other words, they should allow for periods of poor QoS. It is very important to note that most IP traffic is very tolerant of varying QoS; customers who understand this fact will understand that a very weak SLA is satisfactory, even for premium traffic.

Another important aspect of SLAs for IP traffic is the nature of "commitment." In a point-to-point network, it is quite easy to define a committed information rate between two sites. In an IP network, the nature of commitment is different. Consider
Figure 34: even if sites A, B, and C are transmitting packets within their premium-rate access contracts, not all of their packets will be delivered. They are sending packets to D at total of 768 kbps, a level that exceeds the link bandwidth to D, namely 512 kbps. The packet-loss rate in this example will be about 33 percent, even though the access contracts have been met. It is important for IP SLAs to emphasize this fact, and specify that a packet is not "committed" for delivery unless it meets the access contracts, and is sent within the possible receive rate at the destination. In other words, if a customer chooses to send 768 kbps of traffic to a site with a link bandwidth of 512 kbps, then the resulting loss is the customer's responsibility. (The customer may deal with this scenario, for example, by buying more bandwidth on the link to site D).

Figure 34: Committed Delivery in an IP Network

A simple way of structuring SLAs for IP traffic is to use two classes of traffic, resulting in a traffic contract that is similar to a Frame Relay committed information rate. An example of an SLA using this method is considered next.

Service Level Agreement Example Using the Two-Class Model

This white paper presents examples of a type of SLA that a service provider might find to be suitable to offer to its customers for IP traffic. Because the meeting of such a SLA depends in part on use of appropriate planning and monitoring processes by a service provider, the service levels described in this paper are illustrative only and Cisco offers no guarantee that any particular network can meet the service levels described.

The first two clauses define committed and best-effort packets, and the allowable rates for both. Clause 1 excludes traffic that cannot possibly be delivered as discussed previously. Clauses 3 and 4 provide realistic assurance of delivery of packets. Clauses 3 and 4 together allow for relatively poor performance for periods of hours each month, while still assuring adequate delivery performance during bad periods—such a period could indicate some need for improvement in the providers traffic modeling. Note that a provider must provision a network according to an accurate or suitably overestimated traffic model in order to have confidence in meeting an SLA such as this one. Alternatively, it is safer to provision a network using a "maximum possible traffic" model, as discussed earlier. This provisioning would typically be done early in network deployment.

Clauses 5 and 7 specify loose delay bounds, which are straightforward to provide. This feature is discussed in more detail in the section "Delay Limits." Clause 8 is an important provision for IP traffic, stating that packets will be delivered in order. TCP, UDP, and other transport protocols can deal with IP packet misordering, but large amounts of misordering can lead to poor transport-layer performance. Fortunately, the queuing technologies used on Cisco equipment ensure packet ordering within each class of service, except during rerouting¹⁶. Since rerouting is a rare and short-lived event caused only by link or equipment outages, or manual routing changes, it is easy to give a strong assurance of packet ordering. Modern telecommunications networks, including those using Cisco equipment, rarely introduce bit errors. An error provision such as Clause 9 is easy to meet.

Readers familiar with QoS guarantees and SLAs for ATM and Frame Relay traffic may be surprised by the weakness of the guarantees suggested in the example. The numbers suggested are entirely reasonable for IP traffic, because most IP traffic is extremely tolerant of varying QoS. The SLA is also incomparably better than the SLAs offered on most public IP networks today: most public IP networks offer no SLAs at all, and the Internet has been widely successful despite this situation. Providers who offer SLAs of the strength suggested above can capture a large untapped market: customers requiring moderate guarantees for a truly connectionless IP network, but who do not wish to specify all point-to-point bandwidth requirements. Traditional point-to-point QoS guarantees are over engineered for the needs of most IP traffic.

Some IP traffic requires stronger SLAs. Real-time IP traffic is considered in the next example. Stronger, "Frame Relay" quality SLAs may be required for a small minority of IP traffic, and these are considered in the section "Harder Quality of Service in IP+ATM Networks."

Example Service Level Agreement with Provision for Real-Time Traffic

This white paper presents examples of a type of SLA that a service provider might find suitable to offer its customers for IP traffic. Because the meeting of such a SLA depends in part on use of appropriate planning and monitoring processes by a service provider, the service levels described in this paper are illustrative only, and Cisco offers no guarantee that any particular network will be able to meet the service levels described.

If more than two traffic classes are supported, it is necessary to identify the preferred traffic class in some way. Clause 1 defines a means of doing this identification¹⁷. Note that IP Precedence 5 is the default class of voice-over-IP packets sent from Cisco equipment, and that Clause 1 supports any other equipment that marks real-time packets with a precedence greater than 0. Clause 1 strictly limits the real-time traffic that a site may send, and this limiting helps in the engineering of the network to support the real-time traffic class. In order to meet the relatively strong delivery agreements specified in Clauses 4 and 5, it might be necessary to use a "maximum possible traffic" model for the real-time traffic, similar to the one suggested in Figure 30a. The remaining clauses are similar to the previous example. As discussed earlier, a provider must provision a network according to an accurate or suitably conservative traffic model in order to have confidence in meeting an SLA such as this. Alternatively, it is safer to provision both the real-time and committed traffic using a maximum possible traffic model, as discussed in the section "Modeling Network Traffic Flows in Order to Meet Service-Level Agreements."

Admission Control and IP Services

Because IP services are connectionless, normal processes of connection admission control are not of use. The equivalent service admission control is a network management operation. Consider the adding of a new customer site or set of sites. A typical service admission process would do the following:

Step 1.   As an ongoing monitoring operation, maintain a matrix A of the available bandwidth for each CoS between each pair of edge LSRs. This matrix should account for engineering rules; for example, 25 percent of the bandwidth for each CoS on each link should be left unallocated as a statistical reserve. Also maintain a matrix R of bandwidth reserved for services that have been allowed (see Step 3) but not yet activated.

Step 2.   Form a model of the traffic introduced by the new service. Traffic modeling is discussed in the section "Modeling Network Traffic Flows in Order to Meet Service-Level Agreements." This step, results in a matrix N.

Step 3.   Compare the new traffic N to the available bandwidth (A-R). If there is sufficient available bandwidth, that is, (A-R-N)>0, then allow the new service and increase the reserved bandwidth R by the new traffic N. Otherwise, refer the new service request to be dealt with by increasing the provisioning of the network. If increased provisioning is not possible, the service request must be rejected.

Step 4.   After the service is activated, allow some time (weeks or more) for the customer to start using the service. Then decrease the reserved traffic R by N, and deal with any further changes in customers' use of bandwidth by the normal engineering processes.

This process is based on straightforward statistics collection and calculations, and could be added to an existing operations support system. In a manner similar to connection admission control for point-to-point services, service admission control ensures that new services receive its desired quality of service.

What if There Isn't Much Best-Effort Traffic in My Network?

A prevalence of best-effort traffic assists with the engineering of DiffServ networks, by allowing for very good QoS for premium traffic, in the presence of approximate traffic estimates, and without wasting bandwidth. As described previously, best-effort traffic helps cushion premium traffic. If best-effort traffic is not prevalent, it is still possible to use the engineering techniques described previously, but there is an increased need for accuracy in traffic models. Two measures are be useful:

• Use initial, maximum possible traffic models for a longer period while gathering statistics on actual traffic patterns.
  
  
• Use MPLS traffic engineering, as described below.
  
  

Standardization

CAR and the use of Diffserv discussed in the section "The Differential Services Approach to Quality of Service" are based on the forthcoming MPLS and differential services standards from the IETF. The Cisco implementation of these technologies is either fully compliant with the standards (to the extent they are complete), or is compliant with the older IP Precedence definitions and can be upgraded to comply with DiffServ. For relevant IETF documents, see "A Framework for Differentiated Services," draft-ietf-diffserv-framework-02.txt; "An Architecture for Differentiated Services," RFC 2457; "Assured Forwarding PHB Group," RFC 2587; and "MPLS Support of Differentiated Services by ATM LSRs and Frame Relay LSRs," draft-ietf-mpls-diff-ext-00.txt¹⁸. Some of these documents are Working Group Internet Drafts, which are works in progress at IETF Working Groups¹⁹.

The Differential Services Approach to Quality of Service: Summary

The preceding paragraphs have described how good quality of service can be provided to connectionless IP traffic, on MPLS networks in particular. The process involves several steps:

• Enforcing access contracts at the edge of a network using Cisco CAR
  
  
• Using the access contracts as a basis for modeling traffic
  
  
• Optionally refining traffic models based on operation of a network
  
  
• Setting the queuing parameters of the links according to the traffic models
  
  
• Offering SLAs of an appropriate form and strength for a connectionless IP service
  
  
• Service admission control
  
  

The following sections cover traffic engineering of MPLS networks, stronger QoS guarantees, application of QoS to virtual private networks, and other miscellaneous topics.

MPLS Traffic Engineering

The above discussion about quality of service has shown how the forming and measuring of traffic models plays an important role in providing good QoS for connectionless traffic. Cisco is developing tools to assist this process. The first of these, MPLS traffic engineering, is currently available for Cisco router-based MPLS equipment and is being extended to ATM LSRs. MPLS traffic engineering works by evaluating the traffic loads on the links of a network, and then adjusting the routing of traffic to make best use of the available bandwidth.

MPLS traffic engineering has several other uses. It provides support for a full range of operational requirements in IP networks, all related to choosing of traffic routes:

• To "fit" IP traffic to available link bandwidths to make best use of the links, as described above
  
  
• To prepare for operational events such as taking a link out of service, by shifting traffic off the link ahead of time
  
  
• To deal with unexpected events such as a sudden surge of traffic to a particular destination; this process would involve spreading the traffic surge around several alternate paths
  
  
• To select specific routes for certain traffic
  
  

The Cisco implementation of MPLS traffic engineering works by doing the following:

• The network operator identifies candidate origin-destination traffic streams for adjustment by traffic engineering, and specifies a nominal bandwidth for each stream. This bandwidth would typically be derived from previous statistics for source-destination flows in the network.
  
  
• At each label switch router, a nominal load for each link is calculated by Cisco MPLS traffic engineering software. This nominal load is the sum of the nominal bandwidths of the streams which are routed over that link.
  
  
• The link loading information is distributed using the IP routing protocol used inside the network, specifically OSPF or IS-IS²⁰.
  
  
• Using optimization techniques, Cisco MPLS traffic engineering software attempts to find alternate routes for the candidate streams of traffic so that the percentage loads are minimized. In other words, traffic is routed so that it sees the lightest possible loads on the links it traverses.
  
  
• If a stream of traffic is to be sent on a different route to the normal OSPF or IS-IS route, the traffic engineering software sets up an MPLS label-switched path along the alternate route. This path is known as a traffic engineering (TE) tunnel. Careful checks are made to ensure that the TE tunnel does not lead to loops. Specifically, a check is made to ensure that the exit of the tunnel is "closer" to the destination than the ingress. This integrity check is repeated periodically, and a tunnel is disabled if it fails the check, or fails to pass traffic.
  
  

Optimizing traffic routing using MPLS traffic engineering is illustrated in Figure 35. Figure 35a shows an example where the nominal load on the link between nodes E and F is 91 percent. At this load, there is imminent danger that packet loss will occur, if it is not occurring already. MPLS traffic engineering can find candidate streams to reroute away from that link, if possible. For example, the LSPs between edge LSRs A and B may be carrying significant amounts of traffic. MPLS traffic engineering attempts to find an alternative route for one or more LSPs so that the load on the (E,F) link is reduced without increasing the loads on other links to a similarly dangerous level. So, depending on the bandwidth on the LSPs, it may be possible to solve the congestion by rerouting traffic away from (E,F) a different path.

The Cisco implementation of MPLS traffic engineering acts automatically to spread loads around the links of a network as evenly as possible. In other words, it acts to minimize loads, even if links are not currently overloaded. In this way, Cisco MPLS traffic engineering acts to prevents overloads.

Figure 35: Reoptimization of Traffic Using MPLS Traffic Engineering

MPLS traffic engineering is based on calculating the subscribed link loads, in a similar manner to the ATM Forum's PNNI. However, PNNI is not a good routing protocol for IP traffic, as there is no simple way to make PNNI make routing decisions in conjunction with routing information from standard IP routing protocols, namely OSPF and IS-IS²¹. MPLS traffic engineering based on OSPF or IS-IS overcomes these limitations, and supports measurement-based engineering of connectionless traffic. MPLS traffic engineering can support PNNI-style edge-to-edge bandwidth reservations where required, as discussed in the section "Harder Quality of Service in IP+ATM Networks."

Note that these issues are relevant only for routing for MPLS LSPs. Cisco IP+ATM switches use a full-featured PNNI implementation for traditional ATM connections: SVCs, SPVCs, and so on. MPLS and traditional ATM connections can be operated on the same links (recall the descriptions in the section "IP+ATM Capability").

The candidate flows for adjustment using MPLS traffic engineering must be identified ahead of time. Usually, a relatively small percentage of possible origin destination traffic streams account for a large proportion of traffic. For example, a network with 100 edge LSRs has 9900 possible origin destination streams, but typically 90 percent or more of the traffic in the network is on a few hundred of these. In other words, MPLS traffic engineering can successfully optimize the traffic in a network by optimizing routing for a relatively small number of "candidate" origin destination streams. The other streams are left to follow the routes chosen normally by OSPF or IS-IS. Note that this scenario does not imply lower quality of service for the noncandidate streams. They still benefit from the low link loads and even distribution of traffic provided by MPLS traffic engineering.

The signaling used in the Cisco implementation of MPLS traffic engineering is compliant with a traffic engineering method approved by the MPLS working group. Extensions to carry link-loading information in the IS-IS and OSPF routing protocols are works in progress at the IS-IS and OSPF working groups at the IETF.

Harder Quality of Service in IP+ATM Networks

Previous discussions have described the provisioning of SLAs for connectionless traffic in the absence of subscribed, point-to-point bandwidths. In some cases, users may want harder "PVC-like" QoS guarantees for traffic between certain sites. This QoS may be required for critical business applications, disaster recovery, and so on. Cisco IP+ATM can meet these requirements in two ways.

The full MPLS solution shown in Figure 36a is in development now. In this solution, MPLS traffic engineering routes a label-switched path (LSP) with a specific, reserved bandwidth between two edge LSRs. This LSP is reserved for traffic between two particular customer sites. (LSPs may be aggregated to help scalability—this scenario is discussed in the section "Quality of Service for MPLS VPNs") Per-VC queuing is used for this LSP. Because MPLS traffic engineering routes this LSP based on a reserved bandwidth, rather than a measured bandwidth, these point-to-point LSPs give the QoS equivalent to switched PVCs (SPVCs). This setup allows IP SLAs to be extended to include traffic reservations between specific sites.

It was previously described how connectionless IP services require a service admission control process rather than a traditional connection admission control (CAC) process. The point-to-point LSP services, on the other hand, use traditional CAC—the network will reject the connection if insufficient bandwidth is available.

An alternative to point-to-point MPLS links is available now, and it achieves the same results. With Cisco IP+ATM networks, a single customer site, with a single access link, can access both a tradition end-to-end PVC and a connectionless IP service, as shown in Figure 36b. The IP service can be used to deliver the connectionless SLAs shown earlier, and the PVC delivers the extreme reliability already present for Frame Relay and ATM services on Cisco IP+ATM switches. Although the fully integrated solution shown in Figure 36a will be available soon, the IP+ATM method is likely to be widely used for a long time because customers, (such as banks) who already have a Frame Relay or ATM service for existing transaction processing may not want to shift away from Frame Relay or ATM for that traffic. If a customer's established business processes are running on long-used technology, it is quite reasonable for the customer to want to keep that existing infrastructure. Cisco IP+ATM solutions allow existing Frame Relay and ATM connectivity to be retained, while IP services are introduced for any-to-any connectivity for the new IP traffic, as shown in Figure 36c. The switches in the network carry both traditional Frame Relay and ATM services, as well as IP services.

Figure 36: Reserved Point-to-Point Bandwidths in MPLS Networks

Quality of Service for MPLS VPNs

MPLS VPNs have the same QoS options as any other MPLS networks. Sites in VPNs can subscribe to specified rates of specified classes of service, and the provider can offer connectionless service-level agreements for those classes. Cisco CAR is used at provider edge routers to enforce traffic contracts, and differentiated services are used in the network core.

One of the advantages of Cisco MPLS VPNs is that the core LSRs do not have any knowledge or state for individual VPNs. This setup has advantages for class-based service and fairness. In particular, if premium traffic has precedence over best-effort traffic, then this applies, irrespective of which VPNs are the sources of the premium and best-effort traffic.

Site-to-site bandwidth reservations can also be used with MPLS VPNs, as shown in Figure 37b. One refinement is used to help with scalability. If there are two separate point-to-point LSPs with the same originating and destination provider edge routers, as shown in Figure 37b, then they may be aggregated into a single LSP, as shown in Figure 37c. This scenario helps preserve the scalability advantages of MPLS VPNs, specifically the absence of per-VPN state in the network core. When point-to-point LSPs are aggregated, WFQ is used to ensure that each site gets its correct share of the aggregated bandwidth, as shown in Figure 37d. In other words, the QoS achieved with the aggregated LSPs is equivalent to that achieved with separate LSPs.

Figure 37: Quality of Service in Virtual Private Networks

Sometimes it is desirable to provision bandwidth within VPNs to specific users and applications. This scenario can be achieved by running Cisco CAR on CPE routers, as discussed previously. CAR can be used to give precedence or specific bandwidths to specific users or applications, as chosen by the customer. This setup is illustrated in Figure 38.

Figure 38: Providing Bandwidth To Specific Users and Applications in Virtual Private Networks

Other QoS Topics

Discar