Successful Distributed Denial of Service (DDoS) attacks resulted in reported losses in the millions of dollars in 2005 alone. These attacks have also forced companies to shut down Web and/or privately meet extortionists' demands for payment ranging from $10,000 to several millions of dollars.
A number of service providers have reacted to the needs of their customers and currently offer automated, network-based, DDoS services that employ Cisco Systems®Clean Pipes solution. These services have proven to be profitable almost from their inception. However, service providers have also realized a number of supplemental gains in operational savings, bandwidth conservation, and service differentiation that have transformed DDoS mitigation services from tangential to strategic.
This paper, based on interviews with a number of services providers now offering DDoS services based on the Clean Pipes solution, describes not only their service successes, but also the ways they utilize Clean Pipes to increase productivity, reduce operational costs, and enhance the value of a number of business service offerings.
DDOS, A "REFRESHER COURSE"
DDoS attacks have a single aim - to deny legitimate users access to a specific host or service. DDoS attacks accomplish this goal by recruiting a number of agent (slave) machines. These machines are infected with the attack code and are then used to attack the target host or service.
It is currently unclear how many companies have been victimized by DDoS attacks. Many are too embarrassed to report the crime. The lack of reporting has made it difficult for law enforcement agencies to quantify the scope of the problem. Allan Paller, Director of Research for security organization SANS, stated at the SANS Institute's Top 20 Vulnerabilities conference: "The epidemic of cybercrime is growing. You don't hear much about it because it's extortion and people feel embarrassed to talk about it. Hackers use DDoS attacks using botnets to do it. Then they say `pay us $40,000 or we'll do it again.'" In fact, it is estimated that the extortion racket has claimed more than 20 percent of all enterprises as victims and companies pay millions of dollars every year to extortionists.
There are a variety of DDoS attack classifications and methodologies. They are briefly outlined in the table below:
Attack Category
Specific Attack Type
Bandwidth Consumption Attacks
Spoofed and nonspoofed flood attacks:
• TCP Flag (SYN, SYN-ACK, ACK, FIN)
• Internet Control Message Protocol (ICMP)
• User Datagram Protocol (UDP)
Examples include SYN flood, smurf, LAND, and UDP flood attacks.
Zombie/botnet attacks, in which each zombie or bot source opens multiple TCP connections, and sometimes issues repetitive HTTP requests.
DNS attacks such as DNS request flood.
Resource Starvation Attacks
Packet Size attacks, characterized by fragmented or large packets: Examples include teardrop and ping-of-death.
Low-rate zombie/botnet attacks, which are similar to bandwidth consumption attacks except that each attack source sends multiple requests at a low rate.
DNS attacks, with DNS recursive lookup.
The DDoS attack types outlined above, not only open both services providers and their customers to extortion, but also:
• Impact profitability for service providers and their customers - Industry studies have estimated the cost of downtime to range from a low of $330,654 per hour in the hospitality and travel industries to a high of $2,817,846 in the energy industry (telecommunications was a close second at $2,066,245 per hour).
• Influence company reputation and customer retention - Attacks originating with a service provider's infrastructure that affect a customer's business can significantly impact the service provider's reputation. Additionally, increased billing for the traffic generated by a DDoS attack can negatively affect customer relationships.
• Affect compliance with regulatory statutes - The results of DDoS attacks or the exposure of customer or private data to attackers could result in noncompliance with the Sarbanes-Oxley Act, Graham-Leach-Bliley-Act, and other state and federal regulations.
• Impact revenue of existing Internet-based services - DDoS attacks that deny users access to a corporate service or degrade network or hosted server performance could force those services out of compliance with SLAs for network or hosted server availability and performance. Over the last six months, the largest attacks have averaged between 500 Mbps and 1 Gps. Such attacks could force SLA noncompliance and could result in service credits or monetary penalties - in either case, reducing service revenue and profitability.
According to a recent study by Arbor Networks, DDOS attacks ranked as the number one operational problem (average of more than 10 "actionable" attacks that affect customers and 40 overall attacks per month) for 64 percent of all Internet service providers. Protecting network and customer assets is not only prudent, it is a business necessity. However, choosing the right mitigation solution is just as important as choosing to mitigate attacks. A number of service providers have selected Cisco's Clean Pipes solution not only for protection, but for profitability as well.
Cisco®Clean Pipes DDoS protection and mitigation solution is designed to automate the mitigation process. Automating of the mitigation process increases network and host system availability, boosts the productivity of operations personnel, improves customer satisfaction and enables service providers to offer unique and profitable mitigation services, tailored to the specific needs of their customers or target market. The customers interviewed for this paper have implemented the Clean Pipes solution to meet a wide variety of customer requirements - from Web hosting and data center protection to storage networking and hosted IP PBX protection.
The Clean Pipes solution consists of:
• Cisco Guard XT Appliance and Cisco Anomaly Guard Services Module - The Guard XT appliance and Anomaly Guard Services Module (for Cisco Catalyst® 6500 Series Switches and 7600 Series routers) process attack traffic at line rates to 1 Gbps and can be clustered to construct a cleaning center. The Guard XT Appliance and Anomaly Guard Services Module perform the attack analysis, identification, and mitigation services required to block attack traffic and prevent it from disrupting network operations.
• Cisco Traffic Anomaly Detector XT and Cisco Traffic Anomaly Detector Services Module - The Traffic Anomaly Detector XT and Traffic Anomaly Detector Services Module (for Cisco Catalyst 6500 Series Switches and Cisco 7600 Series routers) receive a copy of traffic to a protected zone using the port-mirroring feature or a switch, or by means of splitting. The platforms use behavioral analysis and attack recognition technology to proactively detect and identify assaults. They compile detailed profiles that indicate how individual devices normally behave. When deviations are detected, the detector responds based on user preference - by sending an operator alert, by triggering an existing management system, or by launching the Cisco Guard XT or Cisco Anomaly Guard Services Module to begin immediate mitigation.
• Cisco NetFlow - NetFlow is a widely deployed network traffic flow analysis technology for IP networks. It classifies packets based on seven characteristics:
1. Ingress interface
2. IP protocol type
3. Type-of-service (ToS) byte
4. Source IP address
5. Destination IP address
6. Source port number
7. Destination port number
Since the NetFlow data provides a view into the network traffic, it can be used to monitor and detect DDoS attacks. NetFlow data is used by Arbor Networks Peakflow SP to model normal behavior based on flow data from network routers.
• Arbor Networks Peakflow SP - Peakflow SP consists of three elements: Managed Services (enables service providers to offer customers DDoS protection and traffic management tools), Infrastructure Security (proactive detection and mitigation of networkwide anomalies) and Traffic and Routing (models traffic on the network, enabling service providers to make business decisions on routing, transit, partners, and customers). Peakflow SP gathers NetFlow data out of band to determine a traffic baseline and then compares traffic against the baseline for flagging and anomaly detection. Anomalies are traced back to point of ingress and, in conjunction with Cisco Guard XT, an SSH connection is established to activate the Cisco Guard XT to put the Guard's zone under attack protection.
Mitigation steps to protect a portion of the network are, therefore:
1. Baseline learning - A traffic database is constructed with normal traffic patterns for a zone. Peakflow SP and Cisco Guard XT learn the patterns independently.
2. Detection - Upon completion of the learning process, Peakflow SP and the Cisco Guard XT monitor traffic, sending an alert to an operator or activating the Cisco Guard XT when an anomaly is detected.
3. Diversion - After receiving the request to put an attacked zone into protection mode, the Cisco Guard XT sends a BGP announcement to an upstream router, changing the next-hop address to that of the Cisco Guard XT. A network operator may also order this diversion manually. The upstream router then installs this BGP announcement into its routing table and forwards dirty traffic as well as clean traffic to the Cisco Guard XT.
4. Scrubbing - The Cisco Guard XT analyzes the diverted zone traffic for anomalies. When a flow violates a policy threshold, an anomaly is identified. The Cisco Guard then analyzes the results and creates a set of dynamic filters that continuously adapt to the zone traffic and type of attack. The initial dynamic filter directs traffic to user filters until the Cisco Guard finishes analyzing the flow and creating more dynamic filters to handle the anomaly. The dynamic filters and user filters feed their results into a comparator that chooses the most severe protection measure suggested and directs the traffic to the relevant protection module for authentication. The module drops unauthenticated traffic and the Cisco Guard XT passes the traffic to the rate limiter, which drops traffic that exceeds the defined rate.
5. Injection - The cleaned traffic from the Cisco Guard XT is injected back into the zone. Injection can be based on Policy-Based Routing (PBR), Virtual Routing/Forwarding (VRF), generic routing encapsulation (GRE), and Multiprotocol Label Switching (MPLS) VPN.
6. Completion of Traffic Scrubbing - Dynamic filters on the Cisco Guard XT have a limited lifespan and are erased after the DDoS attack is terminated. By default, the Cisco Guard XT remains in protect mode until a user deactivates it, but it can be set to deactivate protection if no dynamic filters are in use and no new dynamic filter has been added over a predefined period of time. The Cisco Guard XT retracts the previous BGP announcement and traffic resumes on the regular path.
An example of the solution is illustrated below.
Figure 1. DDoS Protection Solution
Many of the service providers interviewed are using Cisco Clean Pipes automated DDoS mitigation solution to protect both customer and service provider network and server resources. Clean Pipes DDoS protection has enabled service providers to:
• Improve profitability for themselves and their customers by maintaining network and server availability. Using the Clean Pipes solution, one operator reduced by sixfold the number of attacks that required direct action.
• Enhance company reputation and customer retention by mitigating attacks before they impact customer resources and billing. (Service providers that bill for transmission or Web hosting found that a DDoS attack could result in an additional $15,000 to $30,000 in monthly billing. Customers are often unwilling to pay the additional fees, or they pay them only reluctantly.)
• Avoid regulatory noncompliance due to a critical service being unavailable.
• Supplement Internet service revenue by offering more stringent (and higher priced) SLAs as well as other "protected" IP-based voice and data services.
For Chris Richter, vice president and general manager for Security Products at Savvis, protecting the SAVVIS network - and customer resources were the key reasons for choosing the Cisco solution: "Everyday over 5000 enterprise customers entrust their business to SAVVIS' ability to deliver all or part of their IT systems and operations, and security is a critical element of our service. We were one of the first network providers to deliver `built-in' security that spanned our network cloud in 45 countries. The challenge has been how to extend that security protection to cover the local loops that connect businesses to our network backbone. We could have used dedicated appliances at the customer's premises, but the equipment was very expensive, and it did not really protect the customer from these attacks at their other facilities where dedicated security appliances were not installed."
Protection is but one benefit that service providers have gained from using Clean Pipes. Service providers are employing the Clean Pipes solution to develop new profit centers from DDoS mitigation services as well.
PROFITS - NEW AND TRANSFORMED SERVICES
The service providers interviewed for this paper all offer profitable, managed services based on Cisco Clean Pipes DDoS mitigation solution. The services and service providers themselves are diverse; however, they can be grouped into three main service and service provider categories:
Service Provider Type
Service Name
Service Description
Service Features
Providers of Internet-Based Data, Voice, and Hosting Services
Managed Network DDoS Protection
Protects "last-mile" bandwidth and data center resources or Enterprise business customers
• Increased network and/or data center availability
• Consistent network and server performance
• Tiered services
• Dedicated and shared models
• Customer or service provider controlled
Providers of Hosting and Content-Based Services
Managed Hosting DDoS Protection
Protects enterprise servers and data center resources
• Increased network and/or data center availability
• Consistent network and server performance
• Bundled and shared offerings
• Service provider controlled
Providers of Wholesale Internet Services
Managed Peering Point DDoS Protection
Protects wholesale connections for downstream ISPs
• Protects routing and transmission infrastructure
Providers of Internet-Based Data, Voice, and Hosting Services
Service providers offering their enterprise customers a number of Internet-based services offer the most variety in the DDoS mitigation services. Driven in many cases purely by customer demand, this group of services and service providers has achieved return on its initial infrastructure investments in one year or less.
Common features across all services include:
• Proactive protection of customer network and server resources from DDoS attacks.
• Provision of equipment, monitoring, and management.
• Customer notification (various methods are used including e-mail, pages, Web-based portals, and advisories.
However, the Clean Pipes solution has also provided service providers with a method to differentiate not only DDoS mitigation services, but also other business services as well. For example, some service providers offer DDoS mitigation services in tiers - offering "basic" services that mitigate attacks with a standard SLA, "enhanced" services that add reporting and attack alert notification with more stringent SLAs, and "premium" services that pass attack control to the customer.
Other service providers have used their DDoS mitigation services to enhance other service offerings and thereby glean additional revenue. DDoS mitigation services, when paired with hosting and network-based storage services, have allowed service providers to offer more stringent network and hosting reliability and network latency SLAs, and higher priced bundled services to customers. Additionally, a number of the service providers interviewed are already planning to marry their DDoS mitigation services to IP-based voice, video, and DNS services.
Stan Quintana, Vice President, AT&T Managed Security Services, states: "Today, many customers are heavily dependent on the WEB to conduct business critical to their corporations. Should denial-of-service attacks occur against these WEB infrastructures, there is potential significant impact on revenue, brand, and/or intellectual property. AT&T has offered a denial-of-service capability for over two years, since which numerous customers have utilized this service and have significantly benefited by having the protection in place, and in many cases, have mitigated attacks that could have rendered significant damage."
Managed Hosting Providers and Content-Based Services
DDoS mitigation solutions are an essential part of any hoster's comprehensive managed security and business continuity. Enterprises have a right to expect that service providers protect them from DDoS attacks when they contract for high-network availability. In addition to customer demand, service providers also use the solutions as a method to protect not only single customers but also the provider's data center, network, and business as a whole.
DDoS mitigation services often increase recurring revenue for the hosting providers. These "bundled" approaches include stringent SLAs, alerts and notifications, and extensive online reporting capabilities. Rackspace Managed Hosting of San Antonio, Texas - a recognized leader in the managed hosting space with more than 20,000 servers under management - earned a spot in the InfoWorld 100 list based on their PrevenTier DDoS mitigation service, which is a combination of the company's own patented technology combined with products from Cisco and Arbor Networks.
"Our customers rely on us to protect them and help keep their business up and running during a DDoS attack. Our solution allows us to quickly and accurately locate suspect traffic at the advent of an attack and sanitize it without disrupting the free flow of legitimate network traffic, while dramatically shortening detection and resolution times," said Paul Froutan, vice president of research and development, Rackspace Managed Hosting. "PrevenTier is a critical component to our comprehensive security offering, enabling Rackspace to deliver maximum uptime to our customers' business-critical applications."
Providers of Wholesale Internet Services
Most wholesale Internet service providers are initially attracted to DDoS mitigation as a means to protect their own valuable routing and transmission infrastructures. However, once the protection infrastructure is in place, a number of them are discovering the opportunity to utilize DDoS to enhance their revenue.
Using the Clean Pipes solution, wholesale ISPs are already protecting "downstream" ISPs - and their customers - from attack. Especially useful to ISPs with customers in vulnerable markets, such as online gaming and electronic-commerce, wholesale providers, such as Cable & Wireless, are expanding their service portfolio while increasing profits and customer loyalty.
"Online retailers are all vulnerable to attack ..., as is anybody who is generating significant revenue online," states Rob Thomas, head of product marketing at Cable & Wireless. "Organized criminals may now be targeting companies who don't have the resources in place to guard against these attacks."
DDoS mitigation services are already delivering new revenue streams for service providers in a variety of markets. However, increased service revenue is only one way that DDoS mitigation services based on Cisco Clean Pipes solution have positively impacted service providers' bottom lines. Service providers have also discovered how Clean Pipes has lowered their operations costs and improved employee productivity as well.
PRODUCTIVE PEOPLE - AND INFRASTRUCTURE
With their Clean Pipes DDoS mitigation solution in place, many service providers discovered that its operational benefits were as important as its protection and profit-generating capabilities. Service providers have discovered a number of operational and productivity benefits, some of which have paid for the Clean Pipes solution in cost savings alone. The list below provides samples of the operational benefits that service providers now gain from this solution.
• Bandwidth Conservation - The rapid and highly automated DDoS mitigation capabilities of the Clean Pipes solution has resulted in a reduction of more than 10 percent (as high as 25 percent in some cases) in network traffic for Clean Pipes users. Although capacity may not be a problem in some portions of a service provider network, some links - including transoceanic, service provider to service provider exchange points, and links that are leased from other service providers - are expensive and inflexible assets. Reduction in transoceanic bandwidth utilization alone produced a return on investment of three months for one service provider.
• Reduced Customer Support Costs - Prior to implementation of Clean Pipes, a number of service providers relied on their customers to notify them of a DDoS attack. This manual method resulted in an escalating number of calls and trouble tickets as the attack spread collateral damage through the customer base. It also resulted in a growing number of calls to customer support as well as interaction with second- and third-level support personnel. Using Clean Pipes, mitigation actions can be automated or performed by the customer, or a message can be triggered to operations to institute mitigation. In all cases, the proactive nature of the solution reduces both the number of calls to customer support and actionable trouble tickets.
• Producing a Profit Center - Before Clean Pipes, the personnel that were responsible for mitigating DDoS attacks were deemed operational overhead by corporate accounting. After implementing services based on Clean Pipes, service providers have transformed their operational "overhead" into revenue production. Some service providers have been able to migrate 40 percent of their security staff from overhead to revenue producing using Clean Pipes-based services.
Service providers who have implemented Cisco Clean Pipes DDoS mitigation solution were often pleasantly surprised by the operational benefits and cost savings the solution provided.
"Clean Pipes customers have already realized significant gains in network reliability and customer satisfaction while reducing the work load on 2nd and 3rd tier support personnel. Additionally, Clean Pipes customers have discovered that operational benefits alone often pay for their Clean Pipes implementations - allowing service revenue to go directly to the bottom line," states Kunjal Trivedi, Cisco Systems.
CLEAN PIPES - PROTECTION, PROFITS, PRODUCTIVITY
The growth in the number and size of DDoS attacks by unscrupulous individuals and organizations places service providers and their customers at significant risk. System and network reliability and performance, regulatory compliance, customer satisfaction, service revenue, and operational costs are all compromised by successful attacks.
A growing number of service providers, including those interviewed here, are utilizing Cisco Clean Pipes DDoS mitigation solution to protect valuable internal infrastructure and customer assets, produce new revenue streams, differentiate services, increase customer satisfaction, and reduce operational costs. They are using Clean Pipes to protect their services people and infrastructures and increase the productivity of these assets, which, in turn, leads to higher profits.
As stated by Michael Halperin, Director of Product Management, Sprint Managed Services, "Our enterprise customers are demanding security-specific service-level agreements for quality and reliability on their Internet services. We respond to these SLA requests with the IP Defender service - coupled with the expertise of our certified security professionals - to prevent and mitigate DDoS attacks coupled with the expertise of our certified security professionals. We also secure our own global networks with this solution."
