Data Disaster Recovery and Business Continuance

Deployment Guidelines

Cisco MDS 9000 Family FCIP WAN Deployment Guidelines

SAN-OS 1.1(2)

Introduction

FCIP, a transparent Fibre Channel tunneling protocol over TCP/IP, can be used for a variety of applications across a variety of network transports and topologies. FCIP can be used across both high and low speed links, and across long and short distances and latencies.

When considering various transports of differing speed and distance, it is important to tune the FCIP transport to ensure expected performance and resiliency. For example, special configuration precautions must be followed when building FCIP solutions over very long, slow WAN facilities.

The following best practices and design guidelines are presented to help the SAN designer to build FCIP-based solutions that are optimized in terms of performance and resiliency. The following design guidelines are based on laboratory testing at Cisco Systems.

Applicable SAN-OS

The following configuration guidelines are based upon MDS9000 SAN-OS system and kickstart version 1.1(2) and higher.

This release introduces TCP packet shaping to FCIP. TCP packet shaping enables the sender to immediately send packets at the configured minimum available bandwidth of the FCIP path.

FCIP Configuration

The following guidelines assume a dedicated FCIP WAN link where the minimum available bandwidth is equal to the maximum available bandwidth (or the speed of the WAN link).

Note that whilst tcp cwm burstsize parameter has no effect when max-bandwidth-kbps = min-available-bandwidth-kbps, it should remain enabled to prevent large bursts at gigabit rate. i.e the shaper is not enabled after partial idle periods unless cwm is enabled.

Guideline configuration for an FCIP entity is shown below.

Notes:

1. The MTU size must be configured to no more than 2500 (this will still hold a maximum size Fibre Channel frame, so anything larger is superfluous anyway).

2. SACK (Selective ACK) must be enabled at each end of the FCIP link (SACK is enabled by default with this release).

Examples:

1. Gigabit Ethernet link at 20ms Round trip delay

2. OC-12 link at 10ms Round trip delay

3. OC-3 link at 10ms Round trip delay

Guidelines for Maximum Fibre Channel Sources at GigE, OC-12, OC-3, and T3 Rates

The following are guidelines based upon laboratory testing for the maximum number of Fibre Channel sources for given link bandwidths and round trip times (RTT).

The following guidelines cover Gigabit Ethernet (1000Mbps), OC-12 (622Mbps), OC-3 (155Mbps), and T3 (45 Mbps) WAN rates at round trip times (RTT) of up to 100ms (50ms each way) and IP WAN packet loss of 0, 0.01%, 0.05%, 0.1%, 0.5% and 1% (random packet loss).

Note that the guideline matrices for packet loss are built around random statistical distributions for packet loss.

For each matrix, the left column shows the number of FC sources or flows, each at 100 MBps (1 Gbps). The top row indicates the RTT (Round trip time) in milliseconds. The intersection of the RTT and flows give the suggested number of Receive BB Credits for ports destined over the FCIP WAN link. Receive BB Credits can be configured through the Device Manager or through the CLI (switchport fcrxbbcredit <xx>). The default number of BB credits allocated to an F_Port or FL_Port is 16 on the Cisco MDS 9000 Family switches and therefore may not require any changes for the following examples. Invalid configurations (configurations that may lead to Fibre Channel frame expiry) are denoted by the shaded boxes labeled "Not Valid").

Gigabit Ethernet

Notes: Each flow generated at 100MBps with 2148 Byte frames (one way)

Packet loss: Zero

One way FC throughput: ~115MBps (@100ms RTT)

Packet loss: 0.01%

One way FC throughput: ~77MBps (@100ms RTT)

Packet loss: 0.05%

One way FC throughput: ~50MBps (@100ms RTT)

Packet loss: 0.1%

One way FC throughput: ~42MBps (@100ms RTT)

Packet loss: 0.5%

One way FC throughput: ~25MBps (@100ms RTT)

Packet loss: 1%

One way FC throughput: ~17MBps (@100ms RTT)

OC-12 (622Mbps)

Notes: Each flow generated at 100MBps with 2148 Byte frames (one way)

Packet loss: Zero

One way FC throughput: ~71MBps (@100ms RTT)

Packet loss: 0.01%

One way FC throughput: ~53MBps (@100ms RTT)

Packet loss: 0.05%

One way FC throughput: ~36MBps (@100ms RTT)

Packet loss: 0.1%

One way FC throughput: ~32MBps (@100ms RTT)

Packet loss: 0.5%

One way FC throughput: ~20MBps (@100ms RTT)

Packet loss: 1%

One way FC throughput: ~14MBps (@100ms RTT)

OC-3 (155Mbps)

Notes: Each flow generated at 100MBps with 2148 Byte frames (one way)

Packet loss: Zero

One way FC throughput: ~17.8MBps (@100ms RTT)

Packet loss: 0.01%

One way FC throughput: ~15.6MBps (@100ms RTT)

Packet loss: 0.05%

One way FC throughput: ~12.8MBps (@100ms RTT)

Packet loss: 0.1%

One way FC throughput: ~11MBps (@100ms RTT)

Packet loss: 0.5%

One way FC throughput: ~7.6MBps (@100ms RTT)

Packet loss: 1%

One way FC throughput: ~6.7MBps (@100ms RTT)

T3 (45Mbps)

Notes: Each flow generated at 100MBps with 2148 Byte frames (one way)

Packet loss: Zero

One way FC throughput: ~5.2MBps (@100ms RTT)

Packet loss: 0.01%

One way FC throughput: ~5.0MBps (@100ms RTT)

Packet loss: 0.05%

One way FC throughput: ~4.5MBps (@100ms RTT)

Packet loss: 0.1%

One way FC throughput: ~4.1MBps (@100ms RTT)

Packet loss: 0.5%

One way FC throughput: ~3.0MBps (@100ms RTT)

Alter default Fibre Channel Receive Performance Buffers for high RTT and # sources as follows for each ingress fc interface:

switchport fcrxbbcredit performance-buffers 4

To return performance buffers to default value, issue:

switchport fcrxbbcredit performance-buffers default

Packet loss: 1%

One way FC throughput: ~2.5MBps (@100ms RTT)

Alter default Fibre Channel Receive Performance Buffers for high RTT and # sources as follows for each ingress fc interface:

switchport fcrxbbcredit performance-buffers 4

To return performance buffers to default value, issue:

switchport fcrxbbcredit performance-buffers default