Enterprise Data Center

Kindred's data center is located at its Kentucky headquarters, with remote servers on a wide area network (WAN) deployed throughout the country. As one of the nation's largest long-term healthcare providers, the company relies heavily on remote and centralized management, using web and terminal-based applications to share patient and corporate information between the data center and the four subsidiary divisions. The main data center, with over 200 Terabytes of storage, is responsible for storing all the data for Kindred's corporate applications, including SAP, ERP, patient records, and clinical applications, and also provides centralized data backup.

In 2004, Kindred's IT staff realized that they would need to expand their storage area network (SAN) to accommodate future growth. They also wanted to include SAN connectivity to their remote disaster recovery site and add iSCSI capabilities for Ethernet connectivity. At that time, the storage environment consisted of two SANs in a core to edge design linking 26 non-modular SAN switches. The SANs were limited because many ports were needed to provide Inter Switch Links (ISLs) between switches, the SANs were somewhat unwieldy to manage, it was difficult to add capacity to the non-modular switches, and the core switches were limited to 1 gigabit per second.

"Traditionally, we have always tried to stay ahead of the curve instead of chasing it," said Tim Hesson, Corporate Manager of Storage Management at Kindred. "So we began planning early on to provide for increased capacity and performance, and we wanted to create a more flexible environment that could grow without impact to our users."

In today's healthcare environment, patient data records are at the heart of the business, as they can literally make the difference between life and death. For this reason, providers need to meet strict federal HIPAA guidelines for safety, security, and availability. "We found that we were at a juncture where we had to make a major investment in order to grow the environment," Hesson said. "We began planning for a SAN core replacement."

Knowing that they faced a significant upgrade, Kindred's IT staff considered their current vendor but also decided to look at other possibilities, including Cisco. The company had been using Cisco in its local area network (LAN) for more than five years, so they had confidence in Cisco as a strategic vendor partner. With an open mind, Kindred's data storage managers investigated all their options with due diligence.

"We did side-by-side product comparisons, overviews of the software, went to executive briefings-and engaged in months of evaluation," Hesson said. "In the end, after selecting Cisco, we realized we had an opportunity to look at the entire storage environment and redesign our SAN with next generation features. We weren't just swapping ports, we were designing for the future."

In planning and designing their new SAN, Kindred's IT staff engaged the help of Cisco Professional Services. "We're a very self sufficient shop, so we like to engage with professional services vendors as partners in a collaborative process from start to finish," Hesson said. "We don't do well with vendors that try to make us fit a standard boilerplate." By taking into account Kindred's unique environment and requirements, the joint team was able to plan and configure a cost effective and flexible design that consolidated 26 switches into two new SANs based on four Cisco MDS 9509 Multilayer Director switches that provide a total of 800 ports. "The consolidation alone freed up 100 ports that had been used just for ISL connectivity," said Kris Kostyo, Senior Systems Programmer at Kindred. "In addition, managing the four new Directors was much simpler and more cost effective than managing 26 switches." With the new design, Kindred was also able to integrate its tape backup system into the SAN fabric.

The new SANs support all of Kindred's mission-critical applications, which include payroll, medical appointment and lab scheduling, financials, web-based applications, and email for all of Kindred's organizations nationwide. With every subsidiary organization depending on them, the new SANs were designed for end-to-end fault tolerance. Each SAN has dual fabrics for redundancy, as well as dual components for everything else, creating a truly fault tolerant environment.

"With the Cisco MDS platform, we can stay ahead of capacity needs, without impacting our customer base," Hesson said. "Our previous non-modular switches required a major restructuring to replace them. Now we can add additional MDS systems by simply plugging them into the fabric; we can snap on additional frames, or use the blades independently - it's a whole lot more flexible."

The actual transfer of data to the new SAN environment was accomplished over a 70 day period, without impacting any of Kindred's users. The migration was assisted by Cisco's virtual SAN (VSAN) capability, which allowed the team to segment the data by function. "For me, one of the most exciting aspects of the Cisco implementation was that no one at Kindred would have been aware of the change if they hadn't been told," Hesson concluded. "The new SANs have been extremely stable and reliable, and never affected productivity at all. We were able to make this transition with virtually zero downtime." With all data moved over to the new SANs, VSANs are now being used to logically separate data for specific applications and functions. Summing up his experiences, Hesson said, "Working with the Cisco team and Cisco Professional Services was probably one of the best vendor engagements I've been exposed to. The level of professionalism, the technical capabilities, the assistance with planning-Cisco was a true partner to us in every sense of the word."

As a long-term healthcare company, Kindred's revenue is based almost entirely on Medicare and Medicaid. As a result, the company places a high value on fiscal responsibility, efficiency, and investment protection, allowing the company to keep the IS budget at a fraction of what most IS organizations spend. Kindred IT staff are already recognizing significant benefits from the new SAN design. For example, investment protection is provided by the MDS modular chassis, which helps to decrease the cost of ownership. Blades can be moved as needed or replaced when expired, without incurring any downtime. The less complex infrastructure has also allowed the IT department to reduce port costs to its in-house customers, at a tangible cost of $200 per port.

The new SAN design also improves IT staff productivity: Kindred's entire storage networking system is successfully managed by an in-house team of just four people. The automation and efficiency of the new design will help the staff to continue adding capacity without adding new head count.

In the near future, the storage team is planning to complete their dual fabric remote disaster recovery SAN, based on two modular Cisco 9216i Multilayer Fabric Switches. With Gigabit Ethernet capability in each SAN, there will be no problems connecting the main SAN with the disaster recovery SAN. In addition, the Kindred team has plans to use Inter VSAN Routing (IVR) to accomplish LAN-free data backups. They will also take advantage of iSCSI capabilities to further reduce their operational costs.

To find out more about Cisco's storage solutions, go to: http://www.cisco.com/go/storagenetworking/