Table Of Contents
Release Notes for Cisco OFED, Release 1.3
Determining the Software Version
Upgrading to a New Software Release
Changes from Release 1.2.5 to 1.3
Changes from Release 1.2 to 1.2.5
Changes from Release 1.1 to 1.2
Obtaining Documentation and Submitting a Service Request
Release Notes for Cisco OFED, Release 1.3
Release Date: May 16, 2008
Part Number: OL-11537-06Contents
This document includes the following sections:
•
Obtaining Documentation and Submitting a Service Request
Introduction
These release notes describe the features and known issues for the Cisco OpenFabrics Enterprise Distribution (OFED) InfiniBand Host Drivers 1.3 for Linux.
System Requirements
This section describes the system requirements for this software release and includes the following topics:
•
•
•
Cisco supports OFED on all existing Cisco InfiniBand HCAs, Cisco InfiniBand switches, and Cisco Ethernet and Fibre Channel Gateways.
Determining the Software Version
If InfiniBand drivers are already installed on the host, they may be installed in one of several locations.
To determine the version of the Cisco InfiniBand host drivers, log in to the host and enter the following commands at the shell prompt. If the first command produces output, the Cisco Commercial InfiniBand host drivers are installed. If the second or third commands produce a version number, OFED host drivers are installed.
host$ rpm -qa | grep topspintopspin-ib-mpi-rhel4-3.2.0-118topspin-ib-mod-rhel4-2.6.9-34.ELsmp-3.2.0-118topspin-ib-rhel4-3.2.0-118host$ ofed_info | grep OFEDOFED-1.2host$ grep OFED /usr/local/ofed/BUILD_IDOFED-1.0Upgrading to a New Software Release
To verify that you are running the latest available release, compare your version against the latest version on the Cisco support website at http://www.cisco.com/cgi-bin/tablebuild.pl/sfs-linux. After registering your product, you should have received a username and password that grant you access to this site.
Switch software and Linux host drivers are released and packaged separately. Cisco OFED 1.3 requires that all switches first be upgraded to TopspinOS 2.4.0 or higher. Switches should be upgraded before the InfiniBand hosts.
Use the ofedinstall command to install Cisco OFED 1.3 on your host. There can only be one set of InfiniBand drivers installed on a host at any time. The ofedinstall command will first uninstall any Cisco Commercial InfiniBand host drivers or other OpenFabrics host drivers. If the host has other non-OpenFabrics drivers, those must be uninstalled first before running ofedinstall. If you are downgrading OFED, you should also uninstall the newer drivers before running ofedinstall.
To enable userspace components such as MPI, ofedinstall adds the following entries to /etc/security/limits.conf. You may tune the value unlimited to a specific amount of RAM if desired.
* soft memlock unlimited* hard memlock unlimitedFor general information about upgrading to a new software release, see the Installing Host Drivers chapter in the Cisco OpenFabrics Enterprise Distribution InfiniBand Host Drivers User Guide for Linux.
Supported Kernels
Cisco OFED 1.3 InfiniBand host drivers are supported on the following kernels:
•
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
•
–
–
–
–
–
–
–
–
•
–
–
–
–
–
–
–
–
–
–
If you are using a different kernel, you will need to compile OFED from source code. To compile OFED source code, unpack the OFED tarball in src/ in the Cisco OFED ISO image and follow the instructions in README.txt. Sample .conf files for install.sh are also provided in src/ in the ISO image. If you compile OFED from source code, we will support you on a best-effort basis, but we recommend that you use the pre-built binary RPMs if possible. More information about OFED is available in the ofed-docs package on a host with the drivers installed, http://www.openfabrics.org, and the OpenFabrics general mailing list and its mail archives.
Supported MPI Implementations
Cisco OFED 1.3 InfiniBand host drivers have been tested with the following MPI implementations:
•
•
•
•
•
Other MPI implementations support OFED, for more information contact the MPI vendor.
Supported Compilers for MPI
Cisco OFED 1.3 InfiniBand host drivers are supported with the following compilers:
•
•
•
Supported HCA Firmware
Cisco OFED 1.3 InfiniBand host drivers include the following InfiniBand HCA firmware, which is based on the Mellanox firmware from http://www.mellanox.com/support/firmware_table.php:
For non-Cisco HCAs with Cisco support (from Dell, HP, and IBM), please contact the HCA vendor for HCA firmware and firmware upgrade instructions (typically using mstflint instead of tvflash).
New and Changed Information
This section describes any new and changed information.
Cisco OFED 1.3 is a major release that introduces significant new features and documentation. It includes the following changes:
•
•
Changes from Release 1.2.5 to 1.3
This section describes the new features and resolved caveats since the Cisco OFED 1.2.5 release.
Note
General
•
•
•
•
•
•
•
tvflash now works on SLES10 ia64.
MPI
•
•
•
•
SDP
•
SDP now supports using the connect() function with AF_INET_SDP.
uDAPL
•
uDAPL is not available on ppc64.
Changes from Release 1.2 to 1.2.5
This section describes the new features and resolved caveats since the Cisco OFED 1.2 release.
General
•
Changes from Release 1.1 to 1.2
This section describes the new features and resolved caveats since the Cisco OFED 1.1 release.
General
•
•
•
•
•
•
•
OFED now installs and uninstalls cleanly when multiple kernel-ib packages are installed.
•
32-bit libraries are now included along with 64-bit libraries on 64-bit Operating Systems. These 32-bit libraries have not been fully tested and are not supported by Cisco.
IPoIB
•
IPoIB CM (Connected Mode) has been added and is now the default. IPoIB CM creates one IB Reliable Connection between pairs of hosts that communicate using IPoIB. IPoIB CM has higher throughput than traditional IPoIB (which uses IB Unreliable Datagram connections). IPoIB CM and non-CM are interoperable, IPoIB CM will transparently fall back to using non-CM communication when necessary. Note that IP multicast still uses non-CM communication.
•
IPoIB High Availability has been added through the IPoIB bonding driver. It is based on the Linux Ethernet Bonding Driver and was adopted to work with IPoIB. The ib-bonding package contains the bonding driver and a utility called ib-bond to manage and control the driver operation. The IPoIB bonding drivers supports active/passive failover, pkey interfaces, multiple slave interfaces and multiple HCAs, and configuration at boot time in openib.conf.
•
A problem has been fixed where the maximum IPoIB multicast throughput was less than 1.5 Gbps. Note that this fix requires new HCA firmware, which is included with Cisco OFED 1.2.
•
IPoIB pkey interfaces can now be configured to start at boot time via ifcfg files.
•
MPI
•
•
•
•
•
•
•
•
Open MPI performance on Pallas/Intel MPI collective benchmarks is now improved.
•
Open MPI now supports MPI-2 Remote Memory Access.
•
Open MPI latency has been improved on ia64.
RDS
•
SDP
•
A new sdpnetstat program has been added that is based on netstat and supports SDP.
•
SDP throughput with messages smaller than 8 KB and larger than 64 KB has been improved.
•
SRP
•
SRP High Availability has been added through the srp_daemon program and Device Mapper Multipath, which is included with both RHEL and SLES. Device Mapper Multipath supports both active/active and active/passive high availability, depending on the capability of the storage device.
•
•
The srp_daemon command is now supported with the Cisco Fibre Channel Gateway.
uDAPL
•
uDAPL test programs dapltest and dtest are now included.
Caveats
This section describes temporary limitations of this release. These restrictions will be resolved in a future release of this product.
General
•
•
•
•
tvflash does not work with HCA recovery jumper.
•
The IB node description will sometimes not have the hostname in it.
•
OFED on IA64 will cause a few benign "unaligned access" messages.
IPoIB
•
IPoIB High Availability may experience slow failover time. This problem does not occur with the Cisco Commercial Linux IB drivers.
•
•
bonding: Ignoring new-style parameters in presence of obsolete onesbonding: Warning: either miimon or arp_interval and arp_ip_target module parametersmust be specified, otherwise bonding will not detect link failures! see bonding.txt for details. ... bonding: bond0: Warning: enslaved VLAN challenged slave ib0. Adding VLANs willbe blocked as long as ib0 is part of bond bond0bonding: bond0: Warning: The first slave device you specified does not supportsetting the MAC address. This bond MAC address would be that of the active slave.
These messages are benign and can be ignored.•
•
IPoIB IP multicast is not supported on RHEL4 U4. U4 IP multicast can only receive from U4 senders, and U4 senders sent traffic to all nodes, not just nodes that have joined the multicast address. U4 multicast traffic can cause a Cisco InfiniBand switch to reboot. This problem is fixed in RHEL4 U5 and RHEL5.
•
•
•
IPoIB High Availability (bonding) does not support IPv6 on RHEL4.
•
IPoIB High Availability (bonding) does not support IPv6 on RHEL4.
•
The ib1 interface cannot be configured at boot time through /etc/sysconfig/network/ifcfg-ib1 on SLES10. This has been fixed in SLES10 SP1.
•
IPoIB non-CM does not interoperate with the mgmt-ib IPoIB management interface on the Cisco SFS 3001 and Topspin 90 switch. Note that IPoIB non-CM is not the default in OFED 1.2.
MPI
•
MPI programs must not call fork() on RHEL4 U5 or earlier, and must be careful in general using fork(). More information is available at http://www.open-mpi.org/faq/?category=openfabrics#ofa-fork.
•
The PGI 7.1 compilers are not supported with Open MPI on SLES10.
•
MPI is not supported on RHEL5 ppc64 on non-ConnectX HCAs.
•
The OSU MPI-2 benchmarks may fail with MVAPICH2 on ppc64.
•
•
•
Intel MPI and HP MPI can be used with IPoIB bonding interfaces, but the bond0 uDAPL provider must be used instead of the ib0 provider.
•
MVAPICH does not support GNU Fortran 90 on ppc64.
SDP
•
SDP is not supported on ppc64.
•
OFED SDP does not interoperate with Cisco Commercial Linux InfiniBand Host Driver SDP.
•
•
•
OFED SDP does not support Linux AIO.
•
There is no #define for AF_INET_SDP, code that uses SDP without libsdp must manually add #define AF_INET_SDP 27 at the beginning of a program.
•
SRP
•
SRP High Availability may experience slow failover time. This problem does not occur with the Cisco Commercial Linux IB drivers.
•
The SRP ib_srp kernel module will often fail to unload manually with rmmod or modprobe -r, even if SRP is not in use. The workaround is to reboot the system.
•
SRP filesystems cannot be mounted from /etc/fstab. The workaround is to manually mount these filesystems in /etc/rc.local.
uDAPL
•
•
Related Documentation
The following list describes the documentation available with TopspinOS 2.9.0, which is available in electronic form and printed form upon request.
Note
You can download the latest documentation updates on the Cisco support site at http://www.cisco.com/en/US/partner/products/ps6418/tsd_products_support_category_home.html.
•
•
•
•
•
•
•
•
•
Service and Support
For additional support, you must first register your product at http://www.cisco.com. After registering, you may contact your supplier for support, or Cisco directly.
Refer to the "Obtaining Documentation and Submitting a Service Request" section in this document.
When you call Cisco Technical Support or use the Cisco Technical Support website at http://www.cisco.com, be prepared to provide the following information to support personnel:
General Information
•
•
•
•
Server Configuration
•
•
•
Chassis Configuration
•
•
Chassis Serial Number
The chassis serial number and corresponding bar code are provided on the serial number label, as shown in this example:
Model: TS360
SN UST323XXXXXXXXX
This chassis serial number can be found on the bottom of the chassis or the outside of the chassis box packaging. It can also be found in the output of the show backplane command.
Obtaining Documentation and Submitting a Service Request
For information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What's New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at:
http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html
Subscribe to the What's New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free service and Cisco currently supports RSS version 2.0.
This document is to be used in conjunction with the documents listed in the "Related Documentation" section.
CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn is a service mark; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick Study, IronPort, the IronPort logo, LightStream, Linksys, MediaTone, MeetingPlace, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.
All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0805R)
Any Internet Protocol (IP) addresses used in this document are not intended to be actual addresses. Any examples, command display output, and figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses in illustrative content is unintentional and coincidental.
© 2008 Cisco Systems, Inc. All rights reserved.
Guest
