Tag: Infiniband

Setting Up Infiniband in Centos 6.7

Installing Infiniband Drivers

In Centos/RHEL, software support for Mellanox infiniband hardware is found in the package group “Infiniband Support”, which can be installed with yum:

$yum -y groupinstall "Infiniband Support"

This will install the required kernel modules, and the infiniband subnet manager opensm.

Several optional packages are also available that make configuring and trouble shooting the network easier:

$yum -y install infiniband-diags perftest gperf

inifiniband-diags is a network diagnostic package containing useful analysis tools such as ibping and ibstat.

perftest and gperf are performance testing packages containing benchmarking tools.

Infinband in Centos uses RDMA (Remote Direct Memory Access, the ability to access the memory of a host without disturbing it’s CPU). Set the rdma and opensm services to start at boot time:

$chkconfig rdma on
$chkconfig opensm on

Restart the system. This will load the required kernel modules and start rdma and opensm.

$reboot

Checking Network Connectivity

After the system reboots, check the status of rdma and opensm (status should be on):

$service rdma status
$service opensm status

Check that any infiniband interfaces are now recognized. They should appear as ib0, ib1, etc. with 20 byte hardware addresses:

$ifconfig

Test status of local infiniband link with ibstat:

$ibstat

Connected ports are in the "LinkUp" state. Each interface will have a Base LID (link id) field by which it will be recognized by other hosts.

To display ib hosts and ib switches visible on the network run the ibhosts and ibswitches commands:

$ibhosts
$ibswitches

To test network connectivity with the ibping command (equivalent to icmp ping command), navigate to the host you wish to ping and start its ibping server:

$ibping -S &

Run ibstat to find the base link id of the connected port. Then navigate the host you wish to ping from and run the command:

$ibping <lid_dest>

You should see a stream of packets traveling to the destination host with transfer times and sequence numbers.

Configuring the Network

Before making any changes to network scripts stop the network service:

$service network stop

Centos 6.7 supports “infiniband over ip” meaning that the infiniband interfaces can be configured with ip addresses just like Ethernet interfaces. Configuration files can be found in the /etc/sysconfig/network-scripts folder, and have ib0, ib1, etc in the file name.

By default, the script should have values similar to:

TYPE=infiniband
BOOTPROTO=dhcp
NAME=ib0
UUID=
NM_CONTROLLED=yes
DEVICE=ib0
ONBOOT=no
To configure a static ip address, change the ONBOOT value to “yes”, the bootproto value to “static”, the NM_CONTROLLED value to “no.” Add the ip address, netmask, and router address. The modified script should look like:
TYPE=infiniband
BOOTPROTO=static
NAME=ib0
UUID=
NM_CONTROLLED=no
DEVICE=ib0
ONBOOT=yes
IPADDR=<address>
GATEWAY=<router address, usually first address of subnet>
NETMASK=<netmask>
 Save changes to the script and start the network service.
$service network start
Run ifconfig again to make sure the infiniband interfaces have static ip addresses.