Setting Up Infiniband in Centos 6.7

Installing Infiniband Drivers

In Centos/RHEL, software support for Mellanox infiniband hardware is found in the package group “Infiniband Support”, which can be installed with yum:

$yum -y groupinstall "Infiniband Support"

This will install the required kernel modules, and the infiniband subnet manager opensm.

Several optional packages are also available that make configuring and trouble shooting the network easier:

$yum -y install infiniband-diags perftest gperf

inifiniband-diags is a network diagnostic package containing useful analysis tools such as ibping and ibstat.

perftest and gperf are performance testing packages containing benchmarking tools.

Infinband in Centos uses RDMA (Remote Direct Memory Access, the ability to access the memory of a host without disturbing it’s CPU). Set the rdma and opensm services to start at boot time:

$chkconfig rdma on
$chkconfig opensm on

Restart the system. This will load the required kernel modules and start rdma and opensm.

$reboot

Checking Network Connectivity

After the system reboots, check the status of rdma and opensm (status should be on):

$service rdma status
$service opensm status

Check that any infiniband interfaces are now recognized. They should appear as ib0, ib1, etc. with 20 byte hardware addresses:

$ifconfig

Test status of local infiniband link with ibstat:

$ibstat

Connected ports are in the "LinkUp" state. Each interface will have a Base LID (link id) field by which it will be recognized by other hosts.

To display ib hosts and ib switches visible on the network run the ibhosts and ibswitches commands:

$ibhosts
$ibswitches

To test network connectivity with the ibping command (equivalent to icmp ping command), navigate to the host you wish to ping and start its ibping server:

$ibping -S &

Run ibstat to find the base link id of the connected port. Then navigate the host you wish to ping from and run the command:

$ibping <lid_dest>

You should see a stream of packets traveling to the destination host with transfer times and sequence numbers.

Configuring the Network

Before making any changes to network scripts stop the network service:

$service network stop

Centos 6.7 supports “infiniband over ip” meaning that the infiniband interfaces can be configured with ip addresses just like Ethernet interfaces. Configuration files can be found in the /etc/sysconfig/network-scripts folder, and have ib0, ib1, etc in the file name.

By default, the script should have values similar to:

TYPE=infiniband
BOOTPROTO=dhcp
NAME=ib0
UUID=
NM_CONTROLLED=yes
DEVICE=ib0
ONBOOT=no
To configure a static ip address, change the ONBOOT value to “yes”, the bootproto value to “static”, the NM_CONTROLLED value to “no.” Add the ip address, netmask, and router address. The modified script should look like:
TYPE=infiniband
BOOTPROTO=static
NAME=ib0
UUID=
NM_CONTROLLED=no
DEVICE=ib0
ONBOOT=yes
IPADDR=<address>
GATEWAY=<router address, usually first address of subnet>
NETMASK=<netmask>
 Save changes to the script and start the network service.
$service network start
Run ifconfig again to make sure the infiniband interfaces have static ip addresses.

How to Install CentOS 6 Minimum Without Internet

When I first installed CentOS 6 minimum, I couldn’t figure out how to install the OS without internet connection, but eventually, I figured it out.

The OS that I’m using is CentOS 6.7 minimum, and normally, you’d need Internet connection while installing the OS because you would select URL for the installation method.

To create the bootable USB flash drive, I’ve created a blog post for the instructions.¬†Insert a bootable USB flash drive into the machine and boot into CentOS 6.7.

IMG_20160413_203819
Install or upgrade an existing system.
IMG_20160413_203906
Select a language.
IMG_20160413_203911
Keyboard type.
IMG_20160413_203920
Select Hard Drive as the Installation Method.
IMG_20160413_203933
Select /dev/sda1 as the Select Partition.
IMG_20160413_204003
Hit Next.
IMG_20160413_204010
Basic Storage Devices.
IMG_20160413_204042
Type a Hostname.
IMG_20160413_204048
Select a time zone.
IMG_20160413_204059
Enter a password.
IMG_20160413_204113
I chose to use all space on the hard drive.
IMG_20160413_204133
Add the Hard Drive. Not the USB Disk.
IMG_20160413_204140
Write changes to disk.
IMG_20160413_204356
Installation happens without Internet.
IMG_20160413_204658
Reboot after CentOS installation is complete.
IMG_20160413_204833
Eject the USB and the computer boots into CentOS 6.7.
IMG_20160413_204840
Use the CentOS 6.7 as expected!