Installing Infiniband Drivers
In Centos/RHEL, software support for Mellanox infiniband hardware is found in the package group “Infiniband Support”, which can be installed with yum:
$yum -y groupinstall "Infiniband Support"
This will install the required kernel modules, and the infiniband subnet manager opensm.
Several optional packages are also available that make configuring and trouble shooting the network easier:
$yum -y install infiniband-diags perftest gperf
inifiniband-diags is a network diagnostic package containing useful analysis tools such as ibping and ibstat.
perftest and gperf are performance testing packages containing benchmarking tools.
Infinband in Centos uses RDMA (Remote Direct Memory Access, the ability to access the memory of a host without disturbing it’s CPU). Set the rdma and opensm services to start at boot time:
$chkconfig rdma on $chkconfig opensm on
Restart the system. This will load the required kernel modules and start rdma and opensm.
$reboot
Checking Network Connectivity
After the system reboots, check the status of rdma and opensm (status should be on):
$service rdma status $service opensm status
Check that any infiniband interfaces are now recognized. They should appear as ib0, ib1, etc. with 20 byte hardware addresses:
$ifconfig
Test status of local infiniband link with ibstat:
$ibstat
Connected ports are in the "LinkUp" state. Each interface will have a Base LID (link id) field by which it will be recognized by other hosts.
To display ib hosts and ib switches visible on the network run the ibhosts and ibswitches commands:
$ibhosts $ibswitches
To test network connectivity with the ibping command (equivalent to icmp ping command), navigate to the host you wish to ping and start its ibping server:
$ibping -S &
Run ibstat to find the base link id of the connected port. Then navigate the host you wish to ping from and run the command:
$ibping <lid_dest>
You should see a stream of packets traveling to the destination host with transfer times and sequence numbers.
Configuring the Network
Before making any changes to network scripts stop the network service:
$service network stop
Centos 6.7 supports “infiniband over ip” meaning that the infiniband interfaces can be configured with ip addresses just like Ethernet interfaces. Configuration files can be found in the /etc/sysconfig/network-scripts folder, and have ib0, ib1, etc in the file name.
By default, the script should have values similar to:
TYPE=infiniband BOOTPROTO=dhcp NAME=ib0 UUID=NM_CONTROLLED=yes DEVICE=ib0 ONBOOT=no
ONBOOT value to “yes”, the bootproto value to “static”, the NM_CONTROLLED value to “no.” Add the ip address, netmask, and router address. The modified script should look like:TYPE=infiniband BOOTPROTO=static NAME=ib0 UUID=NM_CONTROLLED=no DEVICE=ib0 ONBOOT=yes IPADDR=<address> GATEWAY=<router address, usually first address of subnet> NETMASK=<netmask>
$service network start
ifconfig again to make sure the infiniband interfaces have static ip addresses.
