Installing Infiniband Drivers
In Centos/RHEL, software support for Mellanox infiniband hardware is found in the package group “Infiniband Support”, which can be installed with yum:
$yum -y groupinstall "Infiniband Support"
This will install the required kernel modules, and the infiniband subnet manager
Several optional packages are also available that make configuring and trouble shooting the network easier:
$yum -y install infiniband-diags perftest gperf
inifiniband-diags is a network diagnostic package containing useful analysis tools such as
gperf are performance testing packages containing benchmarking tools.
Infinband in Centos uses RDMA (Remote Direct Memory Access, the ability to access the memory of a host without disturbing it’s CPU). Set the
opensm services to start at boot time:
$chkconfig rdma on $chkconfig opensm on
Restart the system. This will load the required kernel modules and start rdma and opensm.
Checking Network Connectivity
After the system reboots, check the status of
opensm (status should be on):
$service rdma status $service opensm status
Check that any infiniband interfaces are now recognized. They should appear as ib0, ib1, etc. with 20 byte hardware addresses:
Test status of local infiniband link with ibstat:
Connected ports are in the
"LinkUp" state. Each interface will have a Base LID (link id) field by which it will be recognized by other hosts.
To display ib hosts and ib switches visible on the network run the
To test network connectivity with the
ibping command (equivalent to icmp ping command), navigate to the host you wish to ping and start its
$ibping -S &
Run ibstat to find the base link id of the connected port. Then navigate the host you wish to ping from and run the command:
You should see a stream of packets traveling to the destination host with transfer times and sequence numbers.
Configuring the Network
Before making any changes to network scripts stop the network service:
$service network stop
Centos 6.7 supports “infiniband over ip” meaning that the infiniband interfaces can be configured with ip addresses just like Ethernet interfaces. Configuration files can be found in the /etc/sysconfig/network-scripts folder, and have ib0, ib1, etc in the file name.
By default, the script should have values similar to:
TYPE=infiniband BOOTPROTO=dhcp NAME=ib0 UUID=
NM_CONTROLLED=yes DEVICE=ib0 ONBOOT=no
ONBOOTvalue to “yes”, the bootproto value to “static”, the
NM_CONTROLLEDvalue to “no.” Add the ip address, netmask, and router address. The modified script should look like:
TYPE=infiniband BOOTPROTO=static NAME=ib0 UUID=
NM_CONTROLLED=no DEVICE=ib0 ONBOOT=yes IPADDR=<address> GATEWAY=<router address, usually first address of subnet> NETMASK=<netmask>
$service network start
ifconfigagain to make sure the infiniband interfaces have static ip addresses.