In today's world, where speed and efficiency of data transfer are becoming increasingly critical, technologies like InfiniBand and Remote Direct Memory Access (RDMA) are proving to be essential for high-speed network transfers. These technologies enable extremely fast data transfer between computers with minimal CPU overhead, making them ideal for demanding computational tasks and data-intensive applications. In this article, we'll look at how to configure the CentOS 7 operating system to support RDMA over InfiniBand.

Preparing the System

Before you begin configuration, ensure your system is up-to-date and has all the necessary packages installed. You can update the system and install the required dependencies by running the following commands:

sudo yum update
sudo yum groupinstall "Development Tools"
sudo yum install rdma-core libibverbs-utils perftest infiniband-diags

Installing and Configuring InfiniBand Drivers

For full RDMA support over InfiniBand, it's necessary to install and correctly configure InfiniBand drivers. CentOS 7 typically includes InfiniBand drivers in the base package, but it's important to verify they are properly installed and loaded.

1. Checking InfiniBand module loading:

   lsmod | grep ib_
   

   If you see output, it means the InfiniBand drivers are loaded.

2. Configuring network interfaces: InfiniBand network interfaces are usually labeled as ib0, ib1, etc. You can configure these interfaces in files located in /etc/sysconfig/network-scripts/. For static IP configuration, create a file named ifcfg-ib0 with the following content:

   TYPE=InfiniBand
   BOOTPROTO=none
   ONBOOT=yes
   IPADDR=192.168.1.1
   NETMASK=255.255.255.0
   ​

3. Restarting network services: After completing the configuration, restart the network services using the command:

   systemctl restart network
   ​

Configuring RDMA

RDMA communication requires proper configuration on both ends of the communication channel. This includes setting up a Subnet Manager on the InfiniBand network, which can be run on any node in the network. For small to medium deployments, it's common to use OpenSM:

sudo yum install opensm
sudo systemctl enable opensm
sudo systemctl start opensm

Ensure that the Subnet Manager is active and running, which you can verify using the systemctl status opensm command.

Testing RDMA Transfer

After configuring, it's crucial to verify that RDMA transfer is functioning correctly. This can be done using the ibv_devinfo tool to check the state of InfiniBand devices and ibping to test connectivity:

1. Checking InfiniBand device status:

   ibv_devinfo
   

   This command should display information about the InfiniBand device, including its state, which should be PORT_ACTIVE if everything is configured correctly.

2. Testing connectivity with ibping: On one node, run ibping in server mode:

   ibping -S
   ​

   On the other node, run ibping in client mode, using the Local Identifier (LID) of the first node as an argument:

   ibping -G <LID_of_first_node>
   

   A successful ping confirms that RDMA communication between the two nodes is working.

Configuring RDMA over InfiniBand on CentOS 7 enables the utilization of high-speed network transfers, crucial for demanding computational tasks and data-intensive applications. RDMA allows applications to access remote memory without CPU involvement, leading to significant performance improvements and latency reduction. The procedure outlined in this article should serve as a basic guide for setting up and testing RDMA over InfiniBand on your CentOS 7 system. It's important to note that each environment may require specific configuration adjustments, so further testing and optimization for your particular needs are always recommended.