InfiniBand is a high-speed network architecture commonly used in supercomputers and data centers for low-latency, high-bandwidth interconnection of compute nodes and storage. This article focuses on configuring and managing an InfiniBand network on the CentOS 7 operating system.

Installation and Configuration

1. System Preparation

   • Before starting, ensure your system is up to date: sudo yum update.
   • Install necessary packages: sudo yum install infiniband-diags perftest qperf.

2. Identifying InfiniBand Devices

   • Use the ibstat command to display available InfiniBand adapters and their status.

3. Configuring IPoIB (IP over InfiniBand)

   • Modify the file /etc/sysconfig/network-scripts/ifcfg-ib0, where ib0 is the name of your InfiniBand interface. Example configuration:

     TYPE=InfiniBand
     BOOTPROTO=none
     ONBOOT=yes
     NETWORK=192.168.100.0
     NETMASK=255.255.255.0
     IPADDR=192.168.100.1
     DEVICE=ib0
     ​

   • Restart network services: sudo service network restart.

Management and Monitoring

1. InfiniBand Network Management

   • Utilize ibv_devinfo for detailed information about InfiniBand devices.
   • Use ibnetdiscover for network topology discovery and troubleshooting.

2. Monitoring

   • The perfquery tool allows monitoring of performance metrics of InfiniBand devices.
   • ibdiagnet provides comprehensive network diagnostics.

3. Security

   • Securing the InfiniBand network involves managing keys for secure data transfers, which can be configured using ibsecurity.

Advanced Configuration

1. RDMA (Remote Direct Memory Access)

   • RDMA enables direct memory access between hosts over the InfiniBand network, minimizing CPU overhead. Configuring RDMA requires installation and configuration of appropriate RDMA packages.

2. Partitioning and Virtualization

   • InfiniBand supports partitioning for isolating network traffic, configured using ibpart.
   • For virtualization, SR-IOV (Single Root I/O Virtualization) can be utilized, allowing the partitioning of a physical InfiniBand adapter into multiple virtual functions.

Proper configuration and management of the InfiniBand network on CentOS 7 can significantly enhance the performance and efficiency of high-speed interconnects in your computing environment. It's important to regularly monitor and maintain the network to ensure optimal functionality and security.