====== Network and Infiniband ======

===== General =====

==== Adjust MTU (optional) ====

It is highly recommended to increase MTU for eth network, as usage on it will be ssh and data, and that increasing MTU will increase data performances without reducing ssh performances. 9000 would be a good value for ethernet, and 65520 for infiniband IPoIB (prefer using directly connected_mode).

To do so, add the following parameter in incfg-enp0sx files:

<code>
MTU=9000
</code>

===== Ethernet =====

==== Bonding - LACP (optional) ====

LACP can be used on pets servers to increase bandwidth (with multi-streams only) and reliability (in case one NIC fail, the other will continue to operate). LACP is often used on master nodes and io nodes. You need first to configure your switch to gather links, then on OS side, do the following (to gather eth0 and eth1 into one LACP):

Edit (in /etc/sysconfig/network-scripts/) ifcfg-eth0:
<code>
DEVICE=eth0
TYPE=Ethernet
MTU=9000
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
SLAVE=yes
MASTER=bond0
</code>

Then edit ifcfg-eth1:
<code>
DEVICE=eth1
TYPE=Ethernet
MTU=9000
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
SLAVE=yes
MASTER=bond0
</code>

And create ifcfg-bond0 with the following content:
<code>
DEVICE=bond0
MTU=9000
ONBOOT=yes
BOOTPROTO=static
USERCTL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPADDR=10.0.0.1
NETMASK=255.255.0.0
BONDING_OPTS="miimon=100 mode=4 lacp_rate=1 xmit_hash_policy=layer3+4"
</code>

Bonding needs a kernel module to be loaded.

/etc/modprobe.d/bonding.conf
<code>
alias netdev-bond0 bonding
</code>

And after reboot your bond should be up.

More : http://backdrift.org/lacp-configure-network-bonding-linux

==== VLAN (optional) ====

VLAN can be used to divide your network, for management or security reasons.\\
VLAN can also be used on a bonding interface (bond0 most of the time).

Here is an example how to configure it, after having done configuration on switch side.

Native VLAN is 10.0.0.0, in **ifcfg-bond0** :
<code>
DEVICE=bond0
MTU=9000
ONBOOT=yes
BOOTPROTO=static
USERCTL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPADDR=10.0.0.1
NETMASK=255.255.0.0
BONDING_OPTS="miimon=100 mode=4 lacp_rate=1 xmit_hash_policy=layer3+4"
</code>

VLAN 10 is 10.10.0.0, in **ifcfg-bond0.10** :
<code>
DEVICE=bond0.10
MTU=9000
ONBOOT=yes
BOOTPROTO=none
IPADDR=10.10.0.1
NETMASK=255.255.0.0
USERCTL=no
IPV6INIT=no
IPV6_AUTOCONF=no
VLAN=yes
</code>

VLAN need a kernel module to be loaded.

/etc/modprobe.d/8021q.conf
<code>
8021q
</code>

And after reboot, your VLANs should be up.

===== Infiniband (optional) =====

To install infiniband, use manufacturer provided modules, or use the RHEL/CENTOS provided (RDMA). See https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/ch-Configure_InfiniBand_and_RDMA_Networks.html

In short, you need to install packages, and reboot system. Diag and utils tools are optional but very useful. We are going here to install official RHEL infiniband.

<code>
yum groupinstall "Infiniband Support"
yum install infiniband-diags ibutils
systemctl enable rdma
reboot
</code>

After reboot, check with ibstatus if interface is "ACTIVE". After that, you just need to configure an ip on it like any other interfaces, taking into account the MTU choose.

Then, you need to start a subnet manager on your IB network (if not, all interfaces will be in PULLING). You can start opensmd on main server, here batman. DO NOT start opensm/opensmd on another server. There can be only one (like highlanders) on the IB network. Beware, some advanced IB switch can also integrate a subnet manager, ensure you only use one.

If you want to start the subnet manager on master node (batman), use:

<code>
systemctl start opensmd
systemctl enable opensmd
</code>

Then I you should do a snapshot of your fabric (see below) to use it regularly to check if something has change in the fabric. This is a very simple and efficient fabric check.

==== Identification ====

Switches

<code> ibswitches </code>

HCAs in the fabric (Host Channel Adapters

<code> ibhosts </code>

==== Display ====

Fabric topology

<code> ibnetdiscover </code>

Route through the fabric (slid is the LID of the source node and dlid is the LID of the destination node in the fabric)

<code> ibtracert slid dlid </code>

Link status of a node
<code> ibportstate lid port </code>

Counters of a node
Note that this command can be used locally without arguments to obtain the informations of local port.
<code> perfquery lid port </code>

Data counters of a node
<code> ibdatacounts lid port </code>

Low level informations of a node
<code> smpquery switchinfo lid </code>

Low level informations of a port
<code> smpquery portinfo lid port </code>

==== Diagnostic ====
   
Check the entire fabric
<code> ibdiagnet -v -r </code>

Check a specific route
<code> ibdiagpath -v -l slid dlid </code>

Take a snapshot of the fabric after installation, and check changes of the stack when something occurs

Create snapshot
<code> osmtest -f c </code>

Compare with snapshot
<code> osmtest -f v </code>

Check which links are experiencing significant errors
<code> ibdiagnet -c 100 -P all=1 </code>

Check all ports (remove -v for only errors output)
<code> ibcheckstate -v </code>

==== Other ====
	
Clearing error counters
<code> ibclearerrors </code>

Clearing data counters
<code> ibclearcounters </code>

Reset a port
<code> ibportstate lid port reset </code>

Setting port speed (value = 1 for low speed, 7 for high)
<code> ibportstate lid port speed <value> </code>

More informations, see :
  * https://docs.oracle.com/cd/E18476_01/doc.220/e18478/fabric.htm
  * https://wiki.archlinux.org/index.php/InfiniBand