Infiniband
From Leo's Notes
This page was last edited on 3 March 2020, at 17:21.
Commands
Here are some commands when working with Infiniband.
| Command | Description |
|---|---|
ibcheckwidth |
|
ibdiagnet |
Shows diagnostic information about the network. Example commands:
# ibdiagnet --pc -P all=1 -get_phy_info --extended_speed all --pm_per_lane \
--get_cable_info --cable_info_disconnected --pm_pause_time 600 -o /tmp/ibdiagnet_ibm
# ibdiagnet -P symbol_error_counter=1
|
ibnetdiscover |
Discovers the network topology. Pipe stdout to /dev/null to look for any errors.
# ibnetdiscover > /dev/null
|
ibqueryerrors |
Looks for any errors.
# ibqueryerrors -s PortXmitWait,LinkErrorRecoveryCounter,PortRcvSwitchRelayErrors,\
LinkDownedCounter,PortXmitDiscards,VL15Dropped,PortRcvErrors,PortRcvRemotePhysicalErrors
|
ibstat |
Shows port statistics |
ibswitches |
Shows switches on the network |
ibv_devinfo |
Shows the Infiniband device information. |
IP over Infiniband (IPoIB)
On Linux, the driver is ib_ipoib. Like all other kernel modules, its options when loading can be set at /etc/modprobe.d/ib_ipoib.conf.
Issues with dropped packets
I've been beating my head trying to determine why an infiniband adapter on one particular server is dropping packets:
# cat /sys/class/net/ib0/statistics/{tx,rx}_dropped
4069
141287
The ring buffer size appears to be smaller than other servers. Other servers that have no issues have a buffer size of 512/512 for rx/tx respectively.
# ethtool -g ib0
Ring parameters for ib0:
Pre-set maximums:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 128
Set the buffer size one time:
# ethtool -G ib0 rx 8192
# ethtool -G ib0 tx 4096
Or, set it in the kernel modprobe.d file:
# echo "options ib_ipoib recv_queue_size=8192 send_queue_size=4096 >> /etc/modprobe.d/ib_ipoib.conf
See Also
- https://ieeexplore.ieee.org/document/7092697
- IP over Infiniband info - https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt