Infiniband: Difference between revisions

From Leo's Notes
This page was last edited on 3 March 2020, at 17:21.
No edit summary
Line 34: Line 34:
|}
|}


== IP over Infiniband (IPoIB) ==
On Linux, the driver is {{code|ib_ipoib}}. Like all other kernel modules, its options when loading can be set at {{code|/etc/modprobe.d/ib_ipoib.conf}}. 
=== Issues with dropped packets ===
I've been beating my head trying to determine why an infiniband adapter on one particular server is dropping packets:
{{highlight|lang=terminal|code=
# cat /sys/class/net/ib0/statistics/{tx,rx}_dropped
4069
141287
}}
The ring buffer size appears to be smaller than other servers. Other servers that have no issues have a buffer size of 512/512 for rx/tx respectively.
{{highlight|lang=terminal|code=
# ethtool -g ib0
Ring parameters for ib0:
Pre-set maximums:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 128
}}
Set the buffer size one time:
{{highlight|lang=terminal|code=
# ethtool -G ib0 rx 8192
# ethtool -G ib0 tx 4096
}}
Or, set it in the kernel {{code|modprobe.d}} file:
{{highlight|lang=terminal|code=
# echo "options ib_ipoib recv_queue_size=8192 send_queue_size=4096 >> /etc/modprobe.d/ib_ipoib.conf
}}


== See Also ==
== See Also ==
* https://ieeexplore.ieee.org/document/7092697
* https://ieeexplore.ieee.org/document/7092697
* IP over Infiniband info - https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt
* IP over Infiniband info - https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt

Revision as of 17:21, 3 March 2020

Commands

Here are some commands when working with Infiniband.

Command Description
ibcheckwidth
ibdiagnet Shows diagnostic information about the network. Example commands:
# ibdiagnet --pc -P all=1 -get_phy_info --extended_speed all --pm_per_lane \
     --get_cable_info --cable_info_disconnected --pm_pause_time 600 -o /tmp/ibdiagnet_ibm
# ibdiagnet -P symbol_error_counter=1
ibnetdiscover Discovers the network topology. Pipe stdout to /dev/null to look for any errors.
# ibnetdiscover > /dev/null
ibqueryerrors Looks for any errors.
# ibqueryerrors -s PortXmitWait,LinkErrorRecoveryCounter,PortRcvSwitchRelayErrors,\
LinkDownedCounter,PortXmitDiscards,VL15Dropped,PortRcvErrors,PortRcvRemotePhysicalErrors
ibstat Shows port statistics
ibswitches Shows switches on the network
ibv_devinfo Shows the Infiniband device information.

IP over Infiniband (IPoIB)

On Linux, the driver is ib_ipoib. Like all other kernel modules, its options when loading can be set at /etc/modprobe.d/ib_ipoib.conf.

Issues with dropped packets

I've been beating my head trying to determine why an infiniband adapter on one particular server is dropping packets:

# cat /sys/class/net/ib0/statistics/{tx,rx}_dropped
4069
141287

The ring buffer size appears to be smaller than other servers. Other servers that have no issues have a buffer size of 512/512 for rx/tx respectively.

# ethtool -g ib0
Ring parameters for ib0:
Pre-set maximums:
RX:		8192
RX Mini:	0
RX Jumbo:	0
TX:		8192
Current hardware settings:
RX:		256
RX Mini:	0
RX Jumbo:	0
TX:		128

Set the buffer size one time:

# ethtool -G ib0 rx 8192
# ethtool -G ib0 tx 4096

Or, set it in the kernel modprobe.d file:

# echo "options ib_ipoib recv_queue_size=8192 send_queue_size=4096 >> /etc/modprobe.d/ib_ipoib.conf

See Also