Infiniband: Difference between revisions
From Leo's Notes
This page was last edited on 3 March 2020, at 17:21.
No edit summary |
|||
| Line 34: | Line 34: | ||
|} | |} | ||
== IP over Infiniband (IPoIB) == | |||
On Linux, the driver is {{code|ib_ipoib}}. Like all other kernel modules, its options when loading can be set at {{code|/etc/modprobe.d/ib_ipoib.conf}}. | |||
=== Issues with dropped packets === | |||
I've been beating my head trying to determine why an infiniband adapter on one particular server is dropping packets: | |||
{{highlight|lang=terminal|code= | |||
# cat /sys/class/net/ib0/statistics/{tx,rx}_dropped | |||
4069 | |||
141287 | |||
}} | |||
The ring buffer size appears to be smaller than other servers. Other servers that have no issues have a buffer size of 512/512 for rx/tx respectively. | |||
{{highlight|lang=terminal|code= | |||
# ethtool -g ib0 | |||
Ring parameters for ib0: | |||
Pre-set maximums: | |||
RX: 8192 | |||
RX Mini: 0 | |||
RX Jumbo: 0 | |||
TX: 8192 | |||
Current hardware settings: | |||
RX: 256 | |||
RX Mini: 0 | |||
RX Jumbo: 0 | |||
TX: 128 | |||
}} | |||
Set the buffer size one time: | |||
{{highlight|lang=terminal|code= | |||
# ethtool -G ib0 rx 8192 | |||
# ethtool -G ib0 tx 4096 | |||
}} | |||
Or, set it in the kernel {{code|modprobe.d}} file: | |||
{{highlight|lang=terminal|code= | |||
# echo "options ib_ipoib recv_queue_size=8192 send_queue_size=4096 >> /etc/modprobe.d/ib_ipoib.conf | |||
}} | |||
== See Also == | == See Also == | ||
* https://ieeexplore.ieee.org/document/7092697 | * https://ieeexplore.ieee.org/document/7092697 | ||
* IP over Infiniband info - https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt | * IP over Infiniband info - https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt | ||
Revision as of 17:21, 3 March 2020
Commands
Here are some commands when working with Infiniband.
| Command | Description |
|---|---|
ibcheckwidth |
|
ibdiagnet |
Shows diagnostic information about the network. Example commands:
# ibdiagnet --pc -P all=1 -get_phy_info --extended_speed all --pm_per_lane \
--get_cable_info --cable_info_disconnected --pm_pause_time 600 -o /tmp/ibdiagnet_ibm
# ibdiagnet -P symbol_error_counter=1
|
ibnetdiscover |
Discovers the network topology. Pipe stdout to /dev/null to look for any errors.
# ibnetdiscover > /dev/null
|
ibqueryerrors |
Looks for any errors.
# ibqueryerrors -s PortXmitWait,LinkErrorRecoveryCounter,PortRcvSwitchRelayErrors,\
LinkDownedCounter,PortXmitDiscards,VL15Dropped,PortRcvErrors,PortRcvRemotePhysicalErrors
|
ibstat |
Shows port statistics |
ibswitches |
Shows switches on the network |
ibv_devinfo |
Shows the Infiniband device information. |
IP over Infiniband (IPoIB)
On Linux, the driver is ib_ipoib. Like all other kernel modules, its options when loading can be set at /etc/modprobe.d/ib_ipoib.conf.
Issues with dropped packets
I've been beating my head trying to determine why an infiniband adapter on one particular server is dropping packets:
# cat /sys/class/net/ib0/statistics/{tx,rx}_dropped
4069
141287
The ring buffer size appears to be smaller than other servers. Other servers that have no issues have a buffer size of 512/512 for rx/tx respectively.
# ethtool -g ib0
Ring parameters for ib0:
Pre-set maximums:
RX: 8192
RX Mini: 0
RX Jumbo: 0
TX: 8192
Current hardware settings:
RX: 256
RX Mini: 0
RX Jumbo: 0
TX: 128
Set the buffer size one time:
# ethtool -G ib0 rx 8192
# ethtool -G ib0 tx 4096
Or, set it in the kernel modprobe.d file:
# echo "options ib_ipoib recv_queue_size=8192 send_queue_size=4096 >> /etc/modprobe.d/ib_ipoib.conf
See Also
- https://ieeexplore.ieee.org/document/7092697
- IP over Infiniband info - https://www.kernel.org/doc/Documentation/infiniband/ipoib.txt