Irqbalance

irqbalance is a utility that helps balance interrupts across all available CPUs on a system. The tool does the balancing by setting the appropriate CPU affinities for each IRQ source.

Interrupts overloading a single CPU

Without irqbalance, you might see a single ksoftirqd process running with a high CPU utilization. On a busy system, you might see this stuck at 100% CPU. To confirm what's happening, run:

$ watch -d -n 1 "cat /proc/interrupts"

## To ignore inactive interrupts
$ watch -d -n 1 "cat /proc/interrupts | grep -vE '^\s*[0-9]+:[ 0]*  [A-Za-z]+'"

Using watch and the --diff option, you can see how fast the number of interrupts are increasing for each IRQ source. If you see large numbers within a single column, you should try running irqbalance to help balance out the IRQ sources across multiple CPU cores.

Under the hood

What irqbalance does is it reads /proc/interrupts and determines the IRQ sources. It then tries to balance out the IRQs across all CPU cores by setting the appropriate CPU affinity flags under /proc/irq/<irq>/smp_affinity.

To see what irqbalance does, you can try using the --oneshot and --debug flags. After sampling for around 10 seconds, it will then apply the CPU affinity changes.

# irqbalance --oneshot --debug
## wait ~10 seconds
-----------------------------------------------------------------------------
Package 0:  numa_node is 0 cpu mask is 00005555 (load 0)
        Cache domain 0:  numa_node is 0 cpu mask is 00005555  (load 0)
                CPU number 0  numa_node is 0 (load 0)
                  Interrupt 157 node_num is 0 (ethernet/0)
                  Interrupt 149 node_num is 0 (ethernet/0)
                  Interrupt 141 node_num is 0 (ethernet/0)
                  Interrupt 127 node_num is 0 (ethernet/0)
                  Interrupt 119 node_num is 0 (ethernet/0)
                  Interrupt 73 node_num is 0 (ethernet/0)
                  Interrupt 61 node_num is 0 (storage/0)
                  Interrupt 53 node_num is 0 (storage/0)
                  Interrupt 101 node_num is 0 (ethernet/0)
                  Interrupt 93 node_num is 0 (ethernet/0)
                  Interrupt 85 node_num is 0 (ethernet/0)
                  Interrupt 77 node_num is 0 (ethernet/0)
                  Interrupt 135 node_num is 0 (ethernet/0)
...

Depending on the workload, you may also choose to run irqbalance in daemon mode where it will constantly check and readjust the affinities to ensure IRQs are balanced.

If you need to manually balance out IRQs, you can turn off irqbalance and tweak the smp affinities yourself.