Multiple Networks on Linux

Issue

Because Linux has only one routing table out of the box, all replies originating from the host will go through the default route defined by the routing table. This is typically not the desired behavior since traffic from a secondary interface on a different network will be replied back on the primary interface.

Solution

The solution is to create a routing table for each additional interface.

For example:

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:0c:29:ea:92:46 brd ff:ff:ff:ff:ff:ff
    inet 136.159.5.40/24 brd 136.159.5.255 scope global dynamic eth0
       valid_lft 77336sec preferred_lft 77336sec
    inet6 fe80::20c:29ff:feea:9246/64 scope link
       valid_lft forever preferred_lft forever
3: ens36: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:ea:92:50 brd ff:ff:ff:ff:ff:ff
    inet 172.17.12.253/24 brd 172.17.12.255 scope global dynamic ens36
       valid_lft 86108sec preferred_lft 86108sec
    inet6 fe80::6cb0:9a02:c755:15e5/64 scope link
       valid_lft forever preferred_lft forever
# echo "1 ens36" >> /etc/iproute2/rt_tables
# ip route add 172.17.12.0/24 dev ens36 src 172.17.12.253 table ens36
# ip route add default via 172.17.12.1 dev ens36 table ens36
# ip rule add from 172.17.12.253/32 table ens36
# ip rule add to 172.17.12.253/32 table ens36

To make this persistent, create a rule- and route- file under /etc/sysconfig/network-scripts.

If you're using a new-ish version of CentOS (7 & 8), you will also need to add NM_CONTROLLED=no to the ifcfg- file for these files to apply. You will also need the network-scripts package installed.

# cat /etc/sysconfig/network-scripts/rule-eno2
from 136.159.79.8/32 lookup eno2
to 136.159.79.8/32 lookup eno2

# cat /etc/sysconfig/network-scripts/route-eno2
136.159.79.0/24 dev eno2 table eno2
default via 136.159.79.1 dev eno2 table eno2

See Also

• http://brainscraps.wikia.com/wiki/Setup_Gateway_Routing_On_Multiple_Network_Interfaces
• https://www.thomas-krenn.com/en/wiki/Two_Default_Gateways_on_One_System