Using Multiple Networks for a Single Host

Configure the Network Interfaces

To make my Ubuntu Server on more robust I have connected it to 2 ISP so that I can have redundant last mile. Also one of my ISP gives me unlimited bandwidth while the other is costlier but more reliable.

1. To make your linux server multihomed, I am assuming that you have atleast 2 network interfaces. In my case both are ethernet (eth0 and eth1)

2. Make sure both the networks are working individually up by setting it on /etc/network/interface
sudo vi /etc/network/interface

auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
        metric 100

auto eth1
iface eth1 inet static
        metric 200
Now if you go to your terminal, you should be able to
>ip route show dev eth1  proto kernel  scope link  src dev eth0  proto kernel  scope link  src
default via dev eth0  metric 100
default via dev eth1  metric 200
With this if your eth0 goes down, your eth1 will take over and vice-versa. But this is still a long way from making your network multihoned where your can do load balancing.

All you need to do is to add multihop route in your /etc/rc.local ip route append default scope global nexthop via dev eth0 weight 5 nexthop via dev eth1 weight 1



A common configuration is the following, in which there are two providers that connect a local network (or even a single machine) to the big Internet.

                                          +------------+        /
                                          |            |       |
                            +-------------+ Provider 1 +-------
        __                  |             |            |     /
    ___/  \_         +------+-------+     +------------+    |
  _/        \__      |     if1      |                      /
 /             \     |              |                      |
| Local network -----+ Linux router |                      |     Internet
 \_           __/    |              |                      |
   \__     __/       |     if2      |                      \
      \___/          +------+-------+     +------------+    |
                            |             |            |     \
                            +-------------+ Provider 2 +-------
                                          |            |       |
                                          +------------+        \________
There are usually two questions given this setup.

4.2.1. Split access

The first is how to route answers to packets coming in over a particular provider, say Provider 1, back out again over that same provider.

Let us first set some symbolical names. Let eth2 be the name of the first interface (if1 in the picture above) and eth2 the name of the second interface. Then let IP1 be the IP address associated with eth2 and IP2 the IP address associated with eth2. Next, let GW1 be the IP address of the gateway at Provider 1, and GW2 the IP address of the gateway at provider 2. Finally, let NET1 be the IP network GW1 is in, and NET2 the IP network GW2 is in.

One creates two additional routing tables, say T1 and T2. These are added in /etc/iproute2/rt_tables. Then you set up routing in these tables as follows:

	  ip route add NET1 dev eth1 src IP1 table T1
	  ip route add default via GW1 table T1
	  ip route add NET2 dev eth2 src IP2 table T2
	  ip route add default via GW2 table T2


You must create the tables in the file: '/etc/iproute2/rt_tables'

# reserved values
255     local
254     main
253     default
0       unspec
# local
#1      inr.ruhep
1       internal           <<--- These lines were added to the default file
2       WAN2               <<--- they define the table by name and number
3       WAN1               <<--- Table 1 is named 'internal' and so on.
More information about rt_tables can be found here: Nothing spectacular, just build a route to the gateway and build a default route via that gateway, as you would do in the case of a single upstream provider, but put the routes in a separate table per provider. Note that the network route suffices, as it tells you how to find any host in that network, which includes the gateway, as specified above.

Next you set up the main routing table. It is a good idea to route things to the direct neighbour through the interface connected to that neighbour. Note the `src' arguments, they make sure the right outgoing IP address is chosen.

	    ip route add NET1 dev eth2 src IP1
	    ip route add NET2 dev eth2 src IP2

Then, your preference for default route:

	    ip route add default via GW1

Next, you set up the routing rules. These actually choose what routing table to route with. You want to make sure that you route out a given interface if you already have the corresponding source address:

	    ip rule add from IP1 table T1
	    ip rule add from IP2 table T2

This set of commands makes sure all answers to traffic coming in on a particular interface get answered from that interface.


Reader Rod Roark notes: 
If NET0 is the local network and eth0 is its interface, 
the following additional entries are desirable:

ip route add NET0     dev eth0 table T1
ip route add NET2     dev eth2 table T1
ip route add dev lo   table T1
ip route add NET0     dev eth0 table T2
ip route add NET1     dev eth2 table T2
ip route add dev lo   table T2                                      

Now, this is just the very basic setup. It will work for all processes running on the router itself, and for the local network, if it is masqueraded. If it is not, then you either have IP space from both providers or you are going to want to masquerade to one of the two providers. In both cases you will want to add rules selecting which provider to route out from based on the IP address of the machine in the local network.

4.2.2. Load balancing

The second question is how to balance traffic going out over the two providers. This is actually not hard if you already have set up split access as above.

Instead of choosing one of the two providers as your default route, you now set up the default route to be a multipath route. In the default kernel this will balance routes over the two providers. It is done as follows (once more building on the example in the section on split-access):

    ip route add default scope global nexthop via GW1 dev eth2 weight 1 \
    nexthop via GW2 dev eth2 weight 1

This will balance the routes over both providers. The weight parameters can be tweaked to favor one provider over the other.

Note that balancing will not be perfect, as it is route based, and routes are cached. This means that routes to often-used sites will always be over the same provider.

Furthermore, if you really want to do this, you probably also want to look at Julian Anastasov's patches at , Julian's route patch page. They will make things nicer to work with.

Last modified:: 2016/03/19 16:03