Linux routing based on IPtables MARK

http://www.linuxhorizon.ro/iproute2.html

Backup:

This page is a small HOWTO about the advanced linux routing...

First of all let me tell you where you can find the best source of information about the advanced routing under Linux. Most of you probably know or heard about the Linux Advanced Routing & Traffic Control site. There you can see a very comprehensive source of knowledge based not only on documentation but by easy to understand examples...
Credits: Linux Advanced Routing & Traffic Control, Thea
Ok, then...
This page will show you how to set a linux box to use 2 different ISPs on the same time...

First example:
Goal: To route packets that came from 4 network to different ISPs

Let's presume that you have two ISPs. In the following examples I'll use RDS and ASTRAL (two large ISPs from my country)
For the ASCII art and lynx console browser fans I'll use this kind of chart:

                                                                   ________
                                           +-------------+        /
                                           |    ISP 1    |       /
                             +-------------+    (RDS)    +------+
                             |             | gw 10.1.1.1 |     /
                      +------+-------+     +-------------+    / 
+----------------+    |     eth1     |                       /
|                |    |              |                      |
| Local networks +----+ Linux router |                      |  Internet cloud
|                |    |              |                      |
+----------------+    |     eth2     |                       \
                      +------+-------+     +-------------+    \
                             |             |    ISP 2    |     \
                             +-------------+  (ASTRAL)   +------+
                                           | gw 10.8.8.1 |       \
                                           +-------------+        \________

We will work only on Linux router box. From the root prompter do:

echo 1 RDS >> /etc/iproute2/rt_tables
echo 2 ASTRAL >> /etc/iproute2/rt_tables

The /etc/iproute2/rt_tables content after previous commands:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
1 RDS
2 ASTRAL

Now we have three routing tables as follows: RDS table, ASTRAL table and the main table...
Let's fill up every table with the defaults routes:

The next step is to have some routing rules and routes:

For the RDS table:

ip route add default via 10.1.1.1 dev eth1 table RDS
ip rule add from 10.11.11.0/24 table RDS
ip rule add from 10.12.12.0/24 table RDS 

For the ASTRAL table:

ip route add default via 10.8.8.1 dev eth2 table ASTRAL
ip rule add from 10.22.22.0/24 table ASTRAL
ip rule add from 10.33.33.0/24 table ASTRAL

To see the routing tables:

ip route show table ASTRAL
ip route show table RDS
ip route show table main  # it's the same as "route -n" but in different format...

To see the routing tables:

ip rule show   # all the rule list
ip rule show | grep ASTRAL # only for ASRAL
ip rule show | grep RDS  # only for RDS

Let me explain the above rules.
The packets that came from the 10.11.11.0/24 and 10.12.12.0/24 networks will go to the RDS routing table and then (because we have a default route) will be passed to the RDS gateway. And similar, the packets that came from the 10.22.22.0/24 and 10.33.33.0/24 network will go to the ASTRAL gateway...
What is happening with the packets that came from other networks that are not shown in the above rules? Well, they just simply go to main routing table and follow the routing rules that reside there... If you want to block them to go to internet just delete the default route from the main table... (of course, doing that your router can not longer go to interent).


Second example:
Goal: To route the packets having the destination port 22/tcp to the RDS and 80/tcp to the ASTRAL (no matter what network generates them).
This example it is almost the same as the first one except that we will use iptables to mark the packets.

Same chart...

                                                                   ________
                                           +-------------+        /
                                           |    ISP 1    |       /
                             +-------------+    (RDS)    +------+
                             |             | gw 10.1.1.1 |     /
                      +------+-------+     +-------------+    / 
+----------------+    |     eth1     |                       /
|                |    |              |                      |
| Local networks +----+ Linux router |                      |  Internet cloud
|                |    |              |                      |
+----------------+    |     eth2     |                       \
                      +------+-------+     +-------------+    \
                             |             |    ISP 2    |     \
                             +-------------+  (ASTRAL)   +------+
                                           | gw 10.8.8.1 |       \
                                           +-------------+        \________

Same /etc/iproute2/rt_tables content:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
1 RDS
2 ASTRAL

Before you start check your iptables configuration. I strongly recommend to read about iptables if you are unsure about what you will doing next.
For more documentation go to iptables home page or you can download a good documentation from this site (Security & Privacy Section) or directly from here.

To mark the packets that have the 22 and 80 as destination port we will use the MANGLE table...

iptables -A PREROUTING -t mangle -i eth0 -p tcp --dport 22 -j MARK --set-mark 1
iptables -A PREROUTING -t mangle -i eth0 -p tcp --dprot 80 -j MARK --set-mark 2

For the RDS table:

ip route add default via 10.1.1.1 dev eth1 table RDS # the same like in the first example

For the ASTRAL table:

ip route add default via 10.8.8.1 dev eth2 table ASTRAL # the same like in the first example

The next step is to have some routing rules based by the marked packets:

For the RDS:

ip rule add from all fwmark 1 table RDS

For the ASTRAL:

ip rule add from all fwmark 2 table ASTRAL

You can use the same commands to see the routing tables and rule lists as in the first example.
Now you have a routing solution based by the destination port...