How to make inbound WAN connections sticky to the interface

From VyOS Wiki
Jump to: navigation, search
Warning sign.png This page is migrated to Readthedocs .
Information found on this page is migrated to readthedocs and information found here could be outdated or misleading.
For a complete status of all migrations, see Project:Migration
Hourglass.png This page is obsolete. It doesn't apply to VyOS versions newer than 1.1.1


  • Inbound connections to a WAN interface are improperly handled when the reply is sent back to the client.

Vyos Sticky WAN Problem.jpg

  • It appears the reply is being load balanced which can cause the reply to be sent out a different interface than the original connection.
  • This problem can be difficult to detect because the reply is set to the correct IP address, but if you look closely the MAC address is set to the interface it was actually sent out.
    • Since it’s going out the wrong interface, the packet simply disappears because it never reaches the next hop.
    • This may not show up consistently
    • If Vyos happened to load balance the reply out the same interface it came in, then it will work correctly. However, subsequent packets / connections may be subject to load balancing depending on the WAN Load Balancing rules.


Run the following commands by substituting the interface name and the mark value with your values for each WAN you want to enable sticky inbound connections:
sudo iptables -t mangle -N ISP_eth1_IN
sudo iptables -t mangle -A ISP_eth1_IN -j CONNMARK --set-xmark 0x1/0xffffffff
sudo iptables -t mangle -A PREROUTING -i eth1 -m state --state NEW -j ISP_eth1_IN


  • Conditions
    • These conditions may not all be required to reproduce the error, but this is what was enabled when the issue occurred.
    • This problem is occuring on a Vyos virtual machine with the following features enabled:
      • WAN Load Balancing with 3 WAN interfaces and 1 LAN interface
        • Each WAN connection has 5+ IP addresses
      • Source NAT
      • Destination NAT
  • This fix has been tested on Vyos Hydrogen (1.02)
  • This was previously a problem/bug in Vyatta 6.4

Full Explanation

  1. Create a new chain in the mangle table.
    1. Make sure the new chain name is unique.
    2. It's suggested to make it similar to the chain created by Vyos for the respective interface
    3. For example, if eth1 is a WAN then it would be formatted as follows:
      sudo iptables -t mangle -N ISP_eth1_IN
  2. Add a new rule into this new chain that will mark the connection with the same value as Vyos does for other connections on this interface
    1. You should check what value Vyos is setting for this by listing the contents of the chain created by Vyos for that interface
      1. For example, if the interface is eth1:
        sudo iptables –t mangle –L ISP_eth1
      2. On the line beginning with MARK, look for the hex value at the end:
    2. Once you have this value, run the following command to add a new rule to mark the inbound connections with this same value:
      sudo iptables -t mangle -A ISP_eth1_IN -j CONNMARK --set-xmark 0x1/0xffffffff
  3. Finally, add a new rule in the mangle table for the PREROUTING chain that says to jump to the new chain we created in step 1
    1. This line has two important values
      1. Actual interface name
      2. Chain name we created in step 1
    2. For example, if our interface is eth1 and our chain name is ISP_eth1_IN:
      sudo iptables -t mangle -A PREROUTING -i eth1 -m state --state NEW -j ISP_eth1_IN
  4. Repeat steps 1 – 3 for each WAN you have
    1. For example, if WAN2 is on eth2, the commands would be as follows:
      sudo iptables -t mangle -N ISP_eth2_IN
      sudo iptables -t mangle -A ISP_eth2_IN -j CONNMARK --set-xmark 0x2/0xffffffff
      sudo iptables -t mangle -A PREROUTING -i eth2 -m state --state NEW -j ISP_eth2_IN