WAN load balancing
Outbound traffic can be balanced between two or more outbound interfaces. If a path fails, traffic is balanced across the remaining healthy paths, a recovered path is automatically added back to the routing table and used by the load balancer. The load balancer automatically adds routes for each path to the routing table and balances traffic across the configured interfaces, determined by interface health and weight.
- 1 Configuration
- 2 Example: Basic WAN load balancer
- 3 Example: Failover based on interface weights
- 4 Example: Failover based on rule order
- 5 Example: Failover based on rule order - priority traffic
- 6 Example: Exclude traffic from load balancing
Handling and monitoring
Enable system-wide WAN load balancing:
# set load-balancing wan
Restart the WAN load balancer:
# restart wan-load-balance
Show WAN load balancer information including test types and targets. A character at the start of each line depicts the state of the test (+ = successful, - = failed, a blank indicates that no test has been carried out:
$ show wan-load-balance
Show connection data of load balanced traffic:
$ show wan-load-balance connection
Show status of WAN load balancing, DNS resolution can be specified with with-dns:
$ show wan-load-balance status [with-dns]
Per default outbound traffic is distributed randomly across available interfaces. Weights can be assigned to interfaces to influence the balancing. Example with different weights on three interfaces and the resulting traffic percentage for each interface:
Configuring interface weights is described in the next part.
Interfaces, their weight and the type of traffic to be balanced are defined in numbered balancing rule sets. The rule sets are executed in numerical order against outgoing packets. In case of a match the packet is sent through an interface specified in the matching rule. If a packet doesn't match any rule it is sent by using the system routing table. Rule numbers can't be changed.
Create a load balancing rule, rule can be a number between 1...4294967295:
# set load-balancing wan rule rule
Add a description to a balancing rule:
# set load-balancing wan rule rule description desctiption
Set a destination address and port as a match criterium:
# set load-balancing wan rule rule destination address address | port port
Supported entries for address and port, port is only available for TCP and UDP protocol:
|ip-address/prefix||IPv4 network address with prefix, 0.0.0.0/0 matches any network|
|ip-address-ip-address||IPv4 address range|
|!ip-address||Matches all IPv4 address, except the one specified|
|!ip-address/prefix||Matches all IPv4 network addresses, except the one specified|
|!ip-address-ip-address||Matches all IPv4 address, except the ones specified in the range|
|port-name||A service name like http or SSH, specified in /etc/services|
|port-number||Specifies a port number. Range: 1...65535|
|start-end||Specifies a port number range.|
Comma separated lists of port definitions are allowed, for example 25,http,513,8001-8005. Single port definitions and lists can be negated as well: !25,http,513,8001-8005
Set an inbound interface, packets arriving on the specified interface will be load balanced according to the matching rule:
# set load-balancing wan rule rule inbound-interface interface
Exclude traffic from load blalancing, traffic matching an exclude rule is not balanced but routed through the system routing table instead:
# set load-balancing wan rule rule exclude
A packet rate limit can be set for a rule to apply the rule to traffic above or below a specified threshold. To configure the rate limiting use:
# set load-balancing wan rule rule number limit parameter
- burst: Number of packets allowed to overshoot the limit within period. Default 5.
- period: Time window for rate calculation. Possible values: second (one second), minute (one minute), hour (one hour). Default is second.
- rate: Number of packets. Default 5.
- threshold: below or above the specified rate limit.
Flow and packet-based balancing
Outgoing traffic is balanced in a flow-based manner; a connection tracking table is used to track flows by their source address, destination address and port. Each flow is assingned to an interface according to the defined balancing rules and subsequent packets are sent through the same interface. This has the advantage that packets always arrive in order if links with different speeds are in use.
Packet-based balancing can lead to a better balance across interfaces when out of order packets are no issue. Per-packet-based balancing can be set for a balancing rule with:
# set load-balancing wan rule rule per-packet-balancing
The health of interfaces and paths assingned to the load balancer is periodically checked by sending ICMP packets (ping) to remote destinations, a TTL test or the execution of a user defined script. If an interface fails the health check it is removed from the load balancer's pool of interfaces. To enable health checking for an interface:
# set load-balancing wan interface-health interface
Set the number of health check failures before an interface is marked as unavailable, range for number is 1...10, default 1:
# set load-balancing wan interface-health interface failure-count number
Set the number of successful health checks before an interface is added back to the interface pool, range for number is 1...10, default 1:
# set load-balancing wan interface-health interface success-count number
Specify nexthop on the path to destination, IPv4 address can be set to dhcp:
# set load-balancing wan interface-health interface nexthop IPv4 address'
Each health check is configured in its own test, tests are numbered and processed in numeric order. For multi target health checking multiple tests can be defined:
# set load-balancing wan interface-health interface test test number
Specify the type of test. type can be ping, ttl or a user defined script:
# set load-balancing wan interface-health interface test test number type type
Set the target to be sent ICMP packets to, address can be an IPv4 address or hostname:
# set load-balancing wan interface-health interface test test number target address
Maximum response time for ping in seconds. Range 1...30, default 5:
# set load-balancing wan interface-health interface test test number resp-time seconds
For the UDP TTL limit test the hop count limit must be specified. The limit must be shorter than the path length, an ICMP time expired message is needed to be returned for a successful test. Default 1:
# set load-balancing wan interface-health interface test test number ttl-limit limit
A user defined script must return 0 to be considered successful and non-zero to fail. Scripts are located in /config/scripts, for different locations the full path needs to be provided:
# set load-balancing wan interface-health interface test test number test-script script name
Source NAT rules
Per default, interfaces used in a load balancing pool replace the source IP of each outgoing packet with its own address to ensure that replies arrive on the same interface. This works through automatically generated source NAT (SNAT) rules, these rules are only applied to balanced traffic. In cases where this behaviour is not desired, the automatic generation of SNAT rules can be disabled:
# set load-balancing wan disable-source-nat
Upon reception of an incoming packet, when a response is sent, it might be desired to ensure that it leaves from the same interface as the inbound one. This can be achieved by enabling sticky connections in the load balancing:
# set load-balancing wan sticky-connections inbound
In failover mode, one interface is set to be the primary interface and other interfaces are secondary or spare. Instead of balancing traffic across all healthy interfaces, only the primary interface is used and in case of failure, a secondary interface selected from the pool of available interfaces takes over. The primary interface is selected based on its weight and health, others become secondary interfaces. Secondary interfaces to take over a failed primary interface are chosen from the load balancer's interface pool, depending on their weight and health. Interface roles can also be selected based on rule order by including interfaces in balancing rules and ordering those rules accordingly. To put the load balancer in failover mode, create a failover rule:
# set load-balancing wan rule rule name failover
Because existing sessions do not automatically fail over to a new path, the session table can be flushed on each connection state change:
# set load-balancing wan flush-connections
Note that flushing the session table will cause other connections to fall back from flow-based to packet-based balancing until each flow is reestablished.
A script can be run when an interface state change occurs. Scripts are run from /config/scripts, for a different location specify the full path:
# set load-balancing wan hook script-name
Two environment variables are available:
- WLB_INTERFACE_NAME=[interfacename]: Interface to be monitored
- WLB_INTERFACE_STATE=[ACTIVE|FAILED]: Interface state
Note: Blocking call with no timeout. System will become unresponsive if script does not return!
Example: Basic WAN load balancer
The following three steps are required to set up a basic WAN load balancer:
- Configure each interface to have at least one target to determine interface/path health.
- Set a nexthop address for each target.
- Add static host route entries for each target.
The diagram shows the setup used in this example:
- All traffic coming in trough eth2 is balanced between eth0 and eth1 on the router.
- Pings will be sent to four targets for health testing (22.214.171.124, 126.96.36.199, 188.8.131.52 and 184.108.40.206).
- All outgoing packets are assigned the source address of the assigned interface (SNAT).
- eth0 is set to be removed from the load balancer's interface pool after 5 ping failures, eth1 will be removed after 4 ping failures.
Create static routes to ping targets
Create static routes through the two ISPs towards the ping targets and commit the changes:
# set protocols static route 220.127.116.11/32 next-hop 18.104.22.168 # set protocols static route 22.214.171.124/32 next-hop 126.96.36.199 # set protocols static route 188.8.131.52/32 next-hop 184.108.40.206 # set protocols static route 220.127.116.11/32 next-hop 18.104.22.168 # commit
Configure the load balancer
Configure the WAN load balancer with the parameters described above:
# set load-balancing wan interface-health eth0 failure-count 5 # set load-balancing wan interface-health eth0 nexthop 22.214.171.124 # set load-balancing wan interface-health eth0 test 10 type ping # set load-balancing wan interface-health eth0 test 10 target 126.96.36.199 # set load-balancing wan interface-health eth0 test 20 type ping # set load-balancing wan interface-health eth0 test 20 target 188.8.131.52 # set load-balancing wan interface-health eth1 failure-count 4 # set load-balancing wan interface-health eth1 nexthop 184.108.40.206 # set load-balancing wan interface-health eth1 test 10 type ping # set load-balancing wan interface-health eth1 test 10 target 220.127.116.11 # set load-balancing wan interface-health eth1 test 20 type ping # set load-balancing wan interface-health eth1 test 20 target 18.104.22.168 # set load-balancing wan rule 10 inbound-interface eth2 # set load-balancing wan rule 10 interface eth0 # set load-balancing wan rule 10 interface eth1 # commit
Example: Failover based on interface weights
The first example was balancing traffic evenly across eth0 and eth1, this example uses failover mode.
In this example eth0 is the primary interface and eth1 is the secondary interface to provide simple failover functionality. If eth0 fails, eth1 takes over.
Create interface weight based configuration
The configuration steps are the same as in the previous example, except rule 10 so we keep the configuration, remove rule 10 and add a new rule for the failover mode:
# delete load-balancing wan rule 10 # set load-balancing wan rule 10 failover # set load-balancing wan rule 10 inbound-interface eth2 # set load-balancing wan rule 10 interface eth0 weight 10 # set load-balancing wan rule 10 interface eth1 weight 1 # commit
Example: Failover based on rule order
The previous example used the failover command to send traffic thorugh eth1 if eth0 fails. In this example failover functionality is provided by rule order.
Two rules will be created, the first rule directs traffic coming in from eth2 to eth0 and the second rule directs the traffic to eth1. If eth0 fails the first rule is bypassed and the second rule matches, directing traffic to eth1.
Create rule order based configuration
We keep the configurtation from the previous example, delete rule 10 and create the two new rules as described:
# delete load-balancing wan rule 10 # set load-balancing wan rule 10 inbound-interface eth2 # set load-balancing wan rule 10 interface eth0 # set load-balancing wan rule 20 inbound-interface eth2 # set load-balancing wan rule 20 interface eth1 # commit
Example: Failover based on rule order - priority traffic
A rule order for prioritising traffic is useful in scenarios where the secondary link has a lower speed and should only carry high priority traffic. It is assumed for this example that eth1 is connected to a slower connection than eth0 and should prioritise VoIP traffic.
The configuration from the previous example will be kept, rule 20 will deleted and replaced with a new rule. This new rule will only allow VoIP traffic through eth1.
We keep the configuration from the previous example, delete rule 20 and create a new rule as described:
# delete load-balancing wan rule 20 # set load-balancing wan rule 20 inbound-interface eth2 # set load-balancing wan rule 20 interface eth1 # set load-balancing wan rule 20 destination port sip # set load-balancing wan rule 20 protocol tcp # set protocols static route 0.0.0.0/0 next-hop 22.214.171.124 # commit
Example: Exclude traffic from load balancing
In this example two LAN interfaces exist in different subnets instead of one like in the previous examples:
Adding a rule for the second interface
Based on the previous example, another rule for traffic from the second interface eth3 can be added to the load balancer. However, traffic meant to flow between the LAN subnets will be sent to eth0 and eth1 as well. To prevent this, another rule is required. This rule excludes traffic between the local subnets from the load balancer. It also excludes locally-sources packets (required for web caching with load balancing). eth+ is used as an alias that refers to all ethernet interfaces:
# set load-balancing wan rule 5 exclude # set load-balancing wan rule 5 inbound-interface eth+ # set load-balancing wan rule 5 destination address 10.0.0.0/8 # commit