WAN load balancing

From VyOS Wiki
Jump to: navigation, search

Outbound traffic can be balanced between two or more outbound interfaces. If a path fails, traffic is balanced across the remaining healthy paths, a recovered path is automatically added back to the routing table and used by the load balancer. The load balancer automatically adds routes for each path to the routing table and balances traffic across the configured interfaces, determined by interface health and weight.

Configuration

Handling and monitoring

Enable system-wide WAN load balancing:

# set load-balancing wan

Restart the WAN load balancer:

# restart wan-load-balance

Show WAN load balancer information including test types and targets. A character at the start of each line depicts the state of the test (+ = successful, - = failed, a blank indicates that no test has been carried out:

$ show wan-load-balance

Show connection data of load balanced traffic:

$ show wan-load-balance connection

Show status of WAN load balancing, DNS resolution can be specified with with-dns:

$ show wan-load-balance status [with-dns]

Interface weighting

Per default outbound traffic is distributed randomly across available interfaces. Weights can be assigned to interfaces to influence the balancing. Example with different weights on three interfaces and the resulting traffic percentage for each interface:

Interface Weight Traffic
eth0 2 50%
eth1 1 25%
eth2 1 25%

Configuring interface weights is described in the next part.

Balancing rules

Interfaces, their weight and the type of traffic to be balanced are defined in numbered balancing rule sets. The rule sets are executed in numerical order against outgoing packets. In case of a match the packet is sent through an interface specified in the matching rule. If a packet doesn't match any rule it is sent by using the system routing table. Rule numbers can't be changed.

Create a load balancing rule, rule can be a number between 1...4294967295:

# set load-balancing wan rule rule 

Add a description to a balancing rule:

# set load-balancing wan rule rule description desctiption

Set a destination address and port as a match criterium:

# set load-balancing wan rule rule destination address address | port port

Supported entries for address and port, port is only available for TCP and UDP protocol:

Address format Explanation
ip-address IPv4 address
ip-address/prefix IPv4 network address with prefix, 0.0.0.0/0 matches any network
ip-address-ip-address IPv4 address range
!ip-address Matches all IPv4 address, except the one specified
!ip-address/prefix Matches all IPv4 network addresses, except the one specified
!ip-address-ip-address Matches all IPv4 address, except the ones specified in the range
Port format Explanation
port-name A service name like http or SSH, specified in /etc/services
port-number Specifies a port number. Range: 1...65535
start-end Specifies a port number range.

Comma separated lists of port definitions are allowed, for example 25,http,513,8001-8005. Single port definitions and lists can be negated as well: !25,http,513,8001-8005


Set an inbound interface, packets arriving on the specified interface will be load balanced according to the matching rule:

# set load-balancing wan rule rule inbound-interface interface

Exclude traffic from load blalancing, traffic matching an exclude rule is not balanced but routed through the system routing table instead:

# set load-balancing wan rule rule exclude

A packet rate limit can be set for a rule to apply the rule to traffic above or below a specified threshold. To configure the rate limiting use:

# set load-balancing wan rule rule number limit parameter 

Available paramters:

burst: Number of packets allowed to overshoot the limit within period. Default 5.
period: Time window for rate calculation. Possible values: second (one second), minute (one minute), hour (one hour). Default is second.
rate: Number of packets. Default 5.
threshold: below or above the specified rate limit.


Flow and packet-based balancing

Outgoing traffic is balanced in a flow-based manner; a connection tracking table is used to track flows by their source address, destination address and port. Each flow is assingned to an interface according to the defined balancing rules and subsequent packets are sent through the same interface. This has the advantage that packets always arrive in order if links with different speeds are in use.

Packet-based balancing can lead to a better balance across interfaces when out of order packets are no issue. Per-packet-based balancing can be set for a balancing rule with:

# set load-balancing wan rule rule per-packet-balancing

Health checks

The health of interfaces and paths assingned to the load balancer is periodically checked by sending ICMP packets (ping) to remote destinations, a TTL test or the execution of a user defined script. If an interface fails the health check it is removed from the load balancer's pool of interfaces. To enable health checking for an interface:

# set load-balancing wan interface-health interface

Set the number of health check failures before an interface is marked as unavailable, range for number is 1...10, default 1:

# set load-balancing wan interface-health interface failure-count number

Set the number of successful health checks before an interface is added back to the interface pool, range for number is 1...10, default 1:

# set load-balancing wan interface-health interface success-count number

Specify nexthop on the path to destination, IPv4 address can be set to dhcp:

# set load-balancing wan interface-health interface nexthop IPv4 address'

Each health check is configured in its own test, tests are numbered and processed in numeric order. For multi target health checking multiple tests can be defined:

# set load-balancing wan interface-health interface test test number

Specify the type of test. type can be ping, ttl or a user defined script:

# set load-balancing wan interface-health interface test test number type type

Set the target to be sent ICMP packets to, address can be an IPv4 address or hostname:

# set load-balancing wan interface-health interface test test number target address

Maximum response time for ping in seconds. Range 1...30, default 5:

# set load-balancing wan interface-health interface test test number resp-time seconds

For the UDP TTL limit test the hop count limit must be specified. The limit must be shorter than the path length, an ICMP time expired message is needed to be returned for a successful test. Default 1:

# set load-balancing wan interface-health interface test test number ttl-limit limit

A user defined script must return 0 to be considered successful and non-zero to fail. Scripts are located in /config/scripts, for different locations the full path needs to be provided:

# set load-balancing wan interface-health interface test test number test-script script name


Source NAT rules

Per default, interfaces used in a load balancing pool replace the source IP of each outgoing packet with its own address to ensure that replies arrive on the same interface. This works through automatically generated source NAT (SNAT) rules, these rules are only applied to balanced traffic. In cases where this behaviour is not desired, the automatic generation of SNAT rules can be disabled:

# set load-balancing wan disable-source-nat

Failover

In failover mode, one interface is set to be the primary interface and other interfaces are secondary or spare. Instead of balancing traffic across all healthy interfaces, only the primary interface is used and in case of failure, a secondary interface selected from the pool of available interfaces takes over. The primary interface is selected based on its weight and health, others become secondary interfaces. Secondary interfaces to take over a failed primary interface are chosen from the load balancer's interface pool, depending on their weight and health. Interface roles can also be selected based on rule order by including interfaces in balancing rules and ordering those rules accordingly. To put the load balancer in failover mode, create a failover rule:

# set load-balancing wan rule rule name failover  

Because existing sessions do not automatically fail over to a new path, the session table can be flushed on each connection state change:

# set load-balancing wan flush-connections

Note that flushing the session table will cause other connections to fall back from flow-based to packet-based balancing until each flow is reestablished.

Script execution

A script can be run when an interface state change occurs. Scripts are run from /config/scripts, for a different location specify the full path:

# set load-balancing wan hook script-name

Two environment variables are available:

WLB_INTERFACE_NAME=[interfacename]: Interface to be monitored
WLB_INTERFACE_STATE=[ACTIVE|FAILED]: Interface state

Note: Blocking call with no timeout. System will become unresponsive if script does not return!


Example: Basic WAN load balancer

The following three steps are required to set up a basic WAN load balancer:

  • Configure each interface to have at least one target to determine interface/path health.
  • Set a nexthop address for each target.
  • Add static host route entries for each target.

The diagram shows the setup used in this example:

Wan load balancing1.png

Overview

  • All traffic coming in trough eth2 is balanced between eth0 and eth1 on the router.
  • Pings will be sent to four targets for health testing (33.44.55.66, 44.55.66.77, 55.66.77.88 and 66.77.88.99).
  • All outgoing packets are assigned the source address of the assigned interface (SNAT).
  • eth0 is set to be removed from the load balancer's interface pool after 5 ping failures, eth1 will be removed after 4 ping failures.

Create static routes to ping targets

Create static routes through the two ISPs towards the ping targets and commit the changes:

# set protocols static route 33.44.55.66/32 next-hop 11.22.33.1
# set protocols static route 44.55.66.77/32 next-hop 11.22.33.1 
# set protocols static route 55.66.77.88/32 next-hop 22.33.44.1
# set protocols static route 66.77.88.99/32 next-hop 22.33.44.1
# commit

Configure the load balancer

Configure the WAN load balancer with the parameters described above:

# set load-balancing wan interface-health eth0 failure-count 5
# set load-balancing wan interface-health eth0 nexthop 11.22.33.1
# set load-balancing wan interface-health eth0 test 10 type ping
# set load-balancing wan interface-health eth0 test 10 target 33.44.55.66
# set load-balancing wan interface-health eth0 test 20 type ping
# set load-balancing wan interface-health eth0 test 20 target 44.55.66.77
# set load-balancing wan interface-health eth1 failure-count 4
# set load-balancing wan interface-health eth1 nexthop 22.33.44.1
# set load-balancing wan interface-health eth1 test 10 type ping
# set load-balancing wan interface-health eth1 test 10 target 55.66.77.88
# set load-balancing wan interface-health eth1 test 20 type ping
# set load-balancing wan interface-health eth1 test 20 target 66.77.88.99
# set load-balancing wan rule 10 inbound-interface eth2
# set load-balancing wan rule 10 interface eth0
# set load-balancing wan rule 10 interface eth1
# commit

Example: Failover based on interface weights

The first example was balancing traffic evenly across eth0 and eth1, this example uses failover mode.

Overview

In this example eth0 is the primary interface and eth1 is the secondary interface to provide simple failover functionality. If eth0 fails, eth1 takes over.

Create interface weight based configuration

The configuration steps are the same as in the previous example, except rule 10 so we keep the configuration, remove rule 10 and add a new rule for the failover mode:

# delete load-balancing wan rule 10
# set load-balancing wan rule 10 failover
# set load-balancing wan rule 10 inbound-interface eth2
# set load-balancing wan rule 10 interface eth0 weight 10
# set load-balancing wan rule 10 interface eth1 weight 1
# commit 

Example: Failover based on rule order

The previous example used the failover command to send traffic thorugh eth1 if eth0 fails. In this example failover functionality is provided by rule order.

Overview

Two rules will be created, the first rule directs traffic coming in from eth2 to eth0 and the second rule directs the traffic to eth1. If eth0 fails the first rule is bypassed and the second rule matches, directing traffic to eth1.

Create rule order based configuration

We keep the configurtation from the previous example, delete rule 10 and create the two new rules as described:

# delete load-balancing wan rule 10
# set load-balancing wan rule 10 inbound-interface eth2
# set load-balancing wan rule 10 interface eth0
# set load-balancing wan rule 20 inbound-interface eth2
# set load-balancing wan rule 20 interface eth1
# commit

Example: Failover based on rule order - priority traffic

A rule order for prioritising traffic is useful in scenarios where the secondary link has a lower speed and should only carry high priority traffic. It is assumed for this example that eth1 is connected to a slower connection than eth0 and should prioritise VoIP traffic.

Overview

The configuration from the previous example will be kept, rule 20 will deleted and replaced with a new rule. This new rule will only allow VoIP traffic through eth1.

Create rule order based configuration with low speed secondary link

We keep the configuration from the previous example, delete rule 20 and create a new rule as described:

# delete load-balancing wan rule 20
# set load-balancing wan rule 20 inbound-interface eth2
# set load-balancing wan rule 20 interface eth1
# set load-balancing wan rule 20 destination port sip
# set load-balancing wan rule 20 protocol tcp
# set protocols static route 0.0.0.0/0 next-hop 11.22.33.1
# commit

Example: Exclude traffic from load balancing

In this example two LAN interfaces exist in different subnets instead of one like in the previous examples:

Wan load balancing exclude1.png

Adding a rule for the second interface

Based on the previous example, another rule for traffic from the second interface eth3 can be added to the load balancer. However, traffic meant to flow between the LAN subnets will be sent to eth0 and eth1 as well. To prevent this, another rule is required. This rule excludes traffic between the local subnets from the load balancer. It also excludes locally-sources packets (required for web caching with load balancing). eth+ is used as an alias that refers to all ethernet interfaces:

# set load-balancing wan rule 5 exclude
# set load-balancing wan rule 5 inbound-interface eth+
# set load-balancing wan rule 5 destination address 10.0.0.0/8
# commit