VXLAN

From VyOS Wiki
Jump to: navigation, search

VXLAN is an overlaying Ethernet over IP protocol.

RFC is here https://datatracker.ietf.org/doc/rfc7348/

If configuring VXLAN in a VyOS virtual machine, ensure that MAC spoofing (Hyper-V) or Forged Transmits (ESX) are permitted, otherwise forwarded frames may be blocked by the hypervisor.

How Multicast VXLAN works

Example Topology:

PC4 - Leaf2 - Spine1 - Leaf3 - PC5

PC4 has IP 10.0.0.4/24 and PC5 has IP 10.0.0.5/24, so they believe they are in the same broadcast domain.

Let's assume PC4 on Leaf2 wants to ping PC5 on Leaf3. Instead of setting Leaf3 as our remote end manually, Leaf2 encapsulates the packet into a UDP-packet and sends it to its designated multicast-address via Spine1. When Spine1 receives this packet it forwards it to all other Leafs who has joined the same multicast-group, in this case Leaf3. When Leaf3 receives the packet it forwards it, while at the same time learning that PC4 is reachable behind Leaf2, because the encapsulated packet had Leaf2's IP-address set as source IP.

PC5 receives the ping echo, responds with an echo reply that Leaf3 receives and this time forwards to Leaf2's unicast address directly because it learned the location of PC4 above. When Leaf2 receives the echo reply from PC5 it sees that it came from Leaf3 and so remembers that PC5 is reachable via Leaf3.

Thanks to this discovery, any subsequent traffic between PC4 and PC5 will not be using the multicast-address between the Leafs as they both know behind which Leaf the PCs are connected. This saves traffic as less multicast packets sent reduces the load on the network, which improves scalability when more Leafs are added.

For optimal scalability Multicast shouldn't be used at all, but instead use BGP to signal all connected devices between leafs. Unfortunately, VyOS does not yet support this.

Configuration commands

interfaces
  vxlan <vxlan[0-16777215]>
    address          # IP address of the VXLAN interface
    bridge-group     # Configure a L2 bridge-group
    description      # Description
    group <ipv4>     # IPv4 Multicast group address (required)
    ip               # IPv4 routing options
    ipv6             # IPv6 routing options
    link <dev>       # IP interface for underlay of this vxlan overlay (optional)
    mtu              # MTU
    policy           # Policy routing options
    remote           # Remote address of the VXLAN tunnel, used for PTP instead of multicast
    vni <1-16777215> # Virtual Network Identifier (required)

Configuration Example

The setup is this:

Leaf2 - Spine1 - Leaf3

Spine1 is a Cisco IOS router running version 15.4

Leaf2 and Leaf3 is each a VyOS router running 1.2.0 beta1

This topology was built using GNS3.

Topology:

Spine1: 
fa0/2 towards Leaf2, IP-address: 10.1.2.1/24
fa0/3 towards Leaf3, IP-address: 10.1.3.1/24

Leaf2:
Eth0 towards Spine1, IP-address: 10.1.2.2/24
Eth1 towards a vlan-aware switch

Leaf3:
Eth0 towards Spine1, IP-address 10.1.3.3/24
Eth1 towards a vlan-aware switch

Spine1 Configuration:

conf t
ip multicast-routing
!
interface fastethernet0/2
 ip address 10.1.2.1 255.255.255.0
 ip pim sparse-dense-mode
!
interface fastethernet0/3
 ip address 10.1.3.1 255.255.255.0
 ip pim sparse-dense-mode
!
router ospf 1
 network 10.0.0.0 0.255.255.255 area 0

Multicast-routing is required for the leafs to forward traffic between each other in a more scalable way. This also requires PIM to be enabled towards the Leafs so that the Spine can learn what multicast groups each Leaf expect traffic from.

Leaf2 configuration:

set interfaces ethernet eth0 address '10.1.2.2/24'
set protocols ospf area 0 network '10.0.0.0/8'

! Our first vxlan interface
set interfaces bridge br241 address '172.16.241.1/24'
set interfaces ethernet eth1 vif 241 bridge-group bridge 'br241'
set interfaces vxlan vxlan241 bridge-group bridge 'br241'
set interfaces vxlan vxlan241 group '239.0.0.241'
set interfaces vxlan vxlan241 link 'eth0'
set interfaces vxlan vxlan241 vni '241'

! Our seconds vxlan interface
set interfaces bridge br242 address '172.16.242.1/24'
set interfaces ethernet eth1 vif 242 bridge-group bridge 'br242'
set interfaces vxlan vxlan242 bridge-group bridge 'br242'
set interfaces vxlan vxlan242 group '239.0.0.242'
set interfaces vxlan vxlan242 link 'eth0'
set interfaces vxlan vxlan242 vni '242'

Leaf3 configuration:

set interfaces ethernet eth0 address '10.1.3.3/24'
set protocols ospf area 0 network '10.0.0.0/8'

! Our first vxlan interface
set interfaces bridge br241 address '172.16.241.1/24'
set interfaces ethernet eth1 vif 241 bridge-group bridge 'br241'
set interfaces vxlan vxlan241 bridge-group bridge 'br241'
set interfaces vxlan vxlan241 group '239.0.0.241'
set interfaces vxlan vxlan241 link 'eth0'
set interfaces vxlan vxlan241 vni '241'

! Our seconds vxlan interface
set interfaces bridge br242 address '172.16.242.1/24'
set interfaces ethernet eth1 vif 242 bridge-group bridge 'br242'
set interfaces vxlan vxlan242 bridge-group bridge 'br242'
set interfaces vxlan vxlan242 group '239.0.0.242'
set interfaces vxlan vxlan242 link 'eth0'
set interfaces vxlan vxlan242 vni '242'

As you can see, Leaf2 and Leaf3 configuration is almost identical.

There are lots of commands above, I'll try to into more detail below, command descriptions are placed under the command boxes:

set interfaces bridge br241 address '172.16.241.1/24'

This commands creates a bridge that is used to bind traffic on eth1 vlan 241 with the vxlan241-interface. The IP-address is not required. It may however be used as a default gateway for each Leaf which allows devices on the vlan to reach other subnets. This requires that the subnets are redistributed by OSPF so that the Spine will learn how to reach it. To do this you need to change the OSPF network from '10.0.0.0/8' to '0.0.0.0/0' to allow 172.16/12-networks to be advertised.

set interfaces ethernet eth1 vif 241 bridge-group bridge 'br241'
set interfaces vxlan vxlan241 bridge-group bridge 'br241'

Binds eth1 vif 241 and vxlan241 to each other by putting them in the same bridge-group. Internal VyOS requirement.

set interfaces vxlan vxlan241 group '239.0.0.241'

The multicast-group used by all Leafs for this vlan extension. Has to be the same on all Leafs that has this interface.

set interfaces vxlan vxlan241 link 'eth0'

Sets the interface to listen for multicast packets on. Could be a loopback, not yet tested.

set interfaces vxlan vxlan241 vni '241'

Sets the unique id for this vxlan-interface. Not sure how it correlates with multicast-address.

Older Examples

Example for bridging normal L2 segment and vxlan overlay network, and using a vxlan interface as routing interface.

interfaces {
     bridge br0 {
     }
     ethernet eth0 {
         address dhcp
     }
     loopback lo {
     }
     vxlan vxlan0 {
         bridge-group {
             bridge br0
         }
         group 239.0.0.1
         vni 0
     }
     vxlan vxlan1 {
         address 192.168.0.1/24
         link eth0
         group 239.0.0.1
         vni 1
     }
 }

Here is a working configuration that creates a VXLAN between two routers. Each router has a VLAN interface (26) facing the client devices and a VLAN interface (30) that connects it to the other routers. With this configuration, traffic can flow between both routers' VLAN 26, but can't escape since there is no L3 gateway. You can add an IP to a bridge-group to create a gateway.

 interfaces {
     bridge br0 {
     }
     ethernet eth0 {
         duplex auto
         smp-affinity auto
         speed auto
         vif 26 {
             bridge-group {
                 bridge br0
             }
         }
         vif 30 {
             address 10.7.50.6/24
         }
     }
     loopback lo {
     }
     vxlan vxlan0 {
         bridge-group {
             bridge br0
         }
         group 239.0.0.241
         vni 241
     }
 }