A tip of the hat to the marvellously clever Jonas Werner over at https://jonamiki.com/posts/layer-2-extension-with-vyos/ who gave me some tips for the start of this journey.
We recently had a customer who needed to stretch several Layer 2 Subnets up to Microsoft Azure for a Nutanix NC2 deployment – but coming from a non Nutanix on-premises infrastructure we had to bring our own VXLAN solution to plug into the Nutanix Gateway.
Standing up a single VyOS appliance in the on-premises ESXi environment was straight forward enough once you consider the needs for promiscuous mode, and keeping an eye on duplicate packet problems but the customer needed to cater for HA.
Along comes VRRP as a way of allowing us to have two independently deployed VyOS appliances each with their own unique mac and IP addresses but to share a bridge mac, and IP addresses for the connection to Nutanix NC2 on Azure.
This allowed us to have the tunnel established using VyOS-1, and then upon reboot it would automatically come up on VyOS-2 with the same IP addresses to allow traffic to continue to pass. When VyOS-1 returned, VRRP would fail back over to it.
vyos-01 (Primary)
set system host-name ‘vyos-01’
set interfaces ethernet eth0 address ‘10.25.0.6/24’
set protocols static route 0.0.0.0/0 next-hop 10.25.0.254
set service ssh
set interfaces vxlan vxlan1 mtu ‘1306’
set interfaces vxlan vxlan1 port ‘4789’
set interfaces vxlan vxlan1 vni ’82’
set interfaces bridge br1 address ‘10.25.1.6/24’
set interfaces bridge br1 aging ’10’
set interfaces bridge br1 ip arp-cache-timeout 10
set interfaces bridge br1 mac ’00:50:56:b6:05:bd’ // We set this to a mac address that can be maintained between both VyOS appliances to reduce ARP delays
set interfaces bridge br1 member interface eth1
set interfaces bridge br1 member interface vxlan1
set interfaces bridge br1 mtu ‘1306’
set high-availability vrrp group vrrp-eth0 interface ‘eth0’
set high-availability vrrp group vrrp-eth0 vrid 100
set high-availability vrrp group vrrp-eth0 address 10.25.0.10/24
set high-availability vrrp group vrrp-eth0 priority ‘150’
set high-availability vrrp group vrrp-eth0 preempt-delay ‘1’
set high-availability vrrp group vrrp-eth0 advertise-interval ‘1’
set high-availability vrrp group vrrp-vxlan1 interface ‘br1’
set high-availability vrrp group vrrp-vxlan1 vrid 1
set high-availability vrrp group vrrp-vxlan1 address 10.25.1.8/24
set high-availability vrrp group vrrp-vxlan1 priority ‘150’
set high-availability vrrp group vrrp-vxlan1 preempt-delay ‘1’
set high-availability vrrp group vrrp-vxlan1 advertise-interval ‘1’
set high-availability vrrp global-parameters garp interval 0.000
set high-availability vrrp global-parameters garp master-delay 1
set interfaces vxlan vxlan1 source-address ‘10.25.0.10’ // This is the address configured on the Nutanix Gateway side so it is always on the ‘active’ VyOS appliance
set interfaces vxlan vxlan1 remote ‘10.90.2.32’ // This is the address of the Nutanix Gateway
vyos-02 (Backup)
set system host-name ‘vyos-02’
set interfaces ethernet eth0 address ‘10.25.0.7/24’
set protocols static route 0.0.0.0/0 next-hop 10.25.0.254
set service ssh
set interfaces vxlan vxlan1 mtu ‘1306’
set interfaces vxlan vxlan1 port ‘4789’
set interfaces vxlan vxlan1 vni ’82’
set interfaces bridge br1 address ‘10.25.1.7/24’
set interfaces bridge br1 aging ’10’
set interfaces bridge br1 ip arp-cache-timeout 10
set interfaces bridge br1 mac ’00:50:56:b6:05:bd’ // We set this to a mac address that can be maintained between both VyOS appliances to reduce ARP delays
set interfaces bridge br1 member interface eth1
set interfaces bridge br1 member interface vxlan1
set interfaces bridge br1 mtu ‘1306’
set high-availability vrrp group vrrp-eth0 interface ‘eth0’
set high-availability vrrp group vrrp-eth0 vrid 100
set high-availability vrrp group vrrp-eth0 address 10.25.0.10/24
set high-availability vrrp group vrrp-eth0 priority ’50’
set high-availability vrrp group vrrp-eth0 preempt-delay ‘5’
set high-availability vrrp group vrrp-eth0 advertise-interval ‘1’
set high-availability vrrp group vrrp-vxlan1 interface ‘br1’
set high-availability vrrp group vrrp-vxlan1 vrid 1
set high-availability vrrp group vrrp-vxlan1 address 10.25.1.8/24
set high-availability vrrp group vrrp-vxlan1 priority ’50’
set high-availability vrrp group vrrp-vxlan1 preempt-delay ‘5’
set high-availability vrrp group vrrp-vxlan1 advertise-interval ‘1’
set high-availability vrrp global-parameters garp interval 0.000
set high-availability vrrp global-parameters garp master-delay 1
set interfaces vxlan vxlan1 source-address ‘10.25.0.10’ // This is the address configured on the Nutanix Gateway side so it is always on the ‘active’ VyOS appliance
set interfaces vxlan vxlan1 remote ‘10.90.2.32’ // This is the address of the Nutanix Gateway
We can add multiple VLAN’s to be stretched by adding an additional virtual nic to the VyOS appliances e.g. eth2 and using br2 together, then eth3 and br3, and so on.
In this deployment we plan to deploy multiple VyOS pairs to reduce the blast RADIUS should a failure occur it only impacts a subset of the stretched networks.