Setting up a Site-to-Site VPN between AWS and on-premises VyOS (IPSec & BGP)

VyOS is an open-source operating system based on Linux that provides software-based networking. There’s an official guide for setting up a Site-to-Site VPN between an on-premises VyOS router and AWS, but it is fairly outdated, as it was written for VyOS 1.3, not for the current LTS: VyOS 1.4 (sagitta). AWS also has the option to download a sample configuration for Vyatta (the project that VyOS was forked from), which is helpful, but also outdated. I’m publishing my notes in case it’s helpful to anyone else.

AWS: Create CGW, VGW/TGW and VPN

This post assumes that you understand IPSec, have some experience in AWS and have already created the CGW, VGW/TGW and VPN. I assume you’re here because the tunnel status says DOWN and now you’re sad because you’re not sure what magical spells to cast towards the VyOS command prompt to make it say UP.

When initially setting op the VPN in AWS, you’ll be asked to choose the type of routing that’s going to be used. The options are fairly self-explanatory:

• Static requires both sides to configure static routes for traffic they wish to route through the IPSec tunnel.
• Dynamic requires establishing a BGP session between both sides through the IPSec tunnel. Since VyOS supports BGP, we’ll be using this option.

In this example, we’ll terminate the VPN connection into a VGW on the AWS side. The VGW is associated with a VPC with CIDR range 10.137.0.0/16 and the route tables of the VPC are configured to have the VGW propagate routes it has discovered into them. You can also use a TGW if you have a more advanced use case.

VyOS: IPSec tunnel

First, let’s collect some important info from the vpn-01 Site-to-Site VPN I created for test purposes in AWS, along with some info about the on-premises side¹.

We’ll start by configuring the encryption parameters. Download the “Generic” example configuration for your VPN from AWS. This will contain the pre-shared keys for the tunnels. Based on NCSC’s recommendations and the example configuration from AWS, I arrived at the following settings for VyOS:

set vpn ipsec esp-group aws-s2s lifetime '3600'
set vpn ipsec esp-group aws-s2s pfs 'enable'
set vpn ipsec esp-group aws-s2s proposal 1 encryption 'aes128gcm128'

set vpn ipsec ike-group aws-s2s dead-peer-detection action 'restart'
set vpn ipsec ike-group aws-s2s dead-peer-detection interval '10'
set vpn ipsec ike-group aws-s2s dead-peer-detection timeout '30'
set vpn ipsec ike-group aws-s2s key-exchange 'ikev2'
set vpn ipsec ike-group aws-s2s lifetime '28800'
set vpn ipsec ike-group aws-s2s proposal 1 dh-group '19'
set vpn ipsec ike-group aws-s2s proposal 1 encryption 'aes128gcm128'
set vpn ipsec ike-group aws-s2s proposal 1 prf 'prfsha256'

# NOTE: Replace the values below with the tunnel 1/2 outside IP's and corresponding pre-shared secrets
set vpn ipsec authentication psk aws-s2s-tunnel1 id '34.193.192.160'
set vpn ipsec authentication psk aws-s2s-tunnel1 secret 'wDRS31kXtW8Rd0o9QJSAKje76WdJyM6.'
set vpn ipsec authentication psk aws-s2s-tunnel2 id '54.92.230.59'
set vpn ipsec authentication psk aws-s2s-tunnel2 secret '20xllijzqZuLy6UogMVAS5VCVjcPwHP0'

Next, we’ll set up our VTI interfaces. AWS assigns a separate /30 CIDR block in the 169.254.0.0/16 range for use inside each tunnel. The first usable address in these networks is the AWS side and the second address is the on-premises side. So in this case that’d be:

Taking the overhead of the IPSec header into account, and assuming an MTU of 1500 on the interface IPSec is bound to, the MTU for the VTI should be 1436. This is all highly dependent on what your specific setup looks like, of course.

Setting up the VTI interfaces:

# NOTE: Replace the addresses
set interfaces vti vti0 address '169.254.18.94/30'
set interfaces vti vti0 mtu '1436'
set interfaces vti vti0 ip adjust-mss 1396
set interfaces vti vti1 address '169.254.107.246/30'
set interfaces vti vti1 mtu '1436'
set interfaces vti vti1 ip adjust-mss 1396

Tunnels:

set vpn ipsec interface 'pppoe0'
set vpn ipsec options disable-route-autoinstall

# NOTE: Replace the remote-id, local-address and remote-address
set vpn ipsec site-to-site peer aws-s2s-tunnel1 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 authentication remote-id '34.193.192.160'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 connection-type 'initiate'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 ike-group 'aws-s2s'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 local-address '1.2.3.4'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 remote-address '34.193.192.160'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 vti bind 'vti0'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 vti esp-group 'aws-s2s'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 authentication remote-id '54.92.230.59'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 connection-type 'initiate'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 ike-group 'aws-s2s'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 local-address '1.2.3.4'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 remote-address '54.92.230.59'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 vti bind 'vti1'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 vti esp-group 'aws-s2s'

I’m configuring IPSec to use the Point-to-Point interface pppoe0 directly, as it has a public IPv4 address assigned by my ISP configured on it. Your interface will probably have a different name.

Next, we’ll update the firewall rules to allow IPSec traffic from AWS. Replace WAN_LOCAL with the name of your WAN ruleset:

# NOTE: Replace the addresses with the outside IP's of your Site-to-Site VPN in AWS
set firewall group address-group aws-s2s-vpn address '34.193.192.160'
set firewall group address-group aws-s2s-vpn address '54.92.230.59'

set firewall ipv4 name WAN_LOCAL rule 30 action 'accept'
set firewall ipv4 name WAN_LOCAL rule 30 description 'AWS Site-to-Site VPN'
set firewall ipv4 name WAN_LOCAL rule 30 destination port '500,4500'
set firewall ipv4 name WAN_LOCAL rule 30 protocol 'udp'
set firewall ipv4 name WAN_LOCAL rule 30 source group address-group 'aws-s2s-vpn'

Now we can commit the changes, wait for a few moments and check that the IPSec tunnel is now up:

$ show vpn ipsec connections
Connection           State    Type    Remote address    Local TS    Remote TS    Local id    Remote id       Proposal
-------------------  -------  ------  ----------------  ----------  -----------  ----------  --------------  ------------------------
aws-s2s-tunnel1      up       IKEv2   34.193.192.160    -           -                        34.193.192.160  AES_GCM/128/None/ECP_256
aws-s2s-tunnel1-vti  up       IPsec   34.193.192.160    0.0.0.0/0   0.0.0.0/0                34.193.192.160  AES_GCM/128/None/None
                                                        ::/0        ::/0
aws-s2s-tunnel2      up       IKEv2   54.92.230.59      -           -                        54.92.230.59    AES_GCM/128/None/ECP_256
aws-s2s-tunnel2-vti  up       IPsec   54.92.230.59      0.0.0.0/0   0.0.0.0/0                54.92.230.59    AES_GCM/128/None/None
                                                        ::/0        ::/0

The AWS console still reports that the tunnel is DOWN, so we’re still a little sad, but the IPSec tunnel is now up:

If you’ve opted to use “Static” routing when creating the VPN in AWS, the status will say UP and you’ll only have to configure some static routes to start using the VPN. BGP is more fun though, so let’s set up dynamic routing.

VyOS: BGP

Finally, we’ll set up BGP to dynamically exchange routes between AWS and on-premises.

# NOTE: Replace ASN as configured on the customer gateway in AWS
set protocols bgp system-as '65111'
set protocols bgp parameters router-id '192.168.0.1'

# NOTE: Replace ASN as configured on the VGW/TGW in AWS
set protocols bgp peer-group aws-s2s-vpn address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp peer-group aws-s2s-vpn remote-as '64512'

# NOTE: Replace the neighbour address with the AWS-side tunnel inside address
# and replace the source address with the on-premises tunnel inside address
set protocols bgp neighbor 169.254.18.93 description 'BGP - AWS tunnel 1'
set protocols bgp neighbor 169.254.18.93 peer-group 'aws-s2s-vpn'
set protocols bgp neighbor 169.254.18.93 timers holdtime '30'
set protocols bgp neighbor 169.254.18.93 timers keepalive '10'
set protocols bgp neighbor 169.254.18.93 update-source '169.254.18.94'
set protocols bgp neighbor 169.254.107.245 description 'BGP - AWS tunnel 2'
set protocols bgp neighbor 169.254.107.245 peer-group 'aws-s2s-vpn'
set protocols bgp neighbor 169.254.107.245 timers holdtime '30'
set protocols bgp neighbor 169.254.107.245 timers keepalive '10'
set protocols bgp neighbor 169.254.107.245 update-source '169.254.107.246'

There are a few ways to configure which routes to advertise. A common one is to include all locally known routes:

set protocols bgp address-family ipv4-unicast redistribute connected

In my case, I only want to advertise a specific list of routes:

set protocols bgp address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp address-family ipv4-unicast network 192.168.200.0/24

Update firewall rules to allow BGP traffic from VTI interfaces:

set firewall ipv4 input filter rule 20 action 'jump'
set firewall ipv4 input filter rule 20 inbound-interface name 'vti0'
set firewall ipv4 input filter rule 20 jump-target 'AWS_S2S_LOCAL'

set firewall ipv4 input filter rule 21 action 'jump'
set firewall ipv4 input filter rule 21 inbound-interface name 'vti1'
set firewall ipv4 input filter rule 21 jump-target 'AWS_S2S_LOCAL'

set firewall ipv4 name AWS_S2S_LOCAL default-action 'drop'
set firewall ipv4 name AWS_S2S_LOCAL default-log

set firewall ipv4 name AWS_S2S_LOCAL rule 10 action 'accept'
set firewall ipv4 name AWS_S2S_LOCAL rule 10 state 'established'
set firewall ipv4 name AWS_S2S_LOCAL rule 10 state 'related'

set firewall ipv4 name AWS_S2S_LOCAL rule 20 action 'accept'
set firewall ipv4 name AWS_S2S_LOCAL rule 20 description 'BGP'
set firewall ipv4 name AWS_S2S_LOCAL rule 20 destination port '179'
set firewall ipv4 name AWS_S2S_LOCAL rule 20 protocol 'tcp'

Now we can commit all configuration changes again, wait for a few moments, and check the BGP status. We can see that we’ve now established a BGP session over the two tunnels:

$ show bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.0.1, local AS number 65111 vrf-id 0
BGP table version 9
RIB entries 5, using 480 bytes of memory
Peers 2, using 40 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
169.254.18.93   4      64512        48        51        9    0    0 00:07:11            1        3 BGP - AWS tunnel 1
169.254.107.245 4      64512        48        51        9    0    0 00:07:11            1        3 BGP - AWS tunnel 2

Total number of neighbors 2

Looking at the BGP route table, we see our own advertised routes and a learned route with two paths that have a different MED for each path:

$ show bgp ipv4
BGP table version is 9, local router ID is 192.168.0.1, vrf id 0
Default local pref 100, local AS 65111
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *  10.137.0.0/16    169.254.18.93          200             0 64512 i
 *>                  169.254.107.245        100             0 64512 i
 *> 192.168.0.0/24   0.0.0.0                  0         32768 i
 *> 192.168.200.0/24 0.0.0.0                  0         32768 i

If you attached the VPN to a TGW instead and have ECMP enabled, you’ll notice that the MED is equal for the two paths, and that multipath is enabled:

$ show bgp ipv4
BGP table version is 19, local router ID is 192.168.0.1, vrf id 0
Default local pref 100, local AS 65111
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *= 10.137.0.0/16    169.254.18.93          100             0 64512 i
 *>                  169.254.107.245        100             0 64512 i
 *> 192.168.0.0/24   0.0.0.0                  0         32768 i
 *> 192.168.200.0/24 0.0.0.0                  0         32768 i

And the Linux route table now includes the newly learned route from AWS:

$ ip r
***
10.137.0.0/16 nhid 232 via 169.254.107.245 dev vti1 proto bgp metric 20
***

If you attached the VPN to a TGW instead and have ECMP enabled, you’ll notice that the route has two paths with equal weights:

$ ip r
***
10.137.0.0/16 nhid 249 proto bgp metric 20
	nexthop via 169.254.107.245 dev vti1 weight 1
	nexthop via 169.254.18.93 dev vti0 weight 1
***

The AWS console now also reports that both tunnels are UP and that it has learned two routes. AWS looks happy, so we’re happy:

The VGW has also propagated the routes into the VPC route tables:

We can also ping back and forth between machines in the AWS VPC and the on-premises network:

$ ping 192.168.0.3
PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
64 bytes from 192.168.0.3: icmp_seq=1 ttl=63 time=96.8 ms
64 bytes from 192.168.0.3: icmp_seq=2 ttl=63 time=96.4 ms
64 bytes from 192.168.0.3: icmp_seq=3 ttl=63 time=96.2 ms
64 bytes from 192.168.0.3: icmp_seq=4 ttl=63 time=96.3 ms

$ ping 10.137.6.139
PING 10.137.6.139 (10.137.6.139) 56(84) bytes of data.
64 bytes from 10.137.6.139: icmp_seq=1 ttl=126 time=96.6 ms
64 bytes from 10.137.6.139: icmp_seq=2 ttl=126 time=96.4 ms
64 bytes from 10.137.6.139: icmp_seq=3 ttl=126 time=96.5 ms
64 bytes from 10.137.6.139: icmp_seq=4 ttl=126 time=96.4 ms

And that’s it!

In case you were just experimenting and want to clean everything up:

delete vpn ipsec
delete protocols bgp
delete interfaces vti vti0
delete interfaces vti vti1

delete firewall group address-group aws-s2s-vpn
delete firewall ipv4 input filter rule 20
delete firewall ipv4 input filter rule 21
delete firewall ipv4 name AWS_S2S_LOCAL
delete firewall ipv4 name WAN_LOCAL rule 30