Alexander Bakker's Blog

Setting up a Site-to-Site VPN between AWS and on-premises VyOS (IPSec & BGP)

Mon, 05 May 2025 00:00:00 +0000

VyOS is an open-source operating system based on Linux that provides software-based networking. There’s an official guide for setting up a Site-to-Site VPN between an on-premises VyOS router and AWS, but it is fairly outdated, as it was written for VyOS 1.3, not for the current LTS: VyOS 1.4 (sagitta). AWS also has the option to download a sample configuration for Vyatta (the project that VyOS was forked from), which is helpful, but also outdated. I’m publishing my notes in case it’s helpful to anyone else.

AWS: Create CGW, VGW/TGW and VPN

This post assumes that you understand IPSec, have some experience in AWS and have already created the CGW, VGW/TGW and VPN. I assume you’re here because the tunnel status says DOWN and now you’re sad because you’re not sure what magical spells to cast towards the VyOS command prompt to make it say UP.

When initially setting op the VPN in AWS, you’ll be asked to choose the type of routing that’s going to be used. The options are fairly self-explanatory:

Static requires both sides to configure static routes for traffic they wish to route through the IPSec tunnel.
Dynamic requires establishing a BGP session between both sides through the IPSec tunnel. Since VyOS supports BGP, we’ll be using this option.

In this example, we’ll terminate the VPN connection into a VGW on the AWS side. The VGW is associated with a VPC with CIDR range 10.137.0.0/16 and the route tables of the VPC are configured to have the VGW propagate routes it has discovered into them. You can also use a TGW if you have a more advanced use case.

VyOS: IPSec tunnel

First, let’s collect some important info from the vpn-01 Site-to-Site VPN I created for test purposes in AWS, along with some info about the on-premises side¹.

	Outside IP	Inside IPv4 CIDR
Tunnel 1	`34.193.192.160`	`169.254.18.92/30`
Tunnel 2	`54.92.230.59`	`169.254.107.244/30`
On-premises	`1.2.3.4`	n/a

We’ll start by configuring the encryption parameters. Download the “Generic” example configuration for your VPN from AWS. This will contain the pre-shared keys for the tunnels. Based on NCSC’s recommendations and the example configuration from AWS, I arrived at the following settings for VyOS:

set vpn ipsec esp-group aws-s2s lifetime '3600'
set vpn ipsec esp-group aws-s2s pfs 'enable'
set vpn ipsec esp-group aws-s2s proposal 1 encryption 'aes128gcm128'

set vpn ipsec ike-group aws-s2s dead-peer-detection action 'restart'
set vpn ipsec ike-group aws-s2s dead-peer-detection interval '10'
set vpn ipsec ike-group aws-s2s dead-peer-detection timeout '30'
set vpn ipsec ike-group aws-s2s key-exchange 'ikev2'
set vpn ipsec ike-group aws-s2s lifetime '28800'
set vpn ipsec ike-group aws-s2s proposal 1 dh-group '19'
set vpn ipsec ike-group aws-s2s proposal 1 encryption 'aes128gcm128'
set vpn ipsec ike-group aws-s2s proposal 1 prf 'prfsha256'

# NOTE: Replace the values below with the tunnel 1/2 outside IP's and corresponding pre-shared secrets
set vpn ipsec authentication psk aws-s2s-tunnel1 id '34.193.192.160'
set vpn ipsec authentication psk aws-s2s-tunnel1 secret 'wDRS31kXtW8Rd0o9QJSAKje76WdJyM6.'
set vpn ipsec authentication psk aws-s2s-tunnel2 id '54.92.230.59'
set vpn ipsec authentication psk aws-s2s-tunnel2 secret '20xllijzqZuLy6UogMVAS5VCVjcPwHP0'

Next, we’ll set up our VTI interfaces. AWS assigns a separate /30 CIDR block in the 169.254.0.0/16 range for use inside each tunnel. The first usable address in these networks is the AWS side and the second address is the on-premises side. So in this case that’d be:

	AWS	On-premises
Tunnel 1	`169.254.18.93`	`169.254.18.94`
Tunnel 2	`169.254.107.245`	`169.254.107.246`

Taking the overhead of the IPSec header into account, and assuming an MTU of 1500 on the interface IPSec is bound to, the MTU for the VTI should be 1436. This is all highly dependent on what your specific setup looks like, of course.

Setting up the VTI interfaces:

# NOTE: Replace the addresses
set interfaces vti vti0 address '169.254.18.94/30'
set interfaces vti vti0 mtu '1436'
set interfaces vti vti0 ip adjust-mss 1396
set interfaces vti vti1 address '169.254.107.246/30'
set interfaces vti vti1 mtu '1436'
set interfaces vti vti1 ip adjust-mss 1396

Tunnels:

set vpn ipsec interface 'pppoe0'
set vpn ipsec options disable-route-autoinstall

# NOTE: Replace the remote-id, local-address and remote-address
set vpn ipsec site-to-site peer aws-s2s-tunnel1 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 authentication remote-id '34.193.192.160'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 connection-type 'initiate'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 ike-group 'aws-s2s'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 local-address '1.2.3.4'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 remote-address '34.193.192.160'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 vti bind 'vti0'
set vpn ipsec site-to-site peer aws-s2s-tunnel1 vti esp-group 'aws-s2s'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 authentication mode 'pre-shared-secret'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 authentication remote-id '54.92.230.59'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 connection-type 'initiate'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 ike-group 'aws-s2s'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 local-address '1.2.3.4'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 remote-address '54.92.230.59'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 vti bind 'vti1'
set vpn ipsec site-to-site peer aws-s2s-tunnel2 vti esp-group 'aws-s2s'

I’m configuring IPSec to use the Point-to-Point interface pppoe0 directly, as it has a public IPv4 address assigned by my ISP configured on it. Your interface will probably have a different name.

Next, we’ll update the firewall rules to allow IPSec traffic from AWS. Replace WAN_LOCAL with the name of your WAN ruleset:

# NOTE: Replace the addresses with the outside IP's of your Site-to-Site VPN in AWS
set firewall group address-group aws-s2s-vpn address '34.193.192.160'
set firewall group address-group aws-s2s-vpn address '54.92.230.59'

set firewall ipv4 name WAN_LOCAL rule 30 action 'accept'
set firewall ipv4 name WAN_LOCAL rule 30 description 'AWS Site-to-Site VPN'
set firewall ipv4 name WAN_LOCAL rule 30 destination port '500,4500'
set firewall ipv4 name WAN_LOCAL rule 30 protocol 'udp'
set firewall ipv4 name WAN_LOCAL rule 30 source group address-group 'aws-s2s-vpn'

Now we can commit the changes, wait for a few moments and check that the IPSec tunnel is now up:

$ show vpn ipsec connections
Connection           State    Type    Remote address    Local TS    Remote TS    Local id    Remote id       Proposal
-------------------  -------  ------  ----------------  ----------  -----------  ----------  --------------  ------------------------
aws-s2s-tunnel1      up       IKEv2   34.193.192.160    -           -                        34.193.192.160  AES_GCM/128/None/ECP_256
aws-s2s-tunnel1-vti  up       IPsec   34.193.192.160    0.0.0.0/0   0.0.0.0/0                34.193.192.160  AES_GCM/128/None/None
                                                        ::/0        ::/0
aws-s2s-tunnel2      up       IKEv2   54.92.230.59      -           -                        54.92.230.59    AES_GCM/128/None/ECP_256
aws-s2s-tunnel2-vti  up       IPsec   54.92.230.59      0.0.0.0/0   0.0.0.0/0                54.92.230.59    AES_GCM/128/None/None
                                                        ::/0        ::/0

The AWS console still reports that the tunnel is DOWN, so we’re still a little sad, but the IPSec tunnel is now up:

If you’ve opted to use “Static” routing when creating the VPN in AWS, the status will say UP and you’ll only have to configure some static routes to start using the VPN. BGP is more fun though, so let’s set up dynamic routing.

VyOS: BGP

Finally, we’ll set up BGP to dynamically exchange routes between AWS and on-premises.

# NOTE: Replace ASN as configured on the customer gateway in AWS
set protocols bgp system-as '65111'
set protocols bgp parameters router-id '192.168.0.1'

# NOTE: Replace ASN as configured on the VGW/TGW in AWS
set protocols bgp peer-group aws-s2s-vpn address-family ipv4-unicast soft-reconfiguration inbound
set protocols bgp peer-group aws-s2s-vpn remote-as '64512'

# NOTE: Replace the neighbour address with the AWS-side tunnel inside address
# and replace the source address with the on-premises tunnel inside address
set protocols bgp neighbor 169.254.18.93 description 'BGP - AWS tunnel 1'
set protocols bgp neighbor 169.254.18.93 peer-group 'aws-s2s-vpn'
set protocols bgp neighbor 169.254.18.93 timers holdtime '30'
set protocols bgp neighbor 169.254.18.93 timers keepalive '10'
set protocols bgp neighbor 169.254.18.93 update-source '169.254.18.94'
set protocols bgp neighbor 169.254.107.245 description 'BGP - AWS tunnel 2'
set protocols bgp neighbor 169.254.107.245 peer-group 'aws-s2s-vpn'
set protocols bgp neighbor 169.254.107.245 timers holdtime '30'
set protocols bgp neighbor 169.254.107.245 timers keepalive '10'
set protocols bgp neighbor 169.254.107.245 update-source '169.254.107.246'

There are a few ways to configure which routes to advertise. A common one is to include all locally known routes:

set protocols bgp address-family ipv4-unicast redistribute connected

In my case, I only want to advertise a specific list of routes:

set protocols bgp address-family ipv4-unicast network 192.168.0.0/24
set protocols bgp address-family ipv4-unicast network 192.168.200.0/24

Update firewall rules to allow BGP traffic from VTI interfaces:

set firewall ipv4 input filter rule 20 action 'jump'
set firewall ipv4 input filter rule 20 inbound-interface name 'vti0'
set firewall ipv4 input filter rule 20 jump-target 'AWS_S2S_LOCAL'

set firewall ipv4 input filter rule 21 action 'jump'
set firewall ipv4 input filter rule 21 inbound-interface name 'vti1'
set firewall ipv4 input filter rule 21 jump-target 'AWS_S2S_LOCAL'

set firewall ipv4 name AWS_S2S_LOCAL default-action 'drop'
set firewall ipv4 name AWS_S2S_LOCAL default-log

set firewall ipv4 name AWS_S2S_LOCAL rule 10 action 'accept'
set firewall ipv4 name AWS_S2S_LOCAL rule 10 state 'established'
set firewall ipv4 name AWS_S2S_LOCAL rule 10 state 'related'

set firewall ipv4 name AWS_S2S_LOCAL rule 20 action 'accept'
set firewall ipv4 name AWS_S2S_LOCAL rule 20 description 'BGP'
set firewall ipv4 name AWS_S2S_LOCAL rule 20 destination port '179'
set firewall ipv4 name AWS_S2S_LOCAL rule 20 protocol 'tcp'

Now we can commit all configuration changes again, wait for a few moments, and check the BGP status. We can see that we’ve now established a BGP session over the two tunnels:

$ show bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 192.168.0.1, local AS number 65111 vrf-id 0
BGP table version 9
RIB entries 5, using 480 bytes of memory
Peers 2, using 40 KiB of memory
Peer groups 1, using 64 bytes of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt Desc
169.254.18.93   4      64512        48        51        9    0    0 00:07:11            1        3 BGP - AWS tunnel 1
169.254.107.245 4      64512        48        51        9    0    0 00:07:11            1        3 BGP - AWS tunnel 2

Total number of neighbors 2

Looking at the BGP route table, we see our own advertised routes and a learned route with two paths that have a different MED for each path:

$ show bgp ipv4
BGP table version is 9, local router ID is 192.168.0.1, vrf id 0
Default local pref 100, local AS 65111
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *  10.137.0.0/16    169.254.18.93          200             0 64512 i
 *>                  169.254.107.245        100             0 64512 i
 *> 192.168.0.0/24   0.0.0.0                  0         32768 i
 *> 192.168.200.0/24 0.0.0.0                  0         32768 i

If you attached the VPN to a TGW instead and have ECMP enabled, you’ll notice that the MED is equal for the two paths, and that multipath is enabled:

$ show bgp ipv4
BGP table version is 19, local router ID is 192.168.0.1, vrf id 0
Default local pref 100, local AS 65111
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

    Network          Next Hop            Metric LocPrf Weight Path
 *= 10.137.0.0/16    169.254.18.93          100             0 64512 i
 *>                  169.254.107.245        100             0 64512 i
 *> 192.168.0.0/24   0.0.0.0                  0         32768 i
 *> 192.168.200.0/24 0.0.0.0                  0         32768 i

And the Linux route table now includes the newly learned route from AWS:

$ ip r
***
10.137.0.0/16 nhid 232 via 169.254.107.245 dev vti1 proto bgp metric 20
***

If you attached the VPN to a TGW instead and have ECMP enabled, you’ll notice that the route has two paths with equal weights:

$ ip r
***
10.137.0.0/16 nhid 249 proto bgp metric 20
	nexthop via 169.254.107.245 dev vti1 weight 1
	nexthop via 169.254.18.93 dev vti0 weight 1
***

The AWS console now also reports that both tunnels are UP and that it has learned two routes. AWS looks happy, so we’re happy:

The VGW has also propagated the routes into the VPC route tables:

We can also ping back and forth between machines in the AWS VPC and the on-premises network:

$ ping 192.168.0.3
PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
64 bytes from 192.168.0.3: icmp_seq=1 ttl=63 time=96.8 ms
64 bytes from 192.168.0.3: icmp_seq=2 ttl=63 time=96.4 ms
64 bytes from 192.168.0.3: icmp_seq=3 ttl=63 time=96.2 ms
64 bytes from 192.168.0.3: icmp_seq=4 ttl=63 time=96.3 ms

$ ping 10.137.6.139
PING 10.137.6.139 (10.137.6.139) 56(84) bytes of data.
64 bytes from 10.137.6.139: icmp_seq=1 ttl=126 time=96.6 ms
64 bytes from 10.137.6.139: icmp_seq=2 ttl=126 time=96.4 ms
64 bytes from 10.137.6.139: icmp_seq=3 ttl=126 time=96.5 ms
64 bytes from 10.137.6.139: icmp_seq=4 ttl=126 time=96.4 ms

And that’s it!

In case you were just experimenting and want to clean everything up:

delete vpn ipsec
delete protocols bgp
delete interfaces vti vti0
delete interfaces vti vti1

delete firewall group address-group aws-s2s-vpn
delete firewall ipv4 input filter rule 20
delete firewall ipv4 input filter rule 21
delete firewall ipv4 name AWS_S2S_LOCAL
delete firewall ipv4 name WAN_LOCAL rule 30

Don’t worry, all of the IP addresses and secrets shown here are either fake or no longer in use by me.

Bypassing app lock in Ente Auth

Wed, 20 Nov 2024 00:00:00 +0000

A couple of months ago I discovered three issues that allow one to bypass app lock in Ente Auth. The issues I found aren’t technically interesting and don’t really deserve a blog post, but since Ente appears to be hesitant to notify their users about them, off we go.

Disclaimer: I’m one of the authors of Aegis Authenticator.

Some background: Aegis’ implementation of automatic lock has a somewhat annoying side effect that makes the app vanish from the recent apps screen in Android. A user recently commented that Ente Auth does not have this issue and suggested that we take some inspiration from them. I decided to take a look, but instead found myself drafting an email to security@ente.io after 10 minutes of playing around with the app.

The issues

These issues were discovered in Ente Auth 3.1.3 for Android, but may be present on other platforms as well. I used the app without signing into an Ente account.

We’re not going to dive into Ente Auth’s code to find the cause of these issues, because I don’t think it would make for interesting reading in this blog post. As you’ll realize while watching the demonstration videos below, the fact that app lock can be bypassed in this fashion tells us that it was implemented as essentially a glorified “if” statement rather than actually requiring the configured credential to decrypt the secrets used to generate OTP’s.

The security of this app lock implementation relies on the absence of bugs in the UI logic. In this case, there are three.

1. Bypassing app lock after the app was automatically locked from the app lock settings screen:

2. Downgrade from password app lock to device credential based app lock by entering the device PIN:

3. Bypassing device credential based app lock entirely:

The second and third video contain a brief flash of blackness, because Android blocks screen recording of the device credential prompt. The touch indicators give a hint of what I’m doing. In video 2 I’m entering the device PIN (not the password configured in Ente) and in video 3 I simply dismiss the prompt.

Timeline

The first two issues were fixed in 18 days. The third one was fixed in 76 days. Here’s the timeline:

2024-09-04 Report submitted via email.
2024-09-05 Ente responds: Thank you for bringing these issues to our attention. We’ll be fixing these problems in the next update.
2024-09-22 I notice that Ente Auth 4.0.0 was released which silently fixes the first two issues. I send an email where I:
- Point out that device credential based app lock can still be bypassed.
- Express that I was a little surprised to see that there’s no announcement urging users to update.
- Mention that I would have appreciated a brief mention crediting me for reporting these issues.
2024-09-25 - 2024-09-26 Some back and forth emails: The dev team is looking into the remaining issue. They explain that they currently don’t have “a credit system” for issues. I clarify that I’m not looking for a monetary reward. They don’t respond to my comment about not announcing the issue to their users.
2024-11-17 I send an email reminding them about this issue and that a little over 2 weeks are left until the 90 day mark. I also share a draft of the blog post I plan to publish on the 3rd of December.
2024-11-19 Ente Auth 4.1.0 is released with a fix for the third issue. It looks like 4.1.0 was released as a pre-release on GitHub on the 5th of November, but it had not been rolled out through all channels (such as Google Play) yet.
2024-11-20 Vishnu from Ente reaches out via email. To summarize:
- “The GitHub release did not get promoted due to human error, so the release did not get pushed to some of our distribution channels. We’ve now appointed a release manager to reduce the possibility of such errors.”
- The changelogs on GitHub have been updated to mention the security issues:
  - https://github.com/ente-io/ente/releases/tag/auth-v4.0.0
  - https://github.com/ente-io/ente/releases/tag/auth-v4.1.0
- The help articles have been updated to explain how app lock works in Ente Auth.
2024-11-20 Since all issues have been fixed, this blog post was published earlier than the 90 day mark.

Parsing Google Authenticator export QR codes

Sat, 23 May 2020 11:35:00 +0000

We recently added support for scanning the new Google Authenticator export QR codes to Aegis Authenticator. The single token URI format is well-documented, but the format of the QR codes displayed in the new export feature of Google Authenticator is not. It’s not immediately obvious how the format works without doing some reverse engineering, so I figured I’d briefly explain it in a blog post.

The QR codes contain a URI with the following format: otpauth-migration://offline?data=... The scheme is otpauth-migration, the host is offline and there is a data query parameter. The data parameter is a base64 encoded Protobuf message with the following format:

syntax = "proto3";

message MigrationPayload {
  enum Algorithm {
    ALGORITHM_UNSPECIFIED = 0;
    ALGORITHM_SHA1 = 1;
    ALGORITHM_SHA256 = 2;
    ALGORITHM_SHA512 = 3;
    ALGORITHM_MD5 = 4;
  }

  enum DigitCount {
    DIGIT_COUNT_UNSPECIFIED = 0;
    DIGIT_COUNT_SIX = 1;
    DIGIT_COUNT_EIGHT = 2;
  }

  enum OtpType {
    OTP_TYPE_UNSPECIFIED = 0;
    OTP_TYPE_HOTP = 1;
    OTP_TYPE_TOTP = 2;
  }

  message OtpParameters {
    bytes secret = 1;
    string name = 2;
    string issuer = 3;
    Algorithm algorithm = 4;
    DigitCount digits = 5;
    OtpType type = 6;
    int64 counter = 7;
  }

  repeated OtpParameters otp_parameters = 1;
  int32 version = 2;
  int32 batch_size = 3;
  int32 batch_index = 4;
  int32 batch_id = 5;
}

The Protobuf message contains a list of OtpParameters. If you’ve worked with HOTP/TOTP before, the fields in OtpParameters should look familiar, so I won’t explain them here. If there are more than a few entries to export from Google Authenticator, they will be split up into multiple QR codes. The batch_size and batch_index fields indicate the number of QR codes and the index of this QR code, respectively. The batch_id is a unique identifier for the export attempt.

I’m not 100% sure if the types of the fields are exactly correct, as I don’t have access to the Protobuf message definition file of Google Authenticator, but they seem to match up well enough.

A mysterious bug in the firmware of Google's Titan M chip (CVE-2019-9465)

Sat, 29 Feb 2020 13:00:00 +0000

Starting with the release of the Pixel 3, all of Google’s Pixel Android smartphones come with the Titan M security chip on board. When I realized the Pixel 3a XL I purchased also had it, I decided to try to take advantage of it in an app I work on. It turned out that using the Titan M chip through the Android Keystore API for AES-GCM in a specific way lead to predictable and bogus ciphertext. This is the story of how I stumbled upon that bug, and why it’s a bit mysterious.

Discussed on HN and Reddit.

The Android Keystore

Let’s start with a short high-level primer on the Android Keystore and StrongBox before we jump into this. If you’re already familiar with these two concepts, you can skip ahead to Discovering the bug.

The Android Keystore is a system service that allows apps to securely generate and use cryptographic keys. While apps can use the keys they generated, they cannot extract them, as they never enter the process of the app. The idea behind this, is that even if the process of the app is compromised, the key is not.

The practical security of this system varies across devices, as the Android Keystore can have various kinds of “backings”. Some devices simply keep the whole system in Android itself, while other devices come with a TEE. Some even come with an entirely separate chip to back the Android Keystore with. The latter offers the best security, because even if an attacker is able to compromise the operating system, they would still need a vulnerability in the HSM to get to the keys in the Android Keystore. The fact that these chips are shipping in more and more smartphones these days is a pretty exciting development, if you ask me.

StrongBox and the Titan M chip

An Android Keystore implementation that falls under the latter category is referred to as a StrongBox Keymaster, and the Titan M chip is one of them. To take advantage of it in an Android app on Pixel devices, one has to indicate a preference for StrongBox during key generation. By simply calling setIsStrongBoxBacked(true) on the KeyGenerator instance, apps that use the Android Keystore get more secure storage for their cryptographic keys, with no apparent downsides. That’s what I thought, anyway.

Discovering the bug

I work on a 2FA app for Android, Aegis Authenticator. The app stores the OTP secrets in an encrypted file (referred to as the vault). The master key used to encrypt/decrypt the vault is managed by a key slot system that’s very similar to LUKS. You can learn more about it here. In short, Aegis has the concept of two types of key slots: for a password and for biometric authentication. The biometric key slot encrypts the master key with a key stored in the Android Keystore, so that the master key can be decrypted after successful biometric authentication by the user.

Here’s what the key generation code for that looks like in Aegis. Excerpt from KeyStoreHandle.java:

KeyGenerator generator = KeyGenerator.getInstance(KeyProperties.KEY_ALGORITHM_AES, STORE_NAME);
generator.init(new KeyGenParameterSpec.Builder(id, KeyProperties.PURPOSE_ENCRYPT | KeyProperties.PURPOSE_DECRYPT)
         .setBlockModes(KeyProperties.BLOCK_MODE_GCM)
         .setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE)
         .setUserAuthenticationRequired(true)
         .setRandomizedEncryptionRequired(true)
         .setKeySize(CryptoUtils.CRYPTO_AEAD_KEY_SIZE * 8)
         .build());

It generates a 256-bit key, to be used with AES in GCM mode, for encryption and decryption. User authentication is required for this key, so that it can only be used after biometric authentication by the user.

This works fine. Let’s switch this over to StrongBox.

diff --git a/app/src/main/java/com/beemdevelopment/aegis/crypto/KeyStoreHandle.java b/app/src/main/java/com/beemdevelopment/aegis/crypto/KeyStoreHandle.java
index cfa1e57..00cae52 100644
--- a/app/src/main/java/com/beemdevelopment/aegis/crypto/KeyStoreHandle.java
+++ b/app/src/main/java/com/beemdevelopment/aegis/crypto/KeyStoreHandle.java
@@ -57,6 +57,7 @@ public class KeyStoreHandle {
                     .setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE)
                     .setUserAuthenticationRequired(true)
                     .setRandomizedEncryptionRequired(true)
+                    .setIsStrongBoxBacked(true)
                     .setKeySize(CryptoUtils.CRYPTO_AEAD_KEY_SIZE * 8)
                     .build());

To test this change, I went through Aegis’ setup process, set a password and enabled biometric unlock. The main view opened and I could start adding tokens. So far so good. However, when I closed the app, reopened it and tried to unlock the vault with biometric authentication, the app showed an error dialog, caused by the following chain of exceptions:

com.beemdevelopment.aegis.db.slots.SlotIntegrityException: javax.crypto.AEADBadTagException
    at com.beemdevelopment.aegis.db.slots.Slot.getKey(Slot.java:57)
    ...
Caused by: javax.crypto.AEADBadTagException
    at android.security.keystore.AndroidKeyStoreCipherSpiBase.engineDoFinal(AndroidKeyStoreCipherSpiBase.java:517)
    at javax.crypto.Cipher.doFinal(Cipher.java:2055)
    ...
Caused by: android.security.KeyStoreException: Signature/MAC verification failed
    at android.security.KeyStore.getKeyStoreException(KeyStore.java:1292)
    at android.security.keystore.KeyStoreCryptoOperationChunkedStreamer.doFinal(KeyStoreCryptoOperationChunkedStreamer.java:224)
    at android.security.keystore.AndroidKeyStoreAuthenticatedAESCipherSpi$BufferAllOutputUntilDoFinalStreamer.doFinal(AndroidKeyStoreAuthenticatedAESCipherSpi.java:373)
    at android.security.keystore.AndroidKeyStoreCipherSpiBase.engineDoFinal(AndroidKeyStoreCipherSpiBase.java:506)
    at javax.crypto.Cipher.doFinal(Cipher.java:2055) 
    ...

Huh. Decryption of the key slot fails, because apparently verification of the MAC failed. How could that be? We only added a single line to enable StrongBox and left the rest of the code alone. I would have attached a debugger at this point to see what’s going on, but that’s not very useful here, as we can’t see what is going on in the HSM, where the cryptographic operation takes place.

After a couple of more tries, it turned out that the kind of exception I was getting was not consistent. If I waited a couple of seconds before authenticating after the biometric prompt was shown, I got an entirely different exception:

java.security.InvalidKeyException: Keystore operation failed
  at com.beemdevelopment.aegis.crypto.KeyStoreHandle.isKeyPermanentlyInvalidated(KeyStoreHandle.java:104)
  at com.beemdevelopment.aegis.crypto.KeyStoreHandle.getKey(KeyStoreHandle.java:83)
  at com.beemdevelopment.aegis.ui.AuthActivity.onCreate(AuthActivity.java:83)
  ...
Caused by: java.security.InvalidKeyException: Keystore operation failed
  at android.security.KeyStore.getInvalidKeyException(KeyStore.java:1362)
  at android.security.KeyStore.getInvalidKeyException(KeyStore.java:1402)
  at android.security.keystore.KeyStoreCryptoOperationUtils.getInvalidKeyExceptionForInit(KeyStoreCryptoOperationUtils.java:54)
  at android.security.keystore.KeyStoreCryptoOperationUtils.getExceptionForCipherInit(KeyStoreCryptoOperationUtils.java:89)
  at android.security.keystore.AndroidKeyStoreCipherSpiBase.ensureKeystoreOperationInitialized(AndroidKeyStoreCipherSpiBase.java:265)
  at android.security.keystore.AndroidKeyStoreCipherSpiBase.engineInit(AndroidKeyStoreCipherSpiBase.java:109)
  at javax.crypto.Cipher.tryTransformWithProvider(Cipher.java:2984)
  at javax.crypto.Cipher.tryCombinations(Cipher.java:2891)
  at javax.crypto.Cipher$SpiAndProviderUpdater.updateAndGetSpiAndProvider(Cipher.java:2796)
  at javax.crypto.Cipher.chooseProvider(Cipher.java:773)
  at javax.crypto.Cipher.init(Cipher.java:1143)
  at javax.crypto.Cipher.init(Cipher.java:1084)
  at com.beemdevelopment.aegis.crypto.KeyStoreHandle.isKeyPermanentlyInvalidated(KeyStoreHandle.java:96)
  ...
Caused by: android.security.KeyStoreException: Invalid key blob
  at android.security.KeyStore.getKeyStoreException(KeyStore.java:1292)
  ...

Digging deeper

At this point, I decided this issue may be worth reporting to Google through the Android Security Rewards Program. In an attempt to get a better grasp on what the issue actually is, I wrote a small configurable PoC app that demonstrates the issue to go along with the report. It is open source and available on GitHub. It has gone through a few iterations since the initial report to Google. The screenshots below show the latest version.

The app sits in a loop, repeatedly encrypts/decrypts some hard coded plaintext and prints the results to the log. As shown in the screenshots above, user authentication, the timeout between tries and some other things are configurable in the options dialog. While we can’t see what going on inside the HSM, this gives us a good look at the behavior we can see on the outside. Looking at the log, it turns out that this bug causes cryptographic operations to produce predictable and bogus ciphertext:

plaintext: 746869732069732061207465737420737472696e67
ciphertext: d62a2349d993632dddabc30a4a2c8ab7ba2608c5f2
tag: fd6b38cba35ea579918f3b5ec1863e4b
nonce: f102f60a0ef39e310c5f9c4c
decrypted: 746869732069732061207465737420737472696e67

plaintext: 746869732069732061207465737420737472696e67
ciphertext: 0dd0adde0dd0adde0dd0adde0dd0adde585301005d
tag: 995d100cc5d068a83e7ecf13c49e92eb
nonce: cffb9148cb5154232c957ab7

Note the ciphertext of the second iteration: 0dd0adde0dd0adde0dd0adde0dd0adde585301005d. That doesn’t look as random as one would expect. In fact, there’s even a sequence that repeats: 0dd0adde. Let’s try that again.

plaintext: 746869732069732061207465737420737472696e67
ciphertext: 789ae5602a997f5ce7768b08fe5db2d1d7139fad30
tag: 12e28ebf0f9492a76a453cd5d3bcf4ff
nonce: e4921e4825ea1ad01881ad82
decrypted: 746869732069732061207465737420737472696e67

plaintext: 746869732069732061207465737420737472696e67
ciphertext: 0dd0adde0dd0adde0dd0adde0dd0adde585301005d
tag: 74eaa7be302f1dce6d53ae5a58dc2408
nonce: a454451014a075e1883f4521

No matter how many times this is repeated, the resulting ciphertext is always the same (while the nonce is always different, even). Now it’s obvious why decryption failed in Aegis before. Trying to decrypt bogus ciphertext like that will certainly result in a MAC verification failure and bogus plaintext. The most dangerous thing about this, is the fact that it fails so spectacularly without throwing an exception. This also completely defeats one of the most important guarantees that symmetric ciphers provide: ciphertext that is indistinguishable from random noise.

Playing around with the app some more, I concluded that waiting ‘too long’ (around 2 seconds) between cipher initialization and using it to encrypt/decrypt something elicits this behavior. This appears to only be an issue with AES in GCM mode. I also tested the other supported ciphers and modes, but those seem to behave correctly.

Unfortunately, this is where my research grinded to a halt. To go further I would need access to the firmware that is running on the Titan M chip, to see what might be going wrong there. While Google has promised to open source the firmware, it has not delivered on that promise to this day. My best guess would be that this is a memory corruption issue, but that’s about as far as I’m going to get without more access. Google doesn’t seem to be willing to discuss the details of the issue either, as you’ll notice while reading the Timeline.

I also took a quick look at the Android source tree to see if I could find any places where AES-GCM was used in combination with StrongBox and the randomness of ciphertext was being relied upon. The latter is the case with some CSPRNG implementations, for example, though I can’t think of a reason why someone would use GCM mode for that purpose. That’s the only attack vector I could come up with, but perhaps someone more knowledgeable in Android’s internals can think of something else.

Timeline

In total, it took 6 months to get this fixed. Here’s the timeline:

2019-05-20 Report submitted.
2019-05-21 Google responds: We’re looking in to it.
2019-05-28 Google responds: Won’t Fix (Infeasible).

The team tells me there are some errors in my implementation that cause the exhibited behavior and that it’s not actually a bug in Android:
1. In the KeygenParameterSpec, setUserAuthenticationRequired(true) was set up, but KeygenParameterSpec was not set on how to authenticate such as: setUserAuthenticationValidityDurationSeconds(int) So the encryption was never authorized.
2. In addition, when instantiating the KeyGenParameterSpec, KeyProperties.PURPOSE_DECRYPT was not added, so decryption failed.
Neither of these suggestions make sense. To be fair, I’ve always found the Android Keystore key generation API to be a bit confusing, but it’s interesting to see members of the Android Security team struggling with it as well.
2019-05-28 I respond, suggesting that they take another look and actually try to run the POC app, instead of trying to find flaws in my use of the API.
2019-06-06 I ask for a status update.
2019-06-11 Google responds: We have no update at this time.
2019-06-14 Google responds: High severity.
2019-07-05 I ask for a status update.
2019-07-08 Google responds: We’re still looking into it.
2019-08-26 I ask for a status update.
2019-08-26 Google responds: apologizing for the delay, saying that the issue is taking longer to remediate than the usual 90 day window. Asks if I intend to disclose, and if so, whether I’d be willing to participate in coordinated disclosure.
2019-08-26 I respond, saying I don’t intend to disclose yet, as I don’t have a patch/solution to go along with a disclosure.
2019-08-28 Google responds, thanking me for the clarification and assuring me that they’ll keep me updated.
2019-10-08 I ask for a status update.
2019-10-14 Google responds: the issue has been fixed and they’re tracking the fix for release in December. They’ll provide details soon regarding CVE and reward eligibility. Also, the severity has been adjusted from High to Critical.
2019-10-15 I respond, expressing that I’m glad it’s been fixed.
2019-11-20 I ask for a status update.
2019-11-25 Google responds: We’d like to acknowledge your contribution publicly. CVE assigned: CVE-2020-0014.
2019-11-25 I respond: Thanks for the status update. I ask if the fix is still on track for release in December and whether this report is eligible for a reward. I tell them I plan on writing a blog post.
2019-11-26 Google responds: Thanks me for sharing my disclosure plans and reassures me that the fix is still on track for release in December and that the report is eligible for a reward.
2019-11-26 I respond: Thanks for the clarification and assuring them that I’ll let them review this blog post first before I publish it.
2019-11-27 Google responds. The CVE number that was given previously is incorrect. The correct one is: CVE-2019-9465.
2019-12-02 Google responds. The rewards committee decided to reward me for reporting this security issue.
2019-12-02 I respond, thanking them for the reward. I also ask for more details about the bug, as I have very little to go on the for the blog post I plan to write.
2019-12-02 Google responds, confirming my suspicion that the bug was in the Titan M firmware, but doesn’t provide any additional information.
2019-12-05 I respond, reporting my findings after installing the December 2019 security update: the bug is still present on my device.

2019-12-07 - 2019-12-24 Google responds, saying that they are unable to reproduce the issue. After that, there was a lot of back and forth to try to get logs from my device. It turned out that Android was not able to update the firmware of the Titan M chip on my device, possibly due to a bad state I had gotten it into while developing the PoC.

12-24 16:22:57.618  1087  1087 I init_citadel: Citadel version: 0.0.3/brick_v0.0.7574-5b47d37e 2019-06-25 19:20:19 gdk@chunky.cam.corp.google.com
12-24 16:22:57.666  1108  1108 I init_citadel: Citadel is running a known older firmware so sending update
12-24 16:22:57.778  1136  1136 I init_citadel: Citadel is C2-PVT, allowing RO updates
12-24 16:23:11.812  2808  2808 I init_citadel: Citadel update loaded
12-24 16:23:11.852  2810  2810 I init_citadel: Could not enable Citadel update: password required
12-24 16:23:12.009  2812  2812 I init_citadel: Citadel rebooted
12-24 16:24:19.263   584   584 I chatty  : uid=1064(hsm) /vendor/bin/hw/citadeld identical 5 lines
12-24 16:24:20.357   806   806 E /vendor/bin/hw/android.hardware.authsecret@1.0-service.citadel: Incorrect Citadel update password
12-24 16:24:29.466   825   825 I /vendor/bin/hw/android.hardware.oemlock@1.0-service.citadel: Running OemLock::setOemUnlockAllowedByCarrier: 1
12-24 16:24:29.473   584   584 I chatty  : uid=1064(hsm) /vendor/bin/hw/citadeld identical 1 line

2020-02-02 I ask for a status update.
2020-02-07 Google responds, recommending me to factory reset my device to resolve the state it is in, while they continue to investigate the issue.

2020-02-23 I respond, saying I factory reset my device and that the security issue is no longer present. I didn’t catch the update process itself, but the firmware version does seem to be newer:

02-23 16:38:11.981   921   921 I init_citadel: Checking citadel version
02-23 16:38:12.279  1027  1027 I init_citadel: Citadel version: 0.0.3/brick_v0.0.7580-2d3a8cfc 2019-09-16 20:42:26 gdk@chunky.cam.corp.google.com
02-23 16:38:12.333  1054  1054 I init_citadel: Citadel isn't running a known older firmware so not updating

A call with Google

After sharing the draft of this blog post with the team, I was asked if I would be willing to join a short call, as they wanted to “provide some details/context about the fix process for this issue prior to the publication” of my blog post. I agreed.

During the call they apologized for the long turnaround time and poor communication. I appreciate that, as that was indeed the main disappointment I had when submitting this as my first Android security issue report. I had to keep asking for status updates and felt like I was being kept out of the loop, while one would expect the team to share new information as it becomes available. The initial draft of the blog post had a fairly snarky comment about the fact that the severity rating changed from Won’t Fix, to High, to Critical. They explained that their guidelines for severity ratings changed while in the process of handling my report and they adjusted the severity rating based on those new guidelines. Fair enough, I removed that comment.

The overall impression I got is that they genuinely care about handling Android security reports well and regret that it didn’t go as well as they would have liked this time around.

Naturally, I also used the opportunity to ask if they could provide some more information about the nature of the actual bug and about the status of open sourcing the Titan M firmware, but they couldn’t comment on that.

Conclusion

I ended up performing a factory reset on my device a few days ago, which seems to have magically fixed the Titan M firmware update issue. The security issue itself also appears to be resolved. I would have loved to try to dig deeper into this, but as said, the source of the Titan M firmware is simply not available. Google appears to have added CTS tests for the exhibited behavior, but other than that, there is no trace of how this bug was fixed.

I also learned some important lessons. Upon initial disclosure to Google, I should have firmly set the standard 90-day deadline, so that it’s clear for all parties what the expectations are. Secondly, despite not setting a deadline upfront, I should have disclosed this publicly much earlier than I ended up doing. It just didn’t feel right to disclose a security issue that I didn’t have a fix or workaround to offer for, but I’ve since come to the realization that getting the information out there is more important.

Notes on PCI Passthrough on NixOS using QEMU and VFIO

Sat, 28 Sep 2019 22:30:00 +0000

With the release of the Ryzen 3000 series CPUs, I decided it was finally time to upgrade from my good old Intel i5 2500K. It served me well for nearly 8 years, but its age was starting to show. While doing the upgrade, I also wanted to address the other two main pain points I had with my previous setup. Long story short: I ended up installing NixOS and setting up PCI passthrough.

The Linux distribution I was using previously, Gentoo Linux, became too much of a burden to maintain across multiple machines and I was looking for an alternative that offers similar flexibility. The declarative configuration system of NixOS appealed to me and I decided to give it a shot after playing around with it for a bit.

The other annoyance I had, was the fact that I had to reboot into Windows to be able to play games. However, with the combination of the IOMMU on modern motherboards and the VFIO driver on Linux, it’s possible to directly and securely access entire groups of PCIe devices from userspace. This allows one to pass through a GPU to a Windows virtual machine, with minimal overhead in terms of performance.

This blog post walks through my specific PCI passthrough setup, with some notes on how to get it working on NixOS. If you’re new to PCI passthrough, this might not be for you. The Arch Linux Wiki has an excellent article that you can refer to if want to get up to speed. I relied on it heavily.

Hardware

The machine I ended up building has the following specifications:

OS: NixOS 19.09
CPU: Ryzen 9 3900X
Motherboard: Gigabyte X570 Aorus Master (rev 1.0)
- UEFI: F7c (AGESA 1.0.0.3 ABBA)
RAM: G.Skill DDR4 Ripjaws-V 2x16GB 3200MHz CL14
GPU 1: Sapphire Nitro+ Radeon RX 580 4GD5
GPU 2: Sapphire Nitro+ Radeon RX 590 8GD5

You’ll want to do some research online before purchasing the hardware for your machine to make sure that it has good IOMMU support. Perhaps someone has set up PCI passthrough on the same hardware before and has posted the list of IOMMU groups somewhere online.

Here’s the list of IOMMU groups on my system: iommu-groups.txt.

UEFI

To start things off, we need to make a couple of changes to the UEFI settings. AMD SVM, IOMMU, ACS and AER support need to be enabled. We also need to make sure to configure the initial display output GPU, so that the guest GPU remains untouched during the boot process. On my setup, the RX 580 will be used by the host and the RX 590 will be passed through to the guest. The setting defaults to PCI slot 1, so I don’t have to make any changes there.

VFIO

The next step is to identify the PCI slot numbers for the GPUs and the IOMMU groups they’re in. Boot your OS and run the following script:

#!/usr/bin/env bash

shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;

You should see two VGA devices, each in a separate IOMMU group, along with an audio device in the same group. Take note of the PCI slot number of the VGA and audio device you want to pass through to the guest. In my case, that’d be 0b:00.0 and 0b:00.1

IOMMU Group 24 0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X] [1002:67df] (rev e7)
IOMMU Group 24 0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
IOMMU Group 25 0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X] [1002:67df] (rev e1)
IOMMU Group 25 0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]

Next, we’ll make some changes to the initrd to ensure that the vfio-pci driver is loaded for our guest GPU, instead of amdgpu. To allow early kernel modesetting to continue working with this setup, we load vfio-pci for the guest GPU in the preDeviceCommand script block, which is executed before udev is started.

boot.initrd.availableKernelModules = [ "amdgpu" "vfio-pci" ];
boot.initrd.preDeviceCommands = ''
  DEVS="0000:0b:00.0 0000:0b:00.1"
  for DEV in $DEVS; do
    echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
  done
  modprobe -i vfio-pci
'';

We’ve already enabled virtualisation and IOMMU support in the UEFI, but we also need pass a couple of parameters to the kernel command line (intel_iommu=on for Intel CPUs and amd_iommu=on for AMD CPUs). We also need to load an extra kernel module to be able to use KVM (kvm-intel for Intel CPUs and kvm-amd for AMD CPUs).

boot.kernelPackages = pkgs.linuxPackages_latest;
boot.kernelParams = [ "amd_iommu=on" "pcie_aspm=off" ];
boot.kernelModules = [ "kvm-amd" ];

I had to add an extra parameter to turn off PCIe ASPM. Without it, the kernel would start kicking out AER errors and the whole system would become unstable shortly afterwards. On a side note: I usually use the latest kernel so that I receive new features (and any VFIO-related bug fixes) quickly, but if you like having a stable system, you might want to comment that part out.

Apply the configuration by running nixos-rebuild switch and reboot.

QEMU and libvirt

Next, we’ll set up libvirt. Libvirt is a tool that abstracts management of virtual machines with support for multiple different hypervisors. This is not strictly necessary, as we could also piece together a raw QEMU command ourselves, but libvirt makes the whole thing a lot more manageable once you start running multiple virtual machines. Add the following to your configuration.nix:

environment.systemPackages = with pkgs; [
  virtmanager
];

virtualisation.libvirtd = {
  enable = true;
  qemuOvmf = true;
  qemuRunAsRoot = false;
  onBoot = "ignore";
  onShutdown = "shutdown";
};

This will install libvirt along with OVMF and configure the libvirtd service to be started on boot. I also set onBoot to ignore and onShutdown to shutdown, so that the virtual machines are always shut down cleanly and never automatically started when the host boots.

From this point, you can install Windows in a virtual machine as you would on any other Linux distribution. As said, I’m not going to go over the whole process in this blog post, as the Arch Wiki already has a comprehensive article on it.

And that’s it! You should now be able to create a PCI passthrough setup on NixOS! My libvirt configuration file is available here for reference: libvirt-nt10.xml.

Bonus

There are some additional pieces of software that you can install to make the whole experience a bit more pleasant.

Looking Glass

Looking Glass is a piece of software that can capture the video output of a GPU passed through to a virtual machine and share it with the host by writing it to an Inter-VM shared memory device. The Looking Glass client can read from that device and display the video output on the host machine.

It is available in the Nix package collection as looking-glass-client. Once you’ve gone through the setup process on Windows, you’ll want to automatically create the SHM file on the NixOS side with some help from systemd-tmpfiles.

systemd.tmpfiles.rules = [
  "f /dev/shm/looking-glass 0660 alex qemu-libvirtd -"
];

And add it to the configuration of your virtual machine.

<shmem name='looking-glass'>
  <model type='ivshmem-plain'/>
  <size unit='M'>32</size>
  <address type='pci' domain='0x0000' bus='0x0b' slot='0x01' function='0x0'/>
</shmem>

You can then launch the client with looking-glass-client -f /dev/shm/looking-glass. Here’s a short clip of Looking Glass in action:

While the performance and latency is really impressive, I wouldn’t recommend using this for very latency-sensitive games like first-person shooters, as the latency Looking Glass adds will still be at least as bad or worse than having Vsync enabled. It’s a great solution for nearly all other use cases, though.

Scream

Scream is a virtual network sound card for Windows. While it’s primary functionality is publishing the audio played through the device on the local network, it also supports IVSHMEM, like Looking Glass. This allows playing the guest audio on the host in a much better way than some of the other solutions I’ve tried.

It is available in the Nix package collection as scream-receivers. Once you’ve set up the Scream driver on Windows, you’ll want to automatically create another SHM file like we did before and define a systemd user service for the Scream client.

systemd.tmpfiles.rules = [
  "f /dev/shm/scream 0660 alex qemu-libvirtd -"
];

systemd.user.services.scream-ivshmem = {
  enable = true;
  description = "Scream IVSHMEM";
  serviceConfig = {
    ExecStart = "${pkgs.scream-receivers}/bin/scream-ivshmem-pulse /dev/shm/scream";
    Restart = "always";
  };
  wantedBy = [ "multi-user.target" ];
  requires = [ "pulseaudio.service" ];
};

And add it to the configuration of your virtual machine:

<shmem name='scream'>
  <model type='ivshmem-plain'/>
  <size unit='M'>2</size>
  <address type='pci' domain='0x0000' bus='0x0b' slot='0x02' function='0x0'/>
</shmem>

Conclusion

I hope this blog post gives you an idea of where to get started to get PCI passthrough working on NixOS. Eventhough my setup is almost certainly not exactly the same as yours, most of the NixOS-specific parts discussed will apply for everyone. To summarize, the final configuration.nix snippet looks like this:

environment.systemPackages = with pkgs; [
  looking-glass-client
  scream-receivers
  virtmanager
];

virtualisation.libvirtd = {
  enable = true;
  qemuOvmf = true;
  qemuRunAsRoot = false;
  onBoot = "ignore";
  onShutdown = "shutdown";
};

boot.kernelPackages = pkgs.linuxPackages_latest;
boot.kernelParams = [ "amd_iommu=on" "pcie_aspm=off" ];
boot.kernelModules = [ "kvm-amd" ];

boot.initrd.availableKernelModules = [ "amdgpu" "vfio-pci" ];
boot.initrd.preDeviceCommands = ''
  DEVS="0000:0b:00.0 0000:0b:00.1"
  for DEV in $DEVS; do
    echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
  done
  modprobe -i vfio-pci
'';

systemd.tmpfiles.rules = [
  "f /dev/shm/scream 0660 alex qemu-libvirtd -"
  "f /dev/shm/looking-glass 0660 alex qemu-libvirtd -"
];

systemd.user.services.scream-ivshmem = {
  enable = true;
  description = "Scream IVSHMEM";
  serviceConfig = {
    ExecStart = "${pkgs.scream-receivers}/bin/scream-ivshmem-pulse /dev/shm/scream";
    Restart = "always";
  };
  wantedBy = [ "multi-user.target" ];
  requires = [ "pulseaudio.service" ];
};

Did cosmic rays break my Linux build?

Sun, 01 Sep 2019 00:00:00 +0000

I think I experienced a random bit flip while updating Linux on one of my machines today. My laptop was humming along happily during compilation until GCC suddenly aborted with an error: invalid preprocessing directive #lefine; did you mean #define?.

Huh, #lefine instead of #define?

Now, GCC is known for its cryptic error messages, but this one left me baffled. Things I tried to figure out what the issue could be:

I first grepped the offending file, drivers/gpu/drm/i915/i915_reg.h, for #lefine to see if a typo somehow slipped through the Linux 4.19.1 release, but got 0 matches.
When I resumed the build, it completed without errors. The kernel also appeared to run fine upon reboot. I ended up recompiling it entirely multiple times to see if I could reproduce the error, but I couldn’t.
I ran a MemTest86 test for 24 hours, but it didn’t find any errors, so the RAM is also unlikely to be the culprit.

I thought I was losing my mind until it dawned on me that this could have been caused by cosmic rays. Looking at ASCII table, we see that ’d’ and ‘l’ are indeed one bit flip away from each other:

char	decimal	binary
d	100	1100100
l	108	1101100

Could that be it? Isn’t that incredibly unlikely? There doesn’t appear to be a consensus in existing literature ¹ ² ³ on how often these errors actually occur, but I did learn that this phenomenon is actually more common than I thought. The chances of flipping a bit that would cause software to break in this fashion, on the other hand, must be incredibly small.

In the end, there’s just no way to prove that this was actually caused by cosmic rays. It could also have been some other extremely rare hardware fluke. Either way, this took me by surprise, to say the least.

Perhaps this is the cosmos’ way of telling me to invest in ECC RAM.

Insecure seed generation in the Nano Android wallet

Mon, 02 Jul 2018 00:00:00 +0000

On the 21th of June 2018, the release of the new wallet applications for Nano was announced on Reddit. Shortly after that, another announcement was made telling users of the Android app to transfer their funds to a wallet with a seed that was not generated by the app. I quickly looked up the source code and found that the app was using a random number generator that is not cryptographically secure. Let’s analyze how bad this really is. Spoiler: it’s bad.

Discussed on Reddit.

The code

Let’s look at the vulnerable piece of code first:

import java.util.Random;

public static String generateSeed() {
    int numchars = 64;
    Random random = new Random();
    StringBuilder sb = new StringBuilder();
    while (sb.length() < numchars) {
        sb.append(Integer.toHexString(random.nextInt()));
    }
    return sb.toString().substring(0, numchars).toUpperCase();
}

As you can see, it’s using java.util.Random to generate the seed. The algorithm that Random uses to generate random numbers is not cryptographically secure. It’s seeded with a 64-bit integer (usually the current time in milliseconds by default if no seed is specified) and given a number of outputs, the following outputs can be predicted without knowing the seed.

Ok, so this sounds bad, but is it actually exploitable in practise?

Difference in API levels

We first need to know what the implementation of java.util.Random looks like on Android. The API of Random is the same across all API levels, but the implementation differs. Not only the default seed is different, but the algorithm to generate random numbers is also different. We’re only going to look at how the default seed is calculated as the underlying algorithm doesn’t really impact our ability to exploit this vulnerability.

The oldest Android API level the app supports is 16, so we’ll start there.

Level 16 - 19

On API levels 16 - 19, the default seed of Random is calculated as follows (source):

public Random() {
    // Note: Using identityHashCode() to be hermetic wrt subclasses.
    setSeed(System.currentTimeMillis() + System.identityHashCode(this));
}

Older versions of Android use a common implementation of Random where the RNG is seeded with the sum of the current time in milliseconds and the hashCode of the Random object.

Time is obviously a bad source for secure randomness. If we wanted to iterate over all timestamps of a particular day, we’d only have 86400000 of them to go through (as that’s the amount of milliseconds a day has).

The default hashCode implementation can differ across JVM implementations, but it usually returns the memory address of the object. This is also the case on Android. That makes it a little harder to guess, but it’s a 32-bit number and therefore also trivially brute-forceable.

Let’s say we wanted to brute-force all seeds that could have been generated during the beta period. The timespan of the beta period from the initial announcement up until the announcement of the vulnerability is about 4 months (10510000000 milliseconds). We need to account for the addition of the 32-bit number, so we’ll subtract 2³¹ from the timestamp we start with and add 2³¹ to the last timestamp. This gives us a total key space of only 14804967296 (less than 2³⁴)!

Level 20 - 23

On API levels 20 - 23, the default seed of Random is calculated as follows (source):

private static volatile long seedBase = 0;

public Random() {
    // Note: Don't use identityHashCode(this) since that causes the monitor to
    // get inflated when we synchronize.
    setSeed(System.nanoTime() + seedBase);
    ++seedBase;
}

This implementation uses the sum of System.nanoTime and an offset to seed the RNG.

Note that nanoTime does not represent the current time. The starting point is unspecified. On Linux, this represents the elapsed time since startup in nanoseconds, excluding the amount of time spent sleeping (CLOCK_MONOTONIC). This is also the case on Android. I’d say this number is a bit harder to guess, but it’s definitely a lot less than 64 bits of entropy.

Integer ‘seedBase’ is incremented to make sure every instance of Random produces a different output. That’s its only purpose, it doesn’t make things any more secure.

To be clear, even if nanoTime wasn’t predictable in any way, a key space of 64 bits is still way too small for comfort. There’s a reason why we only use key sizes of at least 128 bits.

Level 24 - 27

On API levels 24 - 27, the default seed of Random is calculated as follows (source):

public Random() {
    this(seedUniquifier() ^ System.nanoTime());
}

private static long seedUniquifier() {
    // L'Ecuyer, "Tables of Linear Congruential Generators of
    // Different Sizes and Good Lattice Structure", 1999
    for (;;) {
        long current = seedUniquifier.get();
        long next = current * 181783497276652981L;
        if (seedUniquifier.compareAndSet(current, next))
            return next;
    }
}

private static final AtomicLong seedUniquifier
    = new AtomicLong(8682522807148012L);

This looks a bit more complicated, but the principle is basically the same as the previous implementation we looked at. It also uses nanoTime to seed the RNG but it has a different mechanism to make sure all instances of Random produce different output.

Proof of concept attack

To test if my analysis is correct, I decided to write a POC program that generates all seeds that could have been generated during the entire beta period of the app on Android devices with API level 19 or lower.

Here are the steps the program goes through:

Generate the list of seeds, starting with the time of the beta announcement minus 2³¹ and ending with the time of the vulnerability announcement plus 2³¹.
Derive Ed25519 keypairs from the generated seeds. This involves doing some BLAKE2b hashing to derive a private key from the seed and an index number.
Iterate over all public keys of the keypairs and check if any of them exist in the frontier list.
If there’s a match, we know we have found the seed for that particular public key.

There are definitely more intelligent ways to do this attack. This is just a proof of concept. For example, a way to improve the attack could be to reduce the space of the 32-bit number we need to explore by taking memory alignment into account.

Results

After letting the program run for a couple of days, I found a total of 84 existing addresses that I was able to generate a seed for. The addresses held a grand total of around 663 NANO. To be absolutely clear, I’m not going to steal any funds. I deleted the seeds right after the brute-force completed.

This is not a lot of money, but it’s not insignificant either. Keep in mind that we only looked at a very small portion of affected users: the ones running Android KitKat or older. According to Google’s statistics, this amounts to 14.6% of their userbase. We did not look at more recent Android versions, which amount to 84.7%. Surely, we would have found a lot more seeds with existing addresses if we did do that. As far as I can tell that would require quite a bit more computing power though.

Warning users

Eventhough the Nano team repeatedly reached out through their social media channels (Reddit, Twitter, Discord, Telegram and even Blockfolio) telling users to change their seeds, the accounts I found still held a bunch of NANO. Of course not every holder of NANO is active on social media or follows cryptocurrency news, so I wanted to try a different way to warn users before publishing this post.

My first idea was to send transactions to all vulnerable addresses I found and use steganography to encode a message into the balance. While this is a cute way to send messages over the block lattice, it’s very unlikely users will notice this. A friend came up with a much better idea: send small amounts of NANO from 3 vanity addresses that spell out a message:

xrb_1changec1b68rgyrhpbngthck1p3yekcu868chcxnm6zsm6u87u4bx1iw7wj
xrb_1seedmpo9e63piyyo77db44i6hthedmou9chthe3qudswow75q8f7rzfhhrg
xrb_1asap5ufpqzobb1rkdnjx3eieantny16b1dpequy3csrcrsixexa4517xh7y

I sent 0.000001 NANO from each of the above addresses to the vulnerable addresses I found that had a balance of 0.001 or more. Unfortunately the transactions didn’t always arrive in the correct order. Due to the way a block lattice works, wallets have no way of knowing the order in which transactions occurred if the transactions did not originate from a single account, so the order ends up being arbitrary.

But sure enough, one of the accounts responded by moving their funds elsewhere:

Luckily, this happened to be the account that held the largest amount of NANO. This resulted in a big reduction of the amount of NANO these addresses were holding. The total is now around 131 NANO at the time of writing.

Update 2018-07-02: After publishing this post another account appears to have transferred its funds elsewhere: xrb_16dpz3fj8o3i8tizmyx9ataxmcus53bb3tgfy5eu6f86d4ozc5i8tuww7sdn. According to the transaction history, this account has received funds from the address the funds were just sent to before, so we can be reasonably sure that they were not stolen. This brings the total down to about 34 NANO.

Update 2018-07-06: It looks like someone has reproduced the attack and stole the remaining funds. Luckily the majority of the funds on the vulnerable addresses I found were moved elsewhere as a result of my post, but he still got away with 34 NANO. This is the address of the thief: xrb_3tsjtouqoae78e6pba1e1s1amod66fxsswrq4dzkg3ohh1fqgfrykr75xw6w.

Conclusion

This is one of the worst mistakes one can make when using cryptography. Unsurprisingly, this turns out to be very easy to exploit and it’s only a matter of time before someone else exploits this vulnerability and starts stealing the remaining funds. I was going to publish the POC code but decided not to because I’m afraid I’ll receive the blame eventhough the attack I demonstrated is trivial. To be honest, I was very surprised to see that no funds had been stolen yet.

Before publishing this post, I gave the Nano team a chance to read it and to provide feedback. I also kept them up to date on what I was doing and what the results of the proof of concept attack were.

It’s important to note that this vulnerability was patched almost immediately after it was found. Seeds generated with the new version of the app are generated using a CSPRNG and are therefore secure. I spoke with the Nano team and was told that the vast majority of their users are now on version 1.0.2 or newer of the app which means they were forced to migrate to a new seed. However, users who exported their seed and uninstalled the app are still at risk.

So please, if you used the Nano Android wallet, move your funds to an address derived from a different seed if you haven’t done so already.