mTLS in Chromium Headless Mode on Ubuntu

I recently needed to access an application which required mTLS using an automated browser to facilitate security testing. The last time I attempted this (several years ago) it seemed to be impossible, but since then there have been changes to the Chromium browser which make this possible.

It requires conversion of certificates and keys to a PKCS#12 file, creation of NSS DB files, import of the PKCS#12 file, configuration flags for the objects in NSS DB, Chromium Browser>=112, new headless mode, and a policy file to automatically select the certificate. So while not trivial it is fairly easy to do once you know how.

I hope this post will allow others to accomplish this in less time and with less head scratching than I…

What is mTLS?

Transport Layer Security (TLS) is a protocol which provides confidentiality, integrity, and authentication between a client and a server using public key cryptography.

Typically the server identifies itself and the client verifies it – this is what is used by most of the internet for HTTPS.

In mTLS (Mutual TLS) the client also identifies itself and the server verifies the client. This is not generally used for public applications since it is less user friendly as each client must be issued a cryptographic key from a trusted authority. Typically mTLS is only used in Business to Business applications or in corporate environments where the certificate can be deployed to a managed client.

Chromium Headless Mode

Headless mode in Chromium Browser has been available for many years. Essentially it allows you to use a browser without requiring a display, typically via the ChromeDevTools API. This is very useful when programmatically controlling a browser for automated testing of client side issues such as DOM XSS or missing origin validation in HMTL5 Web Messaging (postMessage), or simply crawling JavaScript heavy applications (Single Page App – SPA).

Until Chromium 112 the headless mode available in the chromium binary was actually a different browser implementation to the headful mode. This caused various unexpected behaviours and some features were missing or limited.

The new headless mode introduced in Chromium 112 and default in 128 is the same browser implementation as headful mode with all features available with no limitations!

Refer to https://developer.chrome.com/docs/chromium/headless for further information.

One of the features that was previously unavailable was mTLS since the old headless browser implementation did not load the Network Security Services database – with new headless mode mTLS is now more easily available in an automated browser context.

Network Security Services (NSS)

Network Security Services NSS, is a set of libraries that was originally developed by Netscape when it developed the Secure Socket Layer (SSL) protocol (the predecessor to Transport Layer Security (TLS)); it was later inherited by Mozilla.

NSS supports the cross platform development of client/server applications.

Because NSS is intended to be interoperable and shared applications, it’s sqlite database files (cert9.db and key4.db) are typically stored at a predictable path ~/.pki/nssdb. Although some applications use a custom location, Chromium Browser uses ~/.pki/nssdb/ luckily it honours the HOME environment variable when the Chromium process is launched which provides some flexibility that may be required (see below).

PKCS#12 Certificate and Key Bundle

Public Key Cryptography Standards (PKCS) #12 is an archive file format to store multiple cryptographic objects. It typically has the .p12 or .pfx file extension.

The PKCS#12 file typically contains an x.509 public certificate and associated private key, as well as the chain of trust that bridges the gap between the certificate authority which signed the certificate and a well-known widely trusted authority higher up the chain of trust.

In order to import a certificate and key pair into NSS PKCS#12 must be used, and the key must be encrypted. For compatibility I use the legacy PBE-SHA1-3DES algorithm since I have had problems with the more secure AES-256-CBC. Although Certificate Authority certificates can be imported into NSS DB in the PEM (Privacy-Enhanced Mail) Base64 encoded text format which is often used for X.509 certificates; I typically convert the all three pem files to a PKCS#12 and include friendly names for the client certificate and CA certificate which can be referenced later.

openssl pkcs12 -export \
  -inkey key.pem \
  -in cert.pem \
  -certfile ca.pem \
  -out client-cert.p12 \
  -name "ClientCert" \
  -caname "SomeCA" \
  -passin pass:pemkeypass \
  -passout pass:p12exportpass \
  -keypbe PBE-SHA1-3DES \
  -certpbe PBE-SHA1-3DES

Import Certificate and Configure flags

The certutil utility (from the libnss3-tools package) can be used to create a set of NSS DB files with the following command:

mkdir -p /tmp/browser1/.pki/nssdb
certutil \
  -N \
  -d sql:/tmp/browser1/.pki/nssdb \
  --empty-password

The PKCS#12 file can then be imported using this command:

pk12util \
  -i client-cert.p12 \
  -d sql:/tmp/browser1/.pki/nssdb \
  -W p12exportpass

Once imported, trust flags must be set to configure the CA certificate as a trusted CA imported.

certutil \
  -M \
  -d sql:/tmp/browser1/.pki/nssdb \
  -n "SomeCA" \
  -t "C,,"

Certificate Selection

With the NSS DB files populated the Chromium browser can now be started and have access to the required certificates and keys.

However when prompted by the server for a client certificate for the first time a browser in headful mode will prompt the user interactively to select the certificate they wish to use for the connection; in headless mode the connection fails. In my testing the browser will not automatically select a certificate in headful or headless mode even if only one certificate is available. Additionally there does not appear to be anyway to use CDP to select the certificate or interact with the browser prompt.

Luckily the Chromium browser honours the enterprise policies, and one of those (https://chromeenterprise.google/intl/en_uk/policies/#AutoSelectCertificateForUrls) allows an administrator to configure the certificate which should be used based on the URL.

One limitation of this is that it is a system level policy, therefore you cannot configure a different policy per browser – which is a problem if you need to support independently testing multiple different applications which require mTLS at the same time, or if you want to test the same application with two different mTLS certificates.

For Chromium, the policy file can have any name with a json extension but must be under /etc/chromium/policies/managed and should be world readable but only writable by administrator/root (i.e. chmod 644). I use /etc/chromium/policies/managed/mtls.json

Refer to https://www.chromium.org/administrators/linux-quick-start/ for further information on configuring policies and the differences between Chrome and Chromium and on different operating systems.

Depending on your exact use case, you may be able to update the policy file to specify the URL pattern and certificate to use – however that isn’t likely to be viable in a highly concurrent environment.

Alternatively you may be able to launch the browser in a dedicated docker container and mount the required policy file and NSS DB.

For my use case I must launch multiple browsers in the same docker container but I am only likely to require a single mTLS certificate per browser instance. This means at the system level I can configure a policy to match any HTTPS URL and leave the filter blank; the browser then selects any certificate from the NSS DB. By using a unique directory for the NSS DB for each browser (by configuring a unique directory path in the HOME environment variable when starting the browser) I can effectively force the browser instance to automatically select my desired certificate (the only one available in the NSS DB) without modifying the policy document for each new certificate or URL I need to handle.

{
  "AutoSelectCertificateForUrls": [
    {
      "pattern": "https://*",
      "filter": {}
    }
  ]
}

With all of this configuration in place, the Chromium browser in new headless mode will automatically select the certificate from the NSS DB when directed to a URL and the server sends a CertificateRequest.

Validate

To quickly confirm that everything is setup correctly, without needing to use CDP, I use the following command to load the application (e.g. https://example.com) which requires mTLS in the browser and dump the DOM – this lets me easily see if the application was successfully loaded and therefore determine that the mTLS certificate was used.

HOME=/tmp/browser1 \
chromium \
  --headless \
  --disable-gpu \
  --no-sandbox \
  --disable-software-rasterizer \
  --disable-extensions \
  --disable-background-networking \
  --disable-default-apps \
  --disable-popup-blocking \
  --disable-translate \
  --disable-sync \
  --metrics-recording-only \
  --no-first-run \
  --no-default-browser-check \
  --disable-dev-shm-usage \
  --disable-background-timer-throttling \
  --disable-renderer-backgrounding \
  --mute-audio \
  --hide-scrollbars \
  --window-size=1920,1080 \
  --user-data-dir="/tmp/browser1" \
  --dump-dom \
  "https://example.com"