Trusting Certificates With BOSH

January 21, 2019

Cloud Foundry

In any on-premise deployment there are usually a bunch of self-signed certificate authorities (CA) that need to be trusted from all VMs in the BOSH deployment (and the jobs running on the VMs). This blog post sheds some light on where to provide CA bundles in order to establish trust in BOSH deployments.

Trusting CAs in a BOSH Deployment

First, let's look at adding a CA bundle to the trust store of all VMs deployed by BOSH, as this is the most obvious use case. The CA bundle can in fact be added to the BOSH director configuration so that BOSH will inject the CA bundle to all deployments automatically.

In order to distribute the CA bundle we can configure the director.trusted_certs property when creating the BOSH environment. The property accepts a PEM-encoded certificate bundle as string, so any number of certificates can be added through this property.

To set the property, we need to create an operations file as follows:

# add-trusted-certs.ops.yml
- type: replace
  path: /instance_groups/name=bosh/properties/director/trusted_certs?
  value: ((trusted_certs))

Then we can use the operations file during the bosh create-env command:

bosh create-env bosh.yml \
    -o add-trusted-certs.ops.yml \
    -l trusted-certs.vars.yml \
    # ...

and supply the actual certificates through a variable file:

# trusted-certs.vars.yml
trusted_certs: |
    -----BEGIN CERTIFICATE-----
    MIIDdTCCAl2gAwIBAgILBAAAAAABFUtaw5QwDQYJKoZIhvcNAQEFBQAwVzELMAkGA1UEBhMCQkUx
    ...
    X4XSQRjbgbMEHMUfpIBvFSDJ3gyICh3WZlXi/EjJKSZp4A==
    -----END CERTIFICATE-----
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----

Alternatively, you could also store the certificates in Credhub and reference the Credhub credential name as BOSH variable. The vars file then boils down to:

# trusted-certs.vars.yml
trusted_certs: ((trusted_ca_certs))

And you can set the certificate in Credhub with:

credhub -n /concourse/<pipeline>/trusted_ca_certs -t value -v "$(cat certs.pem)"

Setting the property either way will instruct the BOSH director to inject the CA bundle into the trust store of every BOSH-managed VM. You may find and inspect the trusted certificates on any BOSH-managed VM at the following location:

$ ll /usr/local/share/ca-certificates
-rw-r--r-- 1 root staff 2516 Dec  9 09:46 bosh-trusted-cert-1.crt
-rw-r--r-- 1 root staff 2252 Dec  9 09:46 bosh-trusted-cert-2.crt

Trusting CAs for the BOSH Director VM

The above BOSH director property director.trusted_certs will only provide the CA bundle to all BOSH-managed VMs in the deployments, but not to the BOSH director VM itself. This seems strange first, but makes sense as the bosh create-env command spins up the BOSH director VM using a locally installed CPI for your IaaS of choice.

Adding a CA bundle to the BOSH director may get relevant when releases or stemcells should be downloaded from other internal systems that use self-signed internal CAs, for example. In order to configure a CA bundle for the BOSH director VM we can leverage the os-conf-release release, which is used to customize the VM operating system with certain jobs. One of those jobs can be used to actually trust a CA bundle. The following operations file will add the CA bundle to the BOSH director VM using the os-conf BOSH release:

# add-trusted-certs-to-director-vm.ops.yml
- type: replace
  path: /releases/name=os-conf?
  value:
    name: os-conf
    version: ((osconf_version))
    url: https://bosh.io/d/github.com/cloudfoundry/os-conf-release?v=((osconf_version))
    sha1: ((osconf_sha1))

- type: replace
  path: /instance_groups/name=bosh/jobs/-
  value:
    name: ca_certs
    release: os-conf
    properties:
      certs: ((trusted_certs))

Again, we need to include this operations file during the bosh create-env command and load the trusted certificates from our variable file as follows:

bosh create-env bosh.yml \
    -o add-trusted-certs-to-director-vm.ops.yml \
    -l trusted-certs.vars.yml \
    # ...

Trusting CAs Within Cloud Foundry

Cloud Foundry application containers can also trust certificate bundles if those are provided to the cf-deployment during bosh deploy. The cf-deployment repository already provides a ready-to-use operations file that we can add in order to inject the CA bundle into the Cloud Foundry rootfs:

bosh ... deploy -d cloudfoundry cf-deployment.yml \
    -o operations/use-trusted-ca-cert-for-apps.yml \
    -l operations/example-vars-files/vars-use-trusted-ca-cert-for-apps.yml \
    # ...

By supplying the certificates, the CA bundles are added to every Cloud Foundry application container and thus establish the trust as expected.

Trusting CAs Within Concourse

Concourse VMs do have the BOSH trusted certificates in place, however the Concourse jobs and tasks are launched within a Garden container and do not have access to the BOSH-deployed certificates and thus cannot trust them.

Concourse v3.9.0 introduced certificate propagation for Concourse resources (so not for Concourse tasks!). At least for the resources certificates trusted by BOSH are automatically propagated to the containers pulling in resources.

However, for concourse tasks running in your image of choice you need to supply the certificates by other means and make sure that commands pick up the correct certificates. Usually, this means setting a CA bundle through task parameters, which are then available as environment variables within jobs. Depending on the tooling, the CA bundle in the environment variable is then passed as is or written to a file. You may find lots of real-world examples in the pcf-pipelines repository.

In a nutshell, you supply a custom CA bundle in a Concourse task by first adding a parameter to the task definition:

# my-task.yml

# ...
params:
- CA_BUNDLE:

And then using that within the task script to pass the bundle on the actual tools:

#!/bin/sh

# use the double-quotes to preserve line endings
echo "$CA_BUNDLE" > /ca-bundle.pem
export CA_BUNDLE_FILE='/ca-bundle.pem'

# then use the file to supply the certs to other tools
aws --ca-bundle /ca-bundle.pem s3 ls ...

Using this setup CA bundles can easily be distributed in Concourse tasks, as the actual bundle value is configured by the pipeline. This means that task authors can write generic tasks that even make supplying CA bundles optional by calling the tools appropriately.

Conclusion

Establishing trust through custom certificates in BOSH-managed deployments is easily doable when providing certificate bundles at the right location. By following the techniques of this blog post, skipping SSL validation for a specific tool to talk to a certain target should be a relict of the past in any BOSH deployment!