HashiCorp Vault in Cloud Foundry environment

December 15, 2017
by Zoltan Altfatter

In this blog post, we will describe how you could configure HashiCorp Vault in a Cloud Foundry environment. In Cloud Foundry developers provision service instances and then bind those service instances to an application. The service broker is responsible to provide those service instances by interacting with the Cloud Controller. The service brokers advertise a catalog of service offerings and service plans in the marketplace (e.g. a single node Vault plan or a clustered multinode Vault plan). They also act on requests from the marketplace for provisioning, binding, unbinding, and de-provisioning service instances.

We will leverage the official HashiCorp Vault broker integration which you can find here. The HashiCorp Vault Service Broker does not run a Vault server for you, we need to setup the Vault Cluster ourself. However, in this blog post we will just demonstrate the integration using a single node Vault server running on our laptop exposed via ngrok since we will use Pivotal Cloud Foundry to deploy our example client application.

After installing Vault (with homebrew is as simple as brew install vault) let’s start it with

vault server -config inmemory.conf

where inmemory.conf is

backend "inmem" {
}

listener "tcp" {
 address = "0.0.0.0:8200"
 tls_disable = 1
}

In another terminal, in order to connect to it, we need to set the VAULT_ADDR environment variable

export VAULT_ADDR=http://127.0.0.1:8200

After Vault is started we need to initialize it.

vault init -key-shares=5 -key-threshold=2

Then setting the VAULT_TOKEN to the previously printed out Initial Root Token we can unseal the Vault specifying 2 keys from the previously printed out 5 keys.

vault unseal <key>
vault unseal <key>

After our Vault in unsealed we can write a secret into the **secret **generic backend

vault write secret/vault-demo message='I find your lack of faith disturbing.'

and with a sample application we can verify that we can connect the Vault and print out the secret

export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=<token>
java -jar target/cloudfoundry-with-vault-demo-0.0.1-SNAPSHOT.jar --spring.cloud.vault.token=`echo $VAULT_TOKEN`

The application you can find in this GitHub repository. It is using the spring-cloud-vault project which makes integrating with Vault very easy. The application has a simple REST endpoint which just simply returns the secret.

@RefreshScope
@RestController
class MessageController {

    @Value("${message:n/a}")
    String message;

    @RequestMapping("/")
    String getMessage() {
        return "message=" + message;
    }
}

Let’s verify that it works.

http :8080
message=I find your lack of faith disturbing.

And indeed we get the secret printed out to the console. If we update the secret inside Vault with the command

vault write secret/vault-demo message='Now, young Skywalker, you will die.'

and request the secret with the endpoint again we see that the secret is not updated since it was cached inside the application. We need to tell the application to fetch the secret again from Vault. This we can easily do with the **refresh **actuator endpoint

http post :8080/actuator/refresh

and after that, we will get the updated secret

http :8080
message='Now, young Skywalker, you will die.'

So far so good, let’s deploy the application to Pivotal Cloud Foundry. In order to achieve this, first we need to get the official Vault broker integration and deploy to Pivotal Cloud Foundry

cf login -a api.run.pivotal.io
git clone https://github.com/hashicorp/vault-service-broker
cf push my-vault-broker-service -m 256M --random-route --no-start

The –no-start flag makes sure it is not started after it is deployed since we need to setup the Vault integration first. With ngrok we can easily expose our locally running Vault

ngrok http 8200
Forwarding http://3db1eef2.ngrok.io -> localhost:8200
Forwarding https://3db1eef2.ngrok.io -> localhost:8200

ngrok provides also a web interface (http://localhost:4040) where we can inspect the received HTTP requests.

Next, let’s set the following environment variables

VAULT_ADDR=<ngrok_url>
VAULT_TOKEN=<token>
VAULT_USERNAME=vault
VAULT_PASSWORD=secret

The broker is configured to use basic authentication.

We needed to also change the DefaultServiceId and DefaultServiceName in the main.go file to be a unique value otherwise when creating the service broker with the **cf create-service-broker **we get an error. You can find more details about this here

Next after setting the needed env variables for the service broker application via

$ cf set-env my-vault-broker-service VAULT_ADDR ${VAULT_ADDR}
$ cf set-env my-vault-broker-service VAULT_TOKEN ${VAULT_TOKEN}
$ cf set-env my-vault-broker-service SECURITY_USER_NAME ${VAULT_USERNAME}
$ cf set-env my-vault-broker-service SECURITY_USER_PASSWORD ${VAULT_PASSWORD}

We can start the broker

$ cf start my-vault-broker-service

We make sure the broker starts successfully by checking the logs

$ cf logs --recent my-vault-broker-service

We notice that the Vault received HTTP requests with the help of the ngrok Inspect UI

GET /v1/sys/mounts
PUT /v1/auth/token/renew-self
POST /v1/sys/mounts/cf/broker
GET /v1/cf/broker

As we can see the service broker created a new mount (cf/broker) which we can verify with the vault mounts command.

In order to get the catalog information we can use this command

$ VAULT_BROKER_URL=$(cf app my-vault-broker-service | grep routes: | awk '{print $2}')
$ curl ${VAULT_USERNAME}:${VAULT_PASSWORD}@${VAULT_BROKER_URL}/v2/catalog | jq
{
  "services": [
    {
      "id": "42ff1ff1-244d-413a-87ab-b2334b801134",
      "name": "my-hashicorp-vault",
      "description": "HashiCorp Vault Service Broker",
      "bindable": true,
      "tags": [
        ""
      ],
      "plan_updateable": false,
      "plans": [
        {
          "id": "42ff1ff1-244d-413a-87ab-b2334b801134.shared",
          "name": "shared",
          "description": "Secure access to Vault's storage and transit backends",
          "free": true
        }
      ]
    }
  ]
}

After creating the service broker

$ cf create-service-broker my-vault-service-broker "${VAULT_USERNAME}" "${VAULT_PASSWORD}" "https://${VAULT_BROKER_URL}" --space-scoped

we can verify that the service my-hashicorp-vault is available in the marketplace

$ cf marketplace
service              plans             description
my-hashicorp-vault   shared            HashiCorp Vault Service Broker

When creating a service instance

cf create-service my-hashicorp-vault shared my-vault-service

we see activity again in the ngrok Inspect UI

POST /v1/sys/mounts/cf/0b24f466-9a54-4215-852e-2bcfab428a82/secret
PUT /v1/cf/broker/0b24f466-9a54-4215-852e-2bcfab428a82
GET /v1/sys/mounts
POST /v1/sys/mounts/cf/0b24f466-9a54-4215-852e-2bcfab428a82/transit
POST /v1/sys/mounts/cf/be7eedf8-c813-49e1-98f8-2fc19370ee4d/secret
POST /v1/sys/mounts/cf/5f7b0811-d90a-47f2-a194-951eb324f867/secret
PUT /v1/sys/policy/cf-0b24f466-9a54-4215-852e-2bcfab428a82
PUT /v1/auth/token/roles/cf-0b24f466-9a54-4215-852e-2bcfab428a82

When  a new service instance is provisioned using the broker, the following paths will be mounted:

A policy named cf-<instance_id> is created for this service instance which grants read-only access to cf/<organization_id>/*, read-write access to cf<space_id>/* and full access to cf/<instance_id>/*.

Next, we need to create a service key

$ cf create-service-key my-vault-service my-vault-service-key

Again in the ngrok Inspect UI we can monitor the received requests

PUT /v1/auth/token/renew-self
PUT /v1/auth/token/renew-self
PUT /v1/cf/broker/0b24f466-9a54-4215-852e-2bcfab428a82/5cf104c9-4515-40f3-94de-a63ab77cb84b
POST /v1/auth/token/create/cf-0b24f466-9a54-4215-852e-2bcfab428a82

Finally, we can retrieve the credentials for this service instance which our demo application will use when we bind this service instance to the demo application

$ cf service-key my-vault-service my-vault-service-key

{
 "address": "https://1f81e0d3.ngrok.io/",
 "auth": {
  "accessor": "3705e5b2-c0bb-6398-ecff-e05a9e6a7b28",
  "token": "d5971c27-cf77-6ff0-f5c9-430fdfe07066"
 },
 "backends": {
  "generic": "cf/0b24f466-9a54-4215-852e-2bcfab428a82/secret",
  "transit": "cf/0b24f466-9a54-4215-852e-2bcfab428a82/transit"
 },
 "backends_shared": {
  "organization": "cf/be7eedf8-c813-49e1-98f8-2fc19370ee4d/secret",
  "space": "cf/5f7b0811-d90a-47f2-a194-951eb324f867/secret"
 }
}

In the client application, we need to do the following changes in the bootstrap.yml in order for the Vault token and backend to be used.

spring:
  application:
    name: vault-demo
  cloud:
    vault:
      token: ${vcap.services.my-vault-service.credentials.auth.token}
      uri: ${vcap.services.my-vault-service.credentials.address:http://localhost:8200}
      generic:
        backend: ${vcap.services.my-vault-service.credentials.backends.generic:secret}

Next, after deploying the demo application, binding the service instance to it and starting the demo application

cf push --random-route --no-start 
cf bind-service vault-demo my-vault-service
cf start vault-demo

let’s write a secret into the proper backend

$ vault write cf/0b24f466-9a54-4215-852e-2bcfab428a82/secret/vault-demo message='Vault Rocks'
$ http post https://vault-demo-twiggiest-sennit.cfapps.io/actuator/refresh

and we can verify that the secret is retrieved

$ http get http://vault-demo-twiggiest-sennit.cfapps.io
message=Vault Rocks

If you have got this far, congratulations you have a running Vault configuration in a Cloud Foundry environment.

About the author: Zoltan Altfatter

Zoltan Altfatter is a software craftsman at mimacom. He has been in software engineering for over 10 years. He is passionate about the JVM and the Spring ecosystem.

Comments
Join us
Are you creative and passionate about software development? Do you think unconventionally and act with initiative? Do you want to achieve great things within our team?