Examples

Docker Compose, Vault, cert auth

Configuring Vault with TLS cert-based auth involves a few moving parts. The following example is not really meant for production, but hopefully it makes everything easier to understand by seeing how all the parts fit together.

First, grab the Docker Compose quick-start:

$ wget https://concourse-ci.org/docker-compose.yml

Next, let's generate all the certificates using certstrap like so:

certstrap init --cn vault-ca
certstrap request-cert --domain vault --ip 127.0.0.1
certstrap sign vault --CA vault-ca
certstrap request-cert --cn concourse
certstrap sign concourse --CA vault-ca
mv out vault-certs

Next we'll configure the Vault server to use the certs we made by creating vault-config/config.hcl:

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_cert_file = "/vault/certs/vault.crt"
  tls_key_file = "/vault/certs/vault.key"
}

storage "file" {
  path = "/vault/file"
}

Next, let's create a docker-compose.override.yml override that will add Vault and configure cert-based auth:

version: '3'

services:
  concourse:
    volumes:
    - ./vault-certs:/vault-certs
    environment:
      CONCOURSE_VAULT_URL: https://vault:8200
      CONCOURSE_VAULT_AUTH_BACKEND: cert
      CONCOURSE_VAULT_CA_CERT: /vault-certs/vault-ca.crt
      CONCOURSE_VAULT_CLIENT_CERT: /vault-certs/concourse.crt
      CONCOURSE_VAULT_CLIENT_KEY: /vault-certs/concourse.key

  vault:
    image: vault
    cap_add: [IPC_LOCK]
    ports: ["8200:8200"]
    volumes:
    - ./vault-certs:/vault/certs
    - ./vault-config:/vault/config
    command: server

From here, we just need to spin up the cluster:

$ docker-compose up

And...now everything should start blowing up! Vault is still sealed, so web node can't log in.

Let's initialize Vault:

$ export VAULT_CACERT=$PWD/vault-certs/vault-ca.crt
$ vault operator init

Make note of the 5 unseal keys and the root token, then run the following:

$ vault operator unseal # paste unseal key 1
$ vault operator unseal # paste unseal key 2
$ vault operator unseal # paste unseal key 3
$ vault login           # paste root token

At this point Vault is unsealed and ready to go, except we haven't configured the cert backend yet.

First let's create a policy for Concourse, concourse-policy.hcl:

path "concourse/*" {
  policy = "read"
}

Save this to concourse-policy.hcl, and then run:

$ vault policy write concourse ./concourse-policy.hcl
Success! Uploaded policy: concourse
$ vault auth enable cert
Success! Enabled cert auth method at: cert/
$ vault write auth/cert/certs/concourse \
    policies=concourse \
    certificate=@vault-certs/vault-ca.crt \
    ttl=1h

At this point the web node should be able to log in successfully.

1.9.2.1 The Vault credential manager

Concourse can be configured to pull credentials from a Vault instance.

To configure this, first configure the URL of your Vault server by setting the following env on the web node:

CONCOURSE_VAULT_URL=https://vault.example.com:8200

You may also need to configure the CA cert for Vault:

CONCOURSE_VAULT_CA_CERT=path/to/ca.crt

You'll also need to configure how the web node authenticates with Vault - see Authenticating with Vault for more details as that step is quite involved.

Credential lookup rules

When resolving a parameter such as ((foo_param)), Concourse will look in the following paths, in order:

  • /concourse/TEAM_NAME/PIPELINE_NAME/foo_param

  • /concourse/TEAM_NAME/foo_param

Vault credentials are actually key-value, so for ((foo)) Concourse will default to the field name value. You can specify the field to grab via . syntax, e.g. ((foo.bar)).

If the action is being run in the context of a pipeline (e.g. a check or a step in a build of a job), Concourse will first look in the pipeline path. If it's not found there, it will look in the team path. This allows credentials to be scoped widely if they're common across many pipelines.

If an action is being run in a one-off build, Concourse will only look in the team path.

The leading /concourse can be changed by specifying the following:

CONCOURSE_VAULT_PATH_PREFIX=/some-other-prefix

Configuring the secrets engine

Concourse is currently limited to looking under a single path, meaning only one secrets engine is supported: kv, version 1. This may change in the future - we're still collecting ideas in (RF)RFC #5.

So, let's configure the kv secrets engine and mount it at /concourse:

$ vault secrets enable -version=1 -path=concourse kv

Next, you'll want to create a policy to allow Concourse to read from this path.

path "concourse/*" {
  policy = "read"
}

Save this to concourse-policy.hcl, and then run:

vault policy write concourse ./concourse-policy.hcl

This configuration will allow Concourse to read all credentials under /concourse. This should match your configured path prefix.

Authenticating with Vault

There are many ways to authenticate with a Vault server. The web-node can be configured with either a token or an arbitrary auth backend and arbitrary auth params, so just about all of them should be configurable.

When the web node acquires a token, either by logging in with an auth backend or by being given one directly, it will continuously renew the token to ensure it doesn't expire. The renewal interval is half of the token's lease duration.

Using a periodic token

The simplest way to authenticate is by generating a periodic token:

$ vault token create --policy concourse --period 1h
Key                Value
---                -----
token              s.mSNnbhGAqxK2ZbMasOQ91rIA
token_accessor     0qsib5YcYvROm86cT08IFxIT
token_duration     1h
token_renewable    true
token_policies     [concourse default]

Choose your --period wisely, as the timer starts counting down as soon as the token is created. You should also use a duration long enough to account for any planned web node downtime.

Once you have the token, just set the following env on the web node:

CONCOURSE_VAULT_CLIENT_TOKEN=s.mSNnbhGAqxK2ZbMasOQ91rIA

Periodic tokens are the quickest way to get started, but they have one fatal flaw: if the web node is down for longer than the token's configured period, the token will expire and a new one will have to be created and configured. This can be avoided by using the approle auth backend.

Using the approle auth backend

The approle backend allows for an app (in this case, Concourse) to authenticate with a role pre-configured in Vault.

With this backend, the web node is configured with a role_id corresponding to a pre-configured role, and a secret_id which is used to authenticate and acquire a token.

The approle backend must first be configured in Vault. Vault's approle backend allows for a few parameters which you may want to set to determine the permissions and lifecycle of its issued tokens:

policies=names

This determines the policies (comma-separated) to set on each token. Be sure to set one that has access to the secrets path - see Configuring the secrets engine for more information.

token_ttl=duration

This determines the TTL for each token granted. The token can be continuously renewed, as long as it is renewed before the TTL elapses.

token_max_ttl=duration

This sets a maximum lifetime for each token, after which the token can no longer be renewed.

If configured, be sure to set the same value on the web node so that it can re-auth before this duration is reached:

CONCOURSE_VAULT_AUTH_BACKEND_MAX_TTL=1h
period=duration

If configured, tokens issued will be periodic. Periodic tokens are not bound by any configured max TTL, and can be renewed continuously. It does not make sense to configure both period and token_max_ttl as the max TTL will be ignored.

token_num_uses=count

This sets a limit on how often a token can be used. We do not recommend setting this value, as it will effectively hamstring Concourse after a few credential acquisitions. The web node does not currently know to re-acquire a token when this limit is reached.

secret_id_ttl=duration and secret_id_num_uses=count

These two configurations will result in the secret ID expiring after the configured time or configured number of log-ins, respectively.

You should only set these if you have something periodically re-generating secret IDs and re-configuring your web nodes accordingly.

Given all that, a typical configuration may look something like this:

$ vault auth enable approle
Success! Enabled approle auth method at: approle/
$ vault write auth/approle/role/concourse policies=concourse period=1h
Success! Data written to: auth/approle/role/concourse

Now that the backend is configured, we'll need to obtain the role_id and generate a secret_id:

$ vault read auth/approle/role/concourse/role-id
Key        Value
---        -----
role_id    5f3420cd-3c66-2eff-8bcc-0e8e258a7d18
$ vault write -f auth/approle/role/concourse/secret-id
Key                   Value
---                   -----
secret_id             f7ec2ac8-ad07-026a-3e1c-4c9781423155
secret_id_accessor    1bd17fc6-dae1-0c82-d325-3b8f9b5654ee

These should then be set on the web node like so:

CONCOURSE_VAULT_AUTH_BACKEND="approle"
CONCOURSE_VAULT_AUTH_PARAM="role_id:5f3420cd-3c66-2eff-8bcc-0e8e258a7d18,secret_id:f7ec2ac8-ad07-026a-3e1c-4c9781423155"

Using the cert auth backend

The cert auth method allows authentication using SSL/TLS client certificates.

With this backend, the web node is configured with a client cert and a client key. Vault must be configured with TLS, which you should be almost certainly be doing anyway.

The cert backend must first be configured in Vault. The backend is associated to a policy and a CA cert used to verify the client certificate. It may also be given the client certificate itself.

The cert backend must first be configured in Vault. Vault's cert backend allows for a few parameters which you may want to set to determine the lifecycle of its issued tokens:

policies=names

This determines the policies (comma-separated) to set on each token. Be sure to set one that has access to the secrets path - see Configuring the secrets engine for more information.

ttl=duration

This determines the TTL for each token granted. The token can be continuously renewed, as long as it is renewed before the TTL elapses.

max_ttl=duration

This sets a maximum lifetime for each token, after which the token can no longer be renewed.

If configured, be sure to set the same value on the web node so that it can re-auth before this duration is reached:

CONCOURSE_VAULT_AUTH_BACKEND_MAX_TTL=1h
period=duration

If configured, tokens issued will be periodic. Periodic tokens are not bound by any configured max TTL, and can be renewed continuously. It does not make sense to configure both period and max_ttl as the max TTL will be ignored.

$ vault auth enable cert
Success! Enabled cert auth method at: cert/
$ vault write auth/cert/certs/concourse policies=concourse certificate=@out/vault-ca.crt ttl=1h
Success! Data written to: auth/cert/certs/concourse

Once that's all set up, you'll just need to configure the client cert and key on the web node like so:

CONCOURSE_VAULT_AUTH_BACKEND="cert"
CONCOURSE_VAULT_CLIENT_CERT=vault-certs/concourse.crt
CONCOURSE_VAULT_CLIENT_KEY=vault-certs/concourse.key

In this case no additional auth params are necessary, as the Vault's TLS auth backend will check the certificate against all roles if no name is specified.

Configuring credential caching

By default, credentials are fetched each time they're used. This can add up when many pipelines are configured, resulting in a ton of requests to Vault.

To reduce load on your Vault server you may want to enable caching, by setting the following env on the web node:

CONCOURSE_VAULT_CACHE=true

When enabled, credentials are cached for half of their lease duration.

To set an upper bound and force cache busting after a certain amount of time, set the following env:

CONCOURSE_VAULT_MAX_LEASE=1m

With this set, credentials will only be cached for up to 1 minute.