1.1.3 Running a web node

Ingress

If your web nodes are going to be accessed over the network, you will need to set CONCOURSE_EXTERNAL_URL to a URL accessible by your consumers. If you don't set this property, logins will redirect to its default value of 127.0.0.1.

If your instance is available on the public internet, you may wish to prevent the Concourse UI from being nefariously embedded as an iframe by setting CONCOURSE_X_FRAME_OPTIONS to deny (to prevent any iframe embeddings) or sameorigin (to only allow iframe embeddings in pages served from the same subdomain). This protects against clickjacking.

Note: If setting the value to allow-from, please note that not all browsers support this value and when not supported, the header is ignored by the browser.

Scaling

The web node can be scaled up for high availability. They'll also roughly share their scheduling workloads, using the database to synchronize. This is done by just running more web commands on different machines, and optionally putting them behind a load balancer.

To run a cluster of web nodes, you'll first need to ensure they're all pointing to the same PostgreSQL server.

Furthermore, you may wish to configure the max number of parallel database connections that each node makes. The number you choose should not be greater than the maximum number of simultaneous connections your Postgres server will allow. For example:

CONCOURSE_MAX_OPEN_CONNECTIONS=32

Next, you'll need to configure a peer address. This is a DNS or IP address that can be used to reach this web node from other web nodes. Typically this uses a private IP, like so:

CONCOURSE_PEER_ADDRESS=10.10.0.1

This address will be used for forwarded worker connections, which listen on the ephemeral port range.

Finally, if all of these nodes are going to be accessed through a load balancer, you'll need to configure the external URL that will be used to reach your Concourse cluster:

CONCOURSE_EXTERNAL_URL=https://ci.example.com

Aside from the peer URL, all configuration must be consistent across all web nodes in the cluster to ensure consistent results.

Restarting & Upgrading

The web nodes can be killed and restarted willy-nilly. No draining is necessary; if the web node was orchestrating a build it will just continue where it left off when it comes back or, or the build will be picked up by one of the other web nodes.

To upgrade a web node, stop its process and start a new one using the newly installed concourse. Any migrations will be run automatically on start. If web nodes are started in parallel, only one will run the migrations.

Note that we don't currently guarantee a lack of funny-business if you're running mixed Concourse versions - database migrations can perform modifications that confuse other web nodes. So there may be some turbulence during a rolling upgrade, but everything should stabilize once all web nodes are running the latest version.

Downgrading

If you're stuck in a pinch and need to downgrade from one version of Concourse to another, you can use the concourse migrate command.

First, grab the desired migration version by running the following:

# make sure this is the *old* Concourse binary
$ concourse migrate --supported-db-version
1551110547

That number (yours will be different) is the expected migration version for that version of Concourse.

Next, run the following with the new Concourse binary:

$ concourse migrate --migrate-db-to-version=1551110547

This will need the same CONCOURSE_POSTGRES_* configuration described in Running.

Once this completes, switch all web nodes back to the older concourse binary and you should be good to go.

Giving your cluster a name

If you've got many Concourse clusters that you switch between, you can make it slightly easier to notice which one you're on by giving each cluster a name:

CONCOURSE_CLUSTER_NAME=production

When set, this name will be shown in the top bar when viewing the dashboard.

TLS via Let's Encrypt

Concourse can be configured to automatically acquire a TLS certificate via Let's Encrypt:

# Enable TLS
CONCOURSE_TLS_BIND_PORT=443

# Enable Let's Encrypt
CONCOURSE_ENABLE_LETS_ENCRYPT=true

By default, Concourse will reach out to Let's Encrypt's ACME CA directory. An alernative URL can be configured like so:

CONCOURSE_LETS_ENCRYPT_ACME_URL=https://acme.example.com/directory

In order to negotiate the certificate, your web node must be reachable by the ACME server. There are intentionally no publicly listed IP addresses to whitelist, so this typically means just making your web node publicly reachable.

Enabling audit logs

A very simplistic form of audit logging can be enabled with the following vars:

# Enable auditing for all api requests connected to builds.
CONCOURSE_ENABLE_BUILD_AUDITING=true

# Enable auditing for all api requests connected to containers.
CONCOURSE_ENABLE_CONTAINER_AUDITING=true

# Enable auditing for all api requests connected to jobs.
CONCOURSE_ENABLE_JOB_AUDITING=true

# Enable auditing for all api requests connected to pipelines.
CONCOURSE_ENABLE_PIPELINE_AUDITING=true

# Enable auditing for all api requests connected to resources.
CONCOURSE_ENABLE_RESOURCE_AUDITING=true

# Enable auditing for all api requests connected to system transactions.
CONCOURSE_ENABLE_SYSTEM_AUDITING=true

# Enable auditing for all api requests connected to teams.
CONCOURSE_ENABLE_TEAM_AUDITING=true

# Enable auditing for all api requests connected to workers.
CONCOURSE_ENABLE_WORKER_AUDITING=true

# Enable auditing for all api requests connected to volumes.
CONCOURSE_ENABLE_VOLUME_AUDITING=true

When enabled, API requests will result in an info-level log line like so:

{"timestamp":"2019-05-09T14:41:54.880381537Z","level":"info","source":"atc","message":"atc.audit","data":{"action":"Info","parameters":{},"user":"test"}}
{"timestamp":"2019-05-09T14:42:36.704864093Z","level":"info","source":"atc","message":"atc.audit","data":{"action":"GetPipeline","parameters":{":pipeline_name":["booklit"],":team_name":["main"]},"user":"test"}}