1.1.4 Running a worker node

The worker node registers with the web node and is then used for executing builds and performing resource checks. It doesn't really decide much on its own.

Prerequisites

  • Linux: kernel v3.19 or later with support for user namespaces enabled. (This is off by default in some distributions!)

    To enforce memory limits on tasks, memory + swap accounting must be enabled.

  • Windows/Darwin: no special requirements (that we know of).

    Note that containerization is fairly primitive on these two platforms, so don't expect great support for multi-tenancy.

Running

The concourse CLI can run as a worker node via the worker subcommand.

First, you'll need to configure a directory for the worker to store data:

CONCOURSE_WORK_DIR=/opt/concourse/worker

This is where all the builds run, and where all resources are fetched in to, so make sure it's backed by enough storage. Then, configure it like so:

Next, point the worker at your web node like so:

CONCOURSE_TSA_HOST=127.0.0.1:2222
CONCOURSE_TSA_PUBLIC_KEY=path/to/tsa_host_key.pub
CONCOURSE_TSA_WORKER_PRIVATE_KEY=path/to/worker_key

Finally, run:

# run with -E to forward env config, or just set it all as root
sudo -E concourse worker

Note that the worker must be run as root, as it orchestrates containers.

All logs will be emitted to stdout, with any panics or lower-level errors being emitted to stderr.

Properties

CPU usage: almost entirely subject to pipeline workloads. More resources configured will result in more checking, and in-flight builds will use as much CPU as they want.

Memory usage: also subject to pipeline workloads. Expect usage to increase with the number of containers on the worker and spike as builds run.

Bandwidth usage: again, almost entirely subject to pipeline workloads. Expect spikes from periodic checking, though the intervals should spread out over enough time. Resource fetching and pushing will also use arbitrary bandwidth.

Disk usage: arbitrary data will be written as builds run, and resource caches will be kept and garbage collected on their own life cycle. We suggest going for a larger disk size if it's not too much trouble. All state on disk must not outlive the worker itself; it is all ephemeral. If the worker is re-created (i.e. fresh VM/container and all processes were killed), it should be brought back with an empty disk.

Highly available: not applicable. Workers are inherently singletons, as they're being used as drivers running entirely different workloads.

Horizontally scalable: yes; workers directly correlate to your capacity required by however many pipelines, resources, and in-flight builds you want to run. It makes sense to scale them up and down with demand.

Outbound traffic:

  • external traffic to arbitrary locations as a result of periodic resource checking and running builds

  • external traffic to the web node's configured external URL when downloading the inputs for a fly execute

  • external traffic to the web node's TSA port (2222) for registering the worker

Inbound traffic:

  • various connections from the web node on port 7777 (Garden), 7788 (BaggageClaim), and 7799 (garbage collection)

  • repeated connections to 7777 and 7788 from the web node's TSA component as it heartbeats to ensure the worker is healthy

Operation

The worker nodes are designed to be stateless and as interchangeable as possible. Tasks and Resources bring their own Docker images, so you should never have to install dependencies on the worker.

In Concourse, all important data is represented by Resources, so the workers themselves are dispensible. Any data in the work-dir is ephemeral and should go away when the worker machine is removed - it should not be persisted between worker VM or container re-creates.

Scaling Workers

More workers should be added to accomodate more pipelines. To know when this is necessary you should probably set up Metrics and keep an eye on container counts. If it's starting to approach 200 or so, you should probably add another worker. Load average is another metric to keep an eye on.

To add a worker, just create another machine for the worker and follow the Running instructions again.

Note: it doesn't really make sense to run multiple workers on one machine, since they'll both just be contending for the same physical resources. Workers should be given their own VMs or physical machines.

Worker Heartbeating & Stalling

Workers will continuously heartbeat to the Concourse cluster in order to remain registered and healthy. If a worker hasn't checked in in a while, possibly due to a network partition, being overloaded, or having crashed, its state will transition to stalled and new workloads will not be scheduled on it until it recovers.

If the worker remains in this state and cannot be recovered, it can be removed using the fly prune-worker command.

Restarting a Worker

Workers can be restarted in-place by sending SIGTERM to the worker process and starting it back up. Containers will remain running and Concourse will reattach to builds that were in flight.

This is a pretty aggressive way to restart a worker, and may result in errored builds - there are a few moving parts involved and we're still working on making this airtight.

A safer way to restart a worker is to land it by sending SIGUSR1 to the worker process. This will switch the worker to landing state, and Concourse will stop scheduling new work on it. When all builds running on the worker have finished, the process will exit.

You may want to enforce a timeout for draining - that way a stuck build won't prevent your workers from being upgraded. This can be enforced by common tools like start-stop-daemon:

start-stop-daemon \
  --pidfile worker.pid \
  --stop \
  --retry USR1/300/TERM/15/KILL

This will send SIGUSR1, wait up to 5 minutes, and then send SIGTERM. If it's still running, it will be killed after an additional 15 seconds.

Once the timeout is enforced, there's still a chance that builds that were running will continue when the worker comes back.

Gracefully Removing a Worker

When a worker machine is going away, it should be retired. This is similar to landing, except at the end the worker is completely unregistered, along with its volumes and containers. This should be done when a worker's VM or container is being destroyed.

To retire a worker, send SIGUSR2 to the worker process. This will switch the worker to retiring state, and Concourse will stop scheduling new work on it. When all builds running on the worker have finished, the worker will be removed and the worker process will exit.

Just like with landing, you may want to enforce a timeout for draining - that way a stuck build won't prevent your workers from being upgraded. This can be enforced by common tools like start-stop-daemon:

start-stop-daemon \
  --pidfile worker.pid \
  --stop \
  --retry USR2/300/TERM/15/KILL

This will send SIGUSR2, wait up to 5 minutes, and then send SIGTERM. If it's still running, it will be killed after an additional 15 seconds.

Configuration

If there's something special about your worker and you'd like to target builds at it specifically, you can configure tags like so:

CONCOURSE_TAG=tag-1,tag-2

A tagged worker is taken out of the default placement logic. To run build steps on it, specify the tags step modifier. Or, to perform resource checks on it, specify tags on the resource itself.

Configuring gdn server

On Linux, the concourse binary is packaged alongside a gdn binary. This binary is used for running Guardian, which is a Garden backend implementation which runs containers via runc (the same technology underlying tools like Docker).

The concourse worker command automatically configures and runs gdn, but depending on the environment you're running Concourse in, you may need to pop open the hood and configure a few things.

The gdn server can be configured in two ways:

  1. By creating a config.ini file and passing it as --garden-config (or CONCOURSE_GARDEN_CONFIG).

    The .ini file should look something like this:

    [server]
    flag-name = flag-value
    

    To learn which flags can be set, consult gdn server --help. Each flag listed can be set under the [server] heading.

  2. By setting CONCOURSE_GARDEN_* environment variables.

    This is primarily supported for backwards compatibility, and these variables are not present in concourse web --help. They are translated to flags passed to gdn server by lower-casing the * portion and replacing underscores with hyphens.

Troubleshooting and fixing DNS resolution

By default, containers created by Guardian will carry over the /etc/resolv.conf from the host into the container. This is often fine, but some Linux distributions configure a special 127.x.x.x DNS resolver (e.g. systemd-resolved).

When Guardian copies the resolv.conf over, it removes these entries, as they won't be reachable from the container's network namespace. As a result, your containers may not have any valid nameservers configured.

To diagnose this problem you can fly intercept into a failing container and check which nameservers are in /etc/resolv.conf:

$ fly -t ci intercept -c concourse/concourse
bash-5.0# grep nameserver /etc/resolv.conf
bash-5.0#

In this case it is empty, as the host only listed a single 127.0.0.53 address which was then stripped out. To fix this, you'll just need to explicitly configure DNS instead of relying on the gdn default behavior.

Pointing to external DNS servers

If you have no need for special DNS resolution within your Concourse containers, you can just configure your containers to use specific DNS server addresses external to the VM.

This can be done by listing DNS servers in config.ini like so:

[server]
; configure Google DNS
dns-server = 8.8.8.8
dns-server = 8.8.4.4

To validate whether the changes have taken effect, you can fly intercept into any container and check /etc/resolv.conf once again:

$ fly -t ci intercept -c concourse/concourse
bash-5.0# cat /etc/resolv.conf
nameserver 8.8.8.8
nameserver 8.8.4.4
bash-5.0# ping google.com
PING google.com (108.177.111.139): 56 data bytes
64 bytes from 108.177.111.139: seq=0 ttl=47 time=2.672 ms
64 bytes from 108.177.111.139: seq=1 ttl=47 time=0.911 ms

Using a local DNS server

If you would like to use Consul, dnsmasq, or some other DNS server running on the worker VM, you'll have to configure the internal address of the VM as the DNS server and allow the containers to reach the address, like so:

[server]
; internal IP of the worker machine
dns-server = 10.1.2.3

; allow containers to reach the above IP
allow-host-access = true

Setting allow-host-access will, well, allow containers to access your host VM's network. If you don't trust your container workloads, you may not want to allow this.

To validate whether the changes have taken effect, you can fly intercept into any container and check /etc/resolv.conf once again:

$ fly -t ci intercept -c concourse/concourse
bash-5.0# cat /etc/resolv.conf
nameserver 10.1.2.3
bash-5.0# nslookup concourse-ci.org
Server:         10.1.2.3
Address:        10.1.2.3#53

Non-authoritative answer:
Name:   concourse-ci.org
Address: 185.199.108.153
Name:   concourse-ci.org
Address: 185.199.109.153
Name:   concourse-ci.org
Address: 185.199.110.153
Name:   concourse-ci.org
Address: 185.199.111.153

If nslookup times out or fails, you may need to open up firewalls or security group configuration so that the worker VM can send UDP/TCP packets to itself.