By default Concourse is configured to feel very snappy. This is good for when you are first trying out Concourse or using it on a small team with a few dozen pipelines.
When you begin trying to scale Concourse is where fires can start breaking out. This section will go over some configuration values in Concourse that you can change to make scaling easier.
- 188.8.131.52 The Big Caveat
- 184.108.40.206 Build Logs
- 220.127.116.11 Resource Checking
- 18.104.22.168 Pipeline Management
- 22.214.171.124 Container Placement
- 126.96.36.199 Garbage Collection
- 188.8.131.52 Web To Worker Ratio
Track Metrics! Everything you read next could be all for nothing if you don't have metrics to track where the bottlenecks are in your Concourse system. We highly suggest tracking metrics so you have a clear before and after picture for any changes you make and to clearly see if you're moving things in the right direction.
Is the size of your database growing dramatically? Can't keep up with the storage costs? Then you should probably configure some default log retention settings.
By default Concourse will not delete any of your logs from your pipelines. You have to opt-in to having Concourse automatically delete build logs for you. You can set a time-based retention policy and/or a policy based on the number of logs a job generates.
Determines how many build logs to retain per job by default. If you set this to
10 then any jobs in your pipelines that have more than ten builds will have the extra logs for those builds deleted.
Users can override this value in their pipelines.
Determines how many build logs to retain per job. Users cannot override this setting.
Determines how old build logs have to be before they are deleted. Setting this to a value like
10 will result in any build logs older than 10 days to be deleted.
Users can override this value in their pipelines.
By default Concourse checks any given resource every ~1min. This makes Concourse feel snappy when you first start using it. Once you start trying to scale though the amount of checks can begin to feel aggressive. The following settings can help you reduce the load caused by resource checking.
This is where the default value for 1min checks comes from. Changing this value changes the default checking interval for all resources. Users can override this value when defining a resource with the
Same as the previous var but only applies to resources with webhooks. Could use this to disable resource checking of resources that use webhooks by setting it to a large value like
Maximum number of checks that can be started per second. This will be calculated as (# of resources)/(resource checking interval). If you're finding that too many resource checks are running at once and consuming a lot of resources on your workers then you can use this var to reduce the overall load.
A value of
-1 will remove this maximum limit of checks per second.
Here are some flags you can set on the web node to help manage the amount of resources pipelines consume. These flags are mostly about ensuring pipelines don't run forever without good reason.
This flag takes a number representing the number of days since a pipeline last ran before it's automatically paused. So specifying
90 means any pipelines that last ran 91 days ago will be automatically paused.
For large instances it can be common for users to set a pipeline and then forget about it. The pipeline may never run another job again and be forgotten forever. Even if the jobs in the pipeline never run Concourse will still be running resource checks for that pipeline, if any resources are defined. By setting this flag you can ensure that any pipelines that meet this criteria will be automatically paused and not consume resources long-term. For some large instances this can mean up to 50% of pipelines eventually being paused.
Global defaults for CPU and memory you can set. Only applies to tasks, not resource containers (
check/get/put steps). You can read more about how to set these limits on the
Users can override these values in their pipelines.
If you find that workers keep crashing due to high CPU and/or memory usage then you could try specifying a custom container placement strategy or strategy chain. The Container Placement page has some examples of container placement strategy chains you can use.
When jobs fail or error out in Concourse their resources are not immediately cleaned up. The container and storage space remain on a worker for some period of time before they get garbage collected. If you want to make the garbage collector more aggressive you can change the following settings on your web node:
This env var only applies to containers where the job failed and has the longest grace period among all the other GC grace periods. It has a default value of
120h (five days).
The reason the default value is so long is so users don't feel rushed to investigate their failed job. A job can fail over a weekend and users can investigate the failed jobs containers when they come back on Monday.
Failed containers get GC as soon as a new build of the job is kicked off. So you don't have to worry about failed containers always hanging around for five days. They'll only hang around for that long if they're the most recent build of a job.
If you notice a lot of containers and volumes hanging around that are tied to failed jobs you can try reducing this setting to fewer days or even a few hours.
Depending on what a container was used for and its exit condition, there are various flags you can adjust to make Concourse GC these resources faster or slower. The following env vars cover the cases where you probably don't need the container hanging around for very long. They have a default value of
CONCOURSE_GC_ONE_OFF_GRACE_PERIOD- Period after which one-off build containers will be garbage-collected
CONCOURSE_GC_MISSING_GRACE_PERIOD- Period after which containers and volumes that were created but went missing from the worker will be garbage-collected
CONCOURSE_GC_HIJACK_GRACE_PERIOD- Period after which hijacked containers will be garbage-collected
This is anecdotal and you should adjust based on your metrics of your web nodes. A starting ratio of web to workers is 1:6; one web instance for every six workers.
The core Concourse team runs two web nodes and 16 workers, a 1:8 ratio. We can get away with this lower web to worker ratio because we don't have that many users actively interacting with the web UI on a daily basis; less than 10 active users. Since we're only one team using the instance we have fewer pipelines than an instance supporting multiple teams would.