Resource Checker

Resources represent external state such as a git repository, an s3 bucket, or anything else that changes over time. By modelling these as resources, it allows you to use this external state as inputs (or triggers) to your workloads.

When are resources checked?

The component that schedules and runs the resource checks is called the resource checker. The rate at which these checks happen is called the check interval. There's an obvious tradeoff, whereby the more frequently you poll, the bigger the strain on Concourse (as well as the external source), however if you want to pick up those new commits as quickly as possible, then you need to poll as often as possible.

The resource checker uses the resource.check_every interval in order to figure out if a resource needs to be checked. A resource's check_every interval dictates how often it should be checked for new versions, with a default of 1 minute. If that seems like a lot, it is, but it's how Concourse keeps everything snappy. You can configure this value independently for each resource, or if your external service supports it, you can set resource.webhook_token to basically eliminate the need for polling altogether.

On every interval tick, the resource checker will see if there are any resources that need to be checked. It does this by first finding resources which are used as inputs to jobs, and then comparing the current time against the last time each resource was checked. If it has been longer than a resource's configured check_every interval, a new check will be enqueued. In practice this means that if a resource has a check_every of 1m, it is not guaranteed to be checked precisely every 60 seconds.

What do resource checks produce?

The whole point of running checks is to produce versions. Everything that happens in concourse is centered around the idea of resource versions. It's how concourse determines that something is new and a new build needs to be triggered. The versions produced by each resource all vary to a large degree. The git resource, for example, uses commits hashes, the registry image resource uses the digest sha, and the time resource uses timestamps. Every resource is different, but the end result is the same, a new input for your workload.

Resource checker components

As of Concourse v5.6.0, resource checking has been redesigned to be asynchronous. There are now two processes that dictate how resource checking happens. The scanner, which determines if new checks need to run, and the checker which runs them.

The scanner will run every CONCOURSE_LIDAR_SCANNER_INTERVAL. It's job is to determine if new checks need to run. It will loop over every resource (and resource type) and if the check interval has elapsed since it last ran, it will schedule a new check for that resource.

The checker will run every CONCOURSE_LIDAR_CHECKER_INTERVAL. It's job is to run all the checks that were scheduled, which can be scheduled through the scanner or the manual resource checks.