Tracing in Concourse enables the delivery of traces related to the internal processes that go into running builds, and other internal operations, breaking them down by time, and component.
It leverages the (OpenTelemetry) SDK to allow support for many platforms. Currently tracing can be configured to integrates with:
To export spans to Jaeger, specify the Thrift HTTP endpoint of the Jaeger collector:
To export spans to Google Cloud Trace, specify the GCP Project ID:
Note that suitable GCP credentials must be available, via the usual
GOOGLE_APPLICATION_CREDENTIALS environment variable, the default location that the
gcloud CLI expects, or from GCP's metadata server (if Concourse is deployed on GCP).
To export spans the OpenTelemetry Collector via the OTLP Exporter, specify your collector access endpoint:
To export spans to Lightstep via the OTLP Exporter, specify your collector access token and endpoint:
To export spans to Elastic Observability via the OTLP Exporter, specify your Elastic APM secret token and endpoint:
CONCOURSE_TRACING_OTLP_ADDRESS=elastic-apm-server.example.com:443 CONCOURSE_TRACING_OTLP_HEADERS=Authorization=Bearer your-secret-token
To export spans to Honeycomb.io, specify the API key, dataset and optionally the service name:
CONCOURSE_TRACING_HONEYCOMB_API_KEY=your-honeycomb-api-key CONCOURSE_TRACING_HONEYCOMB_DATASET=your-honeycomb-dataset CONCOURSE_TRACING_HONEYCOMB_SERVICE_NAME=service-name-for-concourse # NOTE: Optional. Defaults to "concourse"
When tracing is enabled, trace context propagation is activated in pipeline tasks thanks to the injection of the environment variable
run commands. The environment variable
TRACEPARENT complies with the W3C Trace Context specification.
Below is a summary of the various operations that Concourse currently traces. They are arranged like a call tree, so that for each operation described, its sub-operations are described indented immediately below.
scanner.Run-- An execution of the Resource Checker, responsible for determining which resources need to be checked.
scanner.check-- This operation simply represents inserting the check in the database.
scheduler.Run-- This represents one tick of the Build Scheduler.
schedule-job-- this is the same operation scoped to a single job.
Algorithm.Compute-- this is where the Algorithm determines inputs for a job. Each of the resolvers below describes a different strategy for determining inputs, depending on the job's config.
groupResolver.Resolve-- This is the juicy part of the algorithm, which deals with
pinnedResolver.Resolve-- This operation is used to determine inputs when Version Pinning is at play.
job.EnsurePendingBuildExists-- This is where a new build, if deemed necessary by scheduling constraints, will be inserted into the database. This operation follows from
checker.Runabove and will appear under the same trace as the check which produced the resource version responsible for triggering the new build.
build-- this is the primary operation performed by the Build Tracker. When a build is automatically triggered, this span follows from the
job.EnsurePendingBuildExistsoperation which created the build, appearing in the same trace.