1.12.2 Tracing

This is an experimental feature.

Tracing in Concourse enables the delivery of traces related to the internal processes that go into running builds, and other internal operations, breaking them down by time, and component.

It leverages the (OpenTelemetry) SDK to allow support for many platforms. Currently tracing can be configured to integrates with:

Configuring Tracing

To export spans to Jaeger, specify the Thrift HTTP endpoint of the Jaeger collector:

CONCOURSE_TRACING_JAEGER_ENDPOINT=http://jaeger:14268/api/traces

To export spans to Google Cloud Trace, specify the GCP Project ID:

CONCOURSE_TRACING_STACKDRIVER_PROJECTID=your-gcp-project-id

Note that suitable GCP credentials must be available, via the usual GOOGLE_APPLICATION_CREDENTIALS environment variable, the default location that the gcloud CLI expects, or from GCP's metadata server (if Concourse is deployed on GCP).

To export spans to Lightstep via the OTLP Exporter, specify your collector access token and endpoint:

CONCOURSE_TRACING_OTLP_ADDRESS=ingest.lightstep.com:443
CONCOURSE_TRACING_OTLP_HEADERS=lightstep-access-token:mysupersecrettoken

To export spans to Honeycomb.io, specify the API key, dataset and
optionally the service name:

\codeblock{bash}{{{
CONCOURSE_TRACING_HONEYCOMB_API_KEY=your-honeycomb-api-key
CONCOURSE_TRACING_HONEYCOMB_DATASET=your-honeycomb-dataset
CONCOURSE_TRACING_HONEYCOMB_SERVICE_NAME=service-name-for-concourse  # NOTE: Optional. Defaults to "concourse"

What's emitted?

Below is a summary of the various operations that Concourse currently traces. They are arranged like a call tree, so that for each operation described, its sub-operations are described indented immediately below.

  • scanner.Run -- One of the Resource checker components responsible for determining which checks need to be run.

    • scanner.check -- This operation simply represents inserting the check in the database.

  • checker.Run -- follows from scanner.check above and will appear under the same trace. This is where the Resource Checker is actually executing a check script.

  • scheduler.Run -- This represents one tick of the Build Scheduler.

    • schedule-job -- this is the same operation scoped to a single job.

      • Algorithm.Compute -- this is where the Algorithm determines inputs for a job. Each of the resolvers below describes a different strategy for determining inputs, depending on the job's config.

        • individualResolver.Resolve -- This is used to determine versions input to a get step without passed constraints.

        • groupResolver.Resolve -- This is the juicy part of the algorithm, which deals with passed constraints.

        • pinnedResolver.Resolve -- This operation is used to determine inputs when Version Pinning is at play.

      • job.EnsurePendingBuildExists -- This is where a new build, if deemed necessary by scheduling constraints, will be inserted into the database. This operation follows from checker.Run above and will appear under the same trace as the check which produced the resource version responsible for triggering the new build.

  • build -- this is the primary operation performed by the Build Tracker. When a build is automatically triggered, this span follows from the job.EnsurePendingBuildExists operation which created the build, appearing in the same trace.