1 Docs
Concourse is a pipeline-based continuous thing-doer.
The word "pipeline" is all the rage in CI these days, so being more specific about this term is kind of important; Concourse's pipelines are significantly different from the rest.
Pipelines are built around Resources, which represent all external state, and Jobs, which interact with them. Concourse pipelines represent a dependency flow, kind of like distributed Makefile
s. Pipelines are designed to be self-contained so as to minimize server-wide configuration. Maximizing portability also mitigates risk, making it easier for projects to recover from CI disasters.
Resources like the git
resource and s3
resource are used to express source code, dependencies, deployments, and any other external state. This interface is also used to model more abstract things like scheduled or interval triggers, via the time
resource.
Resource Types are defined as part of the pipeline itself, making the pipelines more self-contained and keeping Concourse itself small and generic without resorting to a complicated plugin system.
Jobs are sequences of get
, put
, and task
steps to execute. These steps determine the job's inputs and outputs. Jobs are designed to be idempotent and loosely coupled, allowing the pipeline to grow with the project's needs without requiring engineers to keep too much in their head at a time.
Everything in Concourse runs in a container. Instead of modifying workers to install build tools, Tasks describe their own container image (typically using Docker images via the registry-image
resource).
...What?
Concourse admittedly has a steeper learning curve at first, and depending on your background it might be a lot to take in. A core goal of this project is for the curve to flatten out shortly after and result in higher productivity and less stress over time.
If this all sounds like gobbeldigook, that's OK - you may want to just continue on, start kicking the tires a bit, and use the above as a quick reference of the "big picture" as the mental model sets in.
-
1.1
Getting Started
-
1.1.1
Quick Start
- 1.1.1.1 Docker Compose Concourse
- 1.1.1.2 Install Fly
-
1.1.2
Hello World Pipeline
- 1.1.2.1 Creating a Pipeline
- 1.1.2.2 Add a job
- 1.1.2.3 Add a step
- 1.1.2.4 What is a step?
- 1.1.2.5 Fill in the Task Config
- 1.1.2.6 Run the pipeline
-
1.1.3
Inputs and Outputs
- 1.1.3.1 Overview
- 1.1.3.2 What are inputs and outputs?
- 1.1.3.3 Passing outputs to another task
- 1.1.3.4 How does Concourse track artifacts?
-
1.1.4
Resources
- 1.1.4.1 The Heart of Concourse
- 1.1.4.2 Versions
- 1.1.4.3 Resource Interface
- 1.1.4.4 Get Steps
- 1.1.4.5 Get Steps and Inputs
- 1.1.4.6 Checks
- 1.1.4.7 Put Steps
- 1.1.4.8 Using External Resource Types
- 1.1.4.9 Time For Takeoff ✈️
-
1.1.1
Quick Start
-
1.2
Install
-
1.2.1
Running a PostgreSQL node
- 1.2.1.1 Prerequisites
- 1.2.1.2 Running PostgreSQL
- 1.2.1.3 Resource utilization
-
1.2.2
The
concourse
CLI -
1.2.3
Generating Keys
- 1.2.3.1 Generating the Keys
- 1.2.3.2 Multiple Worker Keys
-
1.2.4
Running a
web
node- 1.2.4.1 Prerequisites
-
1.2.4.2
Running
concourse web
- 1.2.4.2.1 Resource utilization
-
1.2.4.3
Operating a
web
node-
1.2.4.3.1
Scaling
- 1.2.4.3.1.1 Database connection pooling
- 1.2.4.3.2 Reloading worker authorized key
- 1.2.4.3.3 Restarting & Upgrading
- 1.2.4.3.4 Downgrading
-
1.2.4.3.1
Scaling
-
1.2.4.4
Configuring the
web
node- 1.2.4.4.1 Giving your cluster a name
- 1.2.4.4.2 Configuring ingress traffic
- 1.2.4.4.3 TLS via Let's Encrypt
- 1.2.4.4.4 Build log retention
- 1.2.4.4.5 Enabling audit logs
- 1.2.4.4.6 Configuring defaults for resource types
-
1.2.5
Running a
worker
node- 1.2.5.1 Prerequisites
-
1.2.5.2
Running
concourse worker
- 1.2.5.2.1 Resource utilization
-
1.2.5.3
Operating a
worker
node-
1.2.5.3.1
Scaling Workers
- 1.2.5.3.1.1 Horizontal vs Vertical Scaling
- 1.2.5.3.2 Worker Heartbeating & Stalling
- 1.2.5.3.3 Restarting a Worker
- 1.2.5.3.4 Gracefully Removing a Worker
-
1.2.5.3.1
Scaling Workers
-
1.2.5.4
Configuring the
worker
node- 1.2.5.4.1 Tagging Workers
-
1.2.5.4.2
Team Workers
- 1.2.5.4.2.1 Tags and Team Workers
- 1.2.5.4.3 Healthcheck Endpoint
-
1.2.5.4.4
Resource Types
- 1.2.5.4.4.1 Bundled Resource Types
- 1.2.5.4.4.2 Installing or Upgrading Bundled Resource Types
-
1.2.5.4.5
Configuring Runtimes
-
1.2.5.4.5.1
containerd
runtime - 1.2.5.4.5.2 Transitioning from Guardian to containerd
-
1.2.5.4.5.3
Guardian
runtime -
1.2.5.4.5.4
Troubleshooting and fixing DNS resolution
- 1.2.5.4.5.4.1 Pointing to external DNS servers
- 1.2.5.4.5.4.2 Using a local DNS server
- 1.2.5.4.5.4.3 A note on allowing host access and DNS proxy
-
1.2.5.4.5.1
-
1.2.5.4.6
Configuring Peer-to-Peer Volume Streaming
- 1.2.5.4.6.1 P2P Worker Configuration
- 1.2.5.4.6.2 P2P Web Configuration
-
1.2.6
Upgrading Concourse
- 1.2.6.1 Upgrading the Web Node
-
1.2.6.2
Upgrading the Worker Node
- 1.2.6.2.1 Linux Workers
- 1.2.6.2.2 Darwin and Windows Workers
-
1.2.1
Running a PostgreSQL node
-
1.3
Auth & Teams
-
1.3.1
The
main
team -
1.3.2
Configuring Auth
- 1.3.2.1 Local User auth
- 1.3.2.2 GitHub auth
- 1.3.2.3 GitLab auth
- 1.3.2.4 BitBucket Cloud auth
- 1.3.2.5 CF/UAA auth
- 1.3.2.6 LDAP auth
- 1.3.2.7 Microsoft auth
- 1.3.2.8 Generic OIDC auth
- 1.3.2.9 Generic oAuth
- 1.3.2.10 Generic SAML auth
-
1.3.3
Managing Teams
-
1.3.3.1
fly set-team
- 1.3.3.1.1 Setting User Roles
-
1.3.3.2
fly active-users
-
1.3.3.3
fly teams
-
1.3.3.4
fly get-team
-
1.3.3.5
fly rename-team
-
1.3.3.6
fly destroy-team
-
1.3.3.1
-
1.3.4
User Roles & Permissions
- 1.3.4.1 Concourse Admin
-
1.3.4.2
owner
role -
1.3.4.3
member
role -
1.3.4.4
pipeline-operator
role -
1.3.4.5
viewer
role - 1.3.4.6 Action Matrix
- 1.3.4.7 Configuring RBAC
- 1.3.5 Pipeline & Build Visibility
- 1.3.6 Security Caveats
-
1.3.1
The
-
1.4
The
fly
CLI-
1.4.1
fly login
-
1.4.2
fly targets
-
1.4.3
fly status
-
1.4.4
fly userinfo
-
1.4.5
fly logout
-
1.4.6
fly edit-target
-
1.4.7
fly delete-target
-
1.4.8
fly sync
-
1.4.9
fly completion
-
1.4.1
-
1.5
Config Basics
-
1.5.1
Intro to YAML
- 1.5.1.1 YAML Tips & Tricks
- 1.5.1.2 YAML Quirks
- 1.5.2 Basic Schemas
-
1.5.1
Intro to YAML
-
1.6
Pipelines
-
1.6.1
Setting Pipelines
-
1.6.1.1
fly set-pipeline
-
1.6.1.2
fly validate-pipeline
-
1.6.1.3
fly format-pipeline
-
1.6.1.1
-
1.6.2
Managing Pipelines
-
1.6.2.1
fly pipelines
-
1.6.2.2
fly rename-pipeline
-
1.6.2.3
fly pause-pipeline
-
1.6.2.4
fly unpause-pipeline
-
1.6.2.5
fly expose-pipeline
-
1.6.2.6
fly hide-pipeline
-
1.6.2.7
fly get-pipeline
-
1.6.2.8
fly destroy-pipeline
-
1.6.2.9
fly order-pipelines
-
1.6.2.10
fly archive-pipeline
-
1.6.2.1
-
1.6.3
Grouping Pipelines
-
1.6.3.1
Managing Instanced Pipelines
-
1.6.3.1.1
fly order-instanced-pipelines
- 1.6.3.1.2 Managing Jobs and Resources
-
1.6.3.1.1
-
1.6.3.1
Managing Instanced Pipelines
-
1.6.1
Setting Pipelines
-
1.7
Vars
-
1.7.1
((var))
syntax -
1.7.2
The "
.
" var source - 1.7.3 Interpolation
- 1.7.4 Static vars
-
1.7.5
Dynamic vars
- 1.7.5.1 Across Step & Dynamic Vars
- 1.7.5.2 Var sources (experimental)
- 1.7.5.3 The cluster-wide credential manager
-
1.7.1
-
1.8
Resources
-
1.8.1
Resource Versions
- 1.8.1.1 Where do they come from and what are they used for?
-
1.8.1.2
Version Pinning
- 1.8.1.2.1 Unpinning
- 1.8.1.3 Disabling a Version
-
1.8.2
Managing Resources
-
1.8.2.1
fly check-resource
-
1.8.2.2
fly pin-resource
-
1.8.2.3
fly enable-resource-version
-
1.8.2.4
fly disable-resource-version
-
1.8.2.5
fly clear-resource-cache
-
1.8.2.1
-
1.8.1
Resource Versions
-
1.9
Resource Types
-
1.9.1
Implementing a Resource Type
-
1.9.1.1
check
: Check for new versions. -
1.9.1.2
in
: Fetch a given resource. -
1.9.1.3
out
: Update a resource. - 1.9.1.4 Metadata
- 1.9.1.5 Certificate Propagation
- 1.9.1.6 Testing resources locally using docker
-
1.9.1.1
-
1.9.2
Managing Resource Types
-
1.9.2.1
fly check-resource-type
-
1.9.2.1
-
1.9.1
Implementing a Resource Type
-
1.10
Jobs
-
1.10.1
Managing Jobs
-
1.10.1.1
fly jobs
-
1.10.1.2
fly trigger-job
-
1.10.1.3
fly rerun-build
-
1.10.1.4
fly pause-job
-
1.10.1.5
fly unpause-job
-
1.10.1.6
fly clear-task-cache
-
1.10.1.1
-
1.10.1
Managing Jobs
-
1.11
Steps
-
1.11.1
get
step -
1.11.2
put
step -
1.11.3
task
step -
1.11.4
set_pipeline
step -
1.11.5
load_var
step -
1.11.6
in_parallel
step -
1.11.7
do
step -
1.11.8
try
step -
1.11.9
Modifier And Hook Steps
-
1.11.9.1
across
Step Modifier -
1.11.9.2
timeout
Step Modifier -
1.11.9.3
attempts
Step Modifier -
1.11.9.4
tags
Step Modifier -
1.11.9.5
on_success
Step Hook -
1.11.9.6
on_failure
Step Hook -
1.11.9.7
on_abort
Step Hook -
1.11.9.8
on_error
Step Hook -
1.11.9.9
ensure
Step Hook
-
1.11.9.1
-
1.11.1
-
1.12
Tasks
-
1.12.1
Running tasks with
fly execute
- 1.12.2 Task runtime environment
-
1.12.1
Running tasks with
-
1.13
Builds
-
1.13.1
Rerunning a Build
- 1.13.1.1 Current caveats with rerunning
-
1.13.2
fly builds
-
1.13.3
fly intercept
-
1.13.4
fly abort-build
-
1.13.5
fly watch
-
1.13.1
Rerunning a Build
-
1.14
How-To Guides
-
1.14.1
Pipeline Guides
- 1.14.1.1 Common Pipeline Practices
- 1.14.1.2 Exploring Task Input and Output Scenarios
- 1.14.1.3 Gated Pipeline Patterns
- 1.14.1.4 Time Triggered Pipeline Patterns
- 1.14.1.5 Manual Approval Step
-
1.14.2
Git Guides
- 1.14.2.1 Basic Git Operations
- 1.14.2.2 Multi-Branch Workflows
-
1.14.3
Container Image Guides
- 1.14.3.1 Building and Pushing an Image
- 1.14.3.2 Building an Image and Using it in a Task
-
1.14.1
Pipeline Guides
-
1.15
Operation
-
1.15.1
Metrics
- 1.15.1.1 Configuring Metrics
- 1.15.1.2 What's emitted?
-
1.15.2
Tracing
- 1.15.2.1 Configuring Tracing
- 1.15.2.2 Trace context propagation
- 1.15.2.3 What's emitted?
-
1.15.3
Encryption
- 1.15.3.1 What's encrypted?
- 1.15.3.2 Enabling Encryption
- 1.15.3.3 Rotating the Encryption Key
- 1.15.3.4 Disabling Encryption
-
1.15.4
Credential Management
- 1.15.4.1 The Vault credential manager
- 1.15.4.2 The CredHub credential manager
- 1.15.4.3 The AWS SSM credential manager
- 1.15.4.4 The AWS Secrets Manager credential manager
- 1.15.4.5 Kubernetes Credential Manager
- 1.15.4.6 The Conjur credential manager
- 1.15.4.7 Caching credentials
- 1.15.4.8 Redacting credentials
- 1.15.4.9 Retrying failed fetches
-
1.15.5
Container Placement
-
1.15.5.1
The
volume-locality
strategy -
1.15.5.2
The
fewest-build-containers
strategy -
1.15.5.3
The
random
strategy -
1.15.5.4
The
limit-active-tasks
strategy -
1.15.5.5
The
limit-active-containers
strategy -
1.15.5.6
The
limit-active-volumes
strategy - 1.15.5.7 Chaining Placement Strategies
-
1.15.5.1
The
-
1.15.6
Open Policy Agent Integration
- 1.15.6.1 Configuring Concourse
- 1.15.6.2 Writing OPA Rules
-
1.15.6.3
Special Actions
-
1.15.6.3.1
UseImage
-
1.15.6.3.2
SetPipeline
-
1.15.6.3.1
-
1.15.7
Performance Tuning
- 1.15.7.1 The Big Caveat
-
1.15.7.2
Build Logs
-
1.15.7.2.1
CONCOURSE_DEFAULT_BUILD_LOGS_TO_RETAIN
-
1.15.7.2.2
CONCOURSE_MAX_BUILD_LOGS_TO_RETAIN
-
1.15.7.2.3
CONCOURSE_DEFAULT_DAYS_TO_RETAIN_BUILD_LOGS
-
1.15.7.2.4
CONCOURSE_MAX_DAYS_TO_RETAIN_BUILD_LOGS
-
1.15.7.2.1
-
1.15.7.3
Resource Checking
-
1.15.7.3.1
CONCOURSE_RESOURCE_CHECKING_INTERVAL
-
1.15.7.3.2
CONCOURSE_RESOURCE_WITH_WEBHOOK_CHECKING_INTERVAL
-
1.15.7.3.3
CONCOURSE_MAX_CHECKS_PER_SECOND
-
1.15.7.3.1
-
1.15.7.4
Pipeline Management
-
1.15.7.4.1
CONCOURSE_PAUSE_PIPELINES_AFTER
-
1.15.7.4.2
CONCOURSE_DEFAULT_TASK_{CPU/MEMORY}_LIMIT
-
1.15.7.4.3
CONCOURSE_DEFAULT_{GET/PUT/TASK}_TIMEOUT
-
1.15.7.4.1
- 1.15.7.5 Container Placement
-
1.15.7.6
Garbage Collection
-
1.15.7.6.1
CONCOURSE_GC_FAILED_GRACE_PERIOD
- 1.15.7.6.2 Other GC Grace Periods
-
1.15.7.6.1
- 1.15.7.7 Web To Worker Ratio
-
1.15.8
Global Resources
- 1.15.8.1 Benefits of Global Resources
- 1.15.8.2 Risks and Side Effects
-
1.15.9
Administration
-
1.15.9.1
fly workers
-
1.15.9.2
fly prune-worker
-
1.15.9.3
fly land-worker
-
1.15.9.4
fly containers
-
1.15.9.5
fly volumes
-
1.15.9.6
fly curl
-
1.15.9.1
-
1.15.1
Metrics
-
1.16
Observation
- 1.16.1 The Dashboard
-
1.16.2
cc.xml
- 1.16.3 Badges
- 1.16.4 Pipeline Visibility
-
1.17
Internals
- 1.17.1 Basic architecture
-
1.17.2
ATC: web UI & build scheduler
- 1.17.2.1 Resource Checker
- 1.17.2.2 Build Scheduler
- 1.17.2.3 Build Tracker
- 1.17.2.4 Garbage Collector
- 1.17.3 TSA: worker registration & forwarding
-
1.17.4
Workers Architecture
- 1.17.4.1 The worker lifecycle