1 Docs

Concourse is a pipeline-based continuous thing-doer.

The word "pipeline" is all the rage in CI these days, so being more specific about this term is kind of important; Concourse's pipelines are significantly different from the rest.

Pipelines are built around Resources, which represent all external state, and Jobs, which interact with them. Concourse pipelines represent a dependency flow, kind of like distributed Makefiles. Pipelines are designed to be self-contained so as to minimize server-wide configuration. Maximizing portability also mitigates risk, making it easier for projects to recover from CI disasters.

Resources like the git resource and s3 resource are used to express source code, dependencies, deployments, and any other external state. This interface is also used to model more abstract things like scheduled or interval triggers, via the time resource.

Resource Types are defined as part of the pipeline itself, making the pipelines more self-contained and keeping Concourse itself small and generic without resorting to a complicated plugin system.

Jobs are sequences of get, put, and task steps to execute. These steps determine the job's inputs and outputs. Jobs are designed to be idempotent and loosely coupled, allowing the pipeline to grow with the project's needs without requiring engineers to keep too much in their head at a time.

Everything in Concourse runs in a container. Instead of modifying workers to install build tools, Tasks describe their own container image (typically using Docker images via the registry-image resource).

...What?

Concourse admittedly has a steeper learning curve at first, and depending on your background it might be a lot to take in. A core goal of this project is for the curve to flatten out shortly after and result in higher productivity and less stress over time.

If this all sounds like gobbeldigook, that's OK - you may want to just continue on, start kicking the tires a bit, and use the above as a quick reference of the "big picture" as the mental model sets in.

Table of contents:
  1. 1.1 Getting Started
    1. 1.1.1 Quick Start
      1. 1.1.1.1 Docker Compose Concourse
      2. 1.1.1.2 Install Fly
    2. 1.1.2 Hello World Pipeline
      1. 1.1.2.1 Creating a Pipeline
      2. 1.1.2.2 Add a job
      3. 1.1.2.3 Add a step
      4. 1.1.2.4 What is a step?
      5. 1.1.2.5 Fill in the Task Config
      6. 1.1.2.6 Run the pipeline
    3. 1.1.3 Inputs and Outputs
      1. 1.1.3.1 Overview
      2. 1.1.3.2 What are inputs and outputs?
      3. 1.1.3.3 Passing outputs to another task
      4. 1.1.3.4 How does Concourse track artifacts?
    4. 1.1.4 Resources
      1. 1.1.4.1 The Heart of Concourse
      2. 1.1.4.2 Versions
      3. 1.1.4.3 Resource Interface
      4. 1.1.4.4 Get Steps
      5. 1.1.4.5 Get Steps and Inputs
      6. 1.1.4.6 Checks
      7. 1.1.4.7 Put Steps
      8. 1.1.4.8 Using External Resource Types
      9. 1.1.4.9 Time For Takeoff ✈️
  2. 1.2 Install
    1. 1.2.1 Running a PostgreSQL node
      1. 1.2.1.1 Prerequisites
      2. 1.2.1.2 Running PostgreSQL
      3. 1.2.1.3 Resource utilization
    2. 1.2.2 The concourse CLI
    3. 1.2.3 Generating Keys
      1. 1.2.3.1 Generating the Keys
      2. 1.2.3.2 Multiple Worker Keys
    4. 1.2.4 Running a web node
      1. 1.2.4.1 Prerequisites
      2. 1.2.4.2 Running concourse web
        1. 1.2.4.2.1 Resource utilization
      3. 1.2.4.3 Operating a web node
        1. 1.2.4.3.1 Scaling
          1. 1.2.4.3.1.1 Database connection pooling
        2. 1.2.4.3.2 Reloading worker authorized key
        3. 1.2.4.3.3 Restarting & Upgrading
        4. 1.2.4.3.4 Downgrading
      4. 1.2.4.4 Configuring the web node
        1. 1.2.4.4.1 Giving your cluster a name
        2. 1.2.4.4.2 Configuring ingress traffic
        3. 1.2.4.4.3 TLS via Let's Encrypt
        4. 1.2.4.4.4 Build log retention
        5. 1.2.4.4.5 Enabling audit logs
        6. 1.2.4.4.6 Configuring defaults for resource types
    5. 1.2.5 Running a worker node
      1. 1.2.5.1 Prerequisites
      2. 1.2.5.2 Running concourse worker
        1. 1.2.5.2.1 Resource utilization
      3. 1.2.5.3 Operating a worker node
        1. 1.2.5.3.1 Scaling Workers
          1. 1.2.5.3.1.1 Horizontal vs Vertical Scaling
        2. 1.2.5.3.2 Worker Heartbeating & Stalling
        3. 1.2.5.3.3 Restarting a Worker
        4. 1.2.5.3.4 Gracefully Removing a Worker
      4. 1.2.5.4 Configuring the worker node
        1. 1.2.5.4.1 Tagging Workers
        2. 1.2.5.4.2 Team Workers
          1. 1.2.5.4.2.1 Tags and Team Workers
        3. 1.2.5.4.3 Healthcheck Endpoint
        4. 1.2.5.4.4 Resource Types
          1. 1.2.5.4.4.1 Bundled Resource Types
          2. 1.2.5.4.4.2 Installing or Upgrading Bundled Resource Types
        5. 1.2.5.4.5 Configuring Runtimes
          1. 1.2.5.4.5.1 containerd runtime
          2. 1.2.5.4.5.2 Transitioning from Guardian to containerd
          3. 1.2.5.4.5.3 Guardian runtime
          4. 1.2.5.4.5.4 Troubleshooting and fixing DNS resolution
            1. 1.2.5.4.5.4.1 Pointing to external DNS servers
            2. 1.2.5.4.5.4.2 Using a local DNS server
            3. 1.2.5.4.5.4.3 A note on allowing host access and DNS proxy
        6. 1.2.5.4.6 Configuring Peer-to-Peer Volume Streaming
          1. 1.2.5.4.6.1 P2P Worker Configuration
          2. 1.2.5.4.6.2 P2P Web Configuration
    6. 1.2.6 Upgrading Concourse
      1. 1.2.6.1 Upgrading the Web Node
      2. 1.2.6.2 Upgrading the Worker Node
        1. 1.2.6.2.1 Linux Workers
        2. 1.2.6.2.2 Darwin and Windows Workers
  3. 1.3 Auth & Teams
    1. 1.3.1 The main team
    2. 1.3.2 Configuring Auth
      1. 1.3.2.1 Local User auth
      2. 1.3.2.2 GitHub auth
      3. 1.3.2.3 GitLab auth
      4. 1.3.2.4 BitBucket Cloud auth
      5. 1.3.2.5 CF/UAA auth
      6. 1.3.2.6 LDAP auth
      7. 1.3.2.7 Microsoft auth
      8. 1.3.2.8 Generic OIDC auth
      9. 1.3.2.9 Generic oAuth
      10. 1.3.2.10 Generic SAML auth
    3. 1.3.3 Managing Teams
      1. 1.3.3.1 fly set-team
        1. 1.3.3.1.1 Setting User Roles
      2. 1.3.3.2 fly active-users
      3. 1.3.3.3 fly teams
      4. 1.3.3.4 fly get-team
      5. 1.3.3.5 fly rename-team
      6. 1.3.3.6 fly destroy-team
    4. 1.3.4 User Roles & Permissions
      1. 1.3.4.1 Concourse Admin
      2. 1.3.4.2 owner role
      3. 1.3.4.3 member role
      4. 1.3.4.4 pipeline-operator role
      5. 1.3.4.5 viewer role
      6. 1.3.4.6 Action Matrix
      7. 1.3.4.7 Configuring RBAC
    5. 1.3.5 Pipeline & Build Visibility
    6. 1.3.6 Security Caveats
  4. 1.4 The fly CLI
    1. 1.4.1 fly login
    2. 1.4.2 fly targets
    3. 1.4.3 fly status
    4. 1.4.4 fly userinfo
    5. 1.4.5 fly logout
    6. 1.4.6 fly edit-target
    7. 1.4.7 fly delete-target
    8. 1.4.8 fly sync
    9. 1.4.9 fly completion
  5. 1.5 Config Basics
    1. 1.5.1 Intro to YAML
      1. 1.5.1.1 YAML Tips & Tricks
      2. 1.5.1.2 YAML Quirks
    2. 1.5.2 Basic Schemas
  6. 1.6 Pipelines
    1. 1.6.1 Setting Pipelines
      1. 1.6.1.1 fly set-pipeline
      2. 1.6.1.2 fly validate-pipeline
      3. 1.6.1.3 fly format-pipeline
    2. 1.6.2 Managing Pipelines
      1. 1.6.2.1 fly pipelines
      2. 1.6.2.2 fly rename-pipeline
      3. 1.6.2.3 fly pause-pipeline
      4. 1.6.2.4 fly unpause-pipeline
      5. 1.6.2.5 fly expose-pipeline
      6. 1.6.2.6 fly hide-pipeline
      7. 1.6.2.7 fly get-pipeline
      8. 1.6.2.8 fly destroy-pipeline
      9. 1.6.2.9 fly order-pipelines
      10. 1.6.2.10 fly archive-pipeline
    3. 1.6.3 Grouping Pipelines
      1. 1.6.3.1 Managing Instanced Pipelines
        1. 1.6.3.1.1 fly order-instanced-pipelines
        2. 1.6.3.1.2 Managing Jobs and Resources
  7. 1.7 Vars
    1. 1.7.1 ((var)) syntax
    2. 1.7.2 The "." var source
    3. 1.7.3 Interpolation
    4. 1.7.4 Static vars
    5. 1.7.5 Dynamic vars
      1. 1.7.5.1 Across Step & Dynamic Vars
      2. 1.7.5.2 Var sources (experimental)
      3. 1.7.5.3 The cluster-wide credential manager
  8. 1.8 Resources
    1. 1.8.1 Resource Versions
      1. 1.8.1.1 Where do they come from and what are they used for?
      2. 1.8.1.2 Version Pinning
        1. 1.8.1.2.1 Unpinning
      3. 1.8.1.3 Disabling a Version
    2. 1.8.2 Managing Resources
      1. 1.8.2.1 fly check-resource
      2. 1.8.2.2 fly pin-resource
      3. 1.8.2.3 fly enable-resource-version
      4. 1.8.2.4 fly disable-resource-version
      5. 1.8.2.5 fly clear-resource-cache
  9. 1.9 Resource Types
    1. 1.9.1 Implementing a Resource Type
      1. 1.9.1.1 check: Check for new versions.
      2. 1.9.1.2 in: Fetch a given resource.
      3. 1.9.1.3 out: Update a resource.
      4. 1.9.1.4 Metadata
      5. 1.9.1.5 Certificate Propagation
      6. 1.9.1.6 Testing resources locally using docker
    2. 1.9.2 Managing Resource Types
      1. 1.9.2.1 fly check-resource-type
  10. 1.10 Jobs
    1. 1.10.1 Managing Jobs
      1. 1.10.1.1 fly jobs
      2. 1.10.1.2 fly trigger-job
      3. 1.10.1.3 fly rerun-build
      4. 1.10.1.4 fly pause-job
      5. 1.10.1.5 fly unpause-job
      6. 1.10.1.6 fly clear-task-cache
  11. 1.11 Steps
    1. 1.11.1 get step
    2. 1.11.2 put step
    3. 1.11.3 task step
    4. 1.11.4 set_pipeline step
    5. 1.11.5 load_var step
    6. 1.11.6 in_parallel step
    7. 1.11.7 do step
    8. 1.11.8 try step
    9. 1.11.9 Modifier And Hook Steps
      1. 1.11.9.1 across Step Modifier
      2. 1.11.9.2 timeout Step Modifier
      3. 1.11.9.3 attempts Step Modifier
      4. 1.11.9.4 tags Step Modifier
      5. 1.11.9.5 on_success Step Hook
      6. 1.11.9.6 on_failure Step Hook
      7. 1.11.9.7 on_abort Step Hook
      8. 1.11.9.8 on_error Step Hook
      9. 1.11.9.9 ensure Step Hook
  12. 1.12 Tasks
    1. 1.12.1 Running tasks with fly execute
    2. 1.12.2 Task runtime environment
  13. 1.13 Builds
    1. 1.13.1 Rerunning a Build
      1. 1.13.1.1 Current caveats with rerunning
    2. 1.13.2 fly builds
    3. 1.13.3 fly intercept
    4. 1.13.4 fly abort-build
    5. 1.13.5 fly watch
  14. 1.14 How-To Guides
    1. 1.14.1 Pipeline Guides
      1. 1.14.1.1 Common Pipeline Practices
      2. 1.14.1.2 Exploring Task Input and Output Scenarios
      3. 1.14.1.3 Gated Pipeline Patterns
      4. 1.14.1.4 Time Triggered Pipeline Patterns
      5. 1.14.1.5 Manual Approval Step
    2. 1.14.2 Git Guides
      1. 1.14.2.1 Basic Git Operations
      2. 1.14.2.2 Multi-Branch Workflows
    3. 1.14.3 Container Image Guides
      1. 1.14.3.1 Building and Pushing an Image
      2. 1.14.3.2 Building an Image and Using it in a Task
  15. 1.15 Operation
    1. 1.15.1 Metrics
      1. 1.15.1.1 Configuring Metrics
      2. 1.15.1.2 What's emitted?
    2. 1.15.2 Tracing
      1. 1.15.2.1 Configuring Tracing
      2. 1.15.2.2 Trace context propagation
      3. 1.15.2.3 What's emitted?
    3. 1.15.3 Encryption
      1. 1.15.3.1 What's encrypted?
      2. 1.15.3.2 Enabling Encryption
      3. 1.15.3.3 Rotating the Encryption Key
      4. 1.15.3.4 Disabling Encryption
    4. 1.15.4 Credential Management
      1. 1.15.4.1 The Vault credential manager
      2. 1.15.4.2 The CredHub credential manager
      3. 1.15.4.3 The AWS SSM credential manager
      4. 1.15.4.4 The AWS Secrets Manager credential manager
      5. 1.15.4.5 Kubernetes Credential Manager
      6. 1.15.4.6 The Conjur credential manager
      7. 1.15.4.7 Caching credentials
      8. 1.15.4.8 Redacting credentials
      9. 1.15.4.9 Retrying failed fetches
    5. 1.15.5 Container Placement
      1. 1.15.5.1 The volume-locality strategy
      2. 1.15.5.2 The fewest-build-containers strategy
      3. 1.15.5.3 The random strategy
      4. 1.15.5.4 The limit-active-tasks strategy
      5. 1.15.5.5 The limit-active-containers strategy
      6. 1.15.5.6 The limit-active-volumes strategy
      7. 1.15.5.7 Chaining Placement Strategies
    6. 1.15.6 Open Policy Agent Integration
      1. 1.15.6.1 Configuring Concourse
      2. 1.15.6.2 Writing OPA Rules
      3. 1.15.6.3 Special Actions
        1. 1.15.6.3.1 UseImage
        2. 1.15.6.3.2 SetPipeline
    7. 1.15.7 Performance Tuning
      1. 1.15.7.1 The Big Caveat
      2. 1.15.7.2 Build Logs
        1. 1.15.7.2.1 CONCOURSE_DEFAULT_BUILD_LOGS_TO_RETAIN
        2. 1.15.7.2.2 CONCOURSE_MAX_BUILD_LOGS_TO_RETAIN
        3. 1.15.7.2.3 CONCOURSE_DEFAULT_DAYS_TO_RETAIN_BUILD_LOGS
        4. 1.15.7.2.4 CONCOURSE_MAX_DAYS_TO_RETAIN_BUILD_LOGS
      3. 1.15.7.3 Resource Checking
        1. 1.15.7.3.1 CONCOURSE_RESOURCE_CHECKING_INTERVAL
        2. 1.15.7.3.2 CONCOURSE_RESOURCE_WITH_WEBHOOK_CHECKING_INTERVAL
        3. 1.15.7.3.3 CONCOURSE_MAX_CHECKS_PER_SECOND
      4. 1.15.7.4 Pipeline Management
        1. 1.15.7.4.1 CONCOURSE_PAUSE_PIPELINES_AFTER
        2. 1.15.7.4.2 CONCOURSE_DEFAULT_TASK_{CPU/MEMORY}_LIMIT
        3. 1.15.7.4.3 CONCOURSE_DEFAULT_{GET/PUT/TASK}_TIMEOUT
      5. 1.15.7.5 Container Placement
      6. 1.15.7.6 Garbage Collection
        1. 1.15.7.6.1 CONCOURSE_GC_FAILED_GRACE_PERIOD
        2. 1.15.7.6.2 Other GC Grace Periods
      7. 1.15.7.7 Web To Worker Ratio
    8. 1.15.8 Global Resources
      1. 1.15.8.1 Benefits of Global Resources
      2. 1.15.8.2 Risks and Side Effects
    9. 1.15.9 Administration
      1. 1.15.9.1 fly workers
      2. 1.15.9.2 fly prune-worker
      3. 1.15.9.3 fly land-worker
      4. 1.15.9.4 fly containers
      5. 1.15.9.5 fly volumes
      6. 1.15.9.6 fly curl
  16. 1.16 Observation
    1. 1.16.1 The Dashboard
    2. 1.16.2 cc.xml
    3. 1.16.3 Badges
    4. 1.16.4 Pipeline Visibility
  17. 1.17 Internals
    1. 1.17.1 Basic architecture
    2. 1.17.2 ATC: web UI & build scheduler
      1. 1.17.2.1 Resource Checker
      2. 1.17.2.2 Build Scheduler
      3. 1.17.2.3 Build Tracker
      4. 1.17.2.4 Garbage Collector
    3. 1.17.3 TSA: worker registration & forwarding
    4. 1.17.4 Workers Architecture
      1. 1.17.4.1 The worker lifecycle