On this page:
# pipeline schema
# group_config schema
# display_config schema
# pipeline.jobs
# pipeline.resources
# pipeline.resource_types
# pipeline.var_sources
# group_config.name
# group_config.jobs
# pipeline.groups
# display_config.background_image
# pipeline.display

1.6 Pipelines

A pipeline is the result of configuring Jobs and Resources together. When you configure a pipeline, it takes on a life of its own, to continuously detect resource versions and automatically queue new builds for jobs as they have new available inputs.

Pipelines are configured via fly set-pipeline or the set_pipeline step as declarative YAML files which conform to the following schema:

pipeline schema

A set of jobs for the pipeline to continuously schedule. At least one job is required for a pipeline to be valid.

A set of resources for the pipeline to continuously check.

A set of resource types for resources within the pipeline to use.

A set of Var sources for the pipeline to use.

A list of job groups to use for organizing jobs in the web UI.

Groups have no functional effect on your pipeline. They are purely for making it easier to grok large pipelines in the web UI.

Note: once you have added groups to your pipeline, all jobs must be in a group.

The following example will make the "tests" group the default view (since it's listed first), separating the later jobs into a "publish" group:

groups:
- name: test
  jobs:
  - unit
  - integration
- name: publish
  jobs:
  - deploy
  - shipit

This would display two tabs at the top of the home page: "test" and "publish".

For a real world example of how groups can be used to simplify navigation and provide logical grouping, see the groups used at the top of the page in the Concourse pipeline.

group_config schema

A unique name for the group. This should be short and simple as it will be used as the tab name for navigation.

A list of jobs that should appear in this group. A job may appear in multiple groups. Neighbours of jobs in the current group will also appear on the same page in order to give context of the location of the group in the pipeline.

You may also use any valid glob to represent several jobs, e.g.:

groups:
- name: develop
  jobs:
  - terraform-*
  - test
  - deploy-{dev,staging}
- name: ship
  jobs:
  - deploy-prod
- name: all
  jobs:
  - "*"

In this example, the develop group will match terraform-apply, terraform-destroy, test, deploy-dev, deploy-staging. The ship group will only match deploy-prod. The all group will match all jobs in the pipeline.

Note that depending on how it's used, *, {, and } have special meaning in YAML, and may need to be quoted (as was done in the all job above)

display was introduced in Concourse v6.6.0. It is considered an experimental feature.

Visual configurations for personalizing your pipeline.

The following example will display an image in the background of the pipeline it is configured on.

display:
  background_image: https://avatars1.githubusercontent.com/u/7809479?s=400&v=4

display_config schema

Allow users to specify a custom background image which is put at 30% opacity, grayscaled and blended into existing background. Must be an http, https, or relative URL.

Table of contents:
  1. 1.6.1 Setting Pipelines
    1. 1.6.1.1 fly set-pipeline
    2. 1.6.1.2 fly validate-pipeline
    3. 1.6.1.3 fly format-pipeline
  2. 1.6.2 Managing Pipelines
    1. 1.6.2.1 fly pipelines
    2. 1.6.2.2 fly rename-pipeline
    3. 1.6.2.3 fly pause-pipeline
    4. 1.6.2.4 fly unpause-pipeline
    5. 1.6.2.5 fly expose-pipeline
    6. 1.6.2.6 fly hide-pipeline
    7. 1.6.2.7 fly get-pipeline
    8. 1.6.2.8 fly destroy-pipeline
    9. 1.6.2.9 fly order-pipelines
    10. 1.6.2.10 fly archive-pipeline
  3. 1.6.3 Grouping Pipelines
    1. 1.6.3.1 Managing Instanced Pipelines
      1. 1.6.3.1.1 fly order-instanced-pipelines
      2. 1.6.3.1.2 Managing Jobs and Resources

Examples

Hello, world!

The following example makes use of an embedded task config to define the smallest possible pipeline.

jobs:
- name: hello-world
  plan:
  - task: say-hello
    config:
      platform: linux
      image_resource:
        type: registry-image
        source: {repository: alpine}
      run:
        path: echo
        args: ["Hello, world!"]

Small Go Project

When it comes to real-world projects, task configs are usually stored alongside the code that it's testing. This makes the pipeline a bit smaller and makes the config able to change without needing to reconfigure the pipeline.

---
resources:
- name: booklit
  type: git
  source:
    uri: https://github.com/concourse/booklit
    branch: master

jobs:
- name: unit
  plan:
  - get: booklit
    trigger: true
  - task: unit
    file: booklit/ci/test.yml

Chaining Jobs

Promoting resources to downstream jobs is done by setting get step passed on a get step.

Note that nothing in unit says anything about triggering build. Job definitions are self-contained; they describe their dependencies and where they come from, which results in a dependency flow that Concourse pushes forward.

---
resources:
- name: booklit
  type: git
  source:
    uri: https://github.com/concourse/booklit
    branch: master

jobs:
- name: unit
  plan:
  - get: booklit
    trigger: true
  - task: run-unit
    file: booklit/ci/test.yml

- name: build
  plan:
  - get: booklit
    passed: [unit]
    trigger: true
  - task: run-build
    file: booklit/ci/build.yml

Using Resource Types

Resource Types can be used to extend the functionality of your pipeline and provide deeper integrations. This example uses one to trigger a job whenever a new Dinosaur Comic is out.

---
resource_types:
- name: rss
  type: registry-image
  source:
    repository: suhlig/concourse-rss-resource
    tag: latest

resources:
- name: dinosaur-comics
  type: rss
  source:
    url: http://www.qwantz.com/rssfeed.php

jobs:
- name: announce
  plan:
  - get: dinosaur-comics
    trigger: true