The Concourse 2018 Community Survey is here and we'd love your feedback!

Concourse

Concourse is an open-source continuous thing-doer.

Built on the simple mechanics of resources, tasks, and jobs, Concourse presents a general approach to automation that makes it great for CI/CD.

Concourse is distributed as a single concourse binary, making it easy to run just about anywhere, especially with Docker.

$ wget https://concourse-ci.org/docker-compose.yml
$ docker-compose up
Creating docs_concourse-db_1 ... done
Creating docs_concourse_1    ... done

Concourse will be running at 127.0.0.1:8080. You can log in with the username/password as test/test.

Next, install fly by downloading it from the web UI and target your local Concourse as the test user:

$ fly -t tutorial login -c http://127.0.0.1:8080 -u test -p test
logging in to team 'main'

target saved

Then, head to Hello World to kick the tires!

Configuration As Code

resources:
- name: booklit
  type: git
  source: {uri: "https://github.com/vito/booklit"}

jobs:
- name: unit
  plan:
  - get: booklit
    trigger: true
  - task: test
    file: booklit/ci/test.yml

You can think of a pipeline as a distributed, higher-level, continuously-running Makefile.

Each entry under resources is a dependency, and each entry under jobs describes a plan to run when the job is triggered (either manually or by a get step).

Jobs can depend on resources that have passed through prior jobs. The resulting sequence of jobs and resources is a dependency graph that continuously pushes your project forward, from source code to production.

Fancy Visualization

Your pipeline configuration is then visualized in the web UI, taking only one click to get from a red (failed) box to seeing why it failed.

The visualization also provides a "gut check" feedback loop - if it looks wrong, it probably is wrong.

CI Under Source Control

$ fly -t ci set-pipeline -p booklit -c pipeline.yml
$ vim pipeline.yml
$ fly -t ci set-pipeline -p booklit -c pipeline.yml
$ git add pipeline.yml
$ git commit -m "initial pipeline"

All administration is done using the fly CLI. The fly set-pipeline command pushes the config up to Concourse. Once it looks good, you can then check the file in to source control. This makes it easy to recover if your Concourse server burns down.

Reproducible, Debuggable Builds

$ fly -t ci intercept -j booklit/unit -s unit
root@2c15ff11:/tmp/build/0df9eea0# ps
    PID TTY          TIME CMD
    171 pts/1    00:00:00 bash
   1876 pts/1    00:00:00 ps
root@2c15ff11:/tmp/build/0df9eea0# ls
depspath  gopath
root@2c15ff11:/tmp/build/0df9eea0# █

Everything runs in containers, ensuring a clean environment on every run. Each task specifies its own image, giving it full control over its dependencies, rather than managing them on your workers.

The fly intercept command will pop you right into one of your build's containers, which can be useful for debugging.

Rapid Local Iteration

~/booklit $ fly -t ci execute -c ci/test.yml
executing build 1 at http://127.0.0.1:8080/builds/1
initializing
booklit: 4.74 MiB/s 0s
running gopath/src/github.com/vito/booklit/ci/test
fetching dependencies...
installing ginkgo...
running tests...
█

The fly execute command executes a task as a one-off build, with your local changes. This will run your code in exactly the same way it would run in your pipeline, without you having to repeatedly push broken commits until it works. Achieve the fabled green build #1!

When a job fails, you can also use fly execute with -j flag to run with the same inputs as the failed job. You can then replace an input with your local changes with -i to test if your fix is valid.

Bring Your Own Integrations

resource_types:
- name: slack
  type: docker-image
  source:
    repository: quay.io/ahelal/bender
    tag: latest

resources:
- name: slack-deploy
  type: slack
  source:
    slack_token: ((slack_token))
    channel: "admin"
    grammar: "dang! deploy it now!"

jobs:
- name: prod-deploy
  plan:
  - get: slack-deploy
    trigger: true
  - # ...

Concourse does not have a complex plugin system. Instead, it has a single strong abstraction.

The resources section of a pipeline lists Resources, which are abstract external locations where your pipeline will monitor for changes, fetch bits from, and push bits to.

For example, a resource with type git refers to a git repository, which will be cloned in a get step and pushed to in a put step. Behind the scenes, Concourse will continuously run git fetch to look for new commits that jobs may want to trigger on.

At its core, though, Concourse knows nothing about Git. It comes with a git resource type out of the box, but you could just as easily bring your own into your pipeline. Resource types are implemented as container images containing scripts - using the docker-image resource type, they can be fetched from a Docker registry.