The smallest configurable unit in a Concourse pipeline is a single task. A task can be thought of as a function from inputs to outputs that can either succeed or fail.

Going a bit further, ideally tasks are pure functions: given the same set of inputs, it should either always succeed with the same outputs or always fail. This is entirely up to your script's level of discipline, however. Flaky tests or dependencies on the internet are the most common source of impurity.

Once you have a running Concourse deployment, you can start configuring your tasks and executing them interactively from your terminal with the Fly commandline tool.

Once you've figured out your tasks's configuration, you can reuse it for a Job in your Pipeline.

Configuring a Task

Conventionally a task's configuration is placed in the same repository as the code it's testing, possibly under some ci directory. For a simple Ruby app with unit tests it may be called ci/unit.yml, and looks something like:

platform: linux

  type: docker-image
    repository: ruby
    tag: '2.1'

- name: my-app

  path: my-app/scripts/test

This configuration specifies that the task must run with the ruby:2.1 Docker image with a my-app input, and when the task is executed it will run the scripts/test script in the same repo.

A task's configuration specifies the following:

platform: string

Required. The platform the task should run on. By convention, windows, linux, or darwin are specified. This determines the pool of workers that the task can run against. The base deployment provides Linux workers.

image_resource: object

Optional. The base image of the container. This style of specifying the base image has the same effect as image: above but uses Concourse resources to download the image.

The configuration has the following attributes:

type: string

Required. The type of the resource. Usually docker-image.

source: object

Required. The configuration for the resource; see source. If not specified, the input's name is used.

Paths are relative to the working directory of the task. Absolute paths are not respected.

params: object

Optional. A map of arbitrary configuration to forward to the resource. Refer to the resource type's documentation to see what it supports.

version: object

Optional. A specific version of the resource to fetch. This should be a map with string keys and values. If not specified, the latest version will be fetched.

The following example configures the task to use the golang:1.6 Docker image:

  type: docker-image
  source: {repository: golang, tag: "1.6"}

...and the following example uses an insecure private Docker registry with a username and password:

  type: docker-image
    repository: my.local.registry:8080/my/image
    insecure_registries: ["my.local.registry:8080"]
    username: myuser
    password: mypass
    email: x@x.com

You can use any resource that returns a filesystem in the correct format (a /rootfs directory and a metadata.json file in the top level) but normally this will be the Docker Image resource. If you'd like to make a resource of your own that supports this please use that as a reference implementation for now.

If you want to use an artifact source within the plan containing an image, you must set the image in the plan step instead.

rootfs_uri: string

Optional. A string specifying the rootfs uri of the container, as interpreted by your worker's Garden backend.

image_resource is a preferred way to specify base image and rootfs_uri is not recommended. With rootfs_uri image fetching is delegated to backend which does not guarantee image caching and might result in some permission errors. You should only use this if you cannot use image_resource for some reason, and you know what you're doing.

inputs: [object]

Optional. The set of artifacts used by task, determining which artifacts will be available in the current directory when the task runs.

These are satisfied by get steps or artifacts produced by outputs of a previous task in a job's build plan. These can also be provided by -i with execute.

This determines which artifacts will propagate into the task, as the build plan executes. If any required inputs are missing at run-time, then the task will error immediately.

Each input has the following attributes:

name: string

Required. The logical name of the input.

path: string

Optional. The path where the input will be placed. If not specified, the input's name is used.

Paths are relative to the working directory of the task. Absolute paths are not respected.

optional: bool

Optional. If true, then the input is not required by the task. The task may run even if this input is missing.

An optional input that is missing will not appear in the current directory of the running task.

outputs: [object]

Optional. The artifacts produced by the task.

Each output configures a directory to make available to later steps in the build plan. The directory will be automatically created before the task runs, and the task should place any artifacts it wants to export in the directory.

Each output has the following attributes:

name: string

Required. The logical name of the output. The contents under path will be made available to the rest of the plan under this name.

path: string

Optional. The path to a directory where the output will be taken from. If not specified, the output's name is used.

Paths are relative to the working directory of the task. Absolute paths are not respected.

Note that this value must not overlap with any other inputs or outputs. Each output results in a new empty directory that your task should place artifacts in; if the path overlaps it'll clobber whatever files used to be there.

For example, the following task and script would be used to propagate a built binary to later steps:

platform: linux

image_resource: # ...

- name: project-src

- name: built-project

  path: project-src/ci/build

...assuming project-src/ci/build looks something like:


set -e -u -x

export GOPATH=$PWD/project-src

go build -o built-project/my-project github.com/concourse/my-project

...this task could then be used in a build plan like so:

- get: project-src
- task: build-bin
  file: project-src/ci/build.yml
- put: project-bin
  params: file: built-project/my-project

caches: [object]

Optional. The cached directories shared between task runs.

On the task's first run, all cache directories will be empty. It is the responsibility of the task to populate these directories with any artifacts to be cached. On subsequent runs, the cached directories will contain those artifacts.

Caches are scoped to the worker the task is run on, so you will not get a cache hit when subsequent builds run on different workers. This also means that caching is not intended to share state between workers, and your task should be able to run whether or not the cache is warmed.

Caches are also scoped to a particular task name inside of a pipeline's job. As a consequence, if the job name, step name or cache path are changed, the cache will not be used. This also means that caches do not exist for one-off builds.

Each cache has the following attributes:

path: string

Required. The path to a directory to be cached.

Paths are relative to the working directory of the task. Absolute paths are not respected.

Note that this value must not overlap with any other caches in the same task. Each cache results in a new empty directory that your task can place artifacts in; if the path overlaps it'll clobber whatever files used to be there.

For example, the following task and script define a node project that takes advantage of task caches for its node modules:

platform: linux

image_resource: # ...

- name: project-src

- path: project-src/node_modules

  path: project-src/ci/build

...assuming project-src/ci/build looks something like:


set -e -u -x

cd project-src
npm install

# ...

...this task would cache the contents of project-src/node_modules between runs of this task on the same worker.

run: object

Required. The command to execute in the container.

path: string

Required. The command to execute, relative to the task's working directory. For a script living in a resource's repo, you must specify the full path to the resource, i.e. my-resource/scripts/test.

Paths are relative to the working directory of the task. Absolute paths are not respected.

args: [string]

Optional. Arguments to pass to the command. Note that when executed with Fly, any arguments passed to Fly are appended to this array.

dir: string

Optional. A directory, relative to the initial working directory, to set as the working directory when running the script.

user: string

Optional. Explicitly set the user to run as. If not specified, this defaults to the user configured by the task's image. If not specified there, it's up to the Garden backend, and may be e.g. root on Linux.

Note that this is not provided as a script blob, but explicit path and args values; this allows fly to forward arguments to the script, and forces your config .yml to stay fairly small.

params: {string: string}

Optional. A key-value mapping of values that are exposed to the task via environment variables.

Use this to provide things like credentials, not to set up the task's Bash environment (they do not support interpolation).

Anatomy of a running task

A task runs in a new container every time, using the image provided by image_resource as its base filesystem (i.e. /).

The command specified by run will be executed in a working directory containing each of the inputs. If any inputs are missing the task will not run (and the container will not even be created).

The working directory will also contain empty directories for each of the outputs. The task must place artifacts in the output directories for them to be exported. This meshes well with build tools with configurable destination paths.

If your build tools don't support output paths you'll have to copy bits around. If it's a git repo that you're modifying you can do a local git clone ./input ./output, which is much more efficient than cp, and then work out of ./output.

Any params configured will be set in the environment for the task's command, along with any environment variables provided by the task's image (i.e. ENV rules from your Dockerfile).

The user the command runs as is determined by the image. If you're using the Docker Image resource, this will be the user set by a USER rule in your Dockerfile, or root if not specified.

Another relevant bit of configuration is privileged, which determines whether the user the task runs as will have full privileges (primarily when running as root). This is intentionally not configurable by the task itself, to prevent privilege escalation by way of pull requests to repositories containing task configs.

Putting all this together, the following task config:

platform: linux

  type: docker-image
    repository: golang
    tag: '1.6'

  SOME_PARAM: some-default-value

- name: some-input
- name: some-input-with-custom-path
  path: some/custom/path

- name: some-output

  path: sh
  - -exc
  - |
    go version
    find .
    touch some-output/my-built-artifact

...will produce the following output:

+ whoami
+ env
+ go version
go version go1.6 linux/amd64
+ find .
+ touch some-output/my-built-artifact

...and propagate my-built-artifact to any later tasks or puts that reference the some-output artifact, in the same way that this task had some-input as an input.

Running tasks with fly

Fly is a command-line tool that can be used to execute a task configuration against a Concourse deployment. This provides a fast feedback loop for iterating on the task configuration and your code.

For more information, see execute.