Concourse

5.2 Tasks

The smallest configurable unit in a Concourse pipeline is a single task. A task can be thought of as a function from inputs to outputs that can either succeed or fail.

Going a bit further, ideally tasks are pure functions: given the same set of inputs, it should either always succeed with the same outputs or always fail. This is entirely up to your script's level of discipline, however. Flaky tests or dependencies on the internet are the most common source of impurity.

Once you have a running Concourse deployment, you can start configuring your tasks and executing them interactively from your terminal with the Fly commandline tool.

Once you've figured out your tasks's configuration, you can reuse it for a Job in your Pipeline.

Conventionally a task's configuration is placed in the same repository as the code it's testing, possibly under some ci directory.

A task's configuration specifies the following:

platform: string

Required. The platform the task should run on. By convention, windows, linux, or darwin are specified. This determines the pool of workers that the task can run against. The base deployment provides Linux workers.

image_resource: resource

Where resource is:

type: string

Required. The type of the resource. Usually docker-image.

source: object

Required. The configuration for the resource; see source.

params: object

Optional. A map of arbitrary configuration to forward to the resource. Refer to the resource type's documentation to see what it supports.

version: object

Optional. A specific version of the resource to fetch. This should be a map with string keys and values. If not specified, the latest version will be fetched.

Optional. The base image of the container, as provided by a resource definition.

You can use any resource that returns a filesystem in the correct format (a /rootfs directory and a metadata.json file in the top level) but normally this will be the Docker Image resource. If you'd like to make a resource of your own that supports this please use that as a reference implementation for now.

If you want to use an artifact source within the plan containing an image, you must set the image in the plan step instead.

rootfs_uri: string

Optional. A string specifying the rootfs uri of the container, as interpreted by your worker's Garden backend.

image_resource is a preferred way to specify base image and rootfs_uri is not recommended. With rootfs_uri image fetching is delegated to backend which does not guarantee image caching and might result in some permission errors. You should only use this if you cannot use image_resource for some reason, and you know what you're doing.

inputs: [input]

Where input is:

name: string

Required. The logical name of the input.

path: string

Optional. The path where the input will be placed. If not specified, the input's name is used.

Paths are relative to the working directory of the task. Absolute paths are not respected.

optional: bool

Optional. If true, then the input is not required by the task. The task may run even if this input is missing.

An optional input that is missing will not appear in the current directory of the running task.

Optional. The set of artifacts used by task, determining which artifacts will be available in the current directory when the task runs.

These are satisfied by get steps or outputs of a previous task. These can also be provided by -i with fly execute.

If any required inputs are missing at run-time, then the task will error immediately.

outputs: [output]

Where output is:

name: string

Required. The logical name of the output. The contents under path will be made available to the rest of the plan under this name.

path: string

Optional. The path to a directory where the output will be taken from. If not specified, the output's name is used.

Paths are relative to the working directory of the task. Absolute paths are not respected.

Note that this value must not overlap with any other inputs or outputs. Each output results in a new empty directory that your task should place artifacts in; if the path overlaps it'll clobber whatever files used to be there.

Optional. The artifacts produced by the task.

Each output configures a directory to make available to later steps in the build plan. The directory will be automatically created before the task runs, and the task should place any artifacts it wants to export in the directory.

caches: [cache]

Where cache is:

path: string

Required. The path to a directory to be cached.

Paths are relative to the working directory of the task. Absolute paths are not respected.

Note that this value must not overlap with any other caches in the same task. Each cache results in a new empty directory that your task can place artifacts in; if the path overlaps it'll clobber whatever files used to be there.

Optional. The cached directories shared between task runs.

On the task's first run, all cache directories will be empty. It is the responsibility of the task to populate these directories with any artifacts to be cached. On subsequent runs, the cached directories will contain those artifacts.

Caches are scoped to the worker the task is run on, so you will not get a cache hit when subsequent builds run on different workers. This also means that caching is not intended to share state between workers, and your task should be able to run whether or not the cache is warmed.

Caches are also scoped to a particular task name inside of a pipeline's job. As a consequence, if the job name, step name or cache path are changed, the cache will not be used. This also means that caches do not exist for one-off builds.

run: run-config

Where run-config is:

path: string

Required. The command to execute.

This is commonly a path to a script provided by one of the task's inputs, e.g. my-resource/scripts/test. It could also be a command like bash (respecting standard $PATH lookup rules), or an absolute path to a file to execute, e.g. /bin/bash.

args: [string]

Optional. Arguments to pass to the command. Note that when executed with Fly, any arguments passed to Fly are appended to this array.

dir: string

Optional. A directory, relative to the initial working directory, to set as the working directory when running the script.

user: string

Optional. Explicitly set the user to run as. If not specified, this defaults to the user configured by the task's image. If not specified there, it's up to the Garden backend, and may be e.g. root on Linux.

Required. The command to execute in the container.

Note that this is not provided as a script blob, but explicit path and args values; this allows fly to forward arguments to the script, and forces your config .yml to stay fairly small.

params: {string: string}

Optional. A key-value mapping of values that are exposed to the task via environment variables.

Use this to provide things like credentials, not to set up the task's Bash environment (they do not support interpolation).

  1. Running Tasks
    1. fly execute
  2. Task Environment

Examples

Ruby App

This configuration specifies that the task must run with the ruby:2.1 Docker image with a my-app input, and when the task is executed it will run the scripts/test script in the same repo.

---
platform: linux

image_resource:
  type: docker-image
  source:
    repository: ruby
    tag: '2.1'

inputs:
- name: my-app

run:
  path: my-app/scripts/test

Producing Outputs

A task can configure outputs to produce artifacts that can then be propagated to a put step or another task step in the same plan. They can also be downloaded with fly execute by passing -o.

---
platform: linux

image_resource: # ...

inputs:
- name: project-src

outputs:
- name: built-project

run:
  path: project-src/ci/build

...assuming project-src/ci/build looks something like:

#!/bin/bash

set -e -u -x

export GOPATH=$PWD/project-src

go build -o built-project/my-project \
  github.com/concourse/my-project

...this task could then be used in a build plan like so:

plan:
- get: project-src
- task: build-bin
  file: project-src/ci/build.yml
- put: project-bin
  params: file: built-project/my-project

Caching Ephemeral State

The following task and script could be used by a Node project to cache the node_modules directory:

---
platform: linux

image_resource: # ...

inputs:
- name: project-src

caches:
- path: project-src/node_modules

run:
  path: project-src/ci/build

...assuming project-src/ci/build looks something like:

#!/bin/bash

set -e -u -x

cd project-src
npm install

# ...

...this task would cache the contents of project-src/node_modules between runs of this task on the same worker.

Private Docker Registry

The following task uses an image from a private registry. Assuming the CA is configured properly on the workers, SSL should Just Work™.

Note that, at present, credentials must be inlined when using image_resource. We recognize that this kind of sucks, since tasks are often checked in. Until we fix this, you can use image in your pipeline instead. (However this will prevent fly execute from having an image available. Bummer.)

---
platform: linux

image_resource:
  type: docker-image
  source:
    repository: my.local.registry:8080/my/image
    username: myuser
    password: mypass
    email: x@x.com

inputs:
- name: my-app

run:
  path: my-app/scripts/test