1.11 Tasks

The smallest configurable unit in a Concourse pipeline is a single task. A task can be thought of as a function from task.inputs to task.outputs that can either succeed or fail.

Going a bit further, ideally tasks are pure functions: given the same set of inputs, it should either always succeed with the same outputs or always fail. This is entirely up to your script's level of discipline, however. Flaky tests or dependencies on the internet are the most common source of impurity.

Once you have a running Concourse deployment, you can start configuring your tasks and executing them interactively from your terminal with the Fly commandline tool.

Once you've figured out your task's configuration, you can reuse it for a Job in your Pipeline.

Conventionally a task's configuration is placed in the same repository as the code it's testing, possibly under some ci directory.

A task's configuration specifies the following:

task schema

The platform the task should run on. This determines the pool of workers that the task can run against.

Technically any string value is allowed so long as a worker advertises the same platform, but in practice only linux, darwin, and windows are in use.

The container image to run with, as provided by an anonymous resource definition.

Whenever the task runs, the anonymous resource will be checked to discover the latest version available. The image will then be fetched onto the worker, if necessary, just prior to running the task.

To use an image provided by a previous step within your build plan, set task step image on the task step instead.

NOTE: This field is only required for tasks targeting the Linux platform. This field will be ignored for Windows and Darwin workers. Windows containers are currently not supported and Darwin does not have native containers. The task will run inside a clean temporary directory on the Windows/Darwin worker with any inputs and outputs copied into the same directory. Any dependencies should be pre-installed on the worker.

The following task config will use the golang Docker image to run go version:

platform: linux

image_resource:
  type: registry-image
  source: {repository: golang}

run:
  path: go
  args: [version]

anonymous_resource schema

The type of the resource. Usually registry-image.

You can use any resource type that returns a filesystem in the correct format: a /rootfs directory containing a full filesystem, and a metadata.json file containing.

The configuration for the resource; see resource.source.

A map of arbitrary configuration to forward to the resource. Refer to the resource type's documentation to see what it supports.

A specific version of the resource to fetch. This should be a map with string keys and values. If not specified, the latest version will be fetched.

The set of artifacts used by task, determining which artifacts will be available in the current directory when the task runs.

These are satisfied by get steps or task.outputs of a previous task. These can also be provided by -i with fly execute.

If any required inputs are missing at run-time, then the task will error immediately.

input schema

The name of the input.

The path where the input will be placed. If not specified, the input's name is used.

Paths are relative to the working directory of the task. Absolute paths are not respected.

Default false. If true, then the input is not required by the task. The task may run even if this input is missing.

An optional input that is missing will not appear in the current directory of the running task.

The artifacts produced by the task.

Each output configures a directory to make available to later steps in the build plan. The directory will be automatically created before the task runs, and the task should place any artifacts it wants to export in the directory.

output schema

The name of the output. The contents under path will be made available to the rest of the plan under this name.

The path to a directory where the output will be taken from. If not specified, the output's name is used.

Paths are relative to the working directory of the task. Absolute paths are not respected.

The cached directories shared between task runs.

On the task's first run, all cache directories will be empty. It is the responsibility of the task to populate these directories with any artifacts to be cached. On subsequent runs, the cached directories will contain those artifacts.

Caches are scoped to the worker the task is run on, so you will not get a cache hit when subsequent builds run on different workers. This also means that caching is not intended to share state between workers, and your task should be able to run whether or not the cache is warmed.

Caches are also scoped to a particular task name inside of a pipeline's job. As a consequence, if the job name, step name or cache path are changed, the cache will not be used. This also means that caches do not exist for one-off builds.

cache schema

The path to a directory to be cached.

Paths are relative to the working directory of the task. Absolute paths are not respected.

A key-value mapping of string keys and values that are exposed to the task via environment variables.

Pipelines can override these params by setting task step params on the task step. This is a common method of providing credentials to a task.

The command to execute in the container.

Note that this is not provided as a script blob, but explicit path and args values; this allows fly to forward arguments to the script, and forces your config .yml to stay fairly small.

command schema

The name of or path to the executable to run.

path is relative to the working directory. If dir is specified to set the working directory, then path is relative to it.

This is commonly a path to a script provided by one of the task's inputs, e.g. my-resource/scripts/test. It could also be a command like bash (respecting standard $PATH lookup rules), or an absolute path to a file to execute, e.g. /bin/bash.

Arguments to pass to the command. Note that when executed with Fly, any arguments passed to fly execute are appended to this array.

A directory, relative to the initial working directory, to set as the working directory when running the script.

Explicitly set the user to run as. If not specified, this defaults to the user configured by the task's image. If not specified there, it's up to the Garden backend, and may be e.g. root on Linux.

A string specifying the rootfs uri of the container, as interpreted by your worker's Garden backend.

task.image_resource is the preferred way to specify base image. You should only use this if you have no other option and you really know what you're doing.

CPU and memory limits to enforce on the task container.

Note that these values, when specified, will override any limits set by passing the --default-task-cpu-limit or --default-task-memory-limit flags to the concourse web command.

container_limits schema

The maximum amount of CPU available to the task container, measured in shares. 0 means unlimited.

The maximum amount of memory available to the task container, measured in bytes. 0 means unlimited.

This configuration specifies that the task must run with the ruby:2.1 Docker image with a my-app input, and when the task is executed it will run the scripts/test script in the same repo.

---
platform: linux

image_resource:
  type: registry-image
  source:
    repository: ruby
    tag: '2.1'

inputs:
- name: my-app

run:
  path: my-app/scripts/test

A task can configure task.outputs to produce artifacts that can then be propagated to a put step or another task step in the same plan. They can also be downloaded with fly execute by passing -o.

---
platform: linux

image_resource: # ...

inputs:
- name: project-src

outputs:
- name: built-project

run:
  path: project-src/ci/build

...assuming project-src/ci/build looks something like:

#!/bin/bash

set -e -u -x

export GOPATH=$PWD/project-src

go build -o built-project/my-project \
  github.com/concourse/my-project

...this task could then be used in a build plan like so:

plan:
- get: project-src
- task: build-bin
  file: project-src/ci/build.yml
- put: project-bin
  params: file: built-project/my-project

The following task and script could be used by a Node project to cache the node_modules directory:

---
platform: linux

image_resource: # ...

inputs:
- name: project-src

caches:
- path: project-src/node_modules

run:
  path: project-src/ci/build

...assuming project-src/ci/build looks something like:

#!/bin/bash

set -e -u -x

cd project-src
npm install

# ...

...this task would cache the contents of project-src/node_modules between runs of this task on the same worker.

The following external task uses an image from a private registry. Assuming the CA is configured properly on the workers, SSL should Just Work™.

External tasks are now fully interpolated using credential manager variables and task step vars, so you can use template variables in an external task:

---
platform: linux

image_resource:
  type: registry-image
  source:
    repository: my.local.registry:8080/my/image
    username: ((myuser))
    password: ((mypass))

inputs:
- name: my-app

run:
  path: my-app/scripts/test
  args: ["Hello, world!", "((myparam))"]
Table of contents:
  1. 1.11.1 Running tasks with fly execute
  2. 1.11.2 Task runtime environment

Running tasks with fly execute

One of the most common use cases of fly is taking a local project on your computer and setting it up with a task configuration to be run inside a container in Concourse. This is useful to build Linux projects on OS X or to avoid all of those debugging commits when something is configured differently between your local and remote setup.

You can execute a task like this:

$ fly -t example execute --config tests.yml

Your files will be uploaded and the task will be executed with them. The working directory name will be used as the input name. If they do not match, you must specify -i name=. instead, where name is the input name from the task configuration.

Fly will automatically capture SIGINT and SIGTERM and abort the build when received. This allows it to be transparently composed with other toolchains.

By default, fly execute will not send extra files or large files in your current directory that would normally be ignored by your version control system. You can use the --include-ignored flag in order to send ignored files to Concourse along with those that are not ignored.

If your task needs to run as root, then you can specify the -p or --privileged flag.

Tasks in Concourse can take multiple inputs. Up until now we've just been submitting a single input (our current working directory) that has the same name as the directory.

Tasks must specify the inputs that they require as task.inputs. For fly to upload these inputs you can use the -i or --input arguments with name and path pairs. For example:

$ fly -t example execute \
    --config build-stemcell.yml \
    --input code=. \
    --input stemcells=../stemcells

This would work together with a build-stemcell.yml if its inputs: section was as follows:

inputs:
- name: code
- name: stemcells

If you specify an input, then the default input will no longer be added automatically, and you will need to explicitly list it (as with the code input above).

This feature can be used to mimic other resources and try out input combinations that would normally not be possible in a pipeline.

If the --inputs-from flag is given, the specified job will be looked up in the pipeline, and the one-off build will base its inputs on those currently configured for the job.

If any --input flags are given (see above), they will override the base set of inputs.

For example:

$ fly -t example execute \
    --config task.yml \
    --inputs-from main/integration \
    --input foo=./foo

This will trigger a one-off-build using the task.yml task config, basing its inputs on the latest candidates for the integration job in the main pipeline, with the foo input overridden to specify local code to run.

This can be used to more closely replicate the state in CI when weeding out flakiness, or as a shortcut for local development so that you don't have to upload every single resource from your local machine.

When using --inputs-from as above, you can additionally specify which input to use as the task's image by passing --image input-name.

For example, the following pipeline fetches an image via a get step and uses it for task step image:

resources:
- name: my-repo
  type: git
  source: {uri: https://example.com}

- name: some-image
  type: registry-image
  source: {repository: ubuntu}

jobs:
- name: integration
  plan:
  - get: my-repo
  - get: some-image
  - task: my-task
    file: my-repo/task.yml
    image: some-image

...so to run the same task with the same image in a one-off build, you would run:

$ fly -t example execute \
    --config task.yml \
    --inputs-from main/integration \
    --image some-image

If a task specifies outputs, then you're able to extract these back out of the build and back to your local system. For example:

$ fly -t example execute \
    --config build-stemcell.yml \
    --input code=. \
    --output stemcell=/tmp/stemcell

This would work together with a build-stemcell.yml, if its outputs: section was as follows:

outputs:
- name: stemcell

This feature is useful to farm work out to your Concourse server to build things in a repeatable manner.

Any params listed in the task configuration can be specified by using environment variables.

So, if you have a task with the following params:

params:
  FOO: fizzbuzz
  BAR:

...and you run:

BAR=hello fly execute

The task would then run with BAR as "hello", and FOO as "fizzbuzz" (its default value).

Task config files can contain Vars which can can be set during fly execute by using the -v, -y and -l flags:

fly -t example execute --config tests.yml \
  -l vars.yml \
  -v some_string="Hello World!" \
  -y some_bool=true

Any variables not satisfied via the above flags will be deferred to the configured credential manager.

To satisfy these vars when running the task in a pipeline, see task step vars.

If you want to execute a task on a worker that has a specific tag, you can do so by passing --tag:

fly -t example execute --config task.yml --tag bar

This will execute the task specified by task.yml on a worker that has been tagged bar.

Task runtime environment

A task runs in a new container every time, using the image provided by task.image_resource as its base filesystem (i.e. /).

The command specified by task.run will be executed in a working directory containing each of the task.inputs. If any input is missing, the task will not run (and the container will not even be created).

The working directory will also contain empty directories for each of the task.outputs. The task must place artifacts in the output directories for them to be exported. This meshes well with build tools with configurable destination paths.

If your build tools don't support output paths, you can configure an input and output with the same path. The directory will be populated by the input, and any changes made to the directory will propagate downstream as an output.

Any task step params configured will be set in the environment for the task's command, along with any environment variables provided by the task's image (i.e. ENV rules from your Dockerfile).

The user the command runs as is determined by the image. If you're using a Docker image, this will be the user set by a USER rule in your Dockerfile, or root, if not specified.

Another relevant bit of configuration is task step privileged, which determines whether the user the task runs as will have full privileges (primarily when running as root). This is intentionally not configurable by the task itself, to prevent privilege escalation by submitting pull requests to repositories that contain task configs.

Putting all this together, the following task config:

---
platform: linux

image_resource:
  type: registry-image
  source:
    repository: golang
    tag: '1.6'

params:
  SOME_PARAM: some-default-value

inputs:
- name: some-input
- name: some-input-with-custom-path
  path: some/custom/path

outputs:
- name: some-output

run:
  path: sh
  args:
  - -exc
  - |
    whoami
    env
    go version
    find .
    touch some-output/my-built-artifact

...will produce the following output:

+ whoami
root
+ env
USER=root
HOME=/root
GOLANG_DOWNLOAD_SHA256=5470eac05d273c74ff8bac7bef5bad0b5abbd1c4052efbdbc8db45332e836b0b
PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
GOPATH=/go
PWD=/tmp/build/e55deab7
GOLANG_DOWNLOAD_URL=https://golang.org/dl/go1.6.linux-amd64.tar.gz
GOLANG_VERSION=1.6
SOME_PARAM=some-default-value
+ go version
go version go1.6 linux/amd64
+ find .
.
./some-input
./some-input/foo
./some
./some/custom
./some/custom/path
./some/custom/path/bar
./some-output
+ touch some-output/my-built-artifact

...and propagate my-built-artifact to any later task steps or put steps that reference the some-output artifact, in the same way that this task had some-input as an input.