1.14.1.2 Common Pipeline Practices

Parallelizing Get Steps in Jobs

All jobs usually have get steps as their first set of steps.

jobs:
- name: awesome-job
  plan:
  - get: cool-code
  - get: funny-binary
  - get: the-weather
  - task: business-stuff

To reduce the waiting time to the length of the longest running get step, put all get steps under an in_parallel step.

jobs:
- name: awesome-job
  plan:
  - in_parallel:
    - get: cool-code
    - get: funny-binary
    - get: the-weather
  - task: business-stuff

Specify Inputs for Put Steps

By default put step's have all artifacts from a job mounted in their resource container. This can result in long initialization times for put steps. It's likely that a put step only needs a subset of all available artifacts generated throughout the job.

There are two ways to specify which artifacts to send to a put step. You can specify detect as the put step inputs or you can pass in an exact list of all artifacts the put step needs.

Using detect:

jobs:
- name: the-job
  plan: # Get some artifacts
  - in_parallel:
    - get: apples
    - get: apple-basket
    - get: monkeys
  # using detect will result in "apples-location" and "basket" being passed in
  # "monkeys" will not be passed in
  - put: apples-in-basket
    inputs: detect
    params:
      apples-location: apples/location # matches the first get step
      basket: apple-basket # matches the second get step

Specifying the exact inputs needed for the put step:

jobs:
- name: the-job
  plan: # Get some artifacts
  - in_parallel:
    - get: apples
    - get: apple-basket
    - get: monkeys
  - put: apples-in-basket
    inputs: [apples, apple-basket] # specify the exact inputs needed
    params:
      apples-location: apples/location
      basket: apple-basket

Putting Task Configs in Files

A lot of the pipeline examples that you will find on this site and in resource repos will embed a task step config directly in the pipeline. This is a nice way of clearly seeing what inputs/outputs the task uses. Tasks are usually designed to be used in multiple places, maybe with slightly different configuration. To support this scenario, most users store task configs in files instead of embedding the config directly in the pipeline.

Here's what this looks like in practice:

platform: linux

image_resource: # define a default image for the task to use
  type: registry-image
  source:
    repository: busybox

run:
  path: date
  args: ["+%Y-%m-%d"]

Using the task in a pipeline:

resources:
- name: ci
  type: git
  source:
    uri: https://github.com/concourse/examples.git

jobs:
- name: the-best-job
  plan:
  - get: ci
  - task: today
    file: ci/tasks/print-date.yml

Get Images for Tasks Instead of using Anonymous Image Resources

It is easy to let Concourse fetch images for tasks right when they are needed by using the task-config.image_resource field in a task config. It's the easy out-of-the-box solution. Another way is to pass the image for a task as an input to the job by setting the task step image field. This also allows you to track the version of the image being used by the task and also avoid getting rate-limited by configuring the resource with credentials.

Before - Anonymous Image Fetching for Tasks

jobs:
- name: job
  plan:
  - task: simple-task
    config:
      platform: linux
      image_resource: # anonymous image resource
        type: registry-image
        source:
          repository: busybox
      run:
        path: echo
        args: ["Hello world!"]

After - Passing Task Image as Job Inputs

resources:
- name: busybox
  type: registry-image
  source:
    repository: busybox
    username: ((docker.user))
    password: ((docker.password))

jobs:
- name: job
  plan:
  - get: busybox # pass image into job
  - task: simple-task
    image: busybox # use image for task. Overrides anonymous image
    config:
      platform: linux
      run:
        path: echo
        args: ["Hello world!"]