The following are practices that we see a lot of people use in their pipelines. These are by no means "Best" practices, they are simply common and may or may not work for you and your team.
All jobs usually have get steps as their first set of steps.
jobs: - name: awesome-job plan: - get: cool-code - get: funny-binary - get: the-weather - task: business-stuff
To reduce the waiting time to the length of longest running get step, put all get steps under an
jobs: - name: awesome-job plan: - in_parallel: - get: cool-code - get: funny-binary - get: the-weather - task: business-stuff
put step's have all artifacts from a job mounted in their resource container. This can result in long initialization times for put steps. It's likely that a
put step only needs a subset of all available artifacts generated throughout the job.
jobs: - name: the-job plan: # Get some artifacts - in_parallel: - get: apples - get: apple-basket - get: monkeys # using detect will result in "apples-location" and "basket" being passed in # "monkeys" will not be passed in - put: apples-in-basket input: detect params: apples-location: apples/location # matches the first get step basket: apple-basket # matches the second get step
Specifying the exact inputs needed for the
jobs: - name: the-job plan: # Get some artifacts - in_parallel: - get: apples - get: apple-basket - get: monkeys - put: apples-in-basket input: [apples, apple-basket] # specify the exact inputs needed params: apples-location: apples/location basket: apple-basket
A lot of the pipeline examples that you will find on this site and in resource repos will embed a
config directly in the pipeline. This is a nice way of clearly seeing what inputs/outputs the task uses. Tasks are usually designed to be used in multiple places, maybe with slightly different configuration. To support this scenario, most users store task configs in files instead of embedding the config directly in the pipeline.
Here's what this looks like in practice:
todays-date.yml, stored in some git repo under a folder named "task"
platform: linux image_resource: # define a default image for the task to use type: registry-image source: repository: busybox run: path: date args: ["+%Y-%m-%d"]
Using the task in a pipeline:
resources: - name: ci type: git source: uri: http://github.com/username/my-ci-repo jobs: - name: the-best-job plan: - get: ci - task: what-day-is-it file: ci/tasks/todays-date.yml
It is easy to let Concourse fetch images for tasks right when they are needed by using the
field in a task config. It's the easy out-of-the-box solution. Another way is to pass the image for a task as an input to the job by setting the
image field. This also allows you to track the version of the image being used by the task and also avoid getting rate-limited by configuring the resource with credentials.
Before - Anonymous Image Fetching for Tasks
jobs: - name: job plan: - task: simple-task config: platform: linux image_resource: # anonymous image resource type: registry-image source: repository: busybox run: path: echo args: ["Hello world!"]
After - Passing Task Image as Job Inputs
resources: - name: busybox type: registry-image source: repository: busybox username: ((docker.user)) password: ((docker.password)) jobs: - name: job plan: - get: busybox # pass image into job - task: simple-task image: busybox # use image for task. Overrides anonymous image config: platform: linux run: path: echo args: ["Hello world!"]