1.1.3 Inputs and Outputs

Overview

This section is going to go over how to pass data between different steps in a job. We'll continue building on our hello-world.yml pipeline.

In the previous section we learned that steps are where we tell Concourse what to run (i.e. run my tests, run this bash script, build this image, etc.). We are going to expand on the concept of steps and show you how to pass artifacts/files between tasks using inputs and outputs.

What are inputs and outputs?

The simple answer is that inputs and outputs are directories that get passed between steps. We'll refer to both inputs and outputs as artifacts.

Let's start exploring how artifacts work by adding a task-config.outputs to our hello-world-task.

jobs:
- name: hello-world-job
  plan:
  - task: hello-world-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: busybox
          tag: latest
      # Adds a "the-artifact" directory to our task
      outputs:
      - name: the-artifact
      run:
        # Change the command to `ls -lF` to see
        # what the task sees in its working directory
        path: ls
        args: ["-lF"]

Update the pipeline and trigger the job:

$ fly -t tutorial set-pipeline -p hello-world -c hello-world.yml
$ fly -t tutorial trigger-job --job hello-world/hello-world-job --watch
...
selected worker: 57d7419112ca
running ls -lF
total 4
drwxr-xr-x    2 root     root          4096 Apr  8 16:42 the-artifact/
succeeded

We can see that in the task's current working directory there is now a folder called the-artifact. Concourse makes output directories for you and will pass any contents inside the folder onto later steps. Let's see how that works next.

Passing outputs to another task

To pass artifacts from one task to another, the first task must declare an output. The second task must then declare an input with the exact same name. Let's update the pipeline to do the following:

• Have the first task create a file inside the-artifact

• Create a second task to read the file inside the-artifact from the previous step

jobs:
- name: hello-world-job
  plan:
  - task: hello-world-task
    config:
      platform: linux
      image_resource: &image # Declaring a YAML anchor
        type: registry-image
        source:
          repository: busybox
          tag: latest
      outputs:
      - name: the-artifact
      run:
        # This is a neat way of embedding a script into a task
        path: sh
        args:
        - -cx
        - |
          ls -l .
          echo "hello from another step!" > the-artifact/message
  # Add a second task that reads the contents of the-artifact/message
  - task: read-the-artifact
    config:
      platform: linux
      image_resource: *image
      # To receive "the-artifact", specify it as an input
      inputs:
      - name: the-artifact
      run:
        path: sh
        args:
        - -cx
        - |
          ls -l .
          cat the-artifact/message

Update the pipeline and trigger the job:

$ fly -t tutorial set-pipeline -p hello-world -c hello-world.yml
$ fly -t tutorial trigger-job --job hello-world/hello-world-job --watch

initializing
selected worker: 57d7419112ca
running sh -cx ls -l .
echo "hello from another step!" > the-artifact/message

+ ls -l .
total 4
drwxr-xr-x    2 root     root          4096 Feb 26 19:09 the-artifact
+ echo 'hello from another step!'
initializing
selected worker: 57d7419112ca
running sh -cx ls -l .
cat the-artifact/message

+ ls -l .
total 4
drwxr-xr-x    1 root     root          4096 Feb 26 19:09 the-artifact
+ cat the-artifact/message
hello from another step!
succeeded

It's a bit hard to see when the first task stops and the second one starts in the terminal. Looking at the build from the web UI makes this clearer:

With the above pipeline we can see that the file made in the first step is made available in the second step via the the-artifact.

How does Concourse track artifacts?

As Concourse is running the steps in your job, it is creating a list of named artifacts. Let's see what that looks like for the pipeline we just ran.

• Concourse runs the task step hello-world-task with output the-artifact

  Concourse creates an empty artifact, assigns it the name the-artifact, and mounts it inside the task container.

• Concourse runs the task step read-the-artifact with input the-artifact

  Concourse looks up, in its list of artifacts for the build, for an artifact named the-artifact, and mounts it inside the task container. If no input with that name is found then the build would fail.

The next section will introduce you to the concept of Resources.

If you have any feedback for this tutorial please share it in this GitHub discussion