1.1.4 Resources

The Heart of Concourse

Resources are the heart of Concourse. Resources make Concourse tick and are the source of automation within all Concourse pipelines. Resources are how Concourse interacts with the outside world. Here's a short list of some things that resources can do:

Resources can do a lot of things! The main goal of resources is to represent some external system or object in your pipeline. That external thing can then be used as a trigger for your Jobs or your Jobs can push back and modify the external system or object. It all depends on the resource you use and what features its author has implemented.

Resources are also how Concourse tries to stay as technology agnostic as possible. For example, Concourse doesn't care what version control system you store your code in, if you deploy apps with Helm or Terraform, or what language your apps are built in. If you can put your latest and hottest tech behind the resource interface then Concourse can understand your workflow.

The Concourse team bundles a few basic resource types that come with the Linux tarball you can download from Github. You'll notice that the Linux tarball is much larger than the macOS or Windows tarball because of all the bundled resources.

You can find out which resources a worker has by running: 

$ fly -t tutorial workers --details

Resources only run on Linux workers because resources are distributed as Linux container images. There are currently no resources for macOS or Windows. Only task steps can run on macOS or Windows workers.

Versions

Resources represent the external system or object to Concourse by emitting versions. When a new version is emitted by a resource, that is how Concourse knows to start trigger jobs.

A version is any way that a resource can uniquely identify the state of the external system or object.

For example, the git resource emits versions based on the SHA of new commits it finds. A single version from the git resource will look like this to Concourse.

{ "ref": "04194bfc880c457a5b00f07db78d0532620414cc" }

Let's start digging into resources a bit more by going over the resource interface.

Resource Interface

Resources are container images that contain three executables. Each executable is run by a different type of step within a pipeline:

  • /opt/resource/check - implicitly run when a job contains a get step. Should return the latest version from the external system or object. Its responsibility is to find new versions.

  • /opt/resource/in - run in a get step. in is given a specific version (generated by a check or put step) and retrieves the files representing that version from the external system or object.

  • /opt/resource/out - run in a put step. Generates a new version, usually based on some input generated by another step in the job. Depending on the resource, this may mean sending something to the external system. For the git resource, this means pushing commits to the external git repository.

That's a high-level overview of the resource interface. Next, we will learn how to add resources to a pipeline.

Get Steps

Let's add some automation to our hello-world.yml pipeline and have it trigger on every new commit from some git repo. We will use the git resource to accomplish this.

Before proceeding, please have a git repository ready that you can push code to. The rest of the tutorial will point to github.com/concourse/examples which you should replace with your own repository or your own fork of concourse/examples.

To start, let's go back to the one step version of the pipeline.

jobs:
- name: hello-world-job
  plan:
  - task: hello-world-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: busybox
      run:
        path: echo
        args: ["Hello World!"]

Now let's add the git resource to the pipeline. First we'll add the resources key to the top-level of our yaml.

resources:

jobs:
- name: hello-world-job
  plan:
  - task: hello-world-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: busybox
      run:
        path: echo
        args: ["Hello World!"]

The resources key takes a list of resources. In our case, we're going to add one item to that list. We will name the resource repo. This is the name we'll use to refer to this specific instance of the resource within our pipeline. We'll define the type as git, which we got from the output of fly -t tutorial workers --details.

resources:
- name: repo
  type: git

jobs:
- name: hello-world-job
  plan:
  - task: hello-world-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: busybox
      run:
        path: echo
        args: ["Hello World!"]

There is one last field we need to fill out, the source field.

The source field contains the configuration for the resource. By convention, documentation for each resource's configuration is found in the README of its git repository. The source field will be different for every resource type, so you'll need to refer to that resource's documentation to determine how to fill out the source field.

If we look at the README for the git resource we'll find that it requires we set the uri field. This should point to the location of the git repository. We can specify it in https:// or ssh@ format. Let's add the uri field and set it to your repository.

If you use the ssh@ format for the uri then you'll likely need to set the private_key field as well.

If you use the https:// format and your repo is public then you don't need to set anything else. If you repository is private then you'll need to set the username and password fields to allow the git resource to authenticate. private_key can also be used for https:// uri's.

resources:
- name: repo
  type: git
  source:
    uri: https://github.com/concourse/examples.git

jobs:
- name: hello-world-job
  plan:
  - task: hello-world-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: busybox
      run:
        path: echo
        args: ["Hello World!"]

Next we need to connect the repo resource to the hello-world-job. We connect the two objects by adding a get step to the job.

resources:
- name: repo
  type: git
  source:
    uri: https://github.com/concourse/examples.git

jobs:
- name: hello-world-job
  plan:
  # Add a get step referencing the resource
  - get: repo
  - task: hello-world-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: busybox
      run:
        path: echo
        args: ["Hello World!"]

The final piece is telling Concourse to trigger the hello-world-job whenever the repo resource emits a new version. We can do that by setting trigger to true for the get step.

resources:
- name: repo
  type: git
  source:
    uri: https://github.com/concourse/examples.git

jobs:
- name: hello-world-job
  plan:
  - get: repo
    trigger: true   # tell Concourse to trigger this job when new versions are emitted
  - task: hello-world-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: busybox
      run:
        path: echo
        args: ["Hello World!"]

Let's set our pipeline and watch it automatically trigger from the web UI.

$ fly -t tutorial set-pipeline -p hello-world -c hello-world.yml

If you click the resource you'll see a list of the versions that it has emitted so far. You can expand each version to see which build it was used in and some metadata if the resource fetched any. Metadata is only populated after a version is fetched by a get step.

By default, resources will only fetch the most recent version available. At this point you can make commits to your repo and watch Concourse find them and trigger the job. If you want to populate the version history with older versions you can use fly check-resource with the --from flag.

Get Steps and Inputs

Let's tie together two concepts that we've learned so far. Get steps and task inputs.

In the previous section we learned that task steps can specify outputs that other task steps can then consume as inputs. Get steps generate one output that can then be consumed by tasks as an input. Get steps always generate output artifacts based on their name.

To find out the structure of the artifact generated by a get step you will have to refer to the resource's documentation. The documentation for the git resource tells us that the in script will clone the git repository.

Let's add repo as an input to the hello-world-task. Let's also update the command the task runs to instead print the contents of some file in the repo.

resources:
- name: repo
  type: git
  source:
    uri: https://github.com/concourse/examples.git

jobs:
- name: hello-world-job
  plan:
  - get: repo
    trigger: true
  - task: hello-world-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: busybox
      inputs: # add the get step as an input to this task
      - name: repo
      run: # read the file from the get step
        path: cat
        args: ["repo/README.md"]

Set the pipeline and we'll manually trigger the job.

$ fly -t tutorial set-pipeline -p hello-world -c hello-world.yml
$ fly -t tutorial trigger-job --job hello-world/hello-world-job --watch
started hello-world/hello-world-job #44

initializing
selected worker: d032d4471e67
running cat repo/README.md
# examples
Examples of Concourse workflows
succeeded

The job should finish successfully and print out the contents of repo/README.md or whichever file you selected from your repo.

Checks

Checks are how Concourse finds new versions of resources. In order to find any new versions Concourse has to run the check scripts for all resources across all pipelines. By default, all resources have a check_every interval of one minute.

The default check_every interval helps keep Concourse feeling snappy and responsive to changes in external systems. The negative side-effect of this is that if you have a lot of resources all hitting the same system (e.g. an internal Git system) you may end up DDOS'ing your external system! In these cases you may want to increase the check_every interval of your resource or disable it and use webhooks instead.

To see the results of the last check that ran, go to the resource's page in the pipeline and expand the top line that starts with check: <resource-name>.

If your resource is having trouble finding new versions or is configured incorrectly the resource will print a message that will be visible on its resource page for you to debug.

If you want to debug a check container the easiest way to do that will be to start a shell session in the container using fly intercept. You can pass in the URL to the resources page to start a shell session on the resource container.

$ fly -t tutorial intercept -u http://localhost:8080/teams/main/pipelines/hello-world/resources/every-1min
root@f5bf10f9-6a95-46e6-4e44-63594dace2e0:/tmp/build/check#

Put Steps

The last piece of the resource interface is put steps. Put steps are generally for pushing some change to the external system or object the resource represents. What this means will vary by resource, so again, you should carefully read the documentation for the specific resource you are using. Each resource is basically its own little application.

In order to "put" something, in the context of the git resource, we'll need to make a commit that can then be "put" to the repo.

If you haven't already, you'll probably need to provide the git resource with your private_key or a username and password, depending on the format of uri you provided.

If you're used to pushing code from your computer you probably have a private key in your ~/.ssh/ directory that you can use.

resources:
- name: repo
  type: git
  source:
    # changed uri to ssh format
    uri: git@github.com:concourse/examples.git
    # specify branch because it's required for the put step
    branch: master
    private_key: |
      -----BEGIN OPENSSH PRIVATE KEY-----
      ...
      -----END OPENSSH PRIVATE KEY-----

jobs:
- name: hello-world-job
  plan:
  - get: repo
    trigger: true
  - task: hello-world-task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: busybox
      inputs:
      - name: repo
      run:
        path: cat
        args: ["repo/README.md"]

Next let's change the hello-world-task to instead create a commit. Since we want the commit to propagate to future steps, specifically the put step we'll be adding soon, we will also need to add the repo as an output of the task.

resources:
- name: repo
  type: git
  source:
    uri: git@github.com:concourse/examples.git
    branch: master
    private_key: |
      -----BEGIN OPENSSH PRIVATE KEY-----
      ...
      -----END OPENSSH PRIVATE KEY-----

jobs:
- name: hello-world-job
  plan:
  - get: repo
    trigger: true
  - task: create-commit # renamed task
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: gitea/gitea # use any image that has the git cli
      inputs:
      - name: repo
      outputs: # Add repo as an output
      - name: repo
      params: # Change these to match your ssh key info
        EMAIL: person@example.com
        GIT_AUTHOR_NAME: Person Doe
      run:
        path: sh
        args:
        - -cx
        # this is just a bash script
        - |
          cd repo
          date +%Y-%m-%d > todays-date
          git add ./todays-date
          git config --global user.email $EMAIL
          git config --global user.name $GIT_AUTHOR_NAME
          git commit -m "Update todays date"

Lastly we can add a put step to push the commit made by the task. Put steps usually have various params that you'll need to set in order run the step correctly. If we read the documentation for the git resource we'll see that we need to specify repository field.

If you're wondering why we don't do git push from the task, that's for two reasons:

  1. The newly made commit represents a new version and Concourse won't capture it correctly if it's pushed from the task, which is outside of the resource interface (check, get, put).

  2. The private_key is not available in the task so you would get an authentication error if you git pushed.

resources:
- name: repo
  type: git
  source:
    uri: git@github.com:concourse/examples.git
    branch: master
    private_key: |
      -----BEGIN OPENSSH PRIVATE KEY-----
      ...
      -----END OPENSSH PRIVATE KEY-----

jobs:
- name: hello-world-job
  plan:
  - get: repo
    trigger: true
  - task: create-commit
    config:
      platform: linux
      image_resource:
        type: registry-image
        source:
          repository: gitea/gitea
      inputs:
      - name: repo
      outputs:
      - name: repo
      params:
        EMAIL: person@example.com
        GIT_AUTHOR_NAME: Person Doe
      run:
        path: sh
        args:
        - -cx
        - |
          cd repo
          date +%Y-%m-%d > todays-date
          git add ./todays-date
          git config --global user.email $EMAIL
          git config --global user.name $GIT_AUTHOR_NAME
          git commit -m "Update todays date"
  - put: repo
    params:
      repository: repo

Set the pipeline and let's watch it trigger and run from the web UI.

$ fly -t tutorial set-pipeline -p hello-world -c hello-world.yml

The job now has the repo resource to the left and right of the hello-world-job. This is how Concourse visually represents the inputs (get steps) and outputs (put steps) of a job.

That covers the basics of resources. It is best of think of each resource as a little application, so carefully read any documentation the resource author has provided. Every resource behaves a little differently based on the external system it is representing.

Using External Resource Types

Concourse comes bundled with a lot of resources that are enough for most people to start using Concourse with. However, users will want to extend Concourse to work with all sorts of systems and that means bringing your own Resource Types.

Adding a resource type to your pipeline looks very similar to adding a resource. You can even override the bundled resource types by re-declaring them in your pipeline.

Remember, a resource is a container image. So to pull in a new resource type you need to tell Concourse where to pull the image from. This is done by using the built-in registry-image resource. The process of adding a resource type is just like adding a regular resource.

Here's an example of using an external resource type to read an rss feed.

resource_types:
# declare the new resource type
- name: rss
  type: registry-image
  source:
    repository: suhlig/concourse-rss-resource
    tag: latest

resources:
# use the resource as usual
- name: dino-feed
  type: rss
  source:
    url: http://www.qwantz.com/rssfeed.php

jobs:
- name: announce
  plan:
  - get: dino-feed
    trigger: true

That's how you add external resource types to your pipeline. If you're looking for more resource types there's a catalog of them at resource-types.concourse-ci.org.

Time For Takeoff ✈️

This brings us to the end of the tutorial. You should have a basic understanding about how to read Concourse pipelines and start creating your own. Here are some other parts of the site to help you take off with Concourse:

Best of luck on your automation journey!

If you have any feedback for this tutorial please share it in this GitHub discussion