Deploying Concourse with BOSH provides a scalable cluster with health management and rolling upgrades.
If you're not yet familiar with BOSH, learning it will be a bit of an investment, but it should pay off in spades. There are a lot of parallels between the philosophy of BOSH and Concourse.
If you've never used BOSH before, you may want to first go through its introductory material:
To tie the terminology together: Concourse is packaged as a release, whose source repository lives at concourse/concourse on GitHub.
Releases and stemcells (operating system images) are put together to form a deployment manifest (a YAML file, similar to a Concourse pipeline, which describes what jobs to run where, and with what properties).
A BOSH director is given a cloud config, which is another YAML file describing IaaS-specific things like network names and VM types. This file is changed a lot less often than your deployment manifests.
The BOSH director is then used to deploy the deployment manifest, which is what brings the "abstract" (releases, deployment manifests, a bunch of YAML) to the "concrete" (software running on VMs running on an IaaS of your choice). The director is then used to maintain the deployment, allowing you to e.g. SSH into VMs, re-create them, perform a rolling upgrade, manage persistent state, recover from IaaS instability, etc. etc.
The quickest and easiest way to get going is to use our
concourse-deployment "cluster" recipes. This repository contains a base manifest, some example cloud configs, and useful Operations files for deploying and maintaining the latest version of Concourse with BOSH. In the future, it will also include useful deployments to provide things like credential management and metrics.
concourse-deployment repo covers the following steps:
Setting up your cloud config: there's a
cloud_configsdirectory containing various examples. Some of them may work verbatim. If none of them apply, though, you're on your own; you should consult bosh.io for IaaS-specific information.
Creating the manifest: that's kind of the point of the repo. Any required property changes will be made to the repository as soon as they're needed, so when a new Concourse comes out you should be able to just re-
Uploading the releases: the manifest will include them by URL, version, and sha1, allowing the director to download them for you.
However, it does not cover the following:
Initializing your BOSH director. To go from nothing to a BOSH managed Concourse, you'll first need to initialize a BOSH director.
These steps are general to anyone using BOSH, and so are not replicated in our documentation.
Another handy reference is the release documentation on bosh.io, which makes it easy to peruse the available properties for each job. We intentionally keep all documentation and examples within the release itself to ensure there is no out-of-date documentation.
This really depends on your infrastructure. If you're deploying to AWS you may want to configure the
web VM type to register with an ELB, mapping port
4443 (if you've configured TLS), and
Otherwise you may want to configure
static_ips for the
web instance group and just reach the web UI directly.
With BOSH, the deployment manifest is the source of truth. This is very similar to Concourse's own philosophy, where all pipeline configuration is defined in a single declarative document.
So, to add more workers or web nodes, just change the
instances value for the instance group and re-run
To upgrade, just upload the new releases and re-run
If you need workers that run outside of your BOSH managed deployment (e.g. for testing with iOS or in some special network), you'll need to make some tweaks to the default configuration of the
The TSA is the entryway for workers to join the cluster. For every new worker key pair, the TSA will be told to authorize its public key, and the workers must also know the TSA's public key ahead of time, so they know who they're connecting to.
Typically the TSA and ATC will both be colocated in the same instance group. This way a single load balancer can be used with the following scheme:
8080(ATC's HTTP port) via SSL
2222(TSA's SSH port) via TCP
Be sure to update any relevant security group rules (or equivalent in non-AWS environments) to permit both access from the outside world to port
2222 on your load balancer, and access from the load balancer to port
2222 on your TSA + ATC instances.
The BOSH deployment manifest would then colocate both jobs together, like so:
instance_groups: - name: web vm_type: web_lb jobs: - name: atc release: concourse # ... - name: tsa release: concourse # ...
In AWS, the
web_lb VM type would then configure
cloud_properties.elbs to auto-register instances of
web with an ELB. See the AWS CPI docs for more information.