Tag Archives: BOSH

Beyond automated deployment

I have been involved in quite a lot of automation projects over the last five years. All of them centered around VMware vRealize Automation and vRealize Orchestrator. During these projects customers throw all kinds of challenges at me. Most of which I can solve. Over the years however I found two challenges that go beyond automated deployment which I can’t really solve using vRA/vRO:

  1. If you update a vSphere template, how do you make sure all machines deployed from that template are also updated?
  2. If you change a blueprint, how do you make sure those changes are also made to existing deployments from that blueprint?

The answer two both really is: you can’t. Not If you’re using vRA/vRO. Dont’ get me wrong. I’m not trying to bash these products here. It’s just a result of how these products are designed and how they work.

In my opinion both problems boil down to the fact that in vRA blueprints you define the initial state of a deployment, not the desired state. So if you deploy a blueprint you get whatever was specified in that blueprint. Which is fine initially. But if you change the blueprint or update the template, nothing will be changed on the existing deployments. The other way around is true as well: If you change/damage your deployment, vRA won’t come in and fix it for you.

Now this seems obvious and not a big problem. After all: getting deployment times down from weeks to minutes using automation tools is a pretty good improvement in its own right. But if you think about it for a minute you’ll realize that when you have automated deployment, now you need to spent the rest of your days automating day 2 operations. After all the tool isn’t doing it for you.

For example you’ll have to introduce a tool which manages patches and updates on existing deployments. You also need to figure out a way to keep your template up-to-date, preferable automated. And if somebody breaks his deployment you need to spent time fixing it.

Now, if you’ve been following my blog recently you probably already guessed the solution to this problem: BOSH :). Here are four reason why BOSH makes your life as a platform operator easier:

  1. In BOSH a template is called a stemcell and stemcells are versioned. You don’t have to make you own, up-to-date versions of CentOS and Ubuntu stemcells are available online at bosh.io.
  2. When you’re using BOSH, software is installed on stemcells by using BOSH releases. Which are versioned, available online and actively maintained.
  3. A BOSH deployment defines a desired state. So if a VM disappears BOSH will just re-create it, re-install the software and attach the persistent disk. Also, when you update the deployment manifest to use a newer stemcell version, BOSH will just swap out the current OS disk with the new one in a few seconds and everything will still work afterwards.
  4.  All these parts can be pushed through a Concourse Pipeline! The pipeline will even trigger automatically when a new stemcell version, release version or deployment manifest version is available. Below is a screenshot of a very simple pipeline I build. This pipeline keeps both the software and the OS of my redis server up-to-date without me ever touching anything.

You can find the source files for this pipeline here. In real life you ‘d probably would want to add a few steps to this pipeline. First you deploy it to a test environment, then do some automated tests and then push it into production.

In summary: If you’re using BOSH not only do you get all the goodness of versioning and desired state config, it also enables you to employ Continuous Deployment for all your servers and software. You can even test new versions automatically so you don’t have to spent all your time just keeping your platform up-to-date.

BOSH on VMware Photon Platform

I explained both BOSH and the Photon platform in previous posts. I never did a post on how to deploy BOSH on vSphere but this document does a very good job describing the process. The only thing I want to add to that is: Don’t use “@” in your passwords! Cost me a day or so to figure out what was going wrong. In this post I will detail how to run BOSH on VMware Photon platform.

Update 19-04-2017: This post was based on Photon platform 1.1.1. As of today the current version is Photon platform 1.2. The steps in this post may or may not work for version 1.2.

Prepare Photon Platform

  1. Install Photon platform. This blog post details how you to do that.
  2. Make sure you have the photon cli installed. Instructions here.
  3. I’m going to assume that you don’t have anything configured on the photon platform yet. If you have you’ll probably already know what to do. I’ll also ussume this is a lab where you have full access.
  4. Connect the photon cli to you photon platform.
  5.  Create a photon tenant and tell the cli to use it (press enter on any questions to use the default)
  6. Create a network. I’m going to assume you use the default portgroup named “VM Network”. If not please substitute your network name in the command below.
  7. Create a resource ticket for the bosh environment. I didn’t find a way to deploy to other projects than the one you deployed the bosh director to. So make sure you create a big enough ticket to also fit the workloads you’ll be deploying with BOSH.
  8. Create a project that consumes the resources.
  9. Add some flavor.  Flavors are types of resources on offer on the Photon platform. It’s like AWS instance types.

Deploy BOSH

Install BOSH cli tools

To be able to install BOSH you’ll need the bosh-init tool. This tool is like a mini BOSH which is able to deploy BOSH. So it’s kinda like BOSH deploys itself. I won’t explain how to install bosh-init, the cloud foundry docs on this are pretty good. Please follow instructions here.

To be able to interact with a BOSH director once it’s deployed you’ll need the BOSH cli itself. In this case you’ll even need it before the BOSH director is running because it’s used to build the Photon CPI release. Again, find the cloud foundry docs on how to install the bosh cli here.

Prepare the Photon CPI

BOSH is able to work with a lot of different cloud (IaaS) providers and platforms. I already mentioned vSphere. But BOSH is also able to use vCloud, AWS, Google and Openstack. The magic that makes this multi-cloud solution possible is the Cloud Provider Interface or CPI.

VMware has published a CPI for Photon. It’s not published on the BOSH website yet but you can find it on github.  To be able to use the CPI you’ll want to install it into a BOSH director. How? Using a BOSH release of course. The Photon CPI BOSH release is here. Since there is no ready build  Photon CPI release we’ll have to build our own. Don’t be scared, it’s not that hard (disclaimer: I’m using Ubuntu. commands on a Mac should be  similar, not sure about window though). Here we go:

  1. Make sure you have the git client installed on your OS
  2. Create a folder to contain the CPI release and your deployment yml. I used ~/my-bosh/photon.
  3. cd into the folder you created
  4. Clone the Photon CPI release git repo, cd into the created folder and create the release:
  5. There’ll be a dev_releases folder in the bosh-photon-cpi-release folder now. Copy the cpi tgz file to ~/my-bosh/photon

Create BOSH manifest

deployments in BOSH are described in so called manifests. These are files in YAML format containing a bunch of settings. Each type of deployment has it’s own manifest and so does the bosh deployment itself.

You can find an example manifest for bosh with the photon CPI in the photon CPI release git repo.  I’ll share my own manifest below so you ‘ll have a feel of what it should look like with all the values populated. If you used the yml from my blog post to deploy photon  then you can use the my bosh manifest with just two changes:

  1. change the network id on line 39. The command to get the id is
  2. Change the photon project id on line 114. The command to get the id is

save the manifest yml to ~/my-bosh/photon/bosh-photon.yml

Run bosh-init deploy

Now you can finally start the deployment. It’s very simple 🙂

And now we wait 🙂

Use BOSH

Now that we deployed BOSH we might want to try to use BOSH for something useful. One of the simplest examples of something useful is deploying a redis server. Here are the steps involved:

  1. On the Photon platform create another resource ticket and a new project consuming the ticket.
  2. Target the bosh cli to the fresh BOSH director and login (if you’re using my yml the password is ‘password’
  3. run bosh status to confirm you’re connected and BOSH is up and running. Screenshot from 2017-04-04 16-44-17
  4. Upload the ubuntu trusty stemcell
  5. Upload the redis release
  6. Create a cloud-config YAML for BOSH. Below is my yml.
    1. Replace the project id on line 17
    2. Configure you ip range in lines 37..41
    3. Replace the network id in line 42
  7. Load the cloud config into bosh
  8. Create the redis deployment yaml. Again, below is my version of it.
    1. Replace the director_uuid. Retrieve the uuid by running bosh status
    2. Store the manifest in ~/my-bosh/photon/redis.yml
  9. Tell the bosh cli to use this manifest
  10. Now deploy redis

After the deployment is finished you can list the deployments and the VMs it deployed but running these commands

The output should be similar to this: Screenshot from 2017-04-04 19-19-46

Pfew….. if you made it this far: Congrats! you’re on your way to being a cloud native :).

What is BOSH?

In a previous post I went into what Cloud Foundry is and why you’d want to use it. What I didn’t go into was some of the magic behind the scenes. For infra minded people like myself this part might be even more exciting than the platform itself.  The thing that makes Cloud Foundry so robust and portal is called BOSH. So What is BOSH?

BOSH is a recursive acronym for “BOSH Outer Shell”. But that doesn’t tell you much about what it does. The bosh.io website explains: “BOSH is an open source tool for release engineering, deployment, lifecycle management, and monitoring of distributed systems.”

What does BOSH do?

It’s kinda hard to put BOSH in a certain box like “cloud management platform” or “software deployment tool”. BOSH does a lot of things: It deploys virtual machines, but it’s not strictly a virtual machine deployment tool. It deploys software but it’s not just a software deployment tool and last but not least it also monitors but it’s definitely not a monitoring tool.

It’s something better. BOSH deploys versioned software into a running infrastructure. The software needs a VM to run on so BOSH also deploys a VM. Once software is deployed it’s important that it keeps running. So BOSH also monitors the software and automatically heals the application when needed. If you accidentally delete a VM that’s part of a software deployment, BOSH will automatically redeploy the VM, install the software and rejoin the cluster.

BOSH components and concepts

A BOSH installation consists of the following componenst:

  • BOSH Director: This is what you could call the “BOSH Server”. It is the main part of the software that is responsible for orchestrating deployments and acting on health events.
  • BOSH Agent: This is a piece of software that runs on every VM deployed by BOSH. It is responsible for all the tasks that happen inside the VM.
  • CPI: The Cloud Provider Interface is a component that implements an API which enables BOSH to communicate with different types of infrastructure. There are CPIs for vSphere, vCloud, Google cloud, AWS and even for rackHD if you want to deploy to phyisical hardware. The CPI basically translates what BOSH wants to do to the specific cloud platform you want to deploy to.

When working with BOSH you’ll use the following constructs:

  • Stemcell: This is a bare bones virtual machine image that includes a BOSH agent. It’s a zip file with some descriptor fields and a machine image. Stemcells are platform specific. So there are stemcells for AWS, vSphere and so on. In the case of a vSphere stemcell you’ll simply find a VMDK packaged in a zip. You can download publicly available stemcells but you can also build your own if you want to.
  • Release: A BOSH release a a bundle of everything that is needed to deploy a specific application excluding the virtual machine templates. So it includes all runtimes, shared libraries and scripts that are needed to get the application running on a stemcell. There are public releases for a lot of opensource software including Cloud foundry.
  • Manifest: This is a YAML file that describes how stemcells and releases will be combined into a deployment. It describes the desired state. If you’re familiar with vRealize Automation, this is basically a blueprint.
  • Deployment: A deployment is basically the execution of a manifest. A deployment can contain many machines. When deploying, BOSH uses the manifest to determine the desired state. If you running the deployment BOSH will determine the current state and will do what is necessary to get to the desired state. This is contrary to what vRealize Automation does. When you change a vRA blueprint, that does not change any of the deployments. But if you change a BOSH manifest and run deploy again for that manifest BOSH will implement whatever changes you made to the desired state.

Can I try it?

Start out with bosh.io The documentation is quite good but the learning curve can be a bit steep. I hope to give you some pointers on how to get it running in another blog post soon.