Matt Bruzek: Writing tests for charms (is not that hard)

To ensure quality of the Juju charm store there are automatic processes that test charms on multiple cloud environments. These automated tests help identify the charms that need to be fixed. This has become so useful that charm tests are a requirement to become a recommended charm in the charm store for the trusty release.

What are the goals of charm testing?

For Juju to be magic, the charms must always deploy, scale and relate as they were designed. The Juju charm store contains over 200 charms and those charms can be deployed to more than 10 different cloud environments. That is a lot of environments to ensure charms work, which is why tests are now required!

Prerequisites

The Juju ecosystem team has created different tools to make writing tests easier. The charm-tools package has code that generates tests for charms. Amulet is a python 3 library that makes it easier to programatically work with units and whole deployments. To get started writing tests you will need to install charm-tools and amulet packages:

sudo add-apt repository -y ppa:juju/stable
sudo apt-get update
sudo apt-get install -y charm-tools amulet

Now that the tools are installed, change directory to the charm directory and run the following command:

juju charm add tests

This command generates two executable files 00-setup and 99-autogen into the tests directory. The test are prefixed with a number so they are run in the correct lexicographical order.

00-setup

The first file is a bash script that adds the Juju PPA repository, updates the package list, and installs the amulet package so the next tests can use the Amulet library.

99-autogen

This file contains python 3 code that uses the Amulet library. The class extends a unittest class that is a standard unit testing framework for python. The charm tool test generator creates a skeleton test that deploys related charms and adds relations, so most of the work done already.

This automated test is almost never a good enough test on its own. Ideal tests do a number of things:

1. Deploy the charm and make sure it deploys successfully (no hook errors)
2. Verify the service is running as expected on the remote unit (sudo service apache2 status).
3. Change configuration values to verify users can set different values and the changes are reflected in the resulting unit.
4. Scale up. If the charm handles the peer relation make sure it works with multiple units.
5. Validate the relationships to make sure the charm works with other related charms.

Most charms will need additional lines of code in the 99-autogen file to verify the service is running as expected. For example if your charm implements the http interface you can use the python 3 requests package to verify a valid webpage (or API) is responding.

def test_website(self):  
    unit = self.deployment.sentry.unit['<charm-name>/0']
    url = 'http://%s' % unit['public-address']
    response = requests.get(url)
    # Raise an exception if the url was not a valid web page.
    response.raise_for_status()

What if I don't know python?

Charm tests can be written in languages other than python. The automated test program called bundletester will run the test target in a Makefile if one exists. Including a 'test' target would allow a charm author to build and run tests from the Makefile.

Bundletester will run any executable files in the tests directory of a charm. There are example tests written in bash in the Juju documentation. A test fails if the executable returns a value other than zero.

Where can I get more information about writing charm tests?

There are several videos on youtube.com about charm testing:

Documentation on charm testing can be found here:
* https://juju.ubuntu.com/docs/authors-testing.html
Documentation on Amulet:
* https://juju.ubuntu.com/docs/tools-amulet.html
Check out the lamp charm as an example of multiple amulet tests:
* http://bazaar.launchpad.net/~charmers/charms/precise/lamp/trunk/files