Towards a Test Driven Development Framework in Vala Part 3. DevOps - Continuous Integration with Jenkins

Continuous Integration or CI is widely used in Test Driven Design for keeping the project's codebase tight, reducing errors and making sure there is always a working build available for deployment. It provides a means to automate the whole build and testing process, so developers can focus on writing their tests and the code that passes them. By setting up a system that builds and tests the software on its supported platforms, deployment issues can be identified early and distribution of new releases automated.

Since once of the objectives of Valadate is to integrate with existing toolchains, and wanting to leverage the numerous benefits of CI for the project itself, I took a short DevOps break to set up a Jenkins based system on my local network. Jenkins is a widely used open source Continuous Integration server written in Java, so it can pretty much run anywhere, providing the system has enough juice. Taking this to its extreme, I decided to install it on a spare Raspberry Pi 2 I had lying around. So why Jenkins and why on a Raspberry Pi?

Firstly, Jenkins is a robust and well maintained platform that is widely used. It has a plethora of plugins that integrate it tightly with Git, Docker, TAP and numerous other CI tools and protocols. It works on the master-slave model, where the master server directs the build operations of any number of slaves. A slave can be any other computer on the network that Jenkins can communicate with, either directly through SSH or with a plugin. It is highly configurable and it just works. It seemed like a good choice to start with.

Secondly, the Raspberry Pi. One of my considerations when setting up the CI system was that the master server should be internet accessible and available 24-7. Given that when it isn't running jobs the server is mostly idle, using a full powered computer would be a waste of electricity and CO2. It occurred to me that one of my spare Rapsberry Pis could do the job, so after a quick Google to confirm that it was possible, I proceeded with the install. The one comprehensive guide I had found had suggested a lot mucking about with downloading source packages, but since it was for the previous version of Raspbian I tried sudo apt-get install jenkins and whaddya know, it just worked.

With the Jenkins server up and running, I added my recent port of Gherkin as a test job and set up a machine running Fedora 23 as a slave and in 5 minutes it had checked out, compiled and run the unit tests on it and...

\O/ \O/ \O/

Despite being relatively low-powered, the Raspberry Pi seems up to the task, as nothing is actually being built on it. Some configuration pages take a while to load, but for ordinary usage it's quite snappy. Not only that, but you can do cool things with it as well.

Emboldened by my initial success, I moved onto setting up a Docker slave. For this setup, I revived an old server that had been mothballed, with the idea that as a build slave it doesn't need to be online all the time and with Wake On Lan (WOL) I can have Jenkins wake the server up when it needs to do a build and put it back to sleep when its done. This is still on the to-do list, but seems fairly straightforward.

In this configuration, the slave is a Docker host that starts up and runs a container built from a Dockerfile in the repositories root. It is this container that runs the build, not the host, so it is possible to test your software on pretty much any platform that can be dockerized. Cool eh? So I set up an Ubuntu container and...

Huh?!? I looked at the log and...

./.libs/libgherkin3.so: undefined reference to `g_value_init_from_instance'

Dammit! In my rush to port Gherkin, I had done it on my new Fedora 23 box and hadn't actually tested it on Ubuntu at all. I checked the docs and sure enough, GLib.Value.init_from_instance() is available from GLib 2.42 on only and Ubuntu 15.04 ships with 2.40. D'oh! So now I either have to refactor the code or declare GLib 2.42 a prerequisite.

This particular case is a really good example of the benefits of Continuous Integration. If I had had the Jenkins server set up before I ported the code, I would have noticed the incompatibility almost immediately and would have been able to deal with it then, rather than refactoring later.

As nice as it would be to ignore the existence of other operating systems, the sad truth is that not everyone uses Linux as their primary desktop, including many people who might want to use my software. With this harsh reality in mind, I decided to set up Windows and Mac OSX slaves to test the cross platform compatibility of my projects.

For the Windows slave, I set up a new Windows 7 VM in VirtualBox, running on the same server as the Docker host. For the build toolchain, I installed MinGW64 and MSYS2 and all of the necessary libraries and voila! Well, not quite voila, the MinGW linker is soooo sloooow that it took quite some time to debug but is now working just fine. The process isn't quite fully automated - I still need to manually spin it up and shut it down. There is a VirtualBox plugin to do this, but it doesn't presently support version 5. I also learned the hard way that you need to disable automatic updating for Windows, otherwise it will get stuck at the failed boot recovery screen. I am also thinking that for speed, I will cross compile the Windows binaries in a Docker container and run the tests in the Windows VM to make sure they work.

Now, if you've been to any major Linux conference in the last few years, you'd be forgiven for thinking you were at WWDC with all the Apple hardware being toted about. Heck, my wife, an Open Source guru, was a long time MacBook Air user until she got a Microsoft Surface. And it's true, it is some of the coolest, most expensive hardware you can run a Linux Virtual Machine on. Don't get me wrong, I have one on my desk, I just mostly use it for email, IRC and the occasional Photoshop session (at least until Gimp gets better tablet support). Unfortunately, it's been a little neglected so it needs a bit of a clean up before it can be pressed into service, which will hopefully be by the start of next week.

Along the way I also discovered that our crappy Comcast provided Cable Modem doesn't support hairpin DNS resolutions when I forwarded the Jenkins server ports. I tried to solve this by setting up dnsmasq on the Raspberry Pi but it still required manually editing the resolv.conf files on each machine. In the end I just put the Comcast Modem into bridge mode and set up a trusty old WRT-54GL running DD-WRT as the new Gateway/Router. It still has some problems with IPv6 DHCP but otherwise is running just fine.

So there you have it, a working cross-platform Continuous Integration system building Vala based projects. It's live on the internet now, so you can check it out here (Github login required).

OK, now we're ready to start building Valadate! Tune in again soon for Part 4. Who tests the tester?