Bullet proof Jenkins setup

In this post, I will describe how a neat setup and some discipline will ensure a Jenkins that can be rolled back and recreated very easily - a bullet proof Jenkins setup.

I have been working on configuring our Jenkins instance. This was the first time I had played around with Jenkins. I am fairly comfortable with Go from ThoughtWorks Studios. All of my past teams used Go as their tool for continuous delivery.

One of the things I found very different from Go in Jenkins is the absence of the notion of a Pipeline as the basic entity of build, as proposed in Continuous Delivery. Although there are plugins to make this available in Jenkins, we decided to go with Jenkins' model of Jobs.

Another difference I spotted is that when a custom task is defined as part of a Job, Jenkins creates a shell script with all the steps while executing the Job. In Go, each of the steps will have to be defined as a custom command.

We wanted to ensure that our Jenkins configuration is version controlled. While this is a huge win, one of the ways this situation deteriorates is when a large number of changes are made to the configuration over a period of time and these is not checked in. So we decided to take this one step further and ensure that these changes are automatically checked in. There are instructions on how to do this, but we had to do some tweaking to get this working for us.

These are the steps to setup a bullet proof Jenkins setup. This assumes that Jenkins is running on a Linux box.

1. Create a Git repository in Jenkins' base directory - This is generally /var/lib/jenkins
2. Create a .gitignore file to exclude Jenkins workspaces. The Jenkins base directory is the home directory for the Jenkins user created to run Jenkins. This means that there will be a number of Linux user specific files like .ssh/ , .gem etc.   These files need to be specified in the .gitignore file. A sample .gitignore file is listed below.


3. Setup a Jenkins job to check in the changed configuration files every day at midnight. (Or whatever time interval you choose). Add a custom task with the following steps:


While this ensures that the configuration is more or less tracked well, there are times when somebody makes a massive change in the configuration. This is where the most important piece of the bullet proof configuration comes in - team discipline. The team should ensure that big changes are checked in as soon add possible. This can be easily done by triggering the Jenkins job manually, without having to ssh in to the Jenkins box.

Credits:
1. The Jenkins community documentation provided a nice starting point for this.
2. The .gitignore file was forked from this gist by sit. I have added some project specific stuff to it.

Why your project should have a Getting Started guide.

My new team at work is writing a bunch of Rails applications. This is one of those codebases that one would call "legacy" without much argument. Most of these apps have their own patched, vendorized Rails versions.

Getting up and running was an absolute pain. This project existed before Bundler and the list of gem dependencies are not checked in. I got the output of running gem list on a colleague's box, wrote a Ruby script to generate a shell script that installs all the gems. When I tried running the tests in one of those apps, I got a nice error.

It looked obvious that we were using patched Rails versions. Surprisingly, theses were not checked in to the vendor/ directory. This was proving to be a pain, and today I sat with a team member and wrote a developer guide to get started on the project. We actually had to pull patched Rails tar balls from a remote box and untar them to vendor/ directory. I was really surprised that these patches had not been checked in. We checked in all the patches. Apart from the Rails patches, there was a gem that had to be checked in.

This is why I think every project needs a Getting Started guide:
1. As a developer joining a new team, I want to look at the code as soon as I can. For me, this involves reading and running the tests.
2. Due to various reasons, a project may have patches and dependencies. Being very specific to the project, there is no way a new developer on the team would figure out these hidden dependencies.
3. There needs to be a single place where all the dependencies are specified.
4. While spoon-feeding someone may not be the best way to get them started, a little bit of hand-holding will not do anyone any harm.

One of the biggest problems I see with things like these guides is that as the application and it's dependencies evolve, the guides are not updated to reflect these changes. The only possible time the guide will be updated is when a new developer joins the team, follows these instructions and finds the need to update the guide.

Debugging: C#'s HttpWebRequest, 100-Continue and nginx

Recently I spent some time debugging an issue our team was facing around some C# code making a request on one of our servers. The request was throwing a "The server committed a protocol violation. Section=ResponseStatusLine" error.

Initial investigation suggested that this could happen if we are making HTTP/1.1 requests to a server configured for HTTP/1.0. Our Rails application runs on Mongrel fronted with nginx 0.6.5. We modified the C# code to use HTTP/1.0 and the error went away. The following line does the trick.

request.ProtocolVersion = HttpVersion.Version10;

But wait! This means that somewhere in the chain, a server is configured to use HTTP/1.0. It looked unlikely and further debugging revealed that it was indeed not the case. Further staring at the Rails logs showed that one of the headers that the app expects was not being set, when the request was done using HTTP/1.1 from the code.

After some time, we figured out[1] that the .Net library throws the "server committed ..." error if it is expecting the HTTP 100 (Continue) response in the wrong way. We set the code to not expect the HTTP 100 response from the server using

      request.ServicePoint.Expect100Continue = false;

and voila, it worked. The Rails app received all the headers it expected and things worked fine. The code looked like this:

So what is happening?

The HTTP 100 status is supposed to work like this. When a client has to send some data, instead of sending it upfront, it can send some headers along with the "Expect:100-Continue" header. The server responds with a 100 if it is willing to accept the request or send a final status. The spec is here[2].

We are using nginx as a proxy. The specification says that the proxy should forward the request if it knows that the next-hop server is HTTP/1.1 compliant. The proxy is supposed to ignore the "Expect:100-Continue" header, if the request came from a client using HTTP/1.0.

In our case, the default behavior of the .Net HTTP client library is to set "Expect:100-Continue" header on every request for HTTP/1.1. So the client sends only some headers and waits for the 100 response from nginx. Nginx sees the request, knows that Mongrel supports HTTP/1.1 and just forwards the request. The app sends a 401 because it could not authenticate. The client is expecting a 100 and gets a 401. It thinks the server committed a protocol violation.

When we ask the client to use HTTP/1.0, the .Net library does not use the Expect header, sends all the headers and nginx forwards the request to Mongrel. The authentication goes through.

When we explicitly set the Expect 100 property of the library to false, it sends all the headers at once and the authentication goes through fine.

Looks like there is a way to tell .Net not to expect 100 from the server through configuration, by putting this in .exe.conf

1. http://stackoverflow.com/questions/2482715/the-server-committed-a-protocol-violation-section-responsestatusline-error

2. http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.2.3

Talk on Football and Politics

I recently did a talk on football and the politics behind it, as part of the Banyan Tree talk series at ThoughtWorks. The slides from the talk can be found here on Speakerdeck.

It was great fun for me. I thought I did well. It is a topic I know my way around. The history behind the El Clasico - the Spanish civil war, Franco, the trade unions and Di Stefano was discussed. We also looked at The Old Firm. I also talked about Athletic Bilbao's use of football as a form of cultural resistance. The last parts of the talk were about multiculturalism, united Europe and football as a globalized game.

 About 20 people turned up and while most of them liked it, some had the opinion that they needed to understand more European history to fully appreciate it.