Configuration Management, pt.1 - A Personal History

First, a look at how we got to now, or rather, how I got to now as told from the perspective of someone who has been working in the web development industry for about two decades... for better or worse.

There has always existed a large variety of approaches with different levels of expertise required and any given stage of development. My first experience is one of a complete novice. Uploading to a shared service like Geocities that included nothing but HTML 4 with inline styles, and eventually tables!  Plenty of dizzying textured site backgrounds, and marquee styled text. No reason to explore this any further, but you have to start somewhere.

After college I began working for a large company that had plenty of check and balances. There were sys admins that kept the servers managed just so. Changes were scripted or performed by hand, and the method of disaster recovery was to have a "Golden Master" image that was everything as it should be (at that time).

After years at this job I went the way of a web contractor who was responsible setting web hosts for the sites I developed. The server configuration was often performed via a UI like Plesk or CPanel. For the average site, no command line work was needed. I want from directly FTPing files to the server to using version control (CVS, SVN). But the server configuration itself was still fragile. Customizations were performed on the fly and was always a moving target. The larger the client the more safety nets put in place, but these safety nets were not automatic and still relied a lot on manual intervention.

As needs changed, there was more custom configuration and less UI configuration. Installs on the server were being done via package managers like aptitude and RPM.

In order to scale the number of clients multiple sites were hosted on a single webserver. This saved a lot of time setting up servers for each new site, and made the live environment predictable. Though the obvious drawback was that sites were coupled together, the requirements for one, were applied to all. Again we ultimately found ourselves coding to the server requirements. Adding server requirements made things riskier not just for the site being developed, but for all the sites sharing the host. We certainly knew this was not ideal, but we were not Sys Admins, nor did we want to be. We did what was required to get our code working. And when things didn't work, or a server went down, things were stressful.

Enter DevOps. We needed servers with some variance in requirements and capabilities, but didn't want to spend all our time following a written script, or running shell scripts to configure things just so. If a site became unavailable we didn't want to spend days trying to recover from that situation, especially since most of this work would be un-billable. It was time for us to invest in a server configuration management that allowed us to write, test, and repeat the server configuration process.

Chef and Puppet were options at the time. We evaluated and decided we would use Chef. And the learning began. There would be plenty of up front time required to get things just so and lots of other considerations. But we would be free from the shackles of poorly written notes and panic filled days that resulted from a server catastrophe.

In the next post I'll describe how we implemented Chef, and some examples relating to hosting Drupal based websites.