Automating TEN7 on Open Source

Deploying Drupal sites is boring. You're just finished committing your code and testing it locally, and now it's ready to live on the server. It could be a staging server to share with your client, or even the production cluster ready and open to the world. But – Ugh! – you have to deploy it first. That means compiling the SASS, copying the files directory, and making sure the settings.php has the right credentials. It's a dull and error prone process that makes going for that third coffee run very inviting.

If you're lucky, you've convinced your client to use Acquia, Pantheon, or one of many PaaS (Platform as a Service) providers that make your life a bit easier. You can just commit to a specially marked branch and let the hosting provider do the work for you. Convincing your clients, however, is always a gamble. You've lost business because of it. You've lost sleep when you were asked, "We don't want to get locked in. What are the alternatives?"

Where We Were

I came to TEN7 several months ago with the mandate to Automate All The Things. We wanted to do it in open source instead of going to a PaaS provider. While I've worked in enterprise environments before, I never really considered myself an operator. It wasn't until I found myself working with a data center automation product that I fell in love with infrastructure at scale. TEN7 provided a different challenge: How to make things as easy as push to deploy, without locking ourselves in, or breaking the bank.

Most of our projects have three environments: live, release and preview. The first two are equivalent to production and develop respectively. Preview is... a little unusual. Since we don't have dynamic environments spun up per branch (yet), preview is a compromise that allows our developers to test their code without it being customer facing.

Instead of using Bitbucket or GitHub, we have an instance of Gitlab's excellent open source repository hosted on our infrastructure. Gitlab not only provides us a nice UI for our code, but it also provides a rudimentary Continuous Integration (CI) server. When I arrived, Gitlab CI was already in use for our release and preview environments. Live, however, was a different matter altogether. Only one of our sites was automated using CI for production. No matter the environment, we also faced several problems that resulted in a failed build.

What we wanted was to move from mostly manual workflows to one taken for granted by PaaS providers: for a developer to commit to a branch and for the site to automatically deploy each time. A simple enough vision, but one that had a lot of little complexities.

Introducing New Tools

Implementing Continuous Integration depends on having repeatable, unattended processes that aren't a nightmare to maintain. Gitlab CI is built around executing shell commands. It's easy to learn, but difficult to make things repeatable. Take a simple operation such as creating a directory. Sure, you can use the mkdir command to create the directory, then chmod and chown to set the permissions. But what happens if the directory is already created? The mkdir command will error out and fail the build. With shell scripts, it's never the actual commands that are complex, it's the programming around them ensuring a consistent, desired state that adds so many ifs, elses, and headaches.

Tools such as Puppet and Chef have been created to solve the consistency problem. While popular, I found them to be rather impenetrable to learn and unpleasant to deploy. Both Puppet and Chef require a special agent to be deployed on the target server. This would work for TEN7’s infrastructure, but some of our clients prefer to remain on their own. Often we only have SSH access to the server. The installation of a central master server was also a stumbling block I wanted to avoid. Fortunately, there was an alternative.

I had seen a demonstration of Ansible the previous year at Midwest Drupal Camp. There, Geerlingguy configured a small cluster of Raspberry Pis into a redundant Drupal server. Ansible’s agentless nature, battery-included philosophy, and no need for a master server made it an ideal solution for our CI problems.

Ansible improves on shell scripting in two key ways: Shell scripts are imperative, saying "do this now". Ansible on the other hand, is descriptive. Using our previous example, I would describe to Ansible where I want a directory, the owner and group, and the permissions. Ansible handles all the conditional logic and ensures the desired state exists. No more ifs, elses, or headaches.

Moving Past Git-Based Deploys

Once Ansible was installed in all environments, we targeted another common reason for failed builds. Most of our existing Gitlab CI scripts performed a git-based deploy. The preview, release and live environments would have a copy of the site’s repository. On commit, the CI process would do a git pull, then build and host the site from the local copy of the repository. This creates a problem for unmanaged items like the files directory and the settings.php file. Yes, you can use a .gitignore to keep them from dirtying the repository, but it’s a game of Whack-a-Mole. The git clean command doesn’t always catch everything, and can leave the repository in an inconsistent – and un-pullable – state. This would fail the build.

The solution was to use what I call a “double deploy”. Instead of hosting the site in the local copy of the git repository, we use Ansible to recreate the site directory on each build:

  1. First, clone or pull the repository into a dedicated directory. Ansible performs a clone or pull in one command since the result of either is, "there is a repository here".

  2. Create a new, empty build directory.

  3. Copy the site files from the repository into the build directory.

  4. Run CSS preprocessors in the build directory, not the git directory. This keeps the git directory clean, while making it easy to blow away failed builds.

  5. Move the files directory from the directory hosted by Apache (the “web” directory) into the build directory. Moving a directory is a much faster operation than copying.

  6. Rename the web directory, saving it as a backup in case of a failed build.

  7. Rename the build directory to the web directory.

The above process would be messy and error prone if done manually. When automated with Ansible, it takes advantage of the nature of file systems to keep the builds quick. Furthermore, it naturally creates a backup just in case the build fails at any point. We've since enhanced this process to also take a backup of the database. Database backups are auto-deleted after a configurable period of time so they don’t eat our disk space.

Automating Database Credentials

The settings.php file, however, presented a problem. We don't want to keep the database credentials in the repository. This means the settings file – from the CI perspective – is an artifact that needs to either be copied on each build, or generated from scratch. We also needed to support different environments (preview, release, live) which each may have different unique settings that do need to be preserved in the repository.

The solution required brainstorming with the entire company – all 8 of us. The database credentials are now stored using Gitlab's secure variable feature. During a build, these secure variables are available to CI scripts as environment variables. Once the build is complete, the shell process started by the build is killed, taking the environment variables with it. Next, we broke up the settings.php into several files – settings.live.php, settings.release.php, and settings.preview.php. This way, each environment may have different, repository-backed configurations when necessary. Finally, we used Ansible's powerful built-in templating engine to populate the database credentials. During the build, the CI process looks for tokens in the settings files such as {{ T7_PREVIEW_DATABASE_NAME }}, replacing them with the matching environment variable set by Gitlab's secure variable feature.

This approach has several advantages. Developers with appropriate access rights may view or change the secure variables. The settings files may be overridden in the repository with hard credentials by simply removing the token. The settings files are updated on each build, ensuring a consistent state. We also added logic to settings.php that falls back to a settings.local.php at any point when it is present, making it transparent to developers working locally.

Live Automated Deployments

Getting automated deployments on our live environment also proved to be a challenge. Adding Memcached and Varnish adds some complexity to the scripts. Buy-in from the company founder was essential in reorienting the permissions on the production cluster. Getting live deployment to work the first time was an arduous task of finding each undocumented and manual step, reorienting the permissions, and adding it to the CI scripts.

Once done, however, we finally got what we wanted. A developer now only needs to commit to master, and CI does the rest.

Summary

The combination of Ansible and Gitlab CI has been a huge benefit to our workflow. It’s allowed us to deploy code faster and with less headaches. And the process is still on-going; we’ve been converting projects one at a time as time allows. Our Ansible code has been per-project for now, but it may soon be abstracted enough to post to Github or Ansible Galaxy. It is possible to build a modern, DevOps workflow using only open source. Gitlab CI and Ansible provide a powerful combination no matter what hosting provider for you and your clients use.

Tess Flynn

DevOps Engineer
 
Image
Tess Flynn

Tess is TEN7’s Swiss Army knife. She’s an ever-present force in Drupal and a frequent speaker at events, where she's known for comic book-style illustrations in her presentations. Her superpower is problem-solving—she’s always finding ways to improve a site’s infrastructure and efficiency, and she has the rare ability to look holistically at a situation through human requirements, not just those of technology and business. She also loves sleuthing out the source of hacks, especially the ugly and ingenious ones. Tess has encyclopedic knowledge of horror/sci-fi ranging from schlocky and campy to highbrow. She loves Star Trek, where the engineers use their skills to help people.