Platform Architect, TEN7
What is Flight Deck
IVAN STEGIC: Hey, everyone! You’re listening to the TEN7 Podcast, where we get together every fortnight to talk about technology, business and the humans in it. I’m your host Ivan Stegic. In this episode of the podcast, Tess Flynn sits down with me to talk about one of TEN7’s internal projects, something we call Flight Deck. Tess welcome to the podcast once more.
TESS FLYNN: Hello!
IVAN: How are you today? It’s Friday afternoon.
TESS: I’m still metabolizing, so I guess that’s good.
IVAN: (laughter) Yes, that’s good. Ok. So, Flight Deck. I really love the name that we have for this internal thing. It wasn’t always called that, it was called DockStack back when we first started this in January of last year, that’s January of 2017. How did Flight Deck come about?
TESS: So, DockStack was the original name because "Docker" and it's also a "stack of containers". It worked. It was the best thing they could come up with at the time that I was writing it, but I never really liked it, it wasn’t particularly creative and like so many other Docker based solutions, it also riffed off of "Docker" the name and I didn’t really like that. Now, some months ago, we were looking at redoing DockStack and bringing in a lot of the features that we are going to put in it, which make it more ready for a production environment, and one thing that we were also working on at the same time is another project internally called "Starbase". Now I won’t get into Starbase, we’ll leave that for another episode, however, when we started talking about these, I kept thinking of starbase as a Death Star, just because, you know, it’s a natural image that comes to my brain. But then you need some place where the work gets done of the starbase, where you get stuff done, where you go places and do things, and because I watch way too much pop sci-fi and cheesy cable TV sci-fi, the first thing that came to my mind was Battlestar Galactica, and suddenly I went “oh, so Flight Deck.” Like the flight deck in a Battlestar, and I rather liked that idea and it really clicked and it was snappy and I liked it.
IVAN: And further to that, so there’s not only the connection between, that you just described, but in this other internal project we call Starbase we actually use Flight Deck internally to that project as well.
TESS: That’s correct.
IVAN: So, there’s a nice link there as well. What is Flight Deck at a high level?
TESS: So, Flight Deck is a collection of Docker containers that are meant to support local Drupal development.
IVAN: So, it’s specifically a way to do development in Drupal on your local machine and probably very easily, right?
TESS: Yeah, that’s the idea. It’s minimal and does not require any kind of additional software overhead other than the basic Docker package that you get from docker.com.
IVAN: So, it’s pretty lightweight.
TESS: It’s just enough to do what you need to do, and nothing else.
IVAN: So, there are other environments, other apps, other things that you can download out there. I’m thinking of Lando, I’m thinking of Docksal, DDEV is another one. Jeff Geerling has his own thing, Drupal VM, we use Flight Deck internally. We developed Flight Deck internally. Why didn’t we just choose one of these other things?
TESS: Mostly because I didn’t like any of them. (laughter)
IVAN: Ok. So, there’s a reason why we didn’t choose any of those, and grant it this was year and a half ago when we were looking. What’s the one thing that’s important about Flight Deck that the others just can’t give us?
TESS: Most of the other ones have kind of a mentality that I found to be a little more inflexible, Dockstall for example the last time I looked at it required you to run a global set of containers for the entire system, and it didn’t allow you to really easily customize those. That was way early in its development, so I’m not sure if its diverged since then. Lando I’ve heard great things about it, but it also I believe is based on Kubernetes and it has its own proprietary format of how you describe the containers and you add that to your system and there’s a global command that you have to install. And, all of this has pluses and minuses, but one of the things that bothers me about most of those approaches is, you either need an additional piece of technology beyond Docker which is necessary to integrate with Docker in order to function, or you need a completely separate set of technology like Drupal VM, obviously needs a hypervisor, so you’ll need VMware or VirtualBox or something else to run that in. I thought that Jeff was working on a Docker variant but still calling it Drupal VM, but I haven’t kept up on that as well. But mostly I just wanted something that was just Docker that used the built-in Docker functionality that would work on any operating system that Docker runs on, so you can take literally the same files you use to run a container set on a Windows machine, and it’ll work on Mac and it’ll work on Linux and it makes no difference, because it’s just Docker.
IVAN: So, I think it sounds like we tried to, or you tried to strip the functionality of running a local environment down to its bare bones, while keeping the number of requirements as low as possible, in this case there’s one requirement, the Docker binary, on whatever system you’re running, and then to try and make it as fast and minimal as possible. Does that sound like a fair summary?
TESS: That sounds about right. Yea.
IVAN: And, everything that we’ve published for Flight Deck is open source, right? There’s no proprietary requirements or downloads that need to happen outside of a download of Docker?
TESS: That’s correct. You don’t even have a binary. There is no Flight Deck binary. All Flight Deck really is, is a series of containers on hub.docker.com and that’s it.
IVAN: So, let’s talk about the components of Flight Deck then. That’s a nice segue. So, you said that Flight Deck is a bunch of containers, they're available on docker.com, hub.docker.com, which containers does Flight Deck have?
TESS: So, the core containers are we have an Apache PHP container, in fact we have a few different versions of that, but we have one of those containers. We have MariaDB, a MySQL container and then everything else is opt-in. You don’t have to use it if you don’t want to, but we provide a relatively full stack of materials in order to run an entire environment that can support a single project. So, we have a custom SOLR container that works out of the box with search API. We have integrations with Mailhog, with Memcache, with Varnish and we even have like Xdebug built-in and Web Profiler built in so you can even do performance profiling in PHP using just containers.
IVAN: And just to be clear, it’s all configured to support Drupal in a local environment, correct?
TESS: That’s correct. It’s optimized for that. It’s not that you can’t use it for other things, but that was the intended use case.
IVAN: And what versions of Drupal will it support?
TESS: This is another thing that’s a little bit unusual about Flight Deck, is that it supports Drupal 8, 7 and 6. It actually works on Drupal 6 too.
IVAN: Wow. So, you can use Flight Deck and you can run any Drupal 6 site locally just by getting the repo from GitHub and then using Docker the binary installed locally?
TESS: That’s correct.
IVAN: Is it a stretch to say that it would support Backdrop as well?
TESS: It probably would. I haven’t tested it, but I don’t see any reason why it wouldn’t.
IVAN: Right. Everything that’s required to run Backdrop is essentially the same requirements for Drupal 6 and 7. Why did we choose the containers we did? So, you talked about us landing on PHP 7.0 and 5.6 for the web containers and for MariaDB, but we're also kind of biased towards Varnish and Memcache. Why did we go that way as opposed to maybe NGINX or Redis?
TESS: So, initially I tried getting NGINX to work but the thing with NGINX is that at the scale that we’re talking here for local development, it doesn’t really bring you any particular advantages, and for Drupal it's even less so. The reason why is because for a Drupal site, NGINX is just a reverse proxy for PHP-FPM, that’s it, that’s all it does. So, the problem is that separating out PHP from the web server means that you have to have two separate containers or kind of hack around it with Supervisord and I do not want to do that either, so, we just used Apache. Apache’s still considered the standard in a lot of environments. There’s nothing wrong with it, it works, and mod_php does not bring you any advantages if you're not on a massive multicore system, or a multi-head system even, off your local development you're kind of still stuck on your own system, so it doesn’t really give you any performance advantages in any sizable or significant way, so why overcomplicate it?
IVAN: Right. What about Redis?
TESS: Well I mean I want to write a Redis container and add that to the stack, but we haven’t had any need for it right now. That might change in the future, but yea, just haven’t had a need for it yet.
IVAN: So, if there’s anyone out there that’s listening that does want Redis, you could likely post an issue on the Flight Deck GitHub page or submit a patch and we could get that into Flight Deck, theoretically.
IVAN: Ok, so, we have these environments, these containers, we use them locally, it seems to be a best practice to be using virtualized environments locally, especially if you’re working on a team of more than one person. MAMP causes and WAMP and LAMP causes a whole bunch of different issues for you when your working locally. How do we share these configurations, and how do we share these local environments amongst the team members? I know that it’s somehow related to the Git repos that we have, but I’m not exactly sure how that works.
TESS: So, like Lando has its own configuration file, and when I read the docs which was quite some time ago, so I might be misremembering, the idea was you add this to the root of your Git repository and then you can run whatever Lando commands that you use in order to stand up a stack. We have the same thing for Flight Deck but we don’t use anything that doesn’t require anything else. We have a Docker composed file that you add to the root of your repository. You can optionally have an override file to customize that further if you’d like. There is a .env file which helps configure the containers, and then we use that mostly because it has a lot of additional documentation we can add to it, so you don’t have to go digging through a webpage to find it. Then really you just add those files to the root of your system and provided that you make Drupal aware of the database configuration, you're good to go.
IVAN: It does sound kind of easy.
TESS: The hard part is getting Drupal to recognize it. The standard best practice for Drupal is to create a settings.local.php which is not tracked in your Git repository, and add the database credentials directly to that. I actually don’t like that approach. I find that it’s a bit more of a golden hammer, and I prefer to actually break out into multiple environment specific settings PHP files, and we actually add a settings.docker.php file to our own internal projects, and it just uses a Git environment function in order to pull the database credentials directly from the .env file. So, there you go.
IVAN: That’s pretty slick. What have been challenges with using Flight Deck internally with the team at TEN7?
TESS: Performance. Performance is the problem and the problem is macOS.
TESS: Yes. It’s macOS. The problem is somewhat present on Windows, but macOS’s file system causes so many problems with how Docker syncs things. Around... which version of Docker was it? 2.17? I forget the number specifically, they added particular synchronization modes to Docker that allow you to specify how much deferment you want between the host operating system file system and what’s in the container and that helped, but performance is still the problem, because Drupal is a very disc bound, and very IO bound kind of system, and that is where Docker on macOS really, really hurts. And, a lot of developers use macOS when they're doing Drupal, so it’s been a kind of a problem.
IVAN: And that’s not necessarily an issue only with Flight Deck. That sounds like an issue that Docker and file system intensive apps have across the board, so Docksal and others would also have those issues.
TESS: Some of them use different approaches in order to work around that issue with different file synchronization systems, but that makes the entire system less portable, and requires custom binaries and custom configurations which can break and be difficult to deploy for other users. And the file synchronization has gotten much better compared to two years ago, where it was downright laughable at times.
IVAN: So, what would you say is the greatest benefit of having and using Flight Deck locally for your own development projects when you are using a Git repo, when you're using best practices and perhaps when you're collaborating with a team?
TESS: So, there’s a number of different advantages that using Flight Deck brings to working with a shared environment team. One, everyone has literally the same set of services on their laptop. It’s not several different versions between MAMP and WAMP and incompatibilities and different versions of PHP, it’s what’s in that file, and that’s what’s in the file, which is a real advantage because sometimes small configuration changes can cause problems. Just today, for example, I ran into an issue where on one environment we weren’t having any issues with a particular implementation, on another one it completely failed out, and it was the same code, the same database, except for one PHP configuration.
IVAN: Uh oh. How did you find that?
TESS: I kind of wandered into it after a little bit of low level debugging.
IVAN: And was that environment configuration that wasn’t on another local machine, it was maybe on a server somewhere?
TESS: It was on a server.
IVAN: So, could we use Flight Deck to configure other environments that are not local?
TESS: So, here’s the thing. Flight Deck is also one thing that a lot of other Docker containers aren’t right now which is our containers do not run as root. By default all Docker containers run as the super-user account on the host operating system. And on Windows, macOS, this isn’t so bad, but on macOS it's worse, on Linux it’s really, really bad because the usernames get mapped between the containers and the host operating system, which means if something runs in a container on Linux it gets mapped to the root user on the host operating system and they get root privileges.
TESS: Flight Deck does not do that. Our containers have been reconfigured in order to run as a non-privileged user, so that they don’t have any access to the underlying host operating system, other than what you give it on disc.
IVAN: That’s a major benefit, which means that we could potentially apply that in environments that are not local.
TESS: That’s correct. The idea is that Flight Deck actually is only a few steps away from running in production.
IVAN: Let’s talk about that a little bit. So, we run our development environment at Linode, and you’ve done an incredible job of automating as much of that deployment and continuous integration as possible. We have our live site up on Linode as well and it’s a set of Linode machines, virtual machines, and you and I have talked about how we could potentially turn this whole Flight Deck experience that we’ve had into supporting something on Linode as well, and that means that we’d have to use the Linode API and perhaps some Ansible code that we’ve already run, but walk me through what your idea is of how we can bridge the gap between running something locally and running something in a staging or a live production environment at a host like Linode and not have these discrepancies in environments and configurations. Have it all be the same thing, essentially.
TESS: So that is possible. The trick is to figure out how to very carefully construct the containers beforehand and that does take a lot of time and effort, and we’ve put that time and effort in. Now, the trick is how do you get this to work in a production environment? This was a little bit more difficult until about nine months ago, when Kubernetes became really, really popular. So, Flight Deck, as just a base Docker installation with no other kind of over-arching need for configuration, would work perfectly fine in a Kubernetes environment. Now, if you take a Linode cluster and you turn that into a Kubernetes cluster, you could run Flight Deck on Kubernetes, on Linode, in production.
IVAN: I love the sound of that.
TESS: I do too, but I haven’t gotten very far with my experiments in setting up Kubernetes on Linode. Mostly because Linode is mostly an infrastructure as a service provider that’s geared towards web traffic. It’s not AWS where it exposes absolutely every control under the sun and then a few that they swept under the rug as well, and it's not Google Cloud where they already provide a Kubernetes service out of the box, but it has some things with the disc storage which require a little bit more thought. All Linode is, is just hardware.
IVAN: So, it would require some work to get the Kubernetes cluster running on Linode and then we could in theory, move the Flight Deck environment over to a Kubernetes powered Linode environment. And, in theory, because all of this is out on the web and available to anyone, anyone could really do it.
TESS: That’s correct, and I actually have been working on setting up all of the Linode setup in Ansible, so it would be easy to configure and stamp out.
IVAN: Wait. So, in Ansible you would have the configuration of each of the Linode virtual machines, and would that in turn be committed to the repo where Flight Deck is and where your site is, or is that different?
TESS: That would be a different place, and this is future plans I’m thinking of, but that would be the idea.
IVAN: So how do the images that we’ve published on Ansible Galaxy correspond to, and are related to, the Docker images that we have that are part of Flight Deck?
TESS: Right now, Flight Deck doesn’t use any of our Ansible stuff internally; it tries to avoid using that because it overcomplicates things. I thought about doing it, but one of the problems is that a lot of what we rely on when we do a hard install of a server, like our Linode infrastructure is today, none of those roles that are open source are freely available by Ansible, support the underlying operating system our containers are based on, which is Alpine Linux and that’s kind of the barrier. It probably would work, but it requires making sure that we check that and test it and submit patches to all of those roles in order to support it.
IVAN: You mentioned Alpine Linux and I know that’s a minimal distribution of Linux. Is that the reason why we chose that particular flavor of Linux to power Flight Deck? Or are there other reasons as well?
TESS: The reason why we settled on Alpine, is a few different ones. One, Alpine Linux is meant for containers. It is an operating system distribution of Linux which is intended to run only containers, which means that it has a lot of other things which make it simple to use. The package manager isn’t complex. It doesn’t have a whole bunch of tools in it that you don’t need. One of the core philosophies of Alpine Linux is you don’t want to give an attack surface you don’t need in the container. So, when you start up an Alpine Linux container you might type in vim and then there’s no vim, there’s no git, there’s no rsync because if you don’t need it, it’s not in the container, and, that philosophy makes the containers both very small, but it also reduces the potential for exploits within each individual container. And, the user benefit is by using a minimal distribution of Linux, whenever you initially download Flight Deck, the amount of storage space that’s required on your system to store that image, is also minimal.
IVAN: That’s an incredible benefit.
TESS: And that can also work at scale as well, because when you have small containers, when you scale it up to production, if you have standardized containers across your stack, then you have less disc space you have to worry about.
IVAN: So, I want to give you a scenario. I am a lone wolf, Drupal developer. I’ve been using MAMP on macOS for the last 10 years to do all my development. I am now looking to get new skills and change the way my workflow is, because it needs to be changed for some reason. How do I bring Flight Deck into my workflow? What do I have to do?
TESS: Ok, so, the first thing you would have to do is, you would have to install Docker for your operating system, for your laptop, if your using Windows laptop, Docker for Windows, macOS – Docker for MAC. If your running it on Linux, consult your distributions documentation to figure out what the instructions are, most of the time it’s just called Docker and whatever the package manager is. Once that’s installed and set up and running...
IVAN: Stop. Stop. Stop. First I have to install Docker is what you're saying and it has to be for the operating system that I’m using—Windows, Linux or Mac. And, to do that, I go to docker.com and find where the link is to download. Or if I’m on Linux I use a package manager and get the Docker binary.
IVAN: Ok. Alright. So, I’ve done that. I install it locally. I have the requirement, the first major requirement for Flight Deck, Docker. What’s next?
TESS: Then what you do, is you go to the Flight Deck Git repository and you want to copy the Docker-compose.yaml, the Docker compose override YAML, and the .env file into the root of your Git project.
IVAN: Ok. So, now I have to get the Flight Deck repo, and to do that I’m going to give you the URL cause you don’t know it, we’re creating it right now, ten7.com/flightdeck. That will give you the Flight Deck repo that’s on GitHub, and you’re saying all you need is three files? Or your saying clone that repo locally somehow?
TESS: Really you just need those three files.
IVAN: Oh, really? Ok.
TESS: Right. Now in the repo there are two different Docker compose files. There’s one that says very obviously D7 and the other one says D8. The D7 version also worked for Drupal 6. So copy the one that you need for your project, which means you can have different ones for different projects and rename it to docker-compose.yaml and once you have that going you can decide if you want to customize it. It will work just like it is, but you might want to like, change the passwords, or change the database name. You might need multiple databases for your particular project. Your web root might be a different name. Our default is src and the root of your Git repository, but it might be, you know, webroot or docroot or public_html or WWW or something. There’s no real standard there. There’s lots of different names. And you might need to change that particular name in your system or in the Docker file at least, and then once you have those saved, you can do a docker-compose up in that directory and you’re up and running.
IVAN: Ok. This sounds really easy. I’m looking at the Docker compose D8 YAML file right now on the GitHub repo that we’ve published, and what I didn’t realize it that it’s actually 38 lines of text that effectively give you a fully configured local environment. That’s really lightweight. I mean this is a web container, database container, Memcache, even PHP MyAdmin for the database container as well as Mailhog all with the ports correctly configured and their relationships correctly configured. I didn’t realize it was that tiny.
TESS: Yea. I mean that is half the idea of only having the configuration file in there. You don’t have to add a bunch of additional scripts. It’s all inside the container. And once you know how to work with Docker itself, it’s not hard.
IVAN: Not hard.
TESS: A few different commands and then you're off and going. You do have to make a settings.local.php or some other equivalent in order to add the database connection information to let Drupal know what those are but after that you're good.
IVAN: Amazing. So, I’m going to recap this. I go to docker.com, I get the Docker binary. I install that on whatever system I’m on. That’s step one. Step two is go to ten7.com/flightdeck. That takes you to the GitHub repo, that’s the open source Flight Deck repo. You don’t actually even have to clone the whole thing. The bare minimum thing you need is one YAML file that’s in the Flight Deck repo, you choose the one you need, you choose D8 for the Drupal 8 site you’re working on or you choose the D7 version for the Drupal 6 or the Drupal 7 site you’re working on. You take that file, you rename it to docker-compose.yaml, and you put it in the parent folder of the source folder of your Docker root for the Drupal site you have. And, once you’ve done that you make sure that the configuration in docker-compose.yaml points to the right docroot and you create the settings.php file as you described and then you basically compile the containers using Docker and boom, you’re done.
TESS: You missed one step.
IVAN: I did? What did I miss?
TESS: You need to copy the .env file in the same directory as the docker-compose.yaml. And those are really the only two files you really need. You don’t need the override.example file, but it can be useful if you do a lot of SSH’ing.
IVAN: And the .env file that’s the configuration file that tells Flight Deck what version of Drupal you’re running, what the Drupal username and password is, some other docroot things, MySQL configuration, Xdebug and so on.
TESS: Mm hmm. And Xdebug is initially set up to work within a cross platform team, so we have one composed file that will work for both macOS and Windows and Linux, and there are instructions to make sure that you can get all the port forwarding set up on your system built in.
IVAN: Wow. Ok. That’s actually relatively easy. So, the thing that’s missing, I think, is the database, right? Because all your files are in that docroot and you have this container that’s running, that presumably now fires up the database container, so I guess the first thing you would have to do it import your current database from whatever local environment you have, it’s probably MAMP, so you’d maybe do a SQL dump out of there, and then you would import into the container. Correct?
TESS: Mm hmm. And this is why we also include phpMyAdmin out of the box, because it's nice and easy and it's not the greatest tool in the world, and it’s not going to wow anybody, but it does get the job done and it works on every platform.
IVAN: Where is the database itself located? Is that in one of the images?
TESS: So, the database is stored in a container and that container is based on an image which provides the database software. Now that container only exists on your laptop when you first start up Flight Deck, and that’s stored somewhere in Docker's internal volume management system.
IVAN: Ok. And if I wanted to share the latest version of my database with a fellow developer, I would have to use phpMyAdmin to export that database. I wouldn’t be using the local install of MySQL on my machine because that database now lives in that virtual container.
TESS: Correct. And we do also provide console access to the container and we also have the default MySQL port open in the container, so that you can actually access everything, like you would from a remote MySQL server.
IVAN: Now, I think we’re coming to an end of this podcast and I love that we’ve gone into the weeds here and spoke about the details of the configuration of Flight Deck. I do have one more question about it. We never talked about Composer or Drush.
TESS: Or Drupal Console.
IVAN: Or Drupal Console. What’s up with that?
TESS: Or SASS, or Compass.
IVAN: No, we didn’t.
TESS: All of that stuff and what about all the extensions. Well guess what? It’s all built in.
IVAN: It’s all there?
TESS: It’s all there. It’s all inside the web container. So the web container also includes all of those tools. You can literally take a clean laptop with nothing else on it, put Docker on it and you can then clone your repo with Flight Deck and get started. You need nothing else.
IVAN: And, further to that, it sounds like I’d be able to make configuration changes to those versions of the containers and maybe to the version of Sass, Composer... I mean there are configurations that I’d be able to make locally in my machine, commit them to the repo, and should another developer pull those updates they would get those configuration changes as well? Is that a good assumption?
TESS: That depends on what kind of configurations you’re talking about.
IVAN: Ok. So, lets not go any further deeper into that. Maybe we’ll leave that for another episode of the podcast. Maybe we could get a little deeper into kind of the nuts and bolts of Flight Deck. What do you think about that?
TESS: Sure. I mean, one thing that is relevant to bring up is that we do provide different versions of some of the pieces of software. We only really provide MariaDB 10 which is equivalent to MySQL 5.5, PHP 5.6 and PHP 7 and we provide two different SOLR versions, I believe...
IVAN:5.5 and 6.6.
TESS: Thank you.
IVAN: And I just want to point out that we could’ve provided MySQL as the database container. but in the interest of being completely open source and without the need to have proprietary licensing inside of Flight Deck, we decided to go with MariaDB.
TESS: That’s correct because MySQL is currently owned by Oracle and Oracle sometimes can have some licensing terms which can be a little frustrating.
IVAN: And, they can change their mind. And, we'd rather not have that issue within Flight Deck.
TESS: That’s correct.
IVAN: Oh my. This has been a great amount of time speaking with you. I really appreciate your time. Thank you so much for spending the time with me. It’s been a pleasure to speak with you. We’ve had a great conversation.
IVAN: For those of you listening, you can find out more about Flight Deck online at ten7.com/flightdeck.
You’ve been listening to the TEN7 podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message, we do love hearing from you.
Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.