Kubernetes: Our Next-Gen Site Hosting
After months deep in the weeds of Kubernetes, our DevOps Engineer Tess Flynn emerged with the best practices for melding Docker, Flight Deck and Kubernetes to create a powerful open source infrastructure for hosting Drupal sites in production (powered by our partner, DigitalOcean). Ivan and Tess take a deep dive into why we chose this combination of tools, our journey to get here, and the nitty gritty of how everything works together.
Tess Flynn, TEN7 DevOps Engineer
- Why offer hosting ourselves now?
- Differences in hosting providers
- The beauty of containerization, and the challenge of containerization
- The best container orchestrator
- What’s with hosting providers and their opaque pricing? (and why we like DigitalOcean)
- Kubernetes’ highly dynamic environment: updated with just a code push
- Flight Deck, the genesis of our journey to Kubernetes
- Docker enables consistent environments
- Flight Deck + Kubernetes + DigitalOcean
- You can do this all yourself! (or we can help you with our training)
- It all runs on Drupal OR other platforms
- In order to enjoy Drupal + Kubernetes, you must let go of your local file system and SSH, and reevaluate your email system
- Complex files vs. static files and S3
- Kubectl! (it sounds cuter when you say it out loud)
- Cron jobs run differently in Kubernetes
- A Tess talk isn’t complete without a car analogy: Kubernetes is like a garage that comes pre-stocked with all the tools you’ll need to work on your car
- our training at BadCamp 2019
- Drupalcorn 2019
- Alpine Linux
- Flight Deck
- Tess's Return of the Clustering talk
IVAN STEGIC: Hey everyone! You’re listening to the TEN7 podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I am your host Ivan Stegic. We’ve talked about DevOps at TEN7 on the show before. We’ve done an episode on why we decided to expand our hosting offering to Linode back at the end of 2017. We’ve talked about why we think it’s important to have a good relationship with your hosting company. And, we’ve written about automation and continuous integration over the years as well.
For the last year or so, we’ve been working on our next generation of hosting service, and our DevOps Engineer, Tess Flynn, has been deep in the weeds with Kubernetes. Today, we’re going to spend some time talking about what we’ve done—and how you could be doing it as well—given that we’ve open sourced all of our work.
We’re also rolling out training at BadCamp this year, that’s in October of 2019, and we’ll be at DrupalCorn as well, in November. So, we’ll talk about that and what you might learn by attending. So, joining me again is our very own Tess Flynn. Hello, socketwench.
TESS FLYNN: Hello.
IVAN: Welcome, welcome. I’m so glad you’re on to talk shop with me. I wanted to start with why. Why are we hosting our own sites and those of our clients? There are so many good options out there for WordPress, for Drupal: you've got Acquia and Pantheon, Blue Host, and others. We typically use the provider that makes the most sense, based on our clients’ needs.
We’ve had a close relationship with ipHouse and their managed hosting services for a long time. But why start hosting now? For us, as an organization, it’s kind of been the perfect storm of circumstances, from the technology being mature, to the cost of it, and the availability of it, to where we are as an organization from a developmental point of view, to even being more conscious of vendor lock in and actively trying to avoid it.
So, I want to talk about technology a little bit more with you, Tess. What’s so different now than it was a few years ago? Why is it suddenly okay for us to be hosting ourselves?
TESS: There’s been kind of an explosion over the last few years of managed Kubernetes hosting providers. Now, we’ve had managed hosting providers forever. We’ve had things that are called Infrastructure as a service (IaaS) provider; that’s going to be things like AWS and Google Compute Cloud, as well as other providers, including DigitalOcean, but also say, Linode and other ones, which just provide raw hardware, virtual machine and root login. Lately, however, a lot of people would rather break up their workloads into containers, using something that’s similar to Docker. And I’ve talked about Docker before, but Docker is an alternative take on virtualization technologies, which works on taking applications and putting them in their own individual, virtual environment. I’m glossing over so many things when I say that, but it gets the general point across, with the two minutes before everybody else falls asleep.
TESS: What’s really nifty about putting applications into a container is that now the container doesn’t really care where it is. You can run it on your system, you can run it somewhere else, you can run it on a hosting provider. And, the great thing about these containers is that you can download ones that other people have created. You can modify them, make your own, and you can string them together to build an entire application service out of them. And that’s really, really great. That’s like infrastructure Legos.
But the problem is, once you get the containers, how do you make sure that they’re on the systems, on the actual hardware where they are supposed to be, in the number of copies that there’s supposed to be, and that they can all talk to each other? And the one’s that aren’t supposed to talk to each other, can’t? That’s a lot trickier. For a long time the problem has been that you really only have two solutions: you do it yourself, or you use something like Docker Swarm. I don’t have the greatest opinion of Docker Swarm. I have worked with it before in a production environment, it’s not my favorite.
IVAN: It’s a little tough, isn’t it? We’ve had a client experience on that.
TESS: It’s a little tough, yeah. It’s not really set up for something like a Drupal workload. It’s set up more for a stateless application. A prototypical example is, you need to calculate the progression of matter within the known galaxy, factoring a certain cosmological constant. Take that variable, set it into a compute grid and go, “Hey, tell me what the results are in 15 years.” But you don’t really do that with Drupal. With Drupal, you’re not just going to send off one thing and always get the same thing back. There’s going to be state, which is preserved. That’s going to be in the databases somewhere, and there are going to be files that are uploaded somewhere. And then you have to get load balancing involved, and then it gets really complicated, and it’s like ugh. I really didn’t like how Swarm did any of this stuff. It was very prescriptive. It was, you do it their way, and nothing else.
IVAN: No flexibility.
TESS: No flexibility at all. It was really, really not fun, and it meant that we had to do a lot of modification of how Drupal works, and incur several single points of failure in our infrastructure, in order to make it work in its form. That whole experience just did not get me interested or excited to make a broader Swarm deployment anywhere else.
Then I ran across Kubernetes, and Kubernetes has a very different mentality around it. Kubernetes has more different options for configurations, and you can tailor how Kubernetes manages your workload, rather than tailoring your workload to work with Docker Swarm. That’s why I really liked it. What's really nifty is, once you have Kubernetes, now you have an open source project, which is platform agnostic, which doesn’t care about which individual hosting provider you’re on, as long as you have containers, and you can send configuration to it somehow, it’s fine, it doesn’t care.
A lot of managed hosting providers are going, “Hey, you know, VMs [virtual machines] were kind of nifty, but we really want to get in on all this container stuff now, too.” “Oh, hey, there’s a container orchestrator,” which is what Kubernetes is, and what Docker Swam is, as well, a container “orchestrator” which does all of the making sure the containers are on the right systems, are running, they can talk to the containers they're supposed to, and can’t talk to containers they're not supposed to.
That made a lot of infrastructure providers go, “This is not really a Platform as a service anymore. This is another form of Infrastructure as a service. As such, that is a segment that we can get into."
So, first it started with Google Kubernetes Engine, which is still considered today the defacto version. Amazon got into it, Azure got into it. And all of these are pretty good, but a lot of these huge cloud service providers, you can’t get clear pricing out of them to save your life.
IVAN: Yeah. That’s so frustrating, as a client, as a business owner. How do you do that? It’s insane.
TESS: I mean, the only way that it seems that is deterministic, in order to figure out what your bill is going to be at the end of the month, is to spend the money and hope that it doesn’t kill your credit card. [laughing]
IVAN: Yeah, right, and then try to figure out what you did, and ways of changing it, and then hell, you’re supposed to be just charged that every month from now on, I suppose.
TESS: It’s just a pain. It wasn’t any fun, whatsoever. So, an alternative approach is, you could actually install Kubernetes yourself on an Infrastructure as a service provider with regular VMs.
IVAN: And, we considered that, right?
TESS: Oh, I considered it, and I even spun that up on a weekend myself. It worked. But the problem is, I’m a colossal cheapskate and I didn’t want to spend $30.00 a month for it. [laughing]
IVAN: [laughing] If only there was a supporting ISP that had free Kubernetes support, and just charged you for the compute engines that you used.
TESS: I was really kind of sad that there wasn’t one, until six or eight months ago, when DigitalOcean announced that they have in beta (now it’s in production) a Kubernetes service, where the pricing was incredibly clear. You go to the cluster page, you select the servers that you want to see (the nodes as it were). I know, Drupal nodes, infrastructure nodes, it’s really confusing. Don’t even get physics people involved, it gets really complicated. [laughing]
IVAN: No, please. No, don’t. [laughing]
TESS: But you select which servers that you want to have in your Kubernetes cluster, the sizing, and the price is just listed, right there, in numbers that you can understand! [laughing]
IVAN: Per month, not per minute.
TESS: I know, per month, not per minute.
IVAN: It’s just the small things. Crazy.
TESS: And, it really targeted the kind of market that we are in for a hosting provider, and it made me really excited, and I really wanted to start putting workloads on it, and that’s what started the entire process.
IVAN: It really was, kind of a fortuitous series of events, and the timing kind of just really worked out. I think one of the biggest things for us, for me, is that with Kubernetes, we don’t have to worry about patching and security updates, and monitoring them, and these large hardware machines that we have to keep patched and updated. Essentially, it’s updated every time we do a code push, right? I mean, we’re still concerned with it, but it’s a much easier burden to bear.
TESS: Right. Now what’s going on is that, every time that we do a push, we’re literally rebuilding every system image necessary to run the underlying application. Which means that if we need to push a system update, it’s really just a matter of updating the underlying container's base image to the newest version. We’re already using Alpine Linux as our base containers, which already is a security-focused minimal container set.
IVAN: So, this is actually a good segue to what I wanted to talk about next. A few years back (as opposed to six to nine months back), which is how we kind of got down the road to get to Kubernetes was, I think the origin of all this really is, Flight Deck, and the desire for us to make it easy for developers who work at TEN7—and anyone else who uses Flight Deck, honestly—to have the same development environment locally. Basically, we wanted to avoid using MAMP and WAMP and different configurations so that we could eliminate that from any of the bug-squashing endeavors that we were going into. So, let’s talk about this started with Docker and led into Flight Deck, and what a benefit it is to have the same environment locally as we do in staging and production.
TESS: So, there’s a joking meme that’s been going around, and DevOp cycles, of a clip of a movie where, I think a father and son are sitting and having a very quiet talk on a bench somewhere in a park, where the kid is saying, “But it works on my machine.” And then the Dad hugs him and says, “Well, then we’ll ship your machine.” [laughing] And, that’s kind of what Docker does. But joking aside, I wanted to get that out of the way so I’m not taking myself too seriously. [laughing]
So, one of the problems with a lot of local development environments—and we still have this problem—is that traditionally we’ve used what I consider a hard-installed hosting product. So, we’re using MAMP or WAMP or Acquia Dev Desktop, or if you’re on Linux you’re just installing Apache directly. And all of those work fine, except when you start working on more than one site and more than one client. So, suddenly you have this one problem where, this one client has this really specific php.ini setting, but this other client can’t have that setting. And MAMP and WAMP work around this through a profile mechanism which, underneath the covers is a huge amount of hyperlinking and weird configurations, and spoofing, and like eww, it makes me shutter.
IVAN: Yeah, it makes me cringe just to talk about it, yeah.
TESS: And, the problem is that, every time you have to do this, every developer has to do this themselves, they can’t just standardize on it. So, if somebody has an individual problem on their system, that only happens on their system at 3:45 on a Thursday, after they’ve had chili for lunch or something or other, then you can’t really reproduce it. So, the solution really is, you need to have replicatable, shareable, consistent development environments across your entire team. And that’s what Docker does.
Docker provides that consistency, that shareability, and makes sure that everybody does, in fact, have the same environment across the board. That’s the entire point of that, and that’s where the whole joke about, “Well, then we’ll ship your machine,” [laughing] because that is in essence what containers are. They are system images that run particular bits of software. Now, once we moved everyone to Docker for development, we now had a consistent environment between all of our systems, so that now we didn’t have to work about a number of different problems.
Another good example is, this site uses PHP 5, this site uses PHP 7—a little out of date now, but it was very relevant two years ago—in which case, how do you make sure you’re on the right version? Well, with Docker, you change a text file, and then you boot the containers up, and that’s it.
IVAN: And that text file lives in a code repository, right? So, everybody else gets that change?
TESS: Mm hmm, because you are literally sharing the same environment; you are enforcing a consistent development environment across your entire team for each individual project. And, if you use that strategy, you have something that is flexible, yet at the same time incredibly consistent.
IVAN: And this is really important across all of our developers, and all of our local development that we do, but the challenge then becomes, how do you consistently replicate this in a staging or in a test environment, and even in production? So, that’s kind of the genesis of how we thought Kubernetes could help us here, right?
IVAN: So, the challenge to you from me was, how do we make this work in production?
TESS: So, the nice thing about Flight Deck is, it was always designed with the intention of being put into production, But the orchestration component just wasn’t there, and the hosting component wasn’t there. Kubernetes showed up, and that solved the orchestration component, and then, eventually DigitalOcean showed up and now we have the hosting component. So, now, we have all the pieces together to create a consistent environment that is literally the same containers, from the first time someone starts working on the project, to when it gets deployed to production. That is the height of continuous integration ideals, to make sure that you have consistency across all of your environments. That you don’t have different, weird shared environments along the way, that everything is exactly the same so that you know that it will work.
IVAN: I want to stop right there, just so our listeners can appreciate the power of what you just said. You basically said, “I’m going to be working on a website, or a web application locally, with some sort of stack of required server components, whose version numbers and installation profile is configured in a certain way. My teammate is able to replicate that environment exactly, to the version, simply by using the same repo, and by using Flight Deck.
Moreover, all of those version numbers and the stack that is being used, is actually also the same now in staging and, most amazingly to me, in production. So, we can guarantee that what container is functioning in production on the Kubernetes cluster, is actually on staging and on everyone else’s machine. We’ve totally eliminated any variability and any chance that the environment is going to be causing an issue that one person may be seeing that another isn’t.
TESS: That’s correct.
IVAN: That’s pretty amazing!
TESS: It’s a really difficult thing to do, but starting with the containers and building that from the base up actually makes it a lot easier, and I don’t think that any other local development environment, even container based local development environment such as DDEV and Lando are doing this quite yet. Last I heard, I think DDEV was working on a production version of their containers, but it’s not the same containers, whereas with Flight Deck, it literally is the same container.
IVAN: It’s the same configuration. Everything is the same. That’s pretty amazing. I’m still kind of really impressed with all of the stuff that we’ve done, that you’ve done. And, honestly, this is all open source too. This is not like TEN7’s proprietary product, right? We’ve open sourced this, this is all on the web, you can download it yourself, you can figure it out yourself, you can do this as well. You can start your own hosting company.
TESS: That’s correct. The key item which puts all this together is, the Ansible role called Flight Deck Cluster. What Flight Deck Cluster does is, it will create a Flight Deck-flavored Kubernetes cluster and it works perfectly well on DigitalOcean. There’s no reason why it can’t work on say, Google Kubernetes Engine or AWS or anyone else. The architecture that Flight Deck Cluster uses is meant to be simple, durable and transportable, which is something that a lot of other architectures that I’ve seen just don’t have.
IVAN: So, we’ve designed a lightweight set of Docker containers called Flight Deck that you can use locally. We’ve evolved them so that they work with Kubernetes, which you can deploy anywhere in staging and production. We’ve open sourced them. And, the fact that it runs Kubernetes, all you need is a service that supports Kubernetes and you should be able to run all of this in those other locations.
So, we’ve talked about how we started with Docker and how that evolved, and I talked about how we've open sourced it and it’s available to you. I want to spend a little bit of time getting into the details, into the nitty gritty of how you would actually do this for yourself. Is there an app I download? Is it all the YML, all the YML files that we’ve open sourced? What would someone who wants to try this themselves, what would they have to do?
TESS: The first thing that I would probably do is, start running Flight Deck locally. Because you don’t need to pay any extra money for it, you just need to use your local laptop, and it’s also a good experience for you to learn how to interact with Docker by itself. That looks good on a résumé and it’s a good skill to actually have.
I have a talk that I used to give about Docker, and I know that there’s a blog post series that I posted somewhere a long time ago, about how Docker actually works under the covers. Both of those are going to be invaluable to understand how to get Flight Deck working on your local environment, and once you have it working on your local environment, then the next problem is to figure out the build chain. Now the way that our build chain works is, that we have another server, which is a build server, and what the build server does, is it’s going to receive a job from Gitlab and that job is going to take all of the files that constitute the site, it will build them into a local file system, and then it will put those inside of a container which is based on Flight Deck. Then it will upload those to a container registry somewhere else. So that we already have a few additional pieces of technology involved. But the nice thing is, Gitlab is open source, Ansible is open source, and all of our build processes are run through Ansible, and the Docker registry is also open source. It's just a container that you can run somewhere. There’s also services that you can buy that will actually provide you a container registry on a fee basis. All of those are definitely options. Once you have the container in a registry somewhere, then you can run Flight Deck Cluster to build out the rest of the cluster itself.
IVAN: You make it sound so easy. [laughing]
TESS: I make it sound easy, but it’s a lot of code, but it is all open source and it is all there for you to use. Right now, our cluster is based on a development version of Flight Deck, which I’ve been calling Flight Deck 4, and this version is intentionally natively designed for a Kubernetes environment. But it still works perfectly fine under Docker Compose locally, and it is literally the containers that we are using in production right now, at this minute. All of those containers have been thoroughly documented. They have nice readmes which describe exactly how you configure each individual container. And the Flight Deck Cluster role on GitHub also has an extensive readme document which describes how every individual piece is supposed to work.
IVAN: So, the easiest way to get to all that documentation into the repo is to simply go to flight-deck.me. That will redirect you to a blog post about Flight Deck on the ten7.com website, and at the bottom of that post you’ll see links to the GitHub repos and all of the other information that you’ll need to get to that.
So, I wanted to talk about the fact that the hosting itself, the Kubernetes hosting that we have, is optimized for Drupal right now—I kind of struggle to say "optimized for Drupal." It’s just configured for Drupal. There’s no reason that Kubernetes is, and what we’ve released, is locked into Drupal. We are hosting our own React app on there. We have a CodeIgniter app that’s running, we even have a Grav CMS site on it. There’s no reason why you couldn’t host WordPress on it, or ExpressionEngine or any other php, MySQL, Apache, Varnish, Stack on it. Right? There’s nothing innately that forces you to be Drupal on this, right?
IVAN: And that’s also from a design perspective. That was always the intention.
TESS: It’s intended to be run for Drupal sites. However, it always keeps an eye towards being as flexible as possible.
IVAN: So, I think that’s an important thing to mention. Let’s talk about some of the challenges of running Kubernetes in a cluster in production. It’s not like running a server with a local file system, is it?
TESS: [laughing] No, it isn’t.
IVAN: [laughing] Okay. Let’s talk about the opportunities of things to learn.
TESS: The biggest, scariest thing about Kubernetes and Drupal is, you have to let go of your local file system. That is the most scary thing that I have to tell people about Kubernetes.
IVAN: So, no file system, huh?
TESS: No file system.
IVAN: Does that make it slow?
TESS: Well, not really. Let me describe why. The problem is, that— and I’ve had this in my Return of the Clustering talk—is that we’re used to something which is called “block storage.” Now, block storage is pretty great. It is a literal attached disk to the server. So, it is mounted on the server, you have direct access to it, and you can store all kinds of things to it. And it’s fast, and it’s right there. It has no failover, it can’t be shared across the systems, but ehhh, whatever, we have one big server, who cares about that.
Then, if you do try building a traditional server cluster, well, you can’t quite do that. So then you get network file system involved, NFS. And then now, all of the file reads and writes occur over the network to some other centralized server. Okay, it still looks like a local block storage, it still works like block storage, so, okay, sure. But the problem with that is that network file systems, by their base nature, introduce a single point of failure.
Now, that’s not good by itself. If the NFS server goes down, your entire site no longer looks or functions correctly. But the problem is, that it also doesn’t scale either. There’s a natural limitation between the number of different replications for frontend server, servers that intercept the actual requests from people, and then send them to the Drupal backend for processing, and then push back their responses. There’s a natural limitation between those systems and those that can access NFS. And as soon as you have too many accesses, suddenly NFS is not going to be keeping up with you and your performance drops to the floor.
Also, NFS is kind of persnickety. You have to tune it. You have to make sure that it has enough RAM, enough bandwidth. You have to make sure it’s physically proximate to the rest of the servers. And, all of this is because it’s trying to replicate block storage. Now, block storage is great for a whole bunch of data, but in a cloud architect's perspective, there are really two different kinds of data. There’s complex data and static data.
And when I tell people about this, they go, “Well, what’s a complex file?” A lot of people will say, “Well, we have a whole bunch of files which are all linked together, that’s complex, right?” Nope. “Well, we have some Excel documents that’s on an NFS file, that’s complex, right?” Not really. So, what is a complex file?
I spent hours, tried to squeeze an answer [laughing] out of the internet for this, and eventually arrived at the answer from a cloud architect's perspective: “complex files, such as the files which constitute the actual underlying disk storage for say, a MySQL database.” Data, which is written sparsely and seemingly randomly in multiple locations at multiple times with strict concurrency requirements. Now when I say that, does that sound like anything that we actually upload in a Drupal site?
TESS: Nope. None of it does. Block storage is required for complex data. But for static data, which is virtually everything that a Drupal site hosts, we don’t need it, it’s too much. It’s way too complicated. And, it doesn’t scale. So, what’s the solution? The solution really is, we need to treat the file system like an API. We need to treat the file system like a database. We don’t care where the database is, as long as you have an IP, a login and the correct credentials to actually get to the database, and then we have multiple readers, multiple writers. That’s what we want for a file system, right? Well, it turns out, there’s a thing that does that already, it’s called S3.
IVAN: Yes, AWS, hello. [laughing]
TESS: And the nice thing about S3 is, it’s perfect for static data. It’s API accessible and it can be made internally redundant. So, it has its own high availability built in that we don’t need to worry about. The nice thing that’s even more than that, is when we say S3, most people go, “Oh, Amazon.” No. S3 is, in fact, a standard. It is not just Amazon’s implementation of S3. There are multiple implementations of S3. So, I usually like saying an S3-compatible hosting provider. And that’s going to include anybody who runs any kind of S3-compatible service. And there’s actually an open source product called Ceph that actually provides an S3 frontend for file storage. And that is actually a service that DigitalOcean also provides. They have DigitalOcean spaces, which provide an S3-compatible static file interface, that’s actually powered by a Ceph cluster underneath the covers. So, open source all the way down to the core.
IVAN: Well, I didn’t know that spaces was Ceph underneath the covers. That’s cool.
TESS: It’s just buried in there. You could find it though.
IVAN: Cool. So, file storage is a challenge, but we fix that by using S3.
TESS: Yep, because Drupal 7 and 8 actually have very good S3 support. There’s S3 FS, that particular module which is excellent for doing Drupal 7 sites. We’ve been using Fly System for Drupal 8 for a few different reasons, but there are reasons that are a little bit easier for us. But your mileage may vary.
IVAN: And, if you’re going to host something that’s not Drupal related, you would need to find some other S3-compatible layer module, right?
TESS: Like for the CodeIgniter application, we are currently looking at implementing that as well.
IVAN: And, there’s a React app as well that we’ve deployed. That uses the underlying Drupal site, though, doesn’t it?
TESS: Yes, that doesn’t actually need a local file system.
IVAN: There’s no SSH access to a cluster of Kubernetes, is there?
TESS: Yes, that’s the other thing. It’s like after I already brutalized you with saying, “No, you can’t have a local file system,” now I take your SSH away as well. [laughing]
IVAN: [laughing] But there is something to use to replace it, right?
TESS: There is. The problem is that, you really, really, really, really, really, really, really shouldn’t use SSH in Kubernetes. SSH is a very dangerous thing to have running anywhere, because it is a potential security access point that can be used and abused, both internally and externally. You really don’t want to have to run it, because if you want to run SSH in Kubernetes, you have to run it in a container. And if you run it in a container, you’re running it as root. And if you’re running it as root, you’re running it as root on the underlying hardware that’s powering the cluster, and that’s bad. [laughing] You don’t want to do that.
So, instead you want to access what is typically called “the backplane.” The backplane is going to be access to the workload via the orchestration system. So, for Kubernetes, the backplane access comes in the form of a command line application called Kubectl or “Kube control” or “Kubey control” or “Kubectl” or like 15 other different names. [laughing] I always thought of Kubectl, that’s my favorite.
IVAN: Let's spell it out. [laughing] I like that one too. k-u-b-e-c-t-l
TESS: And this application not only lets you interact with the orchestrator, but also allows you to directly access individual containers as well. Although getting to an individual container is a little bit more difficult, once you’ve done it a few times, it’s not that hard. Because Kubernetes is so popular, there’s a lot of other command line environments, which will have auto completion assistance for Kubectl as well. So, for me, if I enter in a parameter to Kubectl, say for name space, I can hit tab and it will give me a list of the name spaces that I have. So I don’t actually have to type it out.
IVAN: Pretty slick.
TESS: I use Z Shell (ZSH) but that’s me, I’m weird. Some people like using Fish or some other shell. And I’m sure there’s auto completion mechanisms for your favorite shell somewhere.
IVAN: There’s not a whole lot of challenges then, with Kubernetes. You’ve kind of mentioned a few that are surmountable. Is there anything else, a budding developer, a budding DevOps person should know about, that are looking to start to explore hosting for themselves?
TESS: Well, they should also keep in mind that email is a problem.
IVAN: Yes! We discovered that in the last few weeks, didn’t we?
TESS: Yes, we did.
IVAN: So, we decided that we were going to use an external, transactional email provider. We ended up on SendGrid. But you don’t think of these things once when you’re working on a cluster that’s managed because, hey, these machines all have SendMail on them.
TESS: Yup, and that’s one thing that you really can’t rely on when you start working with a container-based workload. It exposes a lot of these things. But, we’re not where we were two or three years ago where this would’ve been a huge, scary, problem. These things have existing solutions, which are not that difficult to implement, even today.
IVAN: And there are some free tiers as well that you can use, especially if you don’t have a high volume of emails that you’re sending out.
TESS: If you’re only sending 500 emails a day, you can configure your G Suite email as the SMTP provider.
IVAN: Exactly. What about cron? Isn’t that a problem too?
TESS: Cron is a little bit different in Kubernetes. So, the thing with cron is that, in Kubernetes, cron isn’t just something that runs a command. In a traditional server workload, cron is some background process that exists in the system, and when a certain time shows up, it runs a certain command that you tell it to. And, it assumes that you’re running it on literally the same exact system that is running everything else, your web workload. Right?
TESS: That’s not quite the case in Kubernetes. In Kubernetes, a cron job actually runs a container. So, when you actually have your web workload, you’re going to have one container, say, for Apache, somewhere, which is running your site. Then you have a cron job in Kubernetes, and that cron job will literally spin up a completely separate container in order to actually run that process.
So, that’s a bit different.
Now, the only real part of that which gets really confusing is, if you don’t have a nice separation of all of the different infrastructure we just finished talking about, if you don’t have any local disks that you need to worry about, if you don’t have SendMail you have to worry about, if you don’t have any of this stuff and you can scale out your web container to 10 or 20 or more, and not have a problem because they all rely on external API-based providers, then it doesn’t really matter what you do with cron. You just literally run the same container that you run for your web workload, with the same configuration and everything else, but you only tell it run a particular command, instead of "Run Apache." And that’s it. That’s what we do. And, it’s actually not very hard.
IVAN: What’s your favorite thing about Kubernetes? I’m only going to give you five minutes at the most. [laughing]
TESS: [laughing] I think the thing that I like the most about it, is probably the ability to easily scale things. Once you actually have solved all the underlying infrastructure problems, you basically have just a container-based workload that you can say, “I need to run three of these.” Then you can tell it and it will run three of them, and it will just run it, that’s it, you don’t need to worry about it. It already load balances it for you. How can I describe this? Well, let’s go back to the infamous car analogies again.
IVAN: They work.
TESS: They work, but you know they work within a US cultural context of a certain decade period, of a certain geographic location, but let’s put that aside for a second.
So, a car analogy. Let’s say you have a car, and you want to do some work on it. And you go to your garage and what do you see? The car and an empty garage. That’s often what a lot of other systems look like. When you have to do traditional clustering with regular virtual machines, or even self-hosted physical machines, you have to go over to your local hardware store, buy all the tools, buy the car jack, buy an engine lift, buy an air compressor and a whole bunch of other stuff, in order to do your car stuff, and it’s a lot of work and a lot of investment.
With Kubernetes, it’s more like, Okay, I go to my garage and I have Kubernetes. So I have all the tools already. All the tools are just there on the walls, right now. I can just start working. That’s what I really like about Kubernetes. It provides me a room with all the tools for me to actually make this workload do what I want it to do, rather than having to go and grab yet another thing, then another thing, then another thing. Then try to make compromises to make two things, which aren’t the thing that I can’t get right now, but they’re the two I have, to work together.
IVAN: I love the analogy. [laughing] I think that works, Tess. So, what about training? Wouldn’t it be great if, instead of trying to figure this all out yourself (like we did), you could just have us show you how to do it?
TESS: Gee, wouldn’t it? [laughing]
IVAN: Wouldn’t it be great? Well, guess what? That actually exists. We’re going to be doing some free trainings at BadCamp and then at DrupalCorn as well. We’ll be at BadCamp next month, the beginning of October. Now, they’re free trainings, but there is a cost of use to attending the training itself, so I think you have to register and it’s $20, or $10 at DrupalCorn. They’re free as far as we’re concerned.
Can you talk through, just a little bit about the format of the training that we have set up? What are you going to learn and who is it for?
TESS: So, we’ll briefly touch upon different kinds of Kubernetes hosting providers, as well as what Kubernetes actually is and what it does, and what it gives you. Then afterwards, we’re going to start containerizing your particular application. So, we’ll start working with containers, putting them onto Kubernetes, getting used to how to use Kubectl, how to work with individual definitions within Kubernetes, and making all of these pieces work together.
IVAN: And, it’s a four-hour workshop, it’s half a day, you get to spend time with Tess, and I think I’ll be there too. It’s going to be great. So, if you want to contribute to Flight Deck, or to Kubernetes, the Kubernetes Flight Deck Cluster that we have, we’d love it. It’s all online. You can visit ten7.com, and you’ll find it there on the what we give back page and you can also visit us on github.com/ten7, and you’ll see all the repos there. We’d love your help. Thank you, Tess, so much for spending your time with me today. This has been truly great.
TESS: Not a problem.
IVAN: So, if you need help with your own hosting, or figuring out what makes most sense to you, we’d love to be there to help you, whether you’re a developer or a large university, or a small business, it doesn’t matter. We’re happy to provide consulting, whether that means deploying your own Kubernetes or having us do it for you, or even selecting another vendor that makes the most sense to you.
Just send us an email and get in touch. You can reach us at email@example.com. You’ve been listening to the TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is firstname.lastname@example.org. And don’t forget, we’re also doing a survey of our listeners. So, if you’re able to, tell us about what you are and who you are, please take our survey as well at ten7.com/survey. Until next time, this is Ivan Stegic. Thank you for listening.