Software Developer, Author and Creator of the Pi Dramble
Ivan and Jeff geek out over their love of Raspberry Pis
How philosophy relates to programming
Brambles, Drambles and Druplelets, oh my!
How to keep your Raspberry Pis cool
IVAN STEGIC: Hey everyone! You’re listening to the TEN7 podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business and the humans in it. I’m your host Ivan Stegic. My guest today is Jeff Geerling, author and software developer, who is deeply involved in numerous open source development communities. He describes himself as a creative person who builds great software. I first saw his name in the Drupal community, but he’s really active in Ansible and is also the creator of Drupal VM for local Drupal development, as well as the Raspberry Pi Dramble. Hey Jeff, welcome. It’s a great pleasure to have you on the podcast.
JEFF GEERLING: Thank you very much. I’ve been looking forward to talking to you guys on TEN7’s podcast.
IVAN: I’ve been looking forward to talking to you as well. I absolutely love Raspberry Pis. I have, I think, at least 10 of them over here. So I don’t know if that’s catching up to how many you have, [laughing] but it’s definitely something I have tinkered with for the last number of years. And so getting an opportunity to talk to you is very exciting for me.
JEFF: Ten is a good start. You’re going to have to up your game a little bit though [laughing].
IVAN: [laughing] How many do you have Jeff? Come on, let’s be honest.
JEFF: Just looking over in my parts bin I have seven that are not in any active use. I have four in my cluster. I have five hot spares for other purposes and three that are running household projects right now.
IVAN: So, close to 20, if not more.
JEFF: Yeah.
IVAN: That’s a lot of Raspberry Pis.
JEFF: It fluctuates depending on what projects I’m working on.
IVAN: That’s awesome. I want to spend the whole interview talking about Raspberry Pis and the Raspberry Pi Dramble that you have and that you’ve been working on recently, and that’s been going for over four years now, I think. But before we dive deep into that, I’d like to get an understanding just a little bit, about Jeff Geerling, the man. Where do you live in the world? Where did you go to school? What do you do in life? You’re not just playing with Raspberry Pis all the time.
JEFF: [laughing] No. I live in St. Louis, Missouri, which is pretty near the geographical center of the United States. Some people call it flyover country, but because of that we actually now have a Drupal Camp named Flyover Camp.
IVAN: Yes, we do. [laughing] It’s awesome.
JEFF: I got to meet one of TEN7’s employees, Tess Flynn there. She had a really good talk on Kubernetes. But I love living in the Midwest. One of the best advantages is I have a lot of family here. The cost of living is lower than coastal cities, and I get to still do a lot of remote work. The coolest thing is, any flight anywhere in the U.S. is only about two, two and a half hours, whereas a lot of times I get somewhere and they’re like, "Oh, I just got off a six or seven hour flight from the other coast,” so it’s nice to be able to get around anywhere in the U.S. and still have family and all those benefits.
I actually studied philosophy, which you could say it has nothing to do with programming or nothing to do with tech, but really, a lot of the philosophy I studied was logic and reason—which, if you’re a programmer—that’s pretty much all you deal with on a day-to-day basis.
Actually, if you’re dealing with Kubernetes, logic is out the window.
IVAN: [laughing] Yes, it is. It’s a different kind of logic, right?
JEFF: [laughing] Yeah, deep down, there’s ones and zeros somewhere underneath, and you have to understand that. So, I studied philosophy. I actually went to a seminary to be a priest for a few years.
IVAN: Oh.
JEFF: I ended up finding out that was not what I was meant to do. I wanted to get married. So, I ended up leaving the seminary, getting married, having three kids and a wife, and I'm happy to be here in St. Louis. One of the things other than that in my life that’s really affected a lot, especially how I work and the jobs I’ve worked, is my Crohn’s Disease. It’s a chronic illness that I have that has required a few surgeries. It’s required a lot of time in the hospital. In fact, I was just in the hospital a couple weeks ago for it, but because of that I’ve always been extremely grateful to have the skills that I have in tech because I can work. There’s been many times where I’ve been in the hospital, but nobody would even know it because I’m still doing my work, since I can do everything I need to on a computer or an iPad or whatever and work remote.
So, I’m extremely happy and feel very blessed to be able to do that kind of work. In open source everything is asynchronous, so that benefits me, because if I had to do a job where I had to be at a desk or be at a place doing manual labor of certain hours in a day and I had to miss a lot of work, I wouldn’t be a very good employee. But I could be a good employee doing remote work and open source work, even though I have Crohn’s Disease, which impacts my life and a lot of my friends’ lives pretty deeply.
IVAN: I don’t know very much about Crohn’s Disease. I know it’s related to your digestive system and I think to your intestines.
JEFF: Yep.
IVAN: Is it something that either you’re genetically predisposed to have? Is it something that’s hereditary? How did you find out?
JEFF: It’s something that’s still under investigation. One of the difficult things is that Crohn’s is in a family of immune-related diseases and for pretty much all of them, it’s hard to say right now with the research we have, if it’s more genetic or more environmental. There’s a lot of theories, and one other interesting thing—and a good reason why I stay in St. Louis—is we have one of probably the top five or 10 Crohn’s research facilities at our local Washington University Hospital and Medical School.
I’ve been able to talk to some of the best Crohn’s doctors in the world, and all of them have theories, and there’s a lot of research but not a lot of concrete facts other than, throw these medicines at it and hope for the best. Unfortunately, in my life, I’ve tried literally every medicine. I’m not joking. Every medicine that’s ever been made for Crohn’s, I’ve been on it. One of them paralyzed me from the waist down, so I got off that medicine.
IVAN: Oh my God.
JEFF: So, I'm not paralyzed anymore, so that's good. Another one caused anaphylaxis so I’m like, Okay, I’m not going to take that one. So, all these drugs, some of them are life changing, but a lot of them don’t last very long because the way that they operate, your body starts reacting to them no matter what. So, once you hit that, you start doing surgeries—that’s the stage I’m in right now—but the good thing is, in my mind, if you have a positive outlook on it, if you could find ways to work with it, you can bounce back. I’m generally a very optimistic person. I think that’s also how I can survive in the open source world where things can get a little bit dicey sometimes.
IVAN: [laughing] They certainly can. Well, I’m glad that you’re able to talk about Crohn’s so openly, and I hope that your last bout wasn’t a terrible one, a bad one, and I hope you’re on the upswing.
JEFF: I was only in the ICU for one day. That’s a good thing. [laughing]
IVAN: Oh my gosh. Well, I’m glad to be talking to you about it. So, you’re in the Midwest. Working remotely is amazing. You might know we became remote in 2017, and we’re not looking back. It’s definitely opened up so many different avenues for us. So, I’m glad that you have that as well to be able to do what you do at home.
The pidramble.com site—I want to start at the end and then kind of work back to how you are hosting pidramble.com in your basement on a cluster of Raspberry Pis. And they run Ansible and Kubernetes and Drupal and NGINX and all kinds of things, and to someone who heard a whole bunch of jargon right now, and doesn’t understand why this isn’t a really cool thing, can you describe what pidramble.com is?
JEFF: So, most of us who have a Drupal site or WordPress or whatever kind of website we’re running, most of us have a cloud hosting provider, and you never have to worry about things like servers and power and networking and all that kind of stuff, you just go online, buy a service, and you update a Drupal site to it and log in and edit your stuff. I’ve always wanted to have as much control as I can over my hosting and my sites and my performance and all that. So, a few years back when the Raspberry Pi came out, it was interesting.
The first couple versions of the Raspberry Pi were super underpowered, and if you installed Drupal—6 or 7 was the current version at the time—even that version, you’d install Drupal on it, it would take a minute or two for a page to load. They were super slow, and didn’t have much memory or anything. But as time when on, the Raspberry Pi foundation kept introducing newer, faster models, and somewhere around 2014—I think, I don’t remember exactly—they introduced the Raspberry Pi 2, and that model had enough memory to actually run Drupal pretty well. So, I bought a few of those to see if I could set up a Drupal cluster and run my own Drupal website in my own house using a cluster of them.
You could do it on Raspberry Pi, but because of the fact they use micro SD cards which are not as fast or as good as hard drives for longevity, much less SSDs that most of us have nowadays. Because of that, if you just run it on one Raspberry Pi, it’s kind of flaky and it might just blow up and die one day, and then you have to reinstall it and stuff. So, I wanted to see if I could make a robust little cluster of computers to run a Drupal site. And, there was a secondary thing that I was trying to do too, to make a little cluster of computers to demonstrate running Drupal or other applications like it in high availability with multiple servers and have it be able to be in a little box that I could bring with me somewhere.
So, I started doing that. I brought it to Drupal Camp St. Louis, I brought it to DrupalCon. I think the first one was Austin that I brought it to, and Dries [Buytaert] actually took a picture of it. This was before I worked for Acquia—and actually I was hired by Acquia soon after DrupalCon, completely unrelated but it was just an interesting aside. Dries came up and he’s like, “Oh, what is that?” I was like, “Oh, it’s my Raspberry Pi cluster.” He’s like, “Oh, that’s really cool. I gotta take a picture.” So, he and I now have shared more stories about pictures and things. He’s actually a pretty decent photographer.
IVAN: Photographer, right?
JEFF: Yeah. And, I’ve always loved photography. I used to do it semi-professionally with some photojournalism, but don’t get to do it as much nowadays. However, some of my other Raspberry Pi projects do have to do with cameras.
But, getting back to the Pi Dramble, over the years I’ve made it more robust, I’ve done some more work to automate the setup process, I’ve documented everything in excruciating detail. There’s videos on the website of how to do the first version of the cluster. I haven’t set up videos for the current version.
But in 2017 when Kubernetes was getting to be a lot more popular and I was starting to use it for some things, I thought, Maybe it could run on the Pi. It ran, but it just barely ran. I had a lot of trouble because the Raspberry Pi only had one gigabyte of memory, and that was the absolute minimum you could run Kubernetes on. So, I got it working. It was kind of janky and kind of fell apart sometimes, and I was getting really frustrated. But in 2018 the Raspberry Pi—no, was it 2018 or 2019?—was that this year that the Pi 4 came out?
IVAN: Pi 4 came out this year, I believe it was in June. I think they were talking about it for months, but no one ever knew when it would come out.
JEFF: That was a lot more recent that I remembered. But the day it comes out, it’s really hard to get Raspberry Pis right after they’re released, because they are very popular for makers and hackers and people who have fun with computers. And so I placed an order from a company in the UK for one, because most of them have limits on the orders for the first few weeks. So, I did one from the UK, one from a place in the U.S., I got two from Micro Center, they’re usually the best place to get them in the U.S.
IVAN: Yes, they are.
JEFF: So, I got those four together, and I got the 2 gigabyte models, and with 2 gigabytes of RAM, Kubernetes actually ran pretty well. So, that’s the current state of my cluster. I have Kubernetes running, I have Drupal running on Kubernetes, I have the 2 gigabyte Raspberry Pi 4s and I power them using power over ethernet, which means I only have one cable I have to plug into each Pi. If you go to pidramble.com, the picture there shows you how they look and what they do.
IVAN: Now, what the heck is a Dramble? I know what a bramble bush is, but let’s take it down. What’s a Dramble?
JEFF: A Dramble, is, I guess it’s called a portmanteau, when you put together different words. A bramble is a bush of raspberries. So, when you see a raspberry bush, I usually call it a raspberry bush, but a lot of people call it a bramble.
IVAN: I knew that was the case. Bramble is something I knew. [laughing]
JEFF: Yeah. [laughing] So, traditionally from the beginning of the Raspberry Pi, people who made clusters of them called them brambles. That’s not as much the case anymore since Raspberry Pi has gotten super popular. But early on everybody would say, “I have a bramble of Pis.” But my bramble of Pis was running Drupal, and so I took Drupal and bramble and put them together and it came up with "Dramble." Little did I know at the time, that was also when I registered the domain for it, that the word Dramble, if you looked it up and found it on certain urban dictionary-type sites, it had a very different definition. But I think now if you search on Google, my Pi Dramble site might have a higher ranking for it.
IVAN: I just did that. [laughing] I typed in "dramble" because I want to know what it means on urban dictionary, and the first hit was pidramble.com.
JEFF: Drupal SEO is so good after all these years.
IVAN: Anybody listening out there needs Drupal SEO, Jeff is the guy. [laughing]
JEFF: Yeah. So, that was the original reasoning, but somebody asked me more about it one time and I was like, I’ll look up a little more information, and I found out that the generic term, so, Raspberry is our bramble, that’s their type of bush, but all the different types of berries that are like it like raspberries, blackberries, blueberries, all those things, actually grow in clusters, they call them aggregates of drupelets.
IVAN: What!
JEFF: This is like providence, drupelets? So, it’s a Dramble of drupelets. It’s spelled d-r-u-p-e-l-e-t-s, but that is the official, biological term, I guess.
IVAN: You’re making that up. That can’t be possible. Drupelet. That’s basically Drupal.
JEFF: [laughing] Go to Wikipedia and search for "drupe." It has a whole article about drupelets.
IVAN: Oh my gosh, look at that. A stone fruit.
JEFF: Yeah. I don’t think I would want to eat a stone fruit, that doesn’t sound very tasty.
IVAN: No. That does not sound tasty. How interesting. So, what a great name for this little cluster of Drupal technology. Right?
JEFF: Yeah.
IVAN: So, you talked about creating the cluster. Can we go back to the first version of the Raspberry Pi cluster, the Dramble that you created? What were the versions? Was it Raspberry Pi 2 or 2+ or something like that? I’d like to hear about the stack. I think the stack of technology has changed over the years.
JEFF: Yeah. The first version was a very traditional cluster of web server technology. It was based on LAMP, and actually the first ever version which I never actually put up on GitHub, the first version that I had locally was running Linux, Apache, MySQL and PHP. The way I had it set up was, the top server was running Apache, and Apache was set up to redirect web requests to the two PHP servers, server number two and three. Then I had two database servers, there was the database primary or master server, and the secondary server that was set up to replicate the master in case the master goes down.
There’s still a lot of people that run those kind of setups where there is a server for each one of those purposes. You might be in Amazon or somewhere else, or if you’re using a cloud hosting provider for Drupal, some of them still set up their service that way where you have dedicated servers for a database and all that. The cool thing about it is I used Ansible to do everything, and I actually used almost the same kind of Ansible playbook that I use for Drupal VM to manage all those individual servers and get them set up. I timed it one time. In about 35 minutes it took from the time that you have the hardware plugged in to the time it was serving up a Drupal website with a fully redundant, highly available, relatively high performance, because we’re talking about Raspberry Pis, a cluster of computers.
So, that was the first version way, way back. Then soon after that I switched to using NGINX, just because it was a little easier to configure NGINX to be that load balance earlier, and also NGINX has built-in caching. So, instead of having a Varnish server I could use NGINX for that purpose, because it had the basic caching I needed. Varnish is way better for some things, but I switched to NGINX mostly because you just change a couple settings, and it caches images more easily.
IVAN: How many total versions have there been now if we fast forward to the Raspberry Pi for the cluster that you have now? How many versions have there been, and what are the biggest lessons you’ve learned in the evolution of this cluster?
JEFF: There've been four major versions. It hasn’t been like each Pi miles a new version, it’s more like each technological shift has been a new version. The first version had that first two database servers setup and all that. The second version I actually ditched the replica MySQL server for two reasons. One is, it’s not that easy to maintain a replica setup like that with MySQL, especially for people who are newer to the whole LAMP stack and management and things. And even with automation it can be a little difficult.
So, I ditched that just because that was never a problem in the real world for me, and the most important thing was just to have a backup of the primary server, so I have a nightly backup that it does. The second thing was just streamlining things, making it a little simpler, and the first versions I had had a lot of configurability and things, but really you want the server to be easy to set up and easy to manage, And the more complexity you add to a project like this, especially if it’s a hobby project and I’m not earning tons of income from it—in fact, I’ve never earned a dime from it.
IVAN: Not yet.
JEFF: I have gotten to go to a couple Drupal camps and things, and it’s been great to learn from it, but making it simpler is better. So I think version 2 is just simplifying the architecture, making it so that you could use four or five Raspberry Pis instead of requiring 6, things like that. Version 3, might’ve been the first Kubernetes version. I could be wrong there. That was about a year ago that I came out with 3, and I started working on getting everything into Kubernetes which makes it easier to scale up or scale down if you want to.
You could have 100 Raspberry Pis or two Raspberry Pis or four, depending on how redundant you want everything, how scalable, because Kubernetes lets you say, instead of this server is MySQL and this server is PHP, you could say, I want to have three PHP things and two of MySQL things and one load balancer thing and it just puts them on servers. For MySQL I don’t want the two MySQLs to be on the same server because that would be bad for performance.
So, Kubernetes sorts all that out for you and you just say, Hey, Kubernetes, I have these five things I want to run, and I have these four servers, go do your thing. And Kubernetes manages that for you. That’s the cool thing about Kubernetes. It’s more complicated than that in the real world, but it’s getting easier as time goes on. Kubernetes is one technology that started out crazy complicated and has gotten a lot easier over the years as they refine it, as they make things more robust and a little easier to get started with, and as people understand it more.
IVAN: Is everything in the cluster now orchestrated with Ansible, and is everything virtualized as containers inside Kubernetes, or do you still have an ingress that’s a NGINX or on a separate Raspberry Pi, for example?
JEFF: Almost everything. Right now, the ingress is actually running inside of Kubernetes and the way that I have it set up is you just point your DNS at one of the Raspberry Pis. That’s obviously not wonderful. If that one Raspberry Pi goes down, you've got to point your DNS at another Raspberry Pi. So, that’s one slight weakness of the architecture, but as an alternative you could have another Raspberry Pi being in the ingress and be a load balancer, but if that Raspberry Pi goes down you have a problem. That’s one case where having a cloud hosting provider, like Amazon or Google, is really nice because you can have their cloud load balancers. They take care of all the really complicated stuff in terms of when you get a request for your website. What happens if one of the servers that is routing those requests goes down?
All those cloud systems, they kind of self-heal automatically with DNS and with all their different things. When you’re running a website in your basement, you have one IP address and it’s not very reliable usually, especially if you’re using most ISPs in the U.S.A., and you don’t get any more IP addresses. And if that route goes bad and one server goes down, you can have a lot of issues. So, that’s one area. One of the main reasons why I would say if you have a website that sells things or you generate a lot of revenue off your website, you probably don’t want to run it inside your house like the Pi Dramble website, because it does have 10-20 minutes every few days of downtime, when my ISP is like, yeah, You guys aren’t getting internet for right now. Too bad. [laughing] That happens a lot.
IVAN: What do you think the biggest lesson is that you’ve learned going from your first version to the latest one that you have?
JEFF: I think the biggest thing is that managing Kubernetes is difficult, it’s still not easy, but managing a cluster of application stuff with Kubernetes is a lot easier than it was when it was just individual servers. Because you used to have to manage each application on each server and it would take a lot of time to get things set up and to tweak things and make sure all the backups were good. When you standardize in Kubernetes and have everything run in a container, it is more complicated at the start when you’re learning it, because containers mean that you have to build the container and you have to have a place to store the container and all that. But once you have that set up, everything is automated, like out of the box. You don’t have to spend time worrying about, How do I get this to go here and how do I change the configuration? You just say, Deploy this version, and it’s there and it’s happy. And Kubernetes does it all for you.
That was the biggest lesson; sometimes the complexity does save a lot of hassle if you need it. Obviously, there are a lot of people listening to this probably thinking You might not need to be doing your own hosting stuff. I don’t technically need to, but I do like to because a lot of the work that I do does have to be more complex. So, doing these fun side projects for me teaches me things. Another cool lesson was, when you’re running on a server that’s slower, there’s a lot of things you learn to worry about that will save your skin when you run something on even faster servers when they’re under heavy load.
One of the biggest instances of that is, when you have really slow hard drive access, when it takes a long time to write files on the Raspberry Pi—which it does a lot—you start bumping into weird issues that you never see if you only ran your tests on your local computer that has an Intel Core 2 Duo or whatever, the last i9 chip or something. If you’re doing that with an SSD all the time, you’re not going to run into these weird issues. But when you’re on a cloud hosting provider—which most companies do use those—most sites are on cloud hosting now, disk access can sometimes go crazy, and the error message you see and the behaviors you see can get confusing, because you never really notice that and you can’t replicate it locally. But I’ve been able to replicate some of those weird things on a Raspberry Pi, just because it’s so slow, the disk access.
I actually found a bug in Twig, with the way that Twig renders files. If you have multiple computers writing to an NFS storage device and that device is writing slowly, I found in Drupal 8.0, the first version, right around the beta timeframe was when I found the issue and the issue was still there in 8.0, but they fixed it. It was a race condition, when you have multiple servers writing to slow shared storage, and this was good because other people—it’s not just Raspberry Pis—if you’re using cloud storage, a lot of cloud storage providers have throttling, and you can run into this throttling sometimes, so instead of the site going down, it might just be slow. It’s not all for naught and for fun. There was that issue I found that was an actual bug in Drupal that we fixed.
IVAN: It’s amazing the kinds of things you learn when you try to scale down hardware or scale up requests and scale up bandwidth. You just don’t see things in general unless you do something. So, that’s always interesting to me as well. One of the things that I just realized as you were talking was that you actually had to come up with a way to install Kubernetes on Raspbian, and that means that you had to basically either compile Kubernetes for ARM or they’re already precompiled packages, and that never actually dawned on me. So, how hard was it to put Kubernetes on ARM infrastructure?
JEFF: So, it was perfect timing for me to be getting into Kubernetes on Pi around 2017. There was, I think it was a teenager, I don’t remember the guy's name, but there was somebody who just loved Raspberry Pis and loved Kubernetes, and he spent all this time making sure that the Kubernetes build system built and tested ARM architecture stuff for all the Kubernetes releases. And it just got into Kubernetes 1.8 or 1.9, or something like that, right before I got started. So, when I was looking into it, it’s like, Oh, you just install Kubernetes, just like everything else.
However, Kubernetes is one part of the equation when you’re talking about running a cluster that runs Kubernetes, because you also need something to run the containers, and that’s usually Docker, and Docker for ARM is a little bit different. You have to install it differently and there are some things you manage differently. The versions are sometimes older and there’s weird specific version things that cause trouble. So, I ran into some issues there, and I actually created a separate Ansible role for Docker ARM from my normal Docker one, because the ARM one is more complicated and convoluted. But it’s running pretty well now.
Then another thing that you find out if you’re using Raspberry Pis. Since the chip is an ARM processor, it’s not this typical processor in most of our laptops and servers, which is AMD64 or X86. Since it’s a different architecture, everything has to be compiled for that architecture and container images often have to have a specific version of the image built for it.
So, a lot of times you’ll be like, Oh, I want to deploy this to the Raspberry Pi, you needed to play it in the Raspberry Pi, and Kubernetes is like, I’m not going to deploy that. You’re like, Why aren’t you going to do that? So, I was like, Yeah, it’s not for ARM. Then you have to figure out, is it important enough for me to rebuild this image for ARM? So, I actually build and maintain a set of PHP and Apache base images for the Raspberry Pi, and they’re on my Docker Hub account under geerlingguy. Those are the ones that I use on the Raspberry Pi and build on top of. And a lot of images now are supporting ARM because you can get ARM servers from Amazon and from other companies.
IVAN: But it’s also a little more complex than just ARM though, if I’m not mistaken. There are different versions of ARM. Some are 32-bit, and some are 64-bit, and you can say ARM but it’s even harder because you have to know what architecture you’re building for to make sure that when you build it, it actually runs on that ARM processor. So, my guess is, you actually had to update your images when you went from two to three to four.
JEFF: Yeah, so, Raspberry Pi has its own OS that’s kind of the official one, called Raspbian, and it’s based on Debian, but it’s a 32-bit OS. And there’s some initiatives to upgrade to 64 bit, especially now that the Raspberry Pi 4 has more RAM and can use more of that 64-bit power. But, a lot of the things are either ARM V6, ARM V7, or ARM 64, and you have to always figure out which one is which. And I think, the ones that I’m building right now are ARM V7 because that’s what works on Raspbian.
But if you run a different OS on the Pis which you can, there’s a Ubuntu and actual Debian, and Fedora, and some other versions that you can run on Raspberry Pis, you might have to get a different version of a container which may not exist again. It’s not for the faint of heart to get things running on the Pi all the time. Every year it gets easier, because more and more people support Raspberry Pi stuff.
It’s an interesting thing since I do write a lot about Pis, I found out about a lot of companies who you wouldn’t even think about it, who do remote control systems, they do signage, they do radio frequency stuff, logging systems, all kinds of things that are industrial and commercial applications that use Raspberry Pis for everything. So, there’s a lot more support for it nowadays than there was five or 10 years ago when it was only a bunch of people playing around with stuff in their houses. Nowadays the Raspberry Pi is a more serious computing platform.
IVAN: What a great success story, isn’t it?
JEFF: Yeah.
IVAN: That’s cool. So, you mentioned that you had seen Tess’s talk at Flyover Camp, and so you’re probably familiar with Flight Deck and the hosting that we’ve been talking about, hosting live sites using Drupal on Kubernetes. She’s done some great work in developing that for us, and as you know we’ve open sourced all of it. But one of the things it requires is a S3 block storage like the ones from Spaces from Digital Ocean, for example. I wanted to talk to you about two ideas. One, how hard would it be to get Flight Deck-powered Kubernetes hosting onto your cluster? And two, since it requires S3 block storage, have you thought about implementing block storage on your Dramble?
JEFF: That’s an interesting question, because probably of all the issues there are in hosting Drupal on Kubernetes, or any kind of, what we call now cloud-native hosting environments, one of the main issues is always, how do you store files for Drupal? It’s complicated, because in WordPress or a lot of other systems, usually when you ask that question, you’re just talking about media uploads like, I’m creating a blog post and I upload a picture to put in the blog post, where do I put the picture? That’s part of Drupal’s problem.
That one is very perfectly solved by block storage and works great with that, and you can integrate with CDNs and things, there’s a lot of different solutions for that. What I do on the Raspberry Pi right now is I use NFS, which is not block storage, but it’s just a networked file system that is shared among all the servers, and Kubernetes mounts it into Drupal so Drupal can write to it. But when you’re talking about Drupal, you’re also talking about things like the Twig cache files. Every time that you load up a Drupal 8 website it has to write a bunch of Twig cache files that are like compiled PHP, and that currently writes them by default into your public file system. That’s a lot of files that are read, at least once per server. And so having a slow storage solution can cause problems with that and having slow writes can cause problems with that.
Then you also are talking about CSS compilation and JavaScript compilation. So, there are more complicated things with Drupal. And so, I’ve seen some people do S3 block storage, and as you say as long as you have a provider that is compatible with the Amazon S3 kind of API for writing files, it works so you can do it on Digital Ocean. There are open source block storage software that you can do, and if I were to put it on a Raspberry Pi, I’d probably use one of those open source packages and install it on the Pi. Or, if it’s an internet site, available through the internet, you can even use Digital Ocean Spaces or Amazon S3 even though you’re hosting it locally. So, those are options for it. I think you guys might even use Flysystem or something like that to make Drupal integrate with it?
There’s some PHP level stuff that you can do to write files in different places, but that is probably the number one thing that people ask about and debate about, and I’ve gone a hundred which ways. I think I built it five or ten different ways in the real-world clusters that I built that aren’t on Raspberry Pis. So, that is the million-dollar question right now. There’s even a couple issues on Drupal.org exploring how can we make it easier to use a file system in Drupal that’s shared but doesn’t have to make it so complicated.
IVAN: You’re actually right about Flysystem. That’s exactly how Tess implemented the file storage in the solution we have for hosting Drupal on Kubernetes with Flight Deck, and it’s possible to do if you set up the infrastructure so that you have enough caching, those first-time hits don’t seem as bad. And we run our own live sites and other clients sites live, in production with Flight Deck and with block storage. So, we use Flysystem in Drupal 8, and then we’ve also got Drupal 7 sites that we’re running on the Kubernetes infrastructure. But I don’t recall the name of the module that we’re using. It’s not Flysystem because it came along only in Drupal 8.
JEFF: I know I’ve used in the past s3fs as well.
IVAN: Oh, yeah, that’s it. I think that’s what it is.
JEFF: It’s one of those things where for most sites it’ll work perfectly fine, but there’s always going to be some site that’s doing some weird thing and you’re like, you know.
IVAN: There’s nothing like beta storage.
JEFF: There’s no good answer. When you have control over the sites that’s the best. I have control over the Pi Dramble site. So, if you look at it, you’ll see I intentionally kept that thing super simple, because even under load the Raspberry Pis can handle that. If I had commenting in accounts, and all kinds of crazy things going on, and real-time chat and who knows what else that I’ve seen on peoples’ sites, it’s going to fall over.
IVAN: [laughing] Yeah. Well, how taxed do those Raspberry Pi 4s end up being? I’ve seen that you’re writing about the Ice Tower, the need for active cooling on the processors now. How is the cluster handling traffic? And tell me a little about Ice Tower, it sounds so cool?
JEFF: It’s funny to ask about that. Right now, I’m looking at my Mac in front of me, just doing this podcast, the CPU’s at 80% and it’s dying, and the fans are on. And it’s funny because the Pi is sitting over there, the fan's not even on because it’s not hot enough. The Raspberry Pi generally speaking, if you’re not mining bitcoin or something on it, it’s not going to use a whole lot of CPU, and it’s not going to need a whole lot of cooling power. However, when I do these projects I always try to go the extra mile and then an extra ten miles after that, and I always tax them until I can break them, basically. So, I do a lot of performance benchmarking on them, and I found that the Raspberry Pi 3B+ and the Raspberry Pi 4 both, they are not quite as good at keeping their cool.
The Raspberry Pi 4 was even worse than the Raspberry Pi 3, and it was found pretty early on that the reason for that was they had a little flaw in the way that they implemented USB. They added a new USB controller that makes it USB 3, which is awesome. It’s way faster than the old Raspberry Pis with USB 2. You can get an external hard drive and things are a lot faster connection, has gigabit ethernet, that’s awesome, because the old Pis were also constrained to 200 megabits or 100 megabits, way slower. So, it’s really awesome, but the first version of the Pi for ROM or Bootloader, it had a flaw in it that would keep the USB chip in high power mode all the time. So the Raspberry Pi 4, even if you just turned it on and left it sitting there, it would just get hotter and hotter.
If you left it out in the open—which is a bad idea for dust and things like that—if you drop something on it and you short out a circuit or something, so everybody should have a Pi in a case if you’re going to have it running for any period of time. If you have it in a case, it just sits there and turns into a little oven and it cooks itself, and it starts throttling the CPU because it gets so hot. I did an early article about this months ago right after it was released on how it gets super hot, and it’s really bad because if you put it in a case it’ll just cook itself, and that’s without much load. If you put load on it, it just starts throttling right away.
So, I said you basically have to have a fan on the thing. I still recommend having a fan on the new Pi. But they released a firmware update a few week ago that sets the USB chip into the correct mode. It very slightly reduced the performance of USB, but not really in any huge meaningful way for most peoples' usage. But it makes the Pi 8° Celsius cooler, which is hugely significant. That’s a huge difference. I posted one blog post a week or so ago on this, and I’m going to be doing another one because I tested with a couple more cases how that affects the Pi's cooling. Basically, the general thing that I’d say now is, you still need to have some method of cooling the Pis.
So, for my cluster, I have the Raspberry Pi HAT, the power over ethernet HAT that has a fan built into it, and all that fan does is blows a little bit of air onto the processor. That’s enough to keep it cool. You just need some convection over that processor. If you put it in a case, there’s no convection because the case is just going to hold that air inside of it and heat it up. If you have it open air or if you have a fan blowing on it, it will make some convection, it’ll take the air and move it over the processor and take that heat away.
So that’s all that’s really needed. But, a company called Seeed Studio, S2 Pi, they sent me this colossus heat sink, it’s not colossus when you compare it to—I don’t know if anyone ever built a Pentium 2 rig back in the day—you have those giant coolers that are the size of a computer nowadays, and a giant fan or two fans on either side of it, blowing air through it, or If you ever saw the Power Mac G5 quad core. I saw one of those one time, and the cooling system is bigger than everything else on the computer.
It’s kind of the equivalent of that for Raspberry Pi, because Raspberry Pi is a size of a credit card, and this Ice Tower is the size of the whole board, but super tall as well. If you use it, you can’t put it in a case, but it does do an unbelievably good job at cooling Raspberry Pi. It cools the Raspberry Pi so cool that it’s almost like you don’t even have it running when you have the fan on the thing. It does what it says it does, it keeps it ice cold, but at a bit of an expense you can’t use Pi HATs with it, if you use that cooler, you can’t fit it in most conventional cases.
IVAN: You can’t put it in a cluster.
JEFF: Yeah, if you were doing web hosting it wouldn’t be a bad idea to build a custom little case for these things, because that would keep the CPU so cold that it would operate a little better. It would make the Pi operate a little better.
IVAN: Does it have contact with the CPU?
JEFF: Yeah. It comes with a little thermal pad. It’s like a little piece of rubber but it’s thermal rubber and yet you just wedge it between the CPU and the cooler. It has one copper heat pipe that goes up and down, and that copper attaches to cooling fins. The cooling fins distribute the heat. And even if you run it without the fan, it’s going to keep it way cooler than just blowing the fan over a processor. It does a really good job at cooling and the fan makes it do even a better job. The difference was without the Ice Tower it was 60° Celsius, with the Ice Tower it was 30° Celsius. It was a huge, huge difference.
IVAN: Wow. That’s a giant difference. Well, it’s unfortunate you can’t use it in your whole cluster, because that little compact package you have, I would assume the Ice Tower just doesn’t allow for it.
JEFF: Yeah, luckily with the Tower since I switched to the Pi HAT, the power ethernet HAT for it, since it has those built-in fans that’s just enough to keep it from overheating. With Kubernetes running, Kubernetes does take up some CPU constantly, so it does get hotter over time, but it stays under the throttling threshold.
Another option, I actually bought one because I kept having people tell me, “You should try the Flirc case.” I’m like, Ok, whatever. I’ll buy one. So, I bought one this week and I tested it and it doesn’t do as good a job as the Ice Tower. It’s basically a giant aluminum heat sink case for the Pi. The case actually attaches to the CPU, and it does do a pretty good job cooling it. It keeps it down at 40-45° Celsius, which is still way better than just having an open air, having a little fan blowing on it. But, having a cluster of these, they’d probably still get super hot. [laughing] You got to have some sort of fan blowing the heat out otherwise it’s just going to turn into a little oven.
IVAN: Now, in addition to all the hobbies you have, side projects, and so on, you are also an author, and I would love to hear about your latest book and the book that you’ve written on Kubernetes. What are you working on right now?
JEFF: I love writing. I don’t know how many million words I’ve written in my life, on my blog and on other blogs and things, but I love writing. In 2013 or so, I think that’s when I started, I’ve always wanted to write a book my whole life, I want to write a book sometime. I think part of that was jealousy because my brother, when he was a kid, wrote a book and his book, you know, the 15 minutes of fame, his book caught fire and was a local very popular book. He sold maybe 15,000 copies or something. It was pretty cool being the little brother to the brother who wrote that cool book.
But I was also a little jealous, like, I want to do that too. But I also just love writing. I’ve always loved English and literature growing up, and I love reading and I love writing. So I put that together with the fact that Ansible didn’t have a book in 2014. I started in 2013 but in 2014 I’m like, There’s still no book for Ansible and it’s really popular.
So I decided to start writing it with a goal that I would write 100 pages and sell 200 copies. And it was funny because I started writing it on a platform called Leanpub where you can publish it while you’re writing it and sell it while you’re writing it. And by the time I had written about 40 pages I already had sold 200 copies [laughing]. And then fast forward these many years later, it’s 2019, so it’s been in print for five years now, and I now sell it on Amazon and other places and it’s called Ansible for DevOps. And that book has sold over 22,000 copies and it’s now 480 pages, including a chapter on Kubernetes and a chapter on Docker and a couple examples that do Drupal.
One of them was inspired by the Raspberry Pi Dramble cluster. So, that was my first book effort, and it went incredibly well, and I was floored. There’s no word to describe, when you’re like, I want to do this thing my whole life, and this is my goal. And then your goal is surpassed by 50 times over, and you get to meet awesome people because of it. It’s just so many cool things happened because of that book. It also helped my family.
We've wanted to remodel our kitchen and after writing the book and making some profit off of it, I was able to remodel the kitchen four years earlier than we thought we might be able to. That’s a huge change for our life, because our old kitchen was kind of hard with three kids and the way that we live our life and stuff at home, especially since I work remote. And I’m at home all the time. We had an old cramped little kitchen, and we were able to get it better.
So, the book was just awesome. I don’t expect to have the same level of success, but who knows. You never know where it’s going to lead. But I’m working on another book. I actually just finished the first chapter a few nights ago, and I have a structure for the rest of it, and I’m working on examples and chapters.
The next book is going to be called Ansible for Kubernetes, and maybe if Ansible is around in five years and there’s another game-changing cloud infrastructure thing, it’ll be Ansible for that and I’ll have a whole series out. But I’m working on that book. I haven’t published it yet. I probably will pretty soon. Even though it’s not finished, I’ll publish in progress updates on Leanpub, but both of those books, if you go to ansiblefordevops.com or ansibleforkubernetes.com, those are the book sites. I love writing them. And one of the best things about writing them in progress is for both books I’ve had a lot of interaction with the people who read it, and they can help me. If they’re interested in something, I can write about that. Or if they are like, "Your example didn’t work on my computer," I can improve it before I actually make a published printed version that people will buy.
IVAN: I very much appreciate knowing about Ansible for Kubernetes. I didn’t know that was what you were working on. I think we actually bought Ansible for DevOps if I’m not mistaken.
JEFF: Thank you.
IVAN: When Tess first started working for us, we wanted to make a big change to how we were doing things at TEN7 and the custom scripts we had, and we wanted more automation. And Tess was very interested in Ansible being the thing, and we had to learn it. So, you’ve certainly provided a great deal of information and helped TEN7 in that manner as well. We wish you all the best for your new book Ansible for Kubernetes, and we’ll be linking to it from the show notes on the podcast episode as well.
JEFF: That’s great. Thank you.
IVAN: You’re very welcome. Well, I think that’s a wrap. We should say goodbye, and thank you so much for being on the show, it’s been a great pleasure talking to you. I hope to talk to you again soon. And I’m going to go to Micro Center right now on a number of different visits, because you can’t buy more than one at a time and buy some more Raspberry Pis, so I have more than you do. [laughing] I don’t know what I’m going to do with them thought. [laughing]
JEFF: You can always get a bunch of Raspberry Pi Zeros and stick them in peoples stocking stuffers, that kind of thing.
IVAN: Oh, that’s a good idea. Stocking stuffer. Five-dollar Raspberry Pi Zero. Awesome. Thank you so much for spending your time with me today. It’s been great.
Author and software developer Jeff Geerling was my guest today, and you can find him across the web using the handle @geerlingguy. You can also find him online at jeffgeerling.com and of course the Raspberry Pi Dramble is at pidramble.com. If you’re interested in either of those books you should go to ansiblefordevops.com or ansibleforkubernetes.com.
You’ve been listening to the TEN7 podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thank you for listening.