Scientist and Group Lead, Fraunhofer Institute for Communication, Information Processing and Ergonomics FKIE
They tested the latest firmware from 127 routers using the Firmware Analysis and Comparison tool.
Ninety percent of routers use Linux-based firmware, many of these with old, insecure kernels.
Real-time operating systems such as ThreadX and eCos are not used as much as Linux; they are harder to exploit.
The best things you can do as a user to have a secure home router are 1) choose a router company from the report that tends to update its firmware more often, 2) change passwords wherever you can on the router firmware/web UI, and 3) continually look for firmware updates for your router once you get it home, as many don’t automatically update.
IVAN STEGIC: Hey everyone! You’re listening to the TEN7 Podcast, where we get together every fortnight, and sometimes more often, to talk about technology, business, and the humans in it. I’m your host Ivan Stegic. My guest today is Johannes vom Dorp, a scientist and group lead at the Fraunhofer Institute for Communication, Information Processing and Ergonomics FKIE in Germany. He co-led the group at the Institute that published the 2020 Home Router Security Report in June of this year, describing “significant security flaws in 127 different home routers from seven manufacturers that are regularly used across the United States and the world.”
In fact, it’s highly probable that the WiFi you are using right now is connected to the internet using one of these routers. We’ll talk about the study, how it was conducted, the report and what the results are, and most importantly what you can do about it.
Joining me from Gottesberg which is just outside of Bonn, Germany is Johannes. Welcome, it’s wonderful to have you on the show, and hopefully I didn’t butcher the names too badly.
JOHANNES VOM DORP: [laughing] Happy to be on the show. Yeah, no I think you did a great job. I’m known to have a not-so-nice name for English native speakers to pronounce, so really, you’re fine on that front.
IVAN: Oh good, I’m glad. It strikes me that the name is almost Dutch. I don’t think I’ve seen German names like yours before, last names, and actually first names too.
JOHANNES: Yeah, it’s a regular question. I really don’t know what the origin of that is. You have to basically switch out one of the last characters into an “f” instead of a “p,”, and then it’s basically from the village in German. So maybe it was just some area in Germany where that was a dialect, and that is where it originated from. Maybe it's Dutch, but we don’t know.
IVAN: We don’t know. How interesting. Well, let’s start with the Fraunhofer Institute for Communication, Information Processing and Ergonomics FKIE, which is a long name [laughing]. What can you tell me about this Institute? Where is it located? What do you do there?
JOHANNES: The Institute is located in another place near Bonn, it’s called Wachtberg.
JOHANNES: It was a former research institute that originated in the fifties, I think. Building space radar, one of these large golf ball-looking things. Through the years there were three institutes that grew around this radar, and one of them is ours. It is now, I think, a 400-people institute mostly researchers, that is doing a lot of research. Some military, some governmental-issued research contracts.
But through the years really after joining the Fraunhofer group, we now do a lot of industry research as well. In the past, we were more like this government research institute, and now we’re really all through industry, government contracts, whatever, doing stuff with everyone.
IVAN: Is it named after the Fraunhofer that’s famous for the Fraunhofer diffraction?
IVAN: It is? Okay.
JOHANNES: I just want to say Fraunhofer’s like this 35,000-people group in Germany with institutes all over the country, which we are just one of a couple hundred that we have, I think. So it’s really a distributive thing. So we don’t have much to do with the optics guy [laughing], but, more a local thing.
IVAN: And you’re a scientist in the Cyber Analysis and Defense Department that’s a part of the institute.
JOHANNES: That’s correct.
IVAN: What’s your role there?
JOHANNES: A week now, I’m group lead of our research group which is called Applied Systems Analysis. Before that I was a researcher in that exact same research group, and before that even a student assistant. So, I started in 2014, and slowly worked my way up to now being the group lead.
IVAN: Congratulations. You said that’s a week old. You’re back from a paternity leave and already have a promotion.
JOHANNES: Yeah, exactly. So, I think five days after my paternity leave started, my former group lead left us. Actually that’s relevant because it was Peter, who was the co-author of the report that we’re talking about today. And basically out of necessity, I got promoted, and I’m now the one who has to do all the administrative stuff to keep the workforce happy and keep everyone doing his job.
IVAN: And hopefully still get a chance to be in the weeds a little bit, and maybe doing some network analysis as part of that administration.
JOHANNES: Yeah, sure. That was the plan at least. So, I’m hopeful that this will be the way it will work out. That’s nothing that I want to do, only that administrative stuff.
IVAN: Of course. Let’s talk a little bit about the report and the paper that is the central part of our discussion today. I’m going to quote something here that came out of the report. You tested 127 routers, and the quote goes as follows from the report, “Nearly all were found to have security flaws, some of them very severe. The problems range from missing security updates to easily decrypted, hard-coded passwords and known vulnerabilities that should have been patched long ago.” That’s scary and bad. Were you surprised by these results?
JOHANNES: Not really. But that’s because we’re doing research into that area for quite some time now, and I’ve had a look at basically these results in some form or fashion, not for that specific devices that we had in the report, but for others in the same class of devices I guess,
And the picture that painted itself was clear beforehand. That was one of the reasons we actually did the report, because we saw that and we were basically as shocked as you are after reading the report the first time. And we wanted people to know basically.
IVAN: Yeah. So, tell me about how the idea got started that you would test these routers, because you explained how you’ve had some experience before. Where did the light go off where you thought, Oh my goodness, we have to pick a large data set of routers and test them all?
JOHANNES: I would say it was developed over time, it wasn’t like a point in time where we said, “Now we have to do it.” We did some observations on that. We, over the years developed the primary analysis tool that we used for the analyses, Firmware Analysis and Comparison Tool or short, FACT.
IVAN: Nice, very nice FACT. I like it. [laughing]
JOHANNES: Quick aside, I think we chose FACT before we chose the names that built the acronym, because it was formerly known as FAF, just Firmware Analysis Framework, and then we found out that there was another GitHub project with that name, so we had to change. And then we thought about which acronyms would be cool. So, it was FACT at the end.
We developed over the years, and basically is another meta tool that generates all the nice statistics that we could then use in the report. And the idea for long has been to really evaluate the landscape of device security, and over time we thought how could we make this most poignant, most precise, and how could we shape these reports, so that someone would be interested in reading them, and they would be reliable or at least as reliable as possible.
So, in the end we chose to only go for home routers, or router firmware, instead of say also looking at printers which we did a lot, or access points, or other switches, other network hardware, or like say IoT devices such as smart plugs, smart lighting.
So we chose that because then we realized it would be more comparable between each other, and we could, I think, paint a clearer picture regarding that specific class of device. And if I think the most ubiquitous regarding the network hardware that we as humans use in the everyday life.
IVAN: So I have two actual follow-ups to this. So, you mentioned the tool and you mentioned the name was already taken on GitHub. So, does that mean FACT, the tool, the one that you’ve developed is open source and available for anyone to use?
JOHANNES: Yes, you should just not make the mistake of just Googling “FACT” because then it would be on page 200 in Google.
IVAN: What’s the best way to find it?
JOHANNES: You can either search for “FACT core,” like the core of the tool, which is the GitHub name, or you can just google “FACT” and “firmware” and then you will probably find it.
IVAN: FACT and firmware, and of course, we’ll link to this in the show notes on the web, so there are direct notes. That’s wonderful that it’s open source and that it’s out there. I’m definitely going to take a peek at it, maybe clone the repo and see what I can figure out from there as well. But my follow-up question is, the report had the year in the report, so 2020 was in there. Does that mean that you’re planning on rerunning this report again next year, or on some regular basis?
JOHANNES: Yeah, more than likely. So, the plan would be at this point to do it yearly. I think that’s a large and small enough time frame to where it’s not as much overhead to replicate the work, and also both get improvement in the method that we applied, and give the vendors time to periodically update the firmware. Because already when you look at the statistics for release date, and there were a lot of devices that did not get a firmware update in the last year, so then we wouldn’t see any differences in the outcome.
IVAN: It sounds like you’re more optimistic that manufacturers are actually going to update the firmware than I might be, given their past track record.
JOHANNES: I mean sure, we have observed some vendors which will put out software or firmware updates quite regularly, like four times a year say. That is not unseen, so that is depending on which manufacturer you’re purchasing from, that will happen. How much of an impact that makes on the security or specifically the things that we found in our analysis, that I don’t know, because sometimes you only fix usability bugs, sometimes you present new features to your users, and it’s not always that you put out a patch just to close some security issues.
IVAN: I’d like to talk a little bit about the scientific method you used to select the routers that you decided would be a part of the study. Can you talk a little bit about how you selected the pool, whether the routers you selected might change in the next report? And then I’d love to hear about the logistics of getting these things in-house. I assume you actually had the hardware in-house and were testing it there.
JOHANNES: Actually, we did not. At least we didn’t in the most cases. We basically we only analyzed the firmware running on these devices. Since the statistics we generated were all only dependent on the firmware, there was not a direct need for the hardware in the office. And then the thing that you would be interested in doing is to verify as much as you could about the statistical security vulnerabilities that we found.
There I have to say that we basically wouldn’t have time for that, to do that on 127 devices. So, I can go a bit into what we did with some of the devices that we have in-house later on. And I can also start telling you a bit more about how we chose the devices.
IVAN: Yeah, how did you choose the devices? And then we can talk about what hardware you had in-house.
JOHANNES: The methodology was quite naïve at the start. So how did we choose? First, we chose vendors. We looked at some stuff like sales and some of the online sales platforms, which we had access to seeing which devices sold the most, had the most comments from people who bought them, and then also which vendors were seen the most on security news websites regarding issues in the firmware, because obviously that is a point we wanted to look at. And it went from there.
What we really wanted to do was to get some market share documentation, but actually when you take a closer look into that you go into paywalls rather quickly, and these paywalls were high enough say, starting at values that we couldn’t really do it without having some funding for that.
It is something that we want to make sure that we get before we do the 2021 one. I think now having the report from this year, the impact might be motivational for someone to fund that for us. But for this one, which we did basically without any funding, it was impossible basically for us to get a market share documentation.
Then as we had the vendors, which we assumed at the time, or we assume are quite important in the European space, and I think also in the U.S. for the most part, I think excluding one German manufacturer, which is only on the European market, I think we painted a nice picture there. Then we took all devices from all these vendors we selected, which were at the time sold or advertised on their online platforms.
IVAN: And then once you’ve selected all the routers that exist, how do you find them on the web if you don’t have the actual hardware in-house?
JOHANNES: Yeah, like I said, the analysis goes on the firmware of the devices, so the software that is running on them, the operating system of the software.
IVAN: So did you have to download every firmware for each device?
IVAN: I see. How do you get that running, in a virtual machine? Or how does that part of the analysis work?
JOHANNES: Actually, most of the analysis that we do is a static analysis. So, you can, for example, looking at a router when you get to the download portal, you would see a .zip file or the raw archive or whatever. And then in there will be some kind of proprietary archive format, oftentimes called .bin or .image or whatever, which is like a file format that the vendor will probably create or have created in relation to their update procedure.
And what we have to do now is extract the real firmware from that, because of some compressed archive format mostly, and that can be done during some open source tooling, some other stuff that we developed in our house. And then when you get this basically a kernel, so, for Linux firmware, a kernel and a file system. And there you have everything you need to run the firmware, if you have the correct hardware platform for all that.
Because most of the times you would not have binary code that is for your usual desktop PC or notebook, which would run on x86, which on ARM, which runs on your Raspberry Pi, for example or on MIPS processors, which is like another one of these, usually embedded hardware used processors. Then we run different kinds of static analysis on that, and some dynamic as well, for example we tried to emulate the single binaries and get more information from that.
IVAN: You had mentioned earlier that you actually had some hardware that you tested and that you analyzed. And so of the hardware, how many of the 127 routers did you actually have in-house, and did you test compared to how much of it was static analysis of the firmware that you downloaded?
JOHANNES: Actually, everything that is in the report is the static lead or mostly static analysis from using the FACT tool. What we did with the hardware basically was either get the firmware because we couldn’t download it, or basically reverse the update procedure, so we could extract the firmware components.
For example, because one of the manufacturers uses an encryption scheme to protect their firmware, and we were able to reverse the update mechanism, find that the crypto key is hard-coded into the binary. And then we could easily decrypt that router and also other routers from the same vendor to get inside it and have the ability to observe them.
IVAN: Tell us about that manufacturer and why they would consider their firmware proprietary. I would guess it’s not the Linux operating system, it’s something else. Do they have any benefit for the consumer that’s using that router, or is it just a way for them to try to protect their own intellectual property?
JOHANNES: It depends on your viewpoint, I guess. It could be different meanings regarding that. So, encrypting your firmware update when you send them through the network might be a good idea, if you don’t want anyone to have it. But since 95% from the router firmware is open source anyway, and every vendor has to publish the open source, source code for their firmware, I don’t really get the impact from that, because it makes only the life of the analyst or the security researcher a bit harder, and doesn’t bring you really much greater level of security, especially if you have a weak encryption mechanism that can be corrupt by just getting one device.
I would call it a bit of a security through a security scheme, but then again, encryption of firmware might be something that is important in other contexts. For example in critical infrastructure where no one but the guy who is operating the power plant has access to it, but in a world where we’re ready to download on the network, the internet, I don’t know why you would do that.
IVAN: When I read the report it was surprising to me that there were actually operating systems on home routers that weren’t based on Linux. Did you find they were more or less secure? Was there anything you could glean comparatively to Linux? And then after that question let’s talk about Linux as well.
JOHANNES: I don’t want to give you too deep an answer, because it would not be very well-advocated, because after saying that most of our work is done on Linux-based firmware. The other operating systems that we found, I think in the report we had two different ones, ThreadX and eCos, which are both real-time operating systems. They differ from say Linux or Windows or Mac, in that they are basically only one large binary file. So, one large binary file that looks a bit like a kernel, but has all the user space software in it as well. It is something that you see oftentimes in environments where timing and response times are critical of where certification is critical, because no one can certify a Linux kernel, there’s too many lines of code, so that you can certify it regarding something. But these smaller real-time operating systems have a better chance to do that.
The consensus in the industry is that basically it is harder on the staff to explore the real-time operating system, mainly because you don’t have the easy way in, so you have to have more expertise as an analyst to start analyzing that. But since Linux is something that a lot of people are looking at, and these real-time operating systems aren’t, you would think that once you have a bearing of what you’re looking at, then it would be easy, almost probably to exploit them.
IVAN: Yeah, and as far as the Linux kernels are concerned in the routers that you analyze, it was frustrating for me to realize that old kernels are being used. That newer kernels exist and have existed that patch all of the security vulnerabilities that have been announced, and that are known for these older kernels. Tell us a little bit more about that. Why do we have outdated kernels running on these routers?
JOHANNES: There’s some reasons, some of them are things they are okay and then there is some that you cannot really understand. So, some of them you have a development team, and they have a development platform for their router, and that development platform might just support this certain range of Linux kernels. And then upgrading to a newer kernel would make you use or create a new development platform, you develop procedures, and so on and so forth. So, it might be a bit of laziness on that part, and that is something that I think is the problem most of the time.
So, as we know, devices get cheaper and cheaper over time, so the vendors have a problem with putting enough resources into the development process, and that leads to outdated code, outdated software, or just when you set up a new device you just take the firmware from the older device and put some additional features on that instead of starting new from scratch.
Also, a reason which is a better reason for not having the 5.4 Linux kernel in there is driver support, because you don’t have off-the-shelf hardware necessarily in your embedded devices and your router. So you have to see that the kernel also supports the wireless module that you have built in your hardware, and that might necessitate that you go to a bit older kernel version. But there is not a real reason to have a 2.6 kernel running, and that is the most common kernel we found in our results.
IVAN: Wow, 2.6? That’s three major versions old. What year does that go back to?
JOHANNES: I think the 2.6.32 got the last security update more than 10 years ago maybe. But I have to check. I think there is some numbers regarding that in the report. I may be able to scroll to them quickly to get it. But like I said, there is some hardware limitations regarding the kernel version, but not for 2.6.32. According to Wikipedia, I think 2011 was the last time we had a security update for the 2.6 kernel.
IVAN: Wow. Let’s talk about a topic that’s near and dear to my heart – passwords. [laughing] I’m a major proponent of using a password management tool. I try to have a unique password for every single account I have, that’s at least 64 characters long. I shudder when people reuse their passwords in different accounts, and what I’m hearing from you and from the report is that, the passwords in the routers are really easy to crack, sometimes unencrypted and also hard-coded in cases. What were your findings regarding passwords and this set of data?
JOHANNES: So, what we found was that a lot of devices had hard-coded passwords. So, why is that bad?
IVAN: Yeah, why is that bad?
JOHANNES: Hard-coded means it is baked into the firmware of the device, and keeping in mind that most of this firmware is mounted read-only, in your device you can’t change it at runtime. So, you have this hard-coded password, which you could never change, and everyone who has access to the firmware can download it, grab the password and now has your password. It’s one of the ways, for example, the Mirai botnet a couple years ago, exploited home recording systems and routers and things like that, because it just tried some standard credentials, and for a lot of devices you cannot really do a lot about that.
IVAN: And what is the alternative? What would a manufacturer actually be able to do to not hard code a password, and did you find any routers that were like that?
JOHANNES: We did. So, when I said most of the routers had hard-coded credentials, we have to say two things. Yes, there were also routers which did not so, there is obviously ways to not hard code it. Then, not all the passwords that we found were easily crackable. We had like 123456. We had like root:root, stuff like that.
IVAN: ...password:password... admin:password… [groan]
JOHANNES: Yes, stuff like that exactly. So, that gets cracked super-fast. We have devices that use old hashing schemes like 3DES, which are known to be easily broken with magic tables. That said, that’s not the majority of passwords, but also you have to think about what this means, like having hard-coded system passwords for example. It means that if you get into a system, it’s like easy to go in there and mostly you have root access, but it also necessitates that you, for example, have a secure shell alternate open on the device and get into that.
We see a lot of times that the system passwords also are the passwords for the web interface, so your configuration interface for your router, but there’s some times where these passwords differ, or where for example you have a hard-coded password onto the firmware, and the web UI prompts you to change your password for the web UI on first login.
Now we have a way how to not do that, you can either have a process where you can set these passwords on first login. Then you have to have some mechanism where there’s not a read-only value on your system. You can put it someplace on the hardware, where it is not updated on firmware updates. And then you can have it not be in the internet. But then again, you should be keeping in mind that you should have a change procedure for that, because otherwise you will probably have like one password for all your devices, and then it is easy again, because just one researcher has to go and crack open the router and post it to the internet, and your security’s broken again.
IVAN: Your report said that some manufacturers are better than others. And so tell us what better means, and tell us if you had to recommend a manufacturer, or even a router to listeners where would you go?
JOHANNES: Vendors that use newer kernel versions, vendors that don’t use passwords or hard-code their crypto keys like private SSH keys into their firmware, which use more exploit mitigation on their binaries, like stack protection stuff, and which updates firmware regularly. Regarding the vendors in our study, where for example the European-only vendor called AVM, they do a nice job regarding the Linux kernel version for example, so they actually use 4.4 kernel in about half their firmwares, and they don’t have any 2.6 kernel versions in there.
You have for example, Netgear, which does use for example 2.6 kernels in about half of their firmware, but other than that also uses a lot of Linux kernels, 4.4, 4.1 and 3.4 for example. So that is a big thing, because then you drop all these Bluetooth and WiFi vulnerabilities that are basically known for 10 years in the Linux kernel.
What I find really important and what we talked earlier about, the regularity of updating. So, for example, Netgear and ASUS and the European AVM, they do regularly upgrade the firmware for most of their devices. Actually, I think for all these three vendors we have nearly all the devices inside of the last year for updates.
IVAN: There are open source router firmwares like the ones that were based on DD-WRT and OpenWrt. I would guess that those are updated more frequently. Is one of the ways of mitigating the security vulnerabilities in routers for manufacturers to just adopt this open source firmware and make their routers use this firmware, so that we’re using the combined knowledge and the combined open source power of the entire planet to secure routers?
JOHANNES: I would say yes, but that would be a clear guess only when I’m looking out of the glasses of my IT security background. I think you have to always be a bit picky regarding background and basically adversarial model. So, do you fear governmental-type attacks on your system, or are you basically someone who just wants to protect from the next botnet taking over your home router? Because if it’s more than that, I think going the open source route might be possible, but also can lead to some, even I would say, less secure router firmware, because if you’re taking the open source route, you can do a lot wrong.
You can choose bad passwords, have services running on your internet-facing interface that you don’t really need, and there are deeper things that then can be exploited from the outside using these maybe bad passwords that you have set. Or you can even brick your device by applying the firmware wrong. So, there’s a lot of pitfalls there.
I think something everyone can do is look for updates for the systems, apply them yourself if it’s necessary, because not every vendor, or let’s say most vendors do not have an automated update procedure. Then change the access passwords on your firmware at least the one for your user interface, because these ones you can probably always change.
Really important, disable all the debug features of your firmware. That is something that you should be able to all do just using the extended menu of your configuration interface. And it does not need for you to really have IT background. But if you have, then go ahead, try OpenWrt, it is nice, it is updated quite frequently, you can have the most recent Linux kernel version for your firmware. And you could have nice additional features that you don’t even know that your device can offer you.
IVAN: I have an Orbi Mesh and I know you were probing it at the beginning of our discussion. What does your home router situation look like?
JOHANNES: So, I have, from this vendor AVM, a Fritz!Box which is a German word I guess, 7590. It is like one of the high-end devices of this specific vendor, and I think it has gotten a firmware update rather recently, like two months ago or a month ago. It is regarding the spectrum of this report on the upper end, I guess, so it’s one of the devices that makes a lot of things right and not so much things wrong. I also have tried out the OpenWrt. There is the possibility for me to have that on the device, but right now I’m running it basically on stock firmware.
IVAN: Is there something that a user who is worried about their router can do that’s easy to know, apart from actually updating the firmware and changing the passwords as you mentioned earlier? Is there somewhere where they can check a list and look to see where their own router is affected in this list of tested firmwares that you have?
JOHANNES: Since we did not like also post device-by-device list of everything that we found, but only the aggregate that is supplied in the report, we did not offer something like that. What you can do is write us an email with the device that you have, and either this is in our data that we generated already, or we’ll be able to compile some information regarding that in a short span of time, most likely.
That would be one way. But aside from that there is not really this, or to my knowledge at least, there is not like a list of all the nice home routers and 1-to-5 how secure it is. So, you can look at the report, see if it’s one of the vendors that’s doing a better job, let’s say. And then maybe or probably doing all the best practice stuff that I just said before you’ll probably find. But aside from that, it’s up to you to check the security of your system.
IVAN: And what email address should people use if they’re interested in getting help?
JOHANNES: There is a mailing list called [email protected]. I think for the purpose of this podcast it might be most helpful to put it in the description of the podcast.
IVAN: Absolutely. We’ll do that. We’ll make sure we get that published, and the correct email address in the show notes. So, if you’re listening and you’re curious about that, make sure you check the website, and we’ll have it there.
JOHANNES: Yeah, happy to help.
IVAN: It would be great if there was a website you could just go to and it would probe the router that you’re on, and it could tell you what firmware you’re using and even what model router it is, and then a green, yellow, red, whether or not you should be thinking of. That would be amazing. But I understand that that doesn’t exist just yet.
JOHANNES: Maybe that’s just the next research project for our group.
IVAN: Yeah, and maybe you'll get some funding to do the next version of the report, and that might be a part of it, that would be wonderful if that was the case.
JOHANNES: Obviously, we would credit you with the idea.
IVAN: [laughing] Well, thank you very much. This was a really fascinating discussion. I’m so glad to have been able to talk to you, and that you’ve shared this important paper and this important work that you’ve done with us, and with our listeners, and I hope you keep going from strength to strength. And maybe next year when the next report is published you’ll join us again, and we could talk about what’s changed and how much better we are.
JOHANNES: I’d be happy to do just that.
IVAN: Well, thank you very much.
JOHANNES: Thank you.
IVAN: Johannes vom Dorp is a scientist and group lead in the Cyber Analysis and Defense Department at the Fraunhofer Institute for Communication, Information Processing and Ergonomics FKIE in Germany. He co-led that group at the institute that published the 2020 Home Router Security Report.
You’ve been listening to The TEN7 Podcast. Find us online at ten7.com/podcast. And if you have a second, do send us a message. We love hearing from you. Our email address is [email protected]. Until next time, this is Ivan Stegic. Thanks for listening.