Unless you’ve been living under a rock for the past few … forever, you may have noticed that open source is taking took over the world. If software ate the world, open source is the dessert course. As of late there have been an uptick in stories about backdoors in open source software. These backdoors were put there by what is assumed to be “bad people” which is probably accurate since everyone is a villain in some way.
The reactions I’ve seen to these backdoors range from “who cares I don’t use that” to “we should rewrite everything in house and in assembler and go back to using CVS on a private network”. Of course both of those extremes are silly, it’s far better to land somewhere in the middle. And as much fun as writing assembler can be, the linker is probably an open source project.
This brings us to the question what do all these backdoors really mean for open source? It isn’t going to mean anything in most instances. There’s a lot happening that’s not well understood yet, and no doubt we’ll see more changes in the future as we understand the problem better. I think there’s a tendency to try to overcorrect when something new happens, in this case I’m not sure we can overcorrect even if we want to.
The first and most important point is to understand that a huge number of open source projects are a couple of people who are doing this for fun. They’re not security experts, they will never be security experts. They’re also not going to adopt some complex security process. If they get a nice looking pull request they’ll probably merge it. Security isn’t on the top of their list when working on the project. The whole point of their project is to solve some sort of problem. While I’m sure many would love getting a few donations, it’s a steep climb to being able to work on your open source library full time. The reality is these project will always be hobbies.
Secondly, you can’t claim you will only use “trusted” open source. There are now a number of vendors who tell you if you come sit by their fire everything will be OK. They’re a safe space and the open source they have is only the finest quality artisan open source crafted by Himalayan monks, but only on Tuesdays because that’s the day the open source karma is best. I don’t think these vendors are trying to mislead, I think they’re just as confused as the rest of us.
Open source is like an enormous tapestry depicting an epic struggle between evil and something slightly less evil than the first thing. Even the big projects and vendors that everyone thinks have it all together are a part of this tapestry. Everything is connected to everything else, it’s open source all the way down. Sometimes it’s a library. Sometimes it’s part of the build system. Sometimes it’s a tool running on the developer workstations. You can’t ignore the point that everything is connected. Claiming to only use trusted open source is just as realistic as claiming you’ll rewrite it all in assembler.
So what can we do about this problem? The collective “we” probably can’t do much, but the tapestry of open source is doing something, it’s just not super obvious. It’s even likely it’s not even intentional. Backdoors are like insects chewing holes in our wonderful tapestry, how do we get rid of them? We don’t.
We can’t prevent backdoors
This is probably going to a controversial position, but I’m going to say backdoors are going to just be a part of open source. They are here to stay and we can’t stop them from happening. The things that make open source development work are the same things that let backdoors happen. Getting rid of backdoors means getting rid of open source. All the positives of open source drastically outweigh the negatives of backdoors.
I imagine this sounds a bit loony to some, but things are happening that should give us all hope. If you made it past the previous paragraph I’m going to explain why backdoors don’t really matter now, and won’t really matter in the future. A comparison here would be security vulnerabilities in software. We used to think we could get rid of vulnerabilities, we just need more training and forcing people to care. If we take this stance on backdoors we’re in for a decade of disapointment.
How did we find out about the last few backdoors? They were found by the community, generally pretty quickly. The mostly discredited Linus’s Law says “given enough eyeballs, all bugs are shallow”. While I don’t think that’s true, I would be willing to amend it to say “with open source bugs can’t hide for long”. A backdoor needs to hide to be useful. The open source community seems to be pretty good at finding backdoors. And more importantly is when a backdoor is found, it gets fixed usually in a few hours. Being fast is really important. Fixing security vulnerabilities, backdoors, and even bugs is a lot different when you can fix it in a few hours vs a few days.
As our infrastructures grow and evolve, as our development tools get better. As we pay attention to what’s happening in our applications like never before, we are seeing an evolution in computing that’s making a backdoor harder and harder to stay in place for a long period of time. This is what I mean when I say open source is doing something, but doesn’t exactly understand what. We aren’t doing these things to find backdoors, finding backdoors is a side effect.
Researcher are also starting to look for backdoors in open source. If you are a security researcher, start looking for these things. If you find a backdoor you’ll get a ton of free PR. There are some tools that can help do this today, we need a lot more. While some backdoors can hide, once we have more people looking and better tooling in this space, it’s going to get a lot harder. It’s always a game of cat and mouse. The defenders need to catch up a bit, I have no doubt we will.
It’s also important to point out in the case of open source we can find the backdoors. Claiming closed source is some magic bullet is even worse than rewriting everything in assembler. I don’t want to hear it!
Lastly, we can and should also account for these in our risk models. If we know backdoors will happen how will that change our behavior? It certainly should, a backdoor is a pretty big deal if you think about it. How will you architect your networks and applications if there is a chance a backdoor exists somewhere in the software? Anyone interested in this should think about it, implement it, and do some writing. I think it will be a very interesting space in the future.
The most important thing you can take away from this post isn’t that we should all just ignore backdoors. The real purpose of this is to help explain how this crazy thing we call open source is going to grow and evolve. Help it grow. If you’re a researcher, look for backdoors. If you’re an open source project, I guess look too, and keep doing whatever it is you do. If you’re an architect, account for backdoors in your risk models and talk about it. Part of the open source community is sharing what you know and learn. We have a lot of room to learn.