Josh and Kurt talk about a blog post that explains there isn’t really an open source software supply chain. The whole idea of open source being one thing is incorrect, open source is really a lot of little things put together. A lot of companies and organizations get this wrong.
Open source has always been held to a higher standard. It has always surpassed this standard.
I ran across a story recently about a proposed bill in the US Congress that is meant to “help” open source software. The bill lays out steps CISA should take to help secure open source software. This post isn’t meant to argue if open source needs to be fixed (it doesn’t), but rather let’s consider the standards and expectations open source is held to.
In the early days of open source, there was an ENORMOUS amount of attention paid to the fact that open source was built by volunteers. It was amateur software, not like the professional companies of Sun, DEC, and Microsoft. The term FUD was used quite a lot by the open source developers to explain what was going on. And even though all the big commercial companies kept changing the reasons open source couldn’t be trusted, open source just kept exceeding every expectation. Now the FUD slingers are either out of business or have embraced open source.
The events following Log4Shell created whole new industries bent on convincing us open source can’t be trusted. This time instead of the answer being closed source software, it’s some sort of trusted open source, that only their company can sell you, and don’t forget to be afraid of open source! I don’t think humans will ever learn that fear doesn’t work. It might work for a little while during an emergency, but it’s just not a long term strategy. Open source won because it works. No amount of trying to convince us otherwise will change a single mind.
We’re again at a place where there are a lot of governments and companies telling us open source needs to be fixed. We also see attempts at trying to create ways to measure open source so we can determine which bits of open source should be trusted and which should be eschewed. On the surface this seems like a fair idea. Obviously some open source would measure well, some will measure bad (this doesn’t mean a good score is better, more on that another day). All software works like this. There’s no call to measure closed source, consider it homework to figure out why not.
So this brings us to how open source surpasses whatever standards are put in front of it. This will probably keep happening. Open source is easy to pick on. Nobody is in charge. There’s no legal department to come after you for making up lies. It’s just people building software in an amazing way. When someone starts trying to scare off users, they just keep building software and ignore the nonsense. It’s generally very impressive.
If we look at some of the most successful open source companies. Red Hat, Canonical, GitHub, and GitLab to name a few, they aren’t trying to convince you open source is dangerous. Open source is how they build their products and their business. They understand open source and that allows them to reap the benefits. These companies directly benefit from the higher standards of open source. Ironically, their competition created the very environment that let the open source companies win.
This isn’t the risk you are looking for
Now, all this said, there is risk when you use open source. There is risk in everything you do. Crossing the street, driving to fast, eating expired ketchup. However, the risk that comes with open source isn’t what we may think it is. There are many who will tell you some open source is dangerous because it’s old, or has vulnerabilities, or doesn’t have the backing of a foundation, or fill in some metric. These reasons aren’t the risk, they’re marketing. The actual risk is never a simple one line explanation.
Open source is like any asset. If you know you have it, you can make informed decisions. There will always be vulnerabilities. There will always be old open source. There will always be projects run by one person in their basement. None of these things are a problem by themselves if you know about it.
Here’s an easy example that picks on Red Hat and Google. Google has container images called “distroless”. They are designed to be incredibly stripped down, the idea being you should only run what you need. The Red Hat container images are gigantic in comparison. One reason often brought up for running distroless is they have fewer security vulnerabilities, which is completely true. If you have a surface understanding of the issues, this seems like a no brainer, smaller image equals less risk. But nothing is ever this simple. Red Hat has a great team of people that is constantly providing care and feeding of their packages (I used to be one of them). They don’t just patch the security problems, they are in the trenches helping open source projects with the updates and making sure they understand the vulnerabilities better than anyone else. Red Hat knows exactly how a security vulnerability affects them and they fix the things that matter very quickly.
The risk in the container images you run is really about support and tooling. Some projects and companies will benefit more from small container images. Maybe because they have fewer vulnerabilities, or maybe because they want to move at lightning speed. Some companies will benefit more from the slower pace Red Hat affords and they want to have world class support. There’s no right answer, everyone has to decide for themselves what makes sense.
The only constant is FUD
I don’t expect how open source is judged or measured to change anytime soon. It’s just too easy to blame with one hand, and keep using with the other. Only time will tell if and how governments get involved. I’m sure some ideas will be good and some will be bad. Something about a road paved with good intentions maybe.
The one thing that gives me the most solace about all of this is how much open source has won. It hasn’t won by a little bit, software ate the world, then open source ate the software. If it uses electricity, it uses open source.
Josh and Kurt talk about the Time Till Open Source Alternative blog post. The numbers probably don’t mean what we think they mean anymore. A lot of modern open source is really corporate controlled. Just because something carries an open source license doesn’t mean you can contribute to it.
Josh and Kurt talk with Josh Aas from the Internet Security Research Group about Let’s Encrypt, Prossimo, and Divvi Up. A lot has changed since the last time we spoke with Josh. Let’s Encrypt won, and the ISG are working on some really cool new projects.
I take a bike ride every morning, it’s a nice way to think about topics of the day. I’ve been wondering lately why software supply chain security has exploded in popularity in the last year or so. Nothing happens by accident, so there must be some series of events we can point at that has led to everyone suddenly making this a priority. Software supply chain security is not new, I’ve been doing it since about 2002 when I was helping track and coordinate security vulnerabilities in Linux distributions. We didn’t call it a supply chain back then, and nobody really paid attention to it. So what changed between then and now?
All this supply chain interest reminds me of how open source exploded in the early 2000s. Much has been written on why open source won, I won’t rehash it here, but keep it in mind as we think about the software supply chain. I think there are parallels and probably some lessons. The most important parallel is how much collaboration we are seeing around supply chain security. Open source won because everyone worked together. Supply chain security will happen because everyone works together. If you try to do this alone, you will fail.
There are three buckets I think can help explain the importance of the software supply chain. I’m calling these buckets tools, ideas, and events.
Let’s start with Events. If you ask anyone in IT about supply chain attacks in the past ten years, they can probably name more than one. Almost everyone will name Heartbleed first. Heartbleed happened back in 2014, so why did it take until 2021 for supply chain to become a topic everyone is paying attention to? Since Heartbleed there have been quite a few similar problem, from Stuxnet, NotPetya, and even SUNBURST as examples. None of those seem to have created any real disruption. It’s possible they just weren’t big enough to capture attention, but my suspicion is they didn’t affect open source. In every one of those instances you had a vendor to blame. But then there was Log4Shell. Log4Shell will probably be the one we look back on as when it all changed. Everyone knows about Log4Shell, it was a pretty big deal. And there was nobody to blame except ourselves.
I think Log4Shell itself wasn’t the event, the real event was everyone understanding just how much open source they had. Log4j was everywhere, and while looking for it we also noticed all the other open source in use. It was an eye opening moment for a lot of people. Open source won a long time, but it wasn’t until Log4Shell that it was declared the winner. Now that we know open source is everywhere, what are we going to do about it?
In the past, the first step would have been to declare “no more open source!” But in 2022, that’s not even realistic as a joke. There’s no way we can stop using open source. So we had to find another way. This supply chain thing seems like the place to start.
The next step is ideas. Once you know a problem exists, you can start to try solving it. For example the Software Bill Of Materials, or SBOM, is not new. There have been many people working on this idea for at least a decade now. But it’s likely you didn’t hear about SBOM until very recently. In the past a large group of people understood this was important and began working on the details of how SBOM should work. But for a long time it was really just an idea. A good idea, but still an idea. The Log4Shell made us all take notice.
If we didn’t have SBOM in the state it is today, how would we be cataloging the open source in use? It would probably a lot of bespoke solutions, nothing would work together, and there would be no clear path to the future. Software supply chain security would probably be a fragmented disaster with everyone trying to do something different. SBOM is really the idea we needed to rally behind for supply chain security to explode in popularity. Thank goodness others had the foresight to create our current SBOM formats.
Just because an idea is good, that doesn’t mean it will be a success. We need the right series of events to bubble an idea to the surface, then we need the right tools to make it happen. Tools can be software or they can be file formats. Maybe even compliance standards. There’s never just one way to solve a problem. An old saying exists that captures this nicely is “ideas are cheap, execution is everything.” How do we execute on the idea of software supply chain security? Why with the tooling of course!
Long ago all the software supply chain work was done by hand. In text files. Probably checked into CVS. There were no standards. Everything was just made up as we went along. I wish that was a joke. Linux distributions like Debian and Red Hat have been using open source to build larger projects for more than twenty years now. Every Linux distribution has been issuing security advisories since the beginning. This was rudimentary software supply chain management and it succeeded in spite of the tools, not because of the tools.
We are now at a place where we have the will, we have the ideas, and we have the tools to start making supply chain security reality. I think without all three we wouldn’t see the sort of progress we see today. But progress comes with its own set of challenges.
It’s always amusing when a new industry starts to emerge. There will be many people complaining about how useless all the new knowledge and technology is because it’s not perfect. This was just like the early days of open source, there were a lot of naysayers. Some are trying to sell you something, some don’t like change, and some just like to complain. Regardless of the reason, there’s no stopping this one. Software supply chain security is coming like a speeding truck.
A lot of the tools aren’t amazing, but are getting better, and fast. A lot of the ideas need to be expanding, and there are groups of people working on everything. The SBOM formats are still incomplete, but they are getting better every day. The tooling has plenty of gaps, but it continues to get better. Progress isn’t a destination, it’s a journey. And in the supply chain world, there are A LOT of people on this journey.
There is a lack of perfection, but there isn’t a lack of progress.
You should keep in mind that we are the very beginning of this new supply chain journey. It’s not the end of the beginning, we’re at the beginning of the beginning. There is enormous opportunity for those who want to contribute. Everything from corporations, open source projects, non profit foundations, and even the government is helping drive change. It’s an amazing community of collaboration, very much in the way open source functions.
Software supply chain security isn’t coming, it’s here today. Go help make it better!
Or complain, I guess.
Josh and Kurt talk about really weird networking bugs. Josh tells a story about his home network problems that made no sense. There was also a qt5 bug that affected wireless networks that made virtually no sense. What should count as a security vulnerability?
Josh and Kurt talk about the recent National Defense Authorization Act that requires security vulnerabilities to be fixed. What does this mean for us, is it as bad as some people are claiming it is? It’s actually not a huge deal, for most of us it’s really just time to deal with product security.
Josh and Kurt talk to Dustin Childs about the recent ZDI Black Hat talk where they discovered the current trend of security patches not actually fixing the security problem. We talk about what this problem means. Why is it happening, and what ZDI is doing to try nudge the industry in the right direction.
Josh and Kurt talk about our lack of security and some of the data bias problems that can emerge. A lot of what we think is security data is really just biased data. This is OK as long as we understand the data is broken and know this is the first step in a longer journey.