I listen to a lot of podcasts. A lot of podcasts. I was listening to the Dave and Gunnar Show podcast episode 212 with guest David A. Wheeler. The Titanic was used as an example of changing process after a security incident. This opened up a flood of thoughts to me, but not for the reasons intended in the conversation. The point of the suggestion was the Titanic sinking created changes to international requirements to help avoid a similar disaster next time, and we should be viewing SolarWinds in a similar way. The idea being we should use the SolarWinds event to drive meaningful change to make security better. Why no change will come of this is a different conversation: TL;DR it’s because nobody important died from SolarWinds, the Titanic killed a lot of important people. But I think this is an interesting way to talk about how we tend to deal with problems in software and how we deal with them in real life.
I won’t bore you with a bad retelling of what happened to the Titanic. Historians have covered this topic at length and from every possible angle you can imagine. What’s not always covered is what happened after the Titanic had its historic voyage. Wikipedia has a page dedicated to changes in safety the Titanic caused. If you read this list it seems pretty obvious in hindsight. This is how many disasters work. There are a lot of little things that have to go wrong for an accident to turn into a disaster. If we could avoid even one or two of those little things, it can change the result dramatically, changing the story from “disaster” to “accident”.
So what does this have to do with software? When we have failures in the world of software it seems like we rarely change anything. How many organizations have changed the process due to Solar Winds? There was a lot of momentary panic, and A LOT of empty talk, but from what I’ve seen most companies have basically just gone back to doing thing the same old way. Most of the suggestions on what to do was what I like to call “security harder!” This is where leadership says they take security very seriously, but their actions tell a different story.
It seems like if we reacted to the Titanic in the same way we handle software incidents the solutions would be “Titanic harder next time”. We would have launched thousands of boats through iceberg filled waters with varying degrees of success. Every time a boat sank we just told the ship builders to use more Titanic the next time. They of course have no idea what that means, but as long as the checks clear they keep building boats.
I think we lack realistic understandings of what happened to SolarWinds. If you listen to the Dave and Gunnar Show mentioned above, the proposed solution to all of this is something called reproducible builds. I’ve written and talked about reproducible builds at length, they are a modern solution to a malicious build system the same way an iceberg satellite tracking system would have been a solution for the Titanic. It’s not a realistic option because the technology doesn’t exist. I hope it does someday, but it’s just not there right now no matter how hard we want it to be. The tooling doesn’t exist. If we want to have an honest conversation about reproducible builds it needs to be with the creators of build systems. And when I say “conversation” I really mean patches. If this is something society wants, we have to do the work. Talking isn’t work (says the guy with a blog). Anyone who has reproducible builds today have done it in spite of the tools, not because of the tools. The number of organizations and projects that can do this is a very very small number.
The obvious question about all this supply chain talk now is “what can we do?” And there isn’t a great answer to this question today. There is no shortage of opinions about how SolarWinds could have Titanic’d harder. But what there is shortage of is actionable advice the rest of us can use. If you asked someone how you could better secure your supply chain, you aren’t going to get advice, you are going to get guesses. There aren’t a lot of people who actually understand what a modern open source supply chain looks like. There are however plenty of opinions on how to secure a modern open source supply chain.
I of course know how to secure the supply chain because I have a blog named “Open Source Security”. This is sadly the closest thing to a real qualification that exists in this space. While I offer a great deal of advice on various open source topics, the most important takeaway is anything I talk about is verifiable by anyone. This is open source after all. You don’t have to take my word for it, go figure it out for yourself. If you find I’m wrong, let me know so I can change what I’m talking about.
So you don’t want to be the next SolarWinds, but you also don’t know what that means. I just wrote a blog post about not trying to manage your supply chain. Find someone else to help you. There are many companies and projects that are better at supply chain tasks than you will ever be (unless you’re in the business of open source software, don’t spend your valuable time worrying about it). But even that post misses some important points. “What should you do first” is a pretty important question for example.
Lately I’ve been imaging the work I have to do as an apple tree. There are some apples close to the ground, and some high up in the tree that will need a ladder to get. If you’re like any normal person, you pick the low apples first. I don’t even have to explain why (if you’re the sort of person who picks the apples on top first this is why nobody likes you). The work we have to do for making our supply chain secure is a lot like this. There are some tasks we can do right away because they’re the low hanging fruit. Low hanging fruit is easy to pick.
By doing the easy things first we get two huge benefits. The first is we are doing the work we need to do. We’re not talking about it, we’re not building some plan, we’re actually doing work. Work beats talk every day of the week. The second benefit is we start to learn more about the problems. Even if you do a task that ends up failing or not really mattering, you will learn something along the way. The more we learn the more we can decide which apples should be picked next. And when in doubt, just pick the apple closest to you.
Here’s a simple thing you can go do right now that can make a difference. Aqua Security have a scanner tool called Trivy. Go run it against your code (it can also scan containers). Here’s an example that scans the Signal Desktop app.
docker run --rm aquasec/trivy repo https://github.com/signalapp/Signal-Desktop.git
Of course, I’ve also talked about what this scanner output really means, so be sure to take it with a grain of salt 🙂
But the point is if you run this tool and look at the output, you are going to learn new things and have a different understanding for what you’re doing. It will help shape your future work and ideas.
And the absolute most important thing you can do once you start this journey is to talk about it. When you do something that works, tell the rest of us. When you do something that doesn’t work, tell the rest of us. We have enough talking heads giving out unrealistic advice. I want advice from people doing the work. I don’t need to hear what the guy with an expensive suit thinks. He’s the reason we’re in this mess.
Josh and Kurt talk about the Google Project Zero report titled “A Year in Review of 0-days Exploited In-The-Wild in 2020”. It’s a cool report but we don’t agree on the conclusion. The answer isn’t to security harder, it’s to stop using C.
I’ve been thinking about what open source is a lot lately. I mean A LOT, probably more than is healthy. There have been a ton of open source happenings in the world and the discussions around open source licenses have been numerous. There are even a lot of discussions around the very idea of open source itself. What we once thought was simple and clear is not simple or clear it would seem.
Full disclosure. I work at Elastic and if you pay attention to open source you probably hear that Elasticsearch has a new license. I’m not going to discuss open source licenses today, I will soon, but today I want to talk about community because it keeps popping into my brain and clouding other ideas.
The term “community” means different things to different people. I’ve heard some people talk about community as some sort of amorphous blob that will give them free work. Some think it’s a bunch of jobless degenerates who need haircuts. Some think it’s where their friends are. Some think it’s where their enemies are. Some people believe community is a mythical beast, something so fantastical that can’t possibly exist, like unicorns, dragons, or Canadians. When we don’t know what something is, it enters the world of myth and it becomes both everything and nothing at the same time. I think many of us have forgotten what community is.
I’m pretty old so I remember the world of what we now call “open source” when it was less about the license and more about the rebellion. There were only 3 licenses to use, so nobody really paid all that much attention. The companies that are now telling us what open source is or isn’t were the same companies we built communities so we could avoid those companies. Microsoft used to be the bad guy. They were an enemy to be defeated. It was a wild time for sure.
I remember when I got my first job working on open source. It was a little company called Progeny Linux Systems. It’s long gone now, but there was always talk of “the community”. At first I had a sort of awe at this idea. “The community” was going to help drive everything forward. They would give us guidance, and help with development, and even pick up our slack. It was going to be so awesome! Eventually I came to realize there was a lot of misunderstanding about this community thing. The community pretty much did what the community has always done, which is whatever it wants.
What’s the point of all this community talk? While I keep turning the lathe of open source in my brain trying to decide what it all means, the thing I keep coming back to is this concept of “The Community”. There isn’t one open source community, there are probably tens of thousands (maybe even hundreds of thousands). They all exist for their own reasons, and they do whatever they want, because that’s just how it works.
The vast majorities of these communities don’t care what the OSI, or the Linux Foundation, or Microsoft, or Elastic want them to think or do. Every community values something different. For some, open source is a license that OSI has approved. For some it’s about collaboration, maybe even using Windows for that collaboration. Some might just be about sharing dank memes on a forum. There’s no limit, or rules, or order to any of this. The community does whatever it wants.
And this is where I keep seeing the shape of open source emerge. I used to think open source was about a license that OSI told me was a good license. I used to think it was all about the freedom, about the software that was set free because the OSI told me it was. I will say, having a license that encourages collaboration is HUGELY important, I cannot stress that enough. If a license is too restrictive you will not build a community, and it’s all about the community. I have many thoughts on that topic, but not yet.
The end goal in all of this shouldn’t be to use a certain license. Just using an OSI approved license doesn’t mean you are open source. There is plenty of proprietary software with an OSI license. The end state is to build a community that is self sufficient, that grows and changes on its own. If you think you have a community but you’re constantly having to resuscitate it, you don’t have a community. A successful community will take on a life of its own. It will not be steered by a Fortune 500 company, it will be steered by the people who are doing the work. Community is about collaboration, discussion, understanding, and most importantly, empathy.
The license you choose for your projects will either help foster or hinder your community. The license is just one of many tools you have to foster your community. This is also true of the development platform you choose. Or the communication platform, even the rules you create to participate in a community.
I think we became too focused on the singular aspect of open source licenses and forgot about all the other things that really matter. Open source isn’t about a license, open source is about people. We shouldn’t let anyone, especially a company, tell us what open source is or isn’t. We are all part of open source, but only if we’re willing to be a part of it. Open source is what we want it to be, open source is what we build it to be.
What a year it’s been! I feel like 2021 went by like a … it’s still January???
So it’s pretty much impossible to ignore any of the events of the last month. I want to talk about something that’s near and dear to my heart, and in the news, not for a good reason. Software supply chains. This is probably a dreadful topic to most, but I love software supply chains. I was talking about them before anyone was even thinking about them. I’m not going to pull a “get off my lawn” here, I think it’s cool everyone is starting to care. I want to talk about how to be realistic about your supply chain. Almost all advice I’ve seen of the last month has been terrible, so I’m going to also give some terrible advice.
I will start with the usual advice for the worst suggestion every time supply chain discussions happen: You cannot remove all the open source from your supply chain. Even suggesting this would be comparable to making your organization build a coal power plant to go off grid for power. Anyone who suggests this should not be taken seriously.
Now that we have that out of the way, let’s talk about a software supply chain. I wrote a lovely (at least I think it’s lovely) piece about this during Secadvent for DevSecCon. Just go read it, I’m not going to rehash the details here.
I would like to congratulate you on now understanding more about the open source supply chain than the vast majority of people giving comments to the media about this topic.
I’ve had a lot of discussions around supply chain over the last few weeks and there is something bothering me about it all, but I couldn’t figure out what it was, when I figured it out I sat down to write this because I think it’s worth sharing. I realized You cannot manage your supply chain.
I think in the industry we call that “clickbait”. I made it the title of the post in hopes you would end up here, be sure to like and subscribe and leave a 7 star review! But really, the thing is, there are very few companies that can actually manage an open source supply chain. If you have a very small footprint you probably can, but most of us don’t have small footprints. Even a trivial app will have thousands of dependencies. Assuming you could even find the talent to do this work, would you want to hire an army of people just to track your dependencies? What about hiring an army of people to create build infrastructure?
It’s not going to happen. Logistically you aren’t going to manage your supply chain. If you try to do it properly, it will cost so much your company won’t be able to grow. Hiring a developer will make your product better. Hiring a supply chain team will … have no noticeable affect.
I have no doubt there are some people screaming into their safety pillow right now. It’s OK, you fell into my trap! I’m not really suggesting you do nothing, well, I sort of am. I’m really suggesting YOU don’t manage your supply chain. Find someone else to do it. If I wanted to explain how to manage a software supply chain it could fill a very long, very boring book. It’s a huge complex problem with a low return on investment.
As a security person I don’t shut up about risk. Risk this and risk that, risk risk risk. I don’t have many friends for some reason. Your supply chain is really just a collection of risks. What part of it are you most worried about? The open source you use? The build system? The distribution of artifacts to customers? Maybe you worry about your developers. There are entire companies that exists focused on whatever part of your supply chain you think is the biggest risk. And they’re better at what they do than you will ever be.
If I was a very important expert giving advice to a reputable news outlet, right now I would be giving a list of things you should go do RIGHT NOW to make your supply chain more better. I’m telling you not to. Figure out what you care about, then find a way to work with someone who cares about solving that boring problem.
For example. If you care most about your build infrastructure, GitHub as the ability to mange all your CI/CD. If you’re worried about your dependencies, there are companies like Tidelift, Snyk, or JFrog. You need developer security training? Go look at Security Journey. There are hundreds of companies you can work with to solve these problems. Find someone to work with. You don’t generate your own electricity, you don’t run your own Internet. There’s a reason you don’t do those things. The whole point in having a supply chain is you have someone else supply what you need.