When I started Open Source Security I knew one of those topics that could use more attention was the security of CI/CD systems. All the talk about securing the supply chain seems to almost exclusively focus on the development stage as well as the deployment stage. It seems like there’s not enough attention happening to the build stage (spoiler: most of the successful attacks have happened at this stage). When François Proulx reached out to chat about CD/CD systems, I couldn’t say yes fast enough.

This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.

François Proulx is a supply chain security researcher at Boost Security. He did a great job explaining what he’s been working on as well as what some of our challenges look like. He’s been in the weeds of supply chain security for about two and a half years, looking into the ways our build systems can go sideways, he’s not just looking at the usual details like source code and the final artifacts. He’s looking at what happens in between. The thing that keeps him up at night is how our CI systems have become the target that nobody’s watching closely enough. The CI system is the place where everything comes together: code, dependencies, build tools, and usually credentials that can access all the important things.

The attack surface

When we boil it all down, the challenge is really all about build time dependencies and test tools. These are applications and code that aren’t cataloged or paid attention to as part of the “typical” supply chain. For example when we think of something like an SBOM, it’s usually the code we ship, not the things used to build the code we ship. We also love ot obsess over the dependencies the developers use. But we ignore the middle, the CI/CD systems. The code we download to do our builds, the code we download for our tests. Do you know which compiler built your project? What about the versions of the test infrastructure?

GitHub Actions

An easy example for CI systems is GitHub actions. The infrastructure GitHub has created is impressive and is used by a huge number of projects and organizations. François had some really good advice, things I didn’t know about before our conversation. If your organization has been around since before 2022, you’re probably running with insecure (or at least less secure) default permissions for your actions. It’s easy to say we should just change your settings to the more secure default settings used today. But if you have an organization that has thousands of projects, that could break quite a few things.

The GitHub token that gets created during your builds? It’s probably way more powerful than it needs to be. It has the ability to submit or change pull requests, branches, and pretty much anything else in your repository. We’ve even seen attacks where an attacker doesn’t need to create a full pull request. There have been attacks where attackers use draft PRs to run malicious code. GitHub system won’t notify anyone because it’s just a draft.

The XZ attack in recent memory is a great example of attackers using the build and test system to attack the library that was built. There wasn’t a nice way to detect this, and a motivated attacker can find ways to get past code reviews.

What can we do?

It’s easy to say we should all audit and update GitHub organization security settings, particularly token permissions. While that is something we should be doing, it’s not a simple task, especially in a GitHub organization that has a lot of repos. So putting in the work to enable some of the new default security settings, there are other things we can also do.

The first step is to understand the problem. If you’re reading this, you’re on the right path! There’s plenty of nuance to this we didn’t discuss, so by all means keep looking. Taking the full supply chain including build-time dependencies as something you pay attention to when threat modeling. These are really hard solutions though because they need a ton of learning and work. Most of us don’t have a lot of free time.

There are some new automated scanning tools to detect common CI vulnerabilities. François has a tool he works on called Poutine. Automation isn’t a silver bullet, but it’s something that can help chip away at the problem. This is a space I’ll have to do more research about and write something up in the future.

The most useful solution is going to be changing your GitHub defaults. It’s a hard thing to do, but as history has shown us over and over again, you have to make the secure way the default way. Humans love ignoring problems, it’s very easy to make up reasons we don’t really have to care about our CI/CD security.