When I started Open Source Security HeroDevs reached out and asked if I wanted to have a chat. I was pretty interested in this discussion because the work HeroDevs does today is very similar to the work I did at Red Hat for a decade. While what they work on is a bit different than the sort of things we shipped in a Linux distribution, the basic idea is still the same.
I was joined by Aaron Frost, the CEO of HeroDevs. He clearly knows what’s up and has also done this wild backporting work. it’s a lot of fun to chat with folks who have done the same sort of work you have, there are plenty of war stories to swap. Aaron will be coming back pretty soon to chat about some other oddities around backporting security fixes in open source, it’s a topic I love to chat about.
This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.
Open Source End of Life
When we think of open source, I don’t think End of Life is part of what we really consider. But just like all software, open source can go end of life. For example Node.js does a nice job of publishing their support schedule. When a version falls out of support, the Node.js project is done releasing fixes for those versions. Open source is sort of magic here though, the source code is open to all, there’s nothing stopping someone else from picking up an old version and fixing bugs and security vulnerabilities.
The release cycles of open source projects are becoming more aggressive in recent times. If you look at the data showing releases of open source software, it’s continuing to accelerate both in number of projects as well as number fo releases. Aaron pointed out in the discuss a Sonatype report telling the percentage of projects running dependencies with high-risk security vulnerabilities jumped from 54% to 74% between 2022 and 2023. The increase isn’t necessarily showing open source is less secure, it probably suggests more organizations are getting left behind as release cycles move faster.
The Enterprise Dilemma
Aaron cited a few challenges enterprises see. I remember seeing similar problems back in my Red Hat days.
- Organizations that cannot upgrade due to technical or business constraints
- Companies that lack the budget for major upgrades
- Enterprises that are working on an upgrade but need more time
It seems that enterprises are suffering from this more than ever, almost certainly due to regulation being more specific around open source. It used to be easy to ignore the open source, or just hand waive it away. That’s not really something that works anymore. Things like PCI, FedRAMP, and upcoming EU regulations are demanding SBOMs and vulnerability scans. You can’t ignore the open source when you have a document telling you what it is.
But the thing is, not everyone knows they need to care, and not everyone does care. If you only have to worry about EOL software inside the PCI boundary, it’s very likely and possible nobody cares what version of Node.js you’re running outside the boundary. This is just how security works, unless it’s mandatory, it’s easy to just forget to fix all those security problems.
So what is backporting?
Backporting is the practice of taking security fixes, bug fixes, and compatibility updates from newer versions of software and massaging them to work in older versions. It’s not as simple as just copy and pasting code. Each patch must be modified to work with old code, usually. The most common use for backporting is security vulnerabilities (all the Linux distributions do this and have been doing it for decades). The goal of the backport is to allow the old version of something to continue working in a way that doesn’t break everything. In theory you change the smallest bit of code you can which helps avoid introducing bugs or instability.
There are plenty of open source projects that will tell you to “just upgrade”, which seems like reasonable advice, but there are tons of reasons why upgrading can’t be done (some of those reasons are in the above section). In those instances something like a backport can be an easy solution. Paying someone to create a backport could be cheaper than the cost of upgrading.
What’s next?
The challenges of maintaining the open source software in your products and services will never go away. If anything it will get more difficult in the future (probably, I don’t have any evidence to back that up, it just seems like an easy claim). While upgrading to newer versions will always be the correct solution, that’s not always possible. There’s not really an amazing call to action to end this one that isn’t “pay someone a lot of money to maintain old software”. Maintaining old versions of things is really, like REALLY hard. It’s also not fun. When something is hard and not fun, the only way to make it happen is to pay someone to do the hard unfun work. I don’t see EOL open source ever happening for free, so if this is something you need, it’s going to come with a bill.