Josh chats with Jeff Matchett from Cloudsmith about a new report they put out. It has some scary looking statistics in it about how organizations are using dependencies. There are some surprising numbers in there, but the story is really one of defense in depth. There’s no single thing we can do here, it’s all about knowing what you have every step of the way. It’s easy to say, but certainly a challenge to do right. James is a ton of fun to chat with and filled with energy and knowledge.
Episode Links
This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.
Episode Transcript
Josh (00:00) Today, open source security is talking to James Matchett. He’s the head of security at CloudSmith. CloudSmith recently wrote a report. Well, I guess James was one of the authors. It’s called The Confidence Gap: Perceived versus Actual Preparedness Against Dependency Attacks. And it has some lovely, scary statistics in it. And I saw this and I thought, I want to talk to these folks. So, James, welcome to the show, man.
James Matchett (00:20) Thank you, thank you.
Josh (00:21) So, why don’t you tell us who you are and tell us what CloudSmith is, and we’ll kind of jump in right from that.
James Matchett (00:26) Awesome. hey, I’m James. I’m the head of security at CloudSmith, full time nerd. I’ve been in cyber for about eight or nine years at this point. Cloudsmith is fully cloud artifact management platform. We often say that our business is packages go up, mac magic happens, packages come back down.
Now unfortunately that magic is supporting thirty eight different types of artifact with custom processing, security, attestations, all that fun stuff. but it means we’re front and centre for everything that we’re seeing in the news from Shy Halud to all the other scary supply chain attacks that are out there this year.
Josh (00:57) Yeah, yeah. And I know I’ve I’ve had some other folks from Cloudsmith on the show in the past. And and you’re definitely like you’re doing some cool work. So it’s always always fun to have a chat. So all right, I want to dive right into this report because there’s kind of we’ll will lead if it if it bleeds, it leads, so to speak, right? Is there there’s kind of two numbers that I think you guys have that are meant to maybe scare us a little bit. Is the first is you have 73% believe their current security tooling can stop install time attacks, which
I’ll we’ll get to that in a moment, but the second one I think is way more interesting, which is thirty-eight percent of your respondents scan for vulnerabilities before ingesting them, meaning I assume there’s developers doing like pip installs or you know NPM install, you’ve got infrastructure pulling things in. Thirty eight percent feels how would I say this? It’s about what I expect, but it’s way lower than it should be, right?
James Matchett (01:46) Yeah. A
hundred percent. And the interesting thing between the two numbers is actually the gap. And it’s one of those things where you ask people, you know, a common behavior that they should do. Like, of course I put the shopping cart back after I use it or of course I lock the door every time I leave the house. but when you ask them, Do you really? What do you use? Like, do you actually screen your dependencies? That’s when the truth really comes out. you know, companies love to say, Yes, of course we’re protected from this attack, we use this, this, that and the other but engineering teams
If you can’t prove where they’re pulling your artifacts from, it’s open season for these types of supply chain attacks to come from. They’d hope something like NPM audit or NPM install would warn them. You know, they imagine that there would be this dialogue box saying, Whoa, you’re about to install malware, buddy. Would you like to do that? Unfortunately, no such dialogue box exists. And there’s a few reasons for that, which of course we can get into afterwards. But this is really asking every engineering team, every CTO, every CISO, do you actually protect yourself from supply chain attacks or do you just hope that you do?
Josh (02:46) Yeah, yeah. Well, okay. So I I wanna I wanna ask you about the first number, the the seventy-three percent, because you work for a vendor selling a product that’s supposed to help with this stuff, right? And I I used to work with a vendor that sold a product that helps with this stuff. And kind of the the the the thing I’m thinking about here, like when I first saw this was I think most people think they have a tool that can do this, whether they do or not, right?
James Matchett (03:12) Yeah.
A hundred percent. And I think part of the confusion comes from people misunderstanding what SBOMs and these big lovely lists of packages that you’ve installed
Josh (03:21) Yeah.
James Matchett (03:22) actually do. And people think, Well, I’ve got an SBOM I can see what’s installed and therefore I’m safe from this type of attack. But what we’re really seeing over the past, you know, five years is this transition from supply chain attacks that have to be all the way promoted and productionized and deployed for you to be vulnerable.
Versus the type these type of attacks we just have to install the package, not not even use it. And that’s how you become compromised. And it’s this confusion between what SBOMs used to protect you from versus these new types of attacks we’re seeing this year.
Josh (03:52) Right, right. And now it it should be said though that is it is it the new NPM that no longer does the preinstall scripts by default?
James Matchett (04:00) I think that’s the direction they’re going in because preinstall is, you know, a non interactive action. The user has one intention which is to give me my packages. The other intention is obviously run code on my my machine and that’s where the real threat factor comes from.
Josh (04:16) Right, right. Cause when I mean, and this is something I don’t think is super well understood is when you install these packages, there’s often like a hook that runs that can do things with zero interaction from the human that just did like NPM install or or I don’t d did Python get rid of I can’t Anyway, all of the registries are moving away from this behavior for this obvious reason, right?
James Matchett (04:37) Exactly, and it’s compounded by the fact that if you install one package, it usually comes with friends. You have sub
Josh (04:43) Yes.
James Matchett (04:43) dependencies and transitive dependencies and things that you don’t intend to install and even beyond that things you don’t intend to run, such as malware.
Josh (04:51) Yes. Okay. I I wanna latch onto that, because let me let me pull up my notes here. Cause you have a a th a a a line in this report that is not on the like big scary headline page,
James Matchett (05:02) Mm-hmm.
Josh (05:03) but it says, I’m just gonna read it verbatim here. It says, additionally, twenty percent believed that automated policy decisions should cover package selection, but in reality only sixteen percent rely on automation. But here’s the thing, you
James Matchett (05:13) Yeah.
Josh (05:14) just said that you you know, you install the thing and you get more things.
And I also think like twenty percent think automation, like there’s so many dependencies, there’s no way humans can possibly keep track of this stuff.
James Matchett (05:28) Hundred percent. And I always joke, you know, do CTOs and CISOs imagine there’s a security team always watching, seeing every time a developer clicks install, clicking yes, no, yes, no for every package flowing through. I mean that’s thousands of packages, that’s never gonna be the case. And even at that, there’s thousands upon thousands of versions within that, and any one of those can become compromised at a later date. Every decision that you’re making today, right now, is dependent on the current threat intelligence picture. But it’s entirely possible. You download a package,
It goes for your policy engine, you find out later after the fact, actually, that was compromised two hours ago. You know, th this is the direction things are going. And there’s no world in which any human or any team can do this manually.
Josh (06:10) Yeah, yeah, for sure. Well, and like one of my favorites is you might install a dependency that then obviously has its subdependencies, but i these these packages don’t usually say like, I depend on version one dot two dot one. They’ll be like I depend on version one dot whatever. So
James Matchett (06:26) Yeah.
Josh (06:26) if if like one dot three is filled with malware, but you only scanned one dot two and you’re suddenly pulling one dot three into your CI system, now you have a problem, right?
James Matchett (06:36) Hundred percent and sometimes you don’t even know what you’re scanning or what you’re pulling until you go to pull it. And this is why
Josh (06:41) Yeah.
James Matchett (06:41) having this system, this dependency firewall is the term we like to use. Every time a developer goes to the internet to try and pull something is so vital. And we try to encourage this thing called soak policies or delay policies. You know, sometimes if you want to pull latest, we might say, Well, maybe that’s not such a good idea. Maybe you want to install the latest version you can that’s at least seven days old.
Because at least by then the community, the threat community has had a chance to analyze it, tear it apart, and it gives you a limit of protection. But it comes with a caveat. Let’s say you have a soak policy for seven days, and then Defender comes out and says, Here, critical security patch, we’re fixing a massive hole. I want to install that now. But wait, you’ve got a policy for seven days. How do you negotiate that difference where you have this confliction where your security policy is actually keeping you a little bit insecure? And that’s why it’s not just enough to have a policy in place.
You have to practice the exemptions, the edge cases. How do you decide if a package is safe to pull despite your best intended policies you put in place?
Josh (07:40) Yeah, yeah. I mean, that’s like any good security policy, right? You have to have an exemption process because having just a strict security policy with that’s totally rigid creates other, you know, second order problems. I mean my favorite example here actually is the original like SCA tools, right? Like your Black Ducks and things like that. They were created because developers were just copying and pasting open source code into projects because they were told you can’t use these open source repositories. So they were like, Okay, I’ll just
steal the code, you know? And I mean th and that’s that’s my worry always, right? If we put these rigid policies
James Matchett (08:15) hundred percent.
Josh (08:16) in place, now developers are just like, I’ll just copy and paste it. And like that’s worse.
James Matchett (08:19) Definitely.
And we’re seeing this with AI actually. This is the shadow inclusion
Josh (08:22) Yes.
James Matchett (08:23) of packages. So let’s say conventional software development, you find a package you like, you add it to your requirements.txt, and it lives in your project. And it’s really easy and nice to audit because you’ve got this lovely list of everything you’ve installed. But you go to Claude OpenAI and you say, I want to recreate something that’s still on these lines, and instead of it including the package for you, it has this tendency to copy the code. And by proxy.
You now have that dependency in your project, but no way to audit that it exists.
Josh (08:52) That’s right. And who knows what version, right? Because this stuff was trained on the past. So it is completely plausible there have been bug fixes and security issues that have been disclosed since the training data and whatever it knows, which is
James Matchett (09:06) definitely.
Josh (09:07) like mind boggling.
James Matchett (09:08) And it gets even worse with slop squatting, which I gave a talk on recently. you’ve heard of typosquatting before.
Josh (09:12) I love that term.
James Matchett (09:13) You know, typo squatting is an old attack, it’s been around for about a decade at at this point. You find a popular package on PyPI or NPM, you swap one character or add an extra one in, and you hope a developer fat fingers it or mistypes it slightly, and suddenly they’ve downloaded a package which is like for like with the official one, and they’re just waiting for the download count to go up and suddenly that threat actor jumps on it, includes something in the next release
And compromises, well, however many people downloaded that real looking but actually fake package. With AI, and there’s been some really good research into this, it sometimes hallucinates a package name. So what threat actors would do, they would go to the latest model of Claude, put it through a few hundred thousand iterations, and try to find examples where it’s hallucinated a package name that doesn’t exist. They will then go and register it and hope that someone else has the exact same hallucination. And this has been going on for the
better part of a year now. There’s been some really, really good examples of this. Even to the point where someone has made these repositories and then an AI has relearned it and said, That’s definitely the official one And I started writing README’s
Josh (10:16) Yeah.
James Matchett (10:17) that say, actually use this one because well, our last training data said you had to.
Josh (10:21) Yep, yep. I I love that term. It’s like one it’s so disgusting sounding and I love it so much. It’s thank
James Matchett (10:26) Ha ha ha.
Josh (10:27) you, Seth Larson, for making that up. I mean it’s funny. Like Seth
James Matchett (10:29) Ha ha ha.
Josh (10:31) has been on the show before and and I think it was before he recorded, he was like, This is the thing I’m gonna go down to history for. It’s like the worst term ever. But yeah, I
James Matchett (10:38) Ha ha ha.
Josh (10:39) mean, but like on that note though, right? You you’ve got you know, you mentioned cooldowns, I can’t remember the name you used for. It wasn’t cooldown. soak soak time. Right. Now
James Matchett (10:48) s soak policies is another name for it, but but cooldown
policies is our a more preferred term for it.
Josh (10:52) Okay. Yeah, yeah. But you you’ve got it here. 24% of your respondents. That I again, it’s one of those I feel like that is what I would expect. It’s maybe a little higher than I would expect because I do feel like security people kind of grasp this concept, but I know I’ve talked to a lot of just developers I know in the past, and they don’t understand this is a thing or why they should even use it or care. Right? It’s
James Matchett (11:15) Yeah.
Josh (11:16) definitely not like common knowledge.
James Matchett (11:19) Yeah, so 24%. This is a really interesting confluence of whose responsibility is it to keep the packages safe? Is it the developers? Is it the security team? Is it both? And it’s interesting because it’s the developers that choose the packages to install and it’s the security team that ultimately has to decide what is safe to use and what isn’t.
And what I’ve found that works really, really well is the security team comes up with these automated policies to decide what is safe and unsafe to use in different scenarios. So your developers working on their local machine or a dev environment, you know, you can be a little bit more lax about it. Obviously, no malware, nothing with a silly high CVSS score. But other than that, you know, fair game, back on, have fun developing. And then you increase the stringency of the rules as you go up to your pre prod, your staging, your actual production environment, have more stringent rules.
Make sure that your packages are signed, they come from the correct source. You know, it’s not even so much about what the package is, but where did it come from? How did you download it? Did it go through the official repository or did it come from somewhere else? But soak policies are a really, really good way to counteract what is a lack of threat intelligence. You usually don’t find out that something is bad or compromised until several days after the package is released. That’s when the community forums and the threat processors have had a go at it. So my personal recommendation
five to seven days for anything going to production. It gives your dev teams enough time to pool something that hopefully isn’t malware, gives them enough time to test it, and by the time it reaches production you can be fairly confident it’s safe and good to use.
Josh (12:48) Okay, you just mentioned attestation. And I that I that’s one of my notes. So this is a great time to bring that up. It is in your your paper, you say 27% of respondents were unfamiliar with build provenance and the SLSA framework. I feel like it should be way more than 27%. Like when I go talk to people
James Matchett (13:07) Yeah.
Josh (13:08) and I ask about SLSA or attestation or signing or any of this stuff, they’re like, we have no idea what any of the words you just said are.
I I’m I was shocked this wasn’t like closer to fifty or seventy five percent.
James Matchett (13:21) Exactly. And it’s actually becoming a part of the Cyber Resilience Act as well. You’ve got to include your SBOMs and if you can include that information about provenance or attestation, you’re golden. I’m not gonna say you’ll never be attacked, but it means it’s so much easier to track down where the package came from, how many people installed it, where they installed it from, and by having these rules and making sure that you’re pulling the packages from a known source.
I can talk forever about, you know, dependency confusion attacks and multiple upstreams and nerdy stuff like that until I’m blue in the face. But having that information about where a package comes from just keeps you safer.
Josh (13:56) Yeah, I mean, I agree, but I will say that like historically signing anything has been an absolute nightmare. And Sigstore makes the signing easier, but it still hasn’t solved some of the hard problems. Because like one of my favorite examples is so there was a it was I think it was Trivi, if I’m remembering correctly. Trivi’s an SBOM scanner, vulnerability
James Matchett (14:20) Course, yes.
Josh (14:20) scanner, and and a friend of mine
Because they they trivi had just started doing some sort of they they were checking Sigstore signatures from how is it Rekor whatever, whatever the the ledger is. I I don’t remember all the terms at at the moment. And they added a signature to Trivi as them as their their login, right? Because they were like whatever at gmail.com.
James Matchett (14:46) Yeah.
Josh (14:46) And Trivy exploded because it wasn’t expecting like this rando person to have added a signature to this thing it was looking at. And I mean this
Like this is one of the challenges, right? Is is how do we still determine like who’s signing this? Because there’s nothing stopping an attacker from adding a signature, a Sigstore signature to the public ledger, right? For something that they maybe that they have no control over. But now you can say, like, there’s a signature in s in Rekor So it’s fine, right? But
James Matchett (15:13) Hundred percent. And that’s why
we saying Cloudsmith, you customers should bring their own key and therefore set up signing at every stage within Cloudsmith, because therefore if it goes through CloudSmith, you know that it’s safer to use. You know exactly where it’s come from. You don’t have a developer that, you know, might have misconfigured their endpoints. You it used to use Cloudsmith, they did a bash RC or they’ve been compromised, and suddenly there are endpoints pointing at the open source internet again, or even worse, a malicious package registry. So good signing at all stages just keeps you safe, because it’s not so much even
Can you do file hashes to make sure things are integral? You have to have that crypto signature to make sure that everything in there is safe to use.
Josh (15:49) Yeah, yeah. I mean, but I I I still like it’s getting better. But I I do think we have a long road ahead of us for that. Because like maybe five years ago, if you’d have been like, we have to sign everything, I’d be like, no way. Like out of the question.
James Matchett (16:01) Yeah.
Josh (16:03) No. Cause I mean, we were using what PGP signatures back then. It was terrible. It’s better,
James Matchett (16:07) Exactly.
Josh (16:08) but we still we still have work to work to do. But now I will say that some of the package registries are starting to kind of lean into this.
And I think it was, I think it was Packagist the the PHP, I think it’s PHP registry. I talked to them a while back. And like this is one of the things they’re working on. It’s how do we start signing this stuff? Because there’s there’s obviously, like you said, immense value in knowing exactly where it came from. Not necessarily like one of the things we have to be very clear about just because something is signed doesn’t mean it’s secure, right? We can only say, I can
James Matchett (16:37) Yes. A hundred percent.
Josh (16:40) prove this came from the place I think it came from. That’s it. That’s what we’re proving.
James Matchett (16:45) Exactly. And I think that’s how SolarWinds actually came about in twenty eighteen, if I’m not mistaken. You know, the attackers got into the build process, introduced
Josh (16:50) Yes. Yes.
James Matchett (16:52) their own dependency before the signing took place, and then suddenly you’ve got a malicious dependency in your build pipeline that was signed by your same crypto infrastructure and you’re like, Well, of course I trust it, we signed it.
Josh (17:03) Well and they even had they had a piece of it didn’t get signed and so it was flagging it was being flagged. And I remember the the the the support, the the solar wind support like, that’s fine, just hit okay. And like it was not fine. It was it was the opposite of fine.
James Matchett (17:16) Well that’s the
that’s the other side of the coin. You know, what do your security teams and engineering teams do if suddenly something isn’t signed but it appears in a much later stage of your build pipeline? Like practicing that fire drill is such an essential part of any secure software operation. You can put all of these security controls and checks and alerts in place, but if you’ve never practiced on how to respond to them, well you’re about to find out very quickly how to respond to it, you know, in an emergency situation.
Josh (17:40) Yeah. my goodness, I know, right? Like that’s so so I’m sensitive to this though, right? Because I’ve been doing security work like forever. And we plan for certain events, but also i sometimes the thing you plan for isn’t the thing that happens, right? Just because like it’s it’s madness out there and and who knows. Like this is one of our challenges as attack as defenders, right? Is the the attackers by definition are in front of us and and it’s almost
James Matchett (18:06) Always.
Josh (18:07) impossible. Like all we can do is chase them.
It’s just it it’s the nature of the beast.
James Matchett (18:13) Definitely. And I find when you don’t know how you’re going to have to respond to a situation, having more monitoring, visibility and data when the bad thing does happen can guide your response so much easier. So I’m talking about Shai Hulud, for instance. You imagine you’re the CISO or head of security at a software company. You find out that from your SBOMs so it’s already too late, that a number of developers have pulled this package. You know it’s in your final product, you know every developer.
within the last week who’s been at work has probably pulled this package, but you don’t know what version they’ve pulled, you don’t know when they pulled it, they don’t know where you pulled it from. And suddenly you don’t know how many laptops have been compromised, you don’t know how many credentials have been exfiltrated. And that’s why I love having this list of who pulled what when, because it makes your life as a responder so much easier. So rather than having to, you know and I’ve had to do this my my myself before at a in a different life.
deploying a bash script to every laptop looking for file hashes of log4j libraries to try and find out, that’s a project that’s vulnerable, we’ve got to fix that one. it’s more expedited in supply chain attacks now where the pre install script is an actual factor to exfiltrate credentials. You’ve got to run faster. You know, supply chain attacks in the past you could walk at. You could say, Yeah, I’ve got a week to fix this. Hopefully I I’m not compromised before then in production. But now it’s the exfiltration of credentials from laptops. You have to run at this. And that type of data
is gold dust. You you absolutely need it as a defender.
Josh (19:42) Yeah, yeah. Well, I mean, so for anyone who doesn’t know, like everything James just described, like Shai Halud has like a special bit of terror for me because actually I was at an org that we didn’t have some of this data, so we rolled every key. We’re like all the creds, we’re just rolling it all because we’re not sure what happened. And we weren’t affected by anything, but it was just purely, you know, we were we were being very proactive. But what what James is talking about, kind of just you know, putting all the all these pieces together is the attackers, they’re getting
They’re getting tokens for like GitHub and NPM and places like that. And and who even knows where? Cause like the the Trivi attack, the Shia Halut attack, that was like three layers deep because they got credentials off of someone for something else that eventually got them into Trivi and then they got into everything because this was a you know like a hyperinstalled package. And that’s what happens. You might think, this is no big deal, like this one token. But if that token leads to another token leads to another token, now you have an enormous problem, which is like
That’s the thing we often miss.
James Matchett (20:43) one thing that’s compounding this and I completely agree with your point Josh it’s the developers and maintainers of these open source projects essentially volunteers are being targeted by these threat actors they’re being
Josh (20:53) Yes.
James Matchett (20:54) phished their NPM credentials like some of them don’t even have 2FA enabled and at a talk I gave recently I talk about this thing called bus factor. Have you heard of it before Josh? for
Josh (21:03) yes.
James Matchett (21:04) anyone who’s listening who doesn’t know what bus factor is.
now HR refers, we use a different term, but I’ll I’ll give an example. Imagine you’re working on a really, really critical project at your company, and there’s one guy, a principal engineer, who is the guy. He knows everything there is for this project. What if he hit h what if he gets hit by a bus? How much trouble is your project in? So we sometimes describe people or projects as having a really high bus factor because there’s some critical people that if they got hit by a bus, or as HR prefers we say they won the lottery or went on holiday
the project’s in trouble. So I did a little bit of threat hunting myself and I looked for projects on the open source supply chain that had a high bus factor. That is to say, many millions of downloads per week, but only two to three people contributing to it, maintaining it. And you just have to think, are those two to three developers following security best practices? Do they have MFA? What if they got compromised? I mean there’s only one other developer who can then try to restore the project before NPM themselves might have to step in.
And we find NPM and PyPI packages that had hundreds of thousands of downloads per week, but only two people devel developing and defending them. And even worse, if you look at transitive dependencies, there’s these really, really hidden but insanely popular packages that don’t show up on the leaderboards because they’re not directly downloaded. So if I was a threat actor, I’d be going out there, I’d be looking for these projects, I’d be looking at these maintainers, and I’d be targeting them.
And I think we as contributors to the software ecosystem have to be doing more to help these maintainers stay secure.
Josh (22:39) Okay, I wanna address the the single maintainer problem, the bus factor, as you say.
James Matchett (22:43) Okay.
Josh (22:44) It is gigantic in every open source ecosystem. Like I I’ve been looking at this data for literally years at this point, and it is alarming when you look at it. It is almost all all the packages, and it doesn’t even matter how you slice and dice it. I mean, this is one of my favorite things. I’m actually giving a talk next week about this, right? I have some graphs, right? If you look at cause my favorite was this happened, this came up, I think it was at an OpenSSF
Meeting once where I showed a graph that shows basically almost everything is a single maintainer in open source. Like it’s just it’s it’s the reality of open source. And they’re like, but it w I only had NPM data. And then someone was like, that’s just NPM. NPM special. So I found the Python data and pulled it in and the graph is literally identical. Like it’s exactly the same. It’s just the numbers are smaller because Py PI versus NPM.
And then someone’s like, that’s just Python and NPM. Ruby’s better. Nope. Same, same graph. And it literally doesn’t
James Matchett (23:34) Ha ha ha.
Josh (23:36) matter what ecosystem you look at, or even if you look at like the number of downloads where if you say, okay, only give me the top 50%, give me the top 20%, give me the top 10%, literally the graph doesn’t change. It is almost terrifying how much of more modern infrastructure is like it’s like that XKCD cartoon everyone always references.
James Matchett (23:53) yes.
Josh (23:53) Except it’s not one package by a person in Nebraska. It’s like thousands of packages.
By random people scattered across the globe no one knows about. It’s it’s horrifying.
James Matchett (24:05) And something really interesting to look at that might horrify you and your listeners a bit further. Look at how those packages are connected. So if you’re able to compromise
Josh (24:12) yeah.
James Matchett (24:13) one of them, follow that tree down and see where that package you’ve just compromised is a transitive dependency of.
Josh (24:19) Yes.
James Matchett (24:20) And you can see how I mean some in my research, some of these packages had a blast radius of 322%. And you’re probably thinking, well, how can you cover more than 100% of the ecosystem? These packages were referenced multiple times in other packages.
That’s the blast reduce effect of
Josh (24:34) Yeah. yeah. Yeah.
James Matchett (24:36) this.
Josh (24:37) No, for sure. For sure. It’s it’s mind boggling. Andrew Nesbitt, who runs ecosystems, has done some work on this stuff. And and it even gets weirder because when you start crossing ecosystems where, for example, like Debian repackages things, you know, you have like Glib C and a bunch of stuff. OpenSSL is one of my favorite. OpenSSL is in NPM packages, it’s in Python packages, it’s in Ruby packages. It’s everywhere, right? Like everyone’s using open SSL binaries. It’s it’s horrifying. Anyway. Okay. Okay. I have one more data point.
I wanted to bring up before we run ourselves out of time, which we’re doing very quickly, which you have you have currently
James Matchett (25:09) I’ve got time.
Josh (25:10) thirty one percent of teams allow AI agents to pull dependencies without human intervention. I’m like, there is no way it’s only thirty one percent
James Matchett (25:16) Ha ha ha.
Josh (25:17) when I read that. Like no way.
James Matchett (25:19) Absolutely not. And you know, I think this is the same problem we’ve had for the past ten years, only happening faster. And by that I mean the old way of programming used be you hit a problem, you look at Stack Overflow, you copy a code snippet, it says, import this random NPM package or PyPI package, and you don’t think twice. You do the NPM install or pip install and it’s done. This is the same problem but faster. You just ask AI to do it for you. It’s one less click. And this is happening at insane speed.
And I mentioned slop squatting
Josh (25:48) Yes.
James Matchett (25:49) earlier. The one really cool thing is they found the biggest degree of package name hallucinations happened whenever the model confused a package which existed in PyPI for instance but didn’t exist in NPM or vice versa.
Josh (26:00) Mm.
James Matchett (26:01) So it looks like a legitimate package name. And you might say requests, of course I know requests, but request has a different meaning altogether in the NPM world.
Josh (26:09) Right. man, I never even thought of that. But yeah, I mean I I totally get that. And and like NPM at this point is so big. Basically every word is a package. Every English word is a
James Matchett (26:19) Ha ha ha.
Josh (26:20) package name. It’s mind-boggling. But okay. man, that’s anyway, okay. All right. So I wanna we need to end on a happy note because I feel like we’ve just talked about the terrible problems that our ecosystems have. So
So James, give us some confidence. Like tell us the good
James Matchett (26:36) Ha ha.
Josh (26:37) things that this report lets us know, or maybe just some good things that are happening around. And because I will say, even though I do feel like reports like this often are scary, like this
James Matchett (26:47) Ha ha.
Josh (26:47) data is important because we have to understand these problems. But I know
James Matchett (26:51) Yeah.
Josh (26:51) like that’s not the story, right? The story isn’t just like be scared. There’s there’s a lot more happening out there.
James Matchett (26:58) Yeah, I suppose the most reassuring thing is people are doing something about it. And that is
Josh (27:02) Yes.
James Matchett (27:02) a really reassuring space to live in. I recently gave a talk on the Cyber Resilience Act, that’s the act the European Union is bringing in. And the cliff notes are this. If you are a software manufacturer and you sell software into the European Union, there are certain rules that you must follow, kind of like GDPR, but for building software. Now the first phase of this went into effect on the eleventh of September, and that was the early reporting deadline. And all that really means is if you are aware
of an exploit in your software that’s actively being exploited, you have to report it to an organization. Depending on where you’re registered and all that good stuff, that mean might mean different things, but you have to report it to a centralized organization. And then they can use that information. You’ve got 24 hours to do the lightweight report saying, hey, I’m company name XYZ, I’ve been breached, and then you’ve got seventy two hours to fill out the bigger report saying how bad it is, how many customers, things like that. The more interesting thing that’s coming into effect is the annex one requirements of the CRA. That is on the eleventh of December.
And that really means for every build.
Josh (27:58) Yep. Well, eleventh
of December next year, twenty twenty seven, not twenty twenty six.
James Matchett (28:03) I believe so. Otherwise I put my team to a really early deadline. But it’s I’ll need to double check that date. But eleventh of December is the correct one. I’m not sure which year, but I’ll double check that one very quickly. but the cool thing about this is, you know, every build that you make has to have an S-bomb of every package that goes into it. And if it can be at attestated and have provenance with it, fantastic. But you’ve also got to include, you know, every security scan relevant to that build and your decision for why you still think it’s safe for that to go out.
And any market surveillance authority can request that information. And if you don’t have it or you haven’t been doing it, you can expect fines. And this is all in the service of keeping users of software safe. You know, software is a shared village. If you compromise one vendor and you can hop to another vendor and compromise them, everyone suffers. And this is just in the service of making everyone a bit more secure and a bit more cognizant of the fact that if you’re building a software solution,
There is no way that every line of code is your own and you wrote it yourself. Very rarely.
Josh (29:03) Yeah.
goodness. I mean, there’ve been I mean, we’ve seen years and years of reports saying like what eighty percent, ninety percent of applications are open source. And I think that’s completely understood at this point. But yes, I agree with you. I’m optimistic the CRA will help with this because I think to date, most software development has just been absolute, you know, wild west, Mad Max, do whatever you want, no one cares. And and I’m I’m cautiously optimistic. So I hope so. I mean, obviously you’re in Europe.
So you have a different view of it than I do. I’m in the US. It is funny though, because I go talk to companies in the US and I’m like, what are you doing about CRA? And they’re like, What? Like, right.
James Matchett (29:40) Ha ha ha.
Josh (29:42) Then I’m like, explain it. They’re like, that’s Europe. I’m like, do you sell products in Europe? Well, of course we do. Well, you have to care. So it’s gonna be a a hill to climb over here, I think, but I’m I’m hopeful. I’m hopeful. All right.
James Matchett (29:56) Definitely. And
I’m hopeful too because just from my position, we’re seeing a lot more of these companies pay attention to the supply chain. It’s no longer this quirky little thing where they get their useful to use packages from. It is a key and critical part to how they build software and they’re interested in helping keep everyone secure. And as a full time nerd that makes me really happy to see.
Josh (30:15) It’s it is funny we’ve been complaining about this for so long. Like why would anyone care?
James Matchett (30:20) Ha ha.
Josh (30:21) Now they’re caring, it’s like, all right, I guess we have to do something about it now. Like it was much easier to complain than it was to actually do something, but
James Matchett (30:29) Exactly.
Josh (30:30) all right, James, what should anyone do? Let’s let’s let’s land this plane. What are next steps? What do you want people to know before we hit stop?
James Matchett (30:37) Yeah, if you’re a CISO or CTO and you have no supply chain attestation whatsoever, and by that I mean put yourself in the shoes of well, just imagine a situation, a good hypothetical. You just get told that there is a compromise or a package you use and it’s stealing credentials off your developers’ laptops. Go. If you couldn’t
Josh (30:54) Yeah.
James Matchett (30:55) answer the question within fifteen minutes of who in your company pulled that version of that software, look at a decent supply chain solution. It should be set up so that every developer pulls the packages through that system.
You need to have that golden log of who pulled what when. And that gives you enough monitoring, enough response. You can actually mount a semi decent response from that point. And once you get past the threshold of monitoring, you can then move into control. You can set up your policies, you can make the rules, and you can basically say, I’m happy for these things to exist in my world, and I’m not happy for these things to exist in that world. And by that I mean you have a set of rules for development, set of rules for staging, a set of rules for production, and that’s you golden for developing software.
But just to cap it off, set up good signing, generate your SBOMs, actually go through it and understand the format, see what’s in there and how to make sense of it. Because otherwise you’ve just got a fancy collection of text that no one’s going to look at. And r run run a good, you know, fire drill every now and again. Pretend that one of your packages compromised. Pretend five developers have pulled it. What is your response plan? Practice it. I mean, if you have it written down in a conference doc or a notion doc somewhere, you’ve got a nice bit of paper to look at. But until you practice it and rehearse it.
That’s when it gets real.
Josh (32:07) Yeah, for sure. man. I love it. All right. You’ve you’ve restored some of my hope, James. So thank you for that.
James Matchett (32:12) Ha ha.
Josh (32:13) yeah, I guess exciting times. Thank you. Thank you so much. Like this has been an awesome chat. I’ve really enjoyed
James Matchett (32:19) It’s been good fun. Thank you so much.
Josh (32:20) Awesome. Until next time, my friend.