Josh chats with Charlie Erickson, a security researcher at Aikido Security. We discuss the recent NPM supply chain attacks that affect hundreds of packages. Charlie shares his experiences dealing with recent security breaches, the challenges of maintaining trust in open source software, and the importance of proactive measures to safeguard open source. The rapid pace of change is impacting our security practices and what steps can be taken to foster resilience in the face of evolving threats.
Episode Links
This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.
Episode Transcript
Josh Bressers (00:00) Today, open source security is talking to Charlie Erickson. He’s a security researcher at Aikido Security. Charlie wrote a bunch of blog posts during the NPM debacle that right before I hit record, we realized was like two months ago only, a month or two months, like which it feels like it’s been years and years. But I jokingly told Charlie that his blog posts are what kept me sane during this event. So Charlie, welcome to the show, man. I truly appreciate it.
Charlie (00:24) Thanks, and great to hear that somebody stayed sane during that because that was a tough, tough month.
Josh Bressers (00:31) Oh, it was bananas. All right. So let’s just start out. Tell us who you are. Kind of tell us what you’re doing and tell us why you’re here and we’ll go from.
Charlie (00:37) Yeah, so I’m Charlie Erickson. I’m a security researcher, primarily working with Aikido security. Basically what I do these days is dealing with the firehose that is supply chain security, open source security. Basically analyzing all of the stuff that we, all the code coming from all of the open source ecosystems, trying to catch all the bad stuff.
Josh Bressers (00:54) Yeah.
Charlie (01:07) Yeah.
Josh Bressers (01:09) And they’re gigantic. It cannot be overstated how enormous these are. I have a handful of blog posts I’ve written where I take the data from ecosystems, ecosyste.ms Andrew, Andrew Nesbitt’s thing. He’s been on the show. And when you look at all the graphs, they’re all growing at an exponential rate. And there’s like just literally tens of millions of packages now. It’s completely bananas.
Charlie (01:10) It’s, yeah.
Yeah. Yeah, it’s crazy. mean, so also just for background, since I think it’s helpful, my background is actually more on the offensive security, but also teaching developers how to hack. ⁓ So really going back into the more defensive side is a relatively new thing for me, even though, yeah, I did a startup during, yeah, basically teaching to all the top 10 and things like that, working with developers, how to, did a lot of source code review.
Josh Bressers (01:41) Nice.
Charlie (01:58) helping development teams, how to build secure code and things like that. So for me, actually really diving this deep into the supply chain stuff is still something that I just started this year. But ⁓ frankly, at the point in time we are in right now in the world, I’m really seeing how this is becoming really, really important because it plays into so many narratives across the world. And it’s kind of scary once you step back and look at it, how
Josh Bressers (02:20) Yeah, yeah.
Charlie (02:27) big of a thing this is now that actually has a significant also geopolitical impact, unfortunately.
Josh Bressers (02:33) Yes,
yes. And okay, there’s a lot to unpack there, but before we do, why don’t you explain kind of the month of September to anyone who maybe doesn’t know, which we’ll assume most people in the modern times know, but we’ll say for historical archival purposes.
Charlie (02:38) Yeah.
Mm.
S, the story of me almost burning out again. It started off with the NX breach and singularity thing, more or less. I was sitting at Utrecht station and I got a message from somebody on my team going, hey, there’s something big going on. And it was all of the NX packages getting compromised. And NX is a package manager for like AI something,
And seeing, all right, because we kind of started to realize what was going on slowly. Like, ⁓ first it was these packages and it was doing all kinds of things that we really hadn’t seen before. Like stealing GitHub tokens, posting stolen data to GitHub in public repositories. And then we started to see it change. We actually saw multiple variants of it that would also then start to turn.
private repositories in GitHub to public. And that was a really ⁓ scary moment actually, right? Because we’re like, well, shit, what’s gonna be the ramifications of all this data being made public? And this kind of Robin Hood mentality of just like breaking and entering and just smashing everything.
Josh Bressers (04:15) Yeah.
Charlie (04:16) ⁓ and then like, it going to continue? But yeah, soon after that we had, I think what was probably one of the most scary moments. Well, no, actually it wasn’t spoiler alert. ⁓ when we then saw, ⁓ I was one of the first ones to notice, ⁓ a bunch of really big packages on NPM, like a debug and chalk were
Josh Bressers (04:33) It’s coming! ⁓
Charlie (04:45) compromised. Luckily, we managed to get that one on control really quickly because I happened to have a Blue Sky account and the guy that was compromised, Josh Junon ⁓ managed to catch him at the right moment where he saw it within like an hour or two. We did a really, I thought, really fantastic interview with Josh afterwards talking about like the more human side. I really would recommend people watch about what actually happens when a maintainer like this gets compromised. ⁓
Josh Bressers (04:59) I know.
Charlie (05:15) So it was like 2.6 billion weekly downloads that all of these packages had, right? I think by maintainer, he must be like one of the most downloaded maintainers out there. I would imagine. Wiz, they estimated that about 99 % of their cloud environments used one of these packages and about 10%, yeah, and 10 % had a compromised version within hours.
Josh Bressers (05:20) Yes.
Everyone had one of them. Yeah.
Wow.
Charlie (05:45) Yeah. ⁓
Josh Bressers (05:48) And we should add that was like what six hours I think from start to finish from the time you jumped in until NPM had basically gotten rid of all the malicious packages, which six hours is a pretty small window really.
Charlie (05:53) Yeah.
Yeah, it was, but it was still quite a while after that, that Josh eventually got his account back and things like that. So it was still messy. Now, this was one of those incidents, right? Where we made a bunch of fun out of it afterwards because like this was by download count, certainly in sort of open source packages ecosystems, definitely the biggest. And we got really, really lucky. Right. I, that week I spent hours talking to journalists.
Josh Bressers (06:08) Yes, yes.
Charlie (06:32) It was hectic as heck. Because we got lucky in the fact that these attackers were too narrow-minded. Having the NX and singularity situation in mind, given the scope of the impact, like of what was breached, ⁓ if they had done something similar as what singularity guys did, I
Josh Bressers (06:34) man, I can only imagine.
Charlie (07:02) really don’t even want to speculate about the implications of it. ⁓ It would have been ridiculous. So we got really lucky and this was a simple phishing attack, like very simple.
Josh Bressers (07:05) It would have been ridiculous, yes.
Charlie (07:16) And the fact that it took that little and it took just catching the right guy at the right moment. We, I thought like, no, this is ridiculous. We cannot be that vulnerable. Like that is not okay. I got mad. And I wrote like some kind of scaling words in one of my blog posts of like, fuck people like this, this, this can’t be allowed to happen again.
Josh Bressers (07:45) right.
Charlie (07:48) One week later.
I get pinged on Slack again. Something is going on.
And sure enough, this was Shai-Hulud
Josh Bressers (08:04) Yeah, the big one.
Charlie (08:05) Yeah. And ⁓ this was the same attackers as Singularity. So it was the same deal of the malware, right? So they had compromised tokens and I’m pretty sure it was tokens that they had left over from the first attack, I think. And it was uploading credentials. ⁓
Josh Bressers (08:25) Sure. Yeah.
Charlie (08:33) onto GitHub, it was turning repositories public. ⁓ There was some really cool techniques actually in both of them that I thought were novel, right? So in Singularity, they were trying to use ⁓ local like LLMs and stuff like that to iterate your file system and find credentials. Very cool. ⁓ It didn’t really do a lot, I think. In the Shai-Hulud thing, they tried to use TruffleHog which again, legitimate tool, really great tool, open source, all that sort of stuff.
Josh Bressers (08:50) Yes.
Charlie (09:03) to try to find credentials. Now, this is where it gets really scary. They also had a worm in there. and they had the GitHub actions things to try to persist and stuff like that. yeah, I think that was kind of successful actually. I think GitGuardian did a little bit of analysis on it and it definitely had, like it definitely worked. But the worm thing.
Josh Bressers (09:28) Yeah. Yeah.
Charlie (09:31) That was scary because what it would do was that it would look for your NPM token in your environment. It would iterate all packages that you had access to. And it would take the, first it was top 10, but then they changed it to top 20 packages you had by download count. It would download the package, unpack it, put itself into it, update the package.json with a pre-install script, and then upload that.
Josh Bressers (09:58) What does the pre-install script mean in this context?
Charlie (10:01) Yeah, so NPM has what is called lifecycle hooks. So if you are a package that requires some sort of setup before you actually will work, like you need some dependencies or anything like that, you can define a script and the package.json that will get run either pre-install, add install or after installation. A lot of malware will use this, right? With callbacks and things like that. Yeah.
Yeah, so it basically replicated itself, right? Completely classic worm. That’s why it’s also called Shai-Hulud The repositories that it created on GitHub, which is stolen stuff, we call Shai-Hulud. Shai-Hulud is the worm from Dune.
Josh Bressers (10:49) Yes, which I love.
Charlie (10:50) Yeah. Um, and, there I, like that day I actually kind of was like, uh, beating myself a little bit because like, some journalists I had gone, if, if they had taken some of the concepts from like the singularity attack and done that at a larger scale and like smash and grabbed, we would have been in real trouble. And this became so close.
Josh Bressers (11:18) Yeah,
yeah.
Charlie (11:20) Now, what was really interesting, we did a bunch of analysis of it afterwards. It turns out they actually seeded most of the compromised packages themselves because we could tell forensically in the packages, like in the new archives, because there were some bugs in their code.
Josh Bressers (11:28) ⁓ interesting.
Right.
Charlie (11:39) So we could actually see that it was the attack uploading most of the packages and actual community spread was quite minimal.
but now they have even more credentials, right? And you could tell from the timeline of things that they seeded a couple packages where hope and would catch on didn’t catch on. They also found out that they had some bugs in their codes that just didn’t work initially. So they kept trying to get it to actually start catching on, but didn’t manage to do it.
Josh Bressers (11:53) Right, right.
but it’s probably pretty hard to debug a self-replicating worm in a way that doesn’t get you banned,
Charlie (12:21) Yeah, it is, but still, it was some really dumb mistakes.
Josh Bressers (12:25) Yeah. Yeah. And you have a blog post about this and I’ll put links to all of, all of Charlie’s blog posts and all this, which again, like I said, I read them all while this was all ongoing and they were incredibly helpful. like, thank you so much for all that effort. I know it’s a ton of work, but okay. That is a scary story. And I guess here’s the thing I keep thinking of is this was like a month ago ish a little more for us in where we are right now.
Charlie (12:38) Thanks.
Yeah.
Josh Bressers (12:54) And I feel like an enormous amount of the world has been like, thank goodness that’s done. And they’ve kind of gone back to normal. Would you agree with that or not?
Charlie (13:05) One of the blog posts I wrote is, cannot, we cannot go back to acting as business as usual. And frankly, ⁓ I’ll be the first one to say I’ve got ADHD, I am time blind. ⁓ But I just think that we have way too short of an attention span.
Josh Bressers (13:12) we can.
I don’t think we have a short attention span. think this is the, mean, so this is one of the challenges security people have actually. like, is a very human thing to ignore like very bad problems happening around us. We find ways to justify many, many things. I mean, like climate change is the ultimate story here where we’ve convinced ourselves it’s fine for a variety of reasons. And I think this is a similar story where it is not fine.
We have been, mean, obviously security people have been screaming about this for years and years and years at this point, but I don’t know the impetus that is going to create the change here. And that’s the thing that like terrifies me and worries me is I don’t know if anything can happen that will be so bad, we will actually change.
Charlie (14:16) Yeah. and that’s where we have a little bit of a, this, shitty kind of, ⁓ dichotomy in terms of what will actually trick a change, right? Because we got really lucky that this did not break mainstream news. If it had been slightly worse and actually costs like proper outages, like it could have in many cases, like think a crowd strike kind of situation where suddenly systems are crashing.
Josh Bressers (14:45) Right, right.
Charlie (14:46) It could have done that if it had spread more or had a larger reach
Josh Bressers (14:52) Yes. Yes. I mean, this is where I think breaking the barrier to like normal person news is maybe a good, it’s a good way to think about this because like Log4j did that. And I don’t know if I’d say there’ve been massive change, but I do think it drove maybe more attention. I think Heartbleed was a huge one, right? I remember when Heartbleed happened, that drove, I mean, it drove a lot of interest, but again, I don’t think we’ve seen massive change necessarily, but regardless.
I don’t want to dwell on all this. Here’s my question for you, since I have you here right now is like, what, what are the things we can start thinking about that can make a difference in a context like this? Cause when we’re talking about like open source, the power of open source is just that like random people can do things in the ecosystem. Right. But at the same time, the downfall to this is random people can do things in the ecosystem. Right.
Charlie (15:49) Yes. Yeah. And this becomes a really difficult conversation, right? Because to me, there was a friend that pointed out to me from a more corporate perspective, sort of a regulatory or compliance perspective. When you develop software internally, right? You have a lot of requirements around how you do things, but an auditor is not going to come into your company
and look at how you use open source other than maybe SBOMs and some stuff like that, and be able to have any opinion on it because it’s outside of the scope of what they can say anything about. There’s no way an auditor is gonna change, like you can’t use open source packages, right? That’s ridiculous. So yeah, so what it effectively is from a, if you assume that the standard should be what corporations have to do,
Josh Bressers (16:36) my goodness, that would be hilarious.
Charlie (16:47) then effectively open source could be argued to be trust washing, right? You’re basically saying it is not made by us, so we can just ignore it.
Josh Bressers (16:58) which is what they do.
Charlie (17:01) Yeah.
Josh Bressers (17:02) I mean, in fact, so I have a funny story about that for you is there is a, a government program that I worked with at one point where they were taking open source software. were taking commercial software and obviously it did justify security findings in the software, you know, the very normal thing. And one day I was like, Oh, this problem is similar to that problem. This open source thing had, um, can you show me your justification for that? I want to understand what, what, like how this worked out. The justification was.
It’s open source. And I’m like, what? This is ridiculous. Like you’re making me do all this work and you’re just giving open source a blanket like pass. They were like, yeah. And I’m like, I don’t even, I don’t even know what to say, you know, but it happens.
Charlie (17:44) Yeah. ⁓
it, really does. So there is this problem, like to me, having had a lot of discussions around this, I think there are some really legitimate questions that almost goes counter to the whole kind of principle or ethos behind open source, which is what is trust and what does it require for you to have trust in something?
And when it is random people on the internet right now, trust is very hard to build takes a long time and very easy to lose. And it doesn’t just happen. Like once you start to lose trust in open source as a concept, which I think we could be going in that direction, things will have to change. And I don’t know. We’ll like that change.
Josh Bressers (18:39) I don’t think they will. And so there is a fellow I talk to on a pretty regular basis. Are you familiar with the blog post, I am not a supplier by Thomas de Pierre? Okay, I’ll send it to you later. You should definitely read it. This happened during Log4j but I talked to Thomas quite a bit. He’s been on the show a couple of times. And he keeps bringing up a really good point every time. I say things like what you just said about, people are gonna lose faith in open source. And his response is, no, they won’t because they want our free stuff.
Charlie (18:49) No.
Josh Bressers (19:07) And so this comes back to that human ability to ignore problems of the free stuff of open source vastly outweighs any potential like cognitive trust issues I might have. The actual danger we have with open source is developers burning out because they get sick of all the crap, not corporations not trusting open source.
Charlie (19:11) Yeah.
Yeah. ⁓ but I also think that’s not true because if we look at like the Josh Junon debug chalk compromise, they also made mistakes and they broke build pipelines in many cases. So there is a cost to this in terms of cleanup.
Josh Bressers (19:49) Sure. yeah, yeah.
Charlie (19:51) ⁓ there is a cost. So with some of the earlier breaches this year, like much smaller ones, I actually saw at these two projects that were compromised, decide to go, no, I’m done with this. I am just going to deprecate the package and I’m gonna. Yeah. Not going to maintain it anymore. So now you had like fairly popular packages, like, ⁓ from an Aikido perspective, I saw we had fair few customers use this. They now.
Josh Bressers (20:11) Yeah, yeah.
Charlie (20:20) to stay secure potentially, we’ll have to go and rip out these packages and find something new. Like, and also just in terms of externalities or if something does happen, well, the ultimate cost is on you. The corporation actually decides to use this package. So yeah, you don’t have an immediate cost. It is free. But you have a cost that could eventually come to
⁓ fruition because of that decision. It is just latent in time.
Josh Bressers (20:54) It’s technical debt, right? It’s the typical technical debt problem. Just kick the can down the road till you can’t kick it anymore and then we’ll deal with it, which is every tech company I’ve ever worked for ever. mean.
Charlie (20:56) Yes, it is.
Yes.
Exactly.
It’s, it’s a time to market, sort of, like sort of cost benefit analysis. And I think it’s reasonable that we came to this point, right? It makes so much sense. But let’s at least have an honest look at ourselves and go like, with so much changing in technology right now. I mean, I’m relatively young, right? I
I haven’t seen this big of a transformation certainly in my lifetime, but I see what’s happening with like AI, LLMs and stuff like that. And we are, in my opinion, seeing a seismic shift that I don’t think we really comprehend yet because we can make change in the world now at a pace and ⁓ size that is frankly not possible for human to comprehend.
Josh Bressers (22:01) Yes. Yes. Agree. Now I’m, I’m old, so I’ve seen a lot of things over the years and yeah, I mean, you’re absolutely right. It is the, the pace of change is incomprehensible. Now we thought it was bad in the early two thousands. That is nothing compared to what we’re dealing with right now. And I mean, this is part of the problem is like, I think this is part of the reason why we have Shai-Hulud happen. Holy crap. The world is ending. And a month later, like no one even remembers what that is. Right.
Charlie (22:06) Uh-huh.
Josh Bressers (22:31) which is as a security person, I don’t know what to do about that. I don’t think there’s anything I can do, but it’s also, it’s like, it’s frustrating to me, but also I’ve got my new fires to fight. So I’m like, whatever, good enough. I’ll think about that some other day.
Charlie (22:46) Yes.
Exactly. And you can think about it some other day, if everything goes slowly, right? When you decide to upgrade your packages, that is not just done automatically, right? So what we are seeing now, and I think again, trust here becomes a really interesting and relevant concept, because trust is built over time. Trust is consistency in doing what you say you’re going to do over time.
Josh Bressers (22:56) Right, right, right.
Yeah. ⁓
Charlie (23:18) And I think PNPM, they hit the nail on the head very beautifully with the features introduced around the same time as, I think it was the whole Quicks debug chalk situation, where they introduced a setting that allows you to say, I am not going to install a package version if it is less than X amount of seconds old. Right?
Josh Bressers (23:33) Yeah, yeah.
Yep, yep.
All for anyone who doesn’t know, tell us what PNPM is.
Charlie (23:49) It is a package manager. forget if it’s like a wraparound NPM or how it is, but it’s basically an alternative to NPM ⁓ for doing more sort of project or workspace or monorepos basically.
Josh Bressers (24:04) Yeah, yeah, and it has some really cool security features.
Charlie (24:07) Yes. Yeah. I was very impressed. they, those guys, huge props to them. They really, I think everybody should adopt this.
Josh Bressers (24:15) Yeah. And so from my perspective as a security geezer, I can tell you if it’s not like a default in the tool or it’s not built into the technology, people just don’t use it. I mean, this has been the story over, you know, decades of security is saying, you should do this. And they’re like, sure. I’ll do that tomorrow. And then tomorrow and obviously never happens versus if it’s built into the tool and especially if it’s a default, then it creates change. And I think this is one of those examples where PNPM I’m adding this ability is it’s a game changer, quite frankly.
Charlie (24:45) Yes. And what it does for us, it is that it slows everything down. And the, the, the, the sort of the decision that we have to make in my view is either NPM and PyPI Well, I’m saying mostly NPM because they are the biggest by a crazy magnitude. Yeah. ⁓
Josh Bressers (25:10) Yes, yes, it’s ridiculous.
Charlie (25:14) Either they have to really step up their game and rather than having, well, whether it’s like Aikido and all of the 10,000 other startups and companies that are actively investing in this space to try to protect the ecosystem, they will, should engineer stuff to really, I mean, the amount of like really obvious stuff that they can catch with a regex. It’s kind of ridiculous if you ask me, but I understand also from their perspective and especially with the open letter from,
a lot of the ecosystems out there saying, hey guys, you’re using this for free basically. And you’re expecting us to do stuff like when this is a cost center for us, like come on, this is not fair. And I understand that point fully. But at the same time, right, there’s some stuff that just really needs to be done and we need to figure out how do we do it? So either NPM and also ecosystems should…
probably do a little bit more to try to at least catch the most obvious stuff and add in defenses so that the really simple phishing stuff doesn’t happen. To NPM’s credit, I think they have taken some really great steps, but there’s still a lot that I think can be done. The alternative is that if it’s not in the pipeline of publishing a package directly on the sort of supply side, then…
You are relying on all of these companies that are doing this commercially to protect the ecosystem. And for that to really be effective, you need to slow things down to a point where the ecosystem has a chance to defend itself before spread can occur.
Josh Bressers (27:08) Okay. Let me ask you the, the question I hear all the time. If we slow down and people aren’t installing the latest and greatest as fast as possible. Now, how are we going to catch? Cause the reason we catch the attackers quite often is there’s like that, you know, like just initial pool of victims that they get infected or whatever. And then we’re like, something’s wrong. Right. If everyone’s waiting two weeks.
Charlie (27:12) Yes.
Josh Bressers (27:37) install a package, whereas our initial, you know, patient zero, so to speak, to figure this out.
Charlie (27:45) ⁓ well, those are in the ecosystem of, well, how you would implement it is the default for everything would be anything less than a day old. It’s not served by default. You have to opt in, in order to get those packages. So it just be a flag saying, am I willing to, you know, ⁓ fly the edge of my seat?
Or do I want to be a little bit more conservative, right? 24 hours is not a lot.
Josh Bressers (28:18) That’s fair.
Charlie (28:18) So
⁓ the same mechanisms would still be valid, right? So of course, we at Aikido, we basically pull all packages, right? For NPM, we will often front run the registry itself and start to pull packages before they’re actually accessible. That happens really often because there’s a big delay from when there’s a notification on the change stream until you can actually access it. ⁓
Josh Bressers (28:35) Nice.
Yeah.
Charlie (28:46) And we run the static analysis, right? We started to do dynamic analysis. So we will catch most of that. we, I’m really happy with the detection rate that we have. ⁓ Like even stuff like we detected a breach of the XRPL or Ripple ⁓ sort of crypto packages where they had like a backdoor that would exfiltrate private keys.
And that is such a small change that can look really benign, but our stuff caught it. And I was like, wow. It was one of the first ones was like, actually, I wasn’t a hundred percent sold that this was actually doing what we were hoping, but it really does. ⁓ so yeah, there’s some stuff that is harder to detect and you really need to have more eyes on it, but fine. Lots, like all the really, ⁓ the really bad stuff.
We know how to detect right now, as long as it doesn’t change too much.
Josh Bressers (29:45) Sure.
Okay. So this is a fair, well, the attackers will adapt. Obviously they always do. But so you’re saying right now that if we wait a day or two to install the latest update from something, the assumption is that there are companies like Aikido that are looking for malicious things. know like, I talked to Brian Fox from Sonatype a while back and I know Sonatype is doing similar things in the Java ecosystem.
And that is kind of what we’re hoping will stop these attacks is we will have defenders paying attention, then telling us like, Hey, don’t use that. And then if we’ve waited two days, theoretically, we don’t get wormed because the things that need to happen to clean up these messes will have happened by that time, which I mean, that, that feels reasonable to me for the time being. Yeah.
Charlie (30:42) Yeah. Yeah. Of course, there’s a lot of other aspects and of side concerns to ⁓ these kind of ecosystems and how you distribute code. There’s a lot of packages out there that will actually then, when you install it, it then goes and fetches a binary from GitHub. Like, come on, guys.
Josh Bressers (31:04) Yep. Yep.
I mean, that’s like how Python wheels work, right? Where they come with prebuilt binaries inside of the wheel.
Charlie (31:15) Yeah, but like if you if the actual package that you download from the registry is not the actual package
Josh Bressers (31:21) you mean where it’s like wgetting or curling something down during, ⁓ yeah, that’s the worst agreed.
Charlie (31:25) Yeah.
Right. And it also speaks a little bit to sort of the concerns from the NPMs and PyPIs of the world of like, Hey, we’re basically a big old CDN for you. Right. Like we need to be able to reason about where software comes from. And if that is not the packets that you download, and if it’s going and fetching additional code that is not, that is not immutable, we’ve got big problems because we can’t reason about security then.
Josh Bressers (31:38) Yeah.
Yeah. Yeah.
Charlie (32:01) So there’s still a lot of other ways that still you start to run into problems where we won’t be able to detect it because they can change behavior of code through a side channel. Where it’s like, okay, they just like don’t turn on the endpoint for 24 hours and then it will pass right through. So this.
Josh Bressers (32:18) Right, right, right. Which is a conversation.
Let’s save that one for another day, but yes, that is a fascinating problem in itself is some of the crazy things that we see. So, okay, Charlie, we’ve come to the end and I’m not feeling real optimistic right now. So give me the optimism I need in your closing.
Charlie (32:28) Yes.
Okay. So I do also want to tag on for next time, ⁓ talking about obfuscated code. That’s a delicious topic, commercial software on public, ⁓ open source re registries. The really optimistic thing from my perspective is, ⁓ I’ve seen NPM take really big and quite bold steps. And I saw a lot of chatter on LinkedIn, especially when they announced the initial thoughts on what they were going to do, where I saw people going like,
Josh Bressers (32:47) yeah, yeah. Yes.
Charlie (33:12) this is ridiculous, like this is way too, like things are gonna break. And I’m like, well, yeah, this is gonna maybe be a little bit painful, but I think it’s the right thing to do because you have to change status quo. So I’m actually really happy that this ruffled feathers, that people are like, no, you can’t do this. I think that’s a sign that it had to be done.
Josh Bressers (33:15) I know.
Yeah, yeah. I mean, it’s the old if no one’s complaining about what you’re doing, you’re not doing anything interesting, right?
Charlie (33:40) Exactly. Now, is it enough? That is the big question. It’s not, but at least we are seeing that people are willing to change things and actually have a conversation about this. I think having conversations is one of the big tools that we have about this, because how do you change public opinion?
Josh Bressers (33:45) Well, no, no it’s not.
Charlie (34:07) We talked earlier about like if this had broken mainstream news, first of all, it would have been incredibly embarrassing for us as a industry, right? This just looks bad in every single way you can reason about it.
Josh Bressers (34:20) I
feel like we’re pretty shameless at this point, so.
Charlie (34:23) Yeah,
well, okay. I’m not sure that’s a good thing. We maybe should have a little bit more shame. ⁓ But the way that you change, like the way that the world changes is conversation and starts chatting with your friend over a cup of coffee or something like that. And as people open and change perspective, if you’re like, I hadn’t considered looking at it this way.
Josh Bressers (34:26) No, it’s not.
Charlie (34:53) We’re like, ⁓ things could be different. That will start to disseminate into the public consciousness and go like, ⁓ yeah, maybe we need to change something. It actually would be much better for everybody because there are so many things also that are not great now, like talking about like mental health aspect of like, especially like the Josh Junon conversation. Yeah, we are kind of being a little bit masochistic to ourselves.
And I think there are ways where things will have to change. So will just be much better for everybody. It won’t be easy, right? We will have to change our ways. But I think we’re slowly starting to realize like we are rethinking basically how the world works anyway. So I think we will start to see that more as long as the whole dead internet thing doesn’t actually just really make all conversation pointless. Wow. I was supposed to be optimistic. I’m sorry.
Josh Bressers (35:52) No,
that’s fine. That’s what the coffee shop part was for. Like go talk to people in real life, not on the internet.
Charlie (35:55) Yeah.
Yes, I love it. ⁓ Yeah. I think ⁓ I’ve seen so many just conversation, change things on a small scale and start to ripple out into the real world. That’s where I see optimism. is human connection.
Josh Bressers (36:12) That’s fair. I agree with this. And I think you’re reminding me of something I forget all the time, but I say it all the time as well, is that as security people, we are notorious for saying it’s not perfect, so it’s worthless. And this is one of those examples where it’s going to be 7,000 tiny steps. And eventually, like, we won’t even realize when things have finally started to change because it’s going to feel like nothing’s going on. But you are right. Like NPM, even PyPy, Java Maven all of these repositories are
Charlie (36:26) Yes.
Josh Bressers (36:42) doing things. It’s just it’s not always obvious and it doesn’t always feel effective. But in a couple of years, hopefully we look back and be like, holy crap, can you believe that even happened? So here’s hoping man. All right, Charlie, this has been a treat, man. I truly appreciate the time. Yeah. And thank you for all the work you do. Keep writing those blog posts. Like keep me sane, man. I love it.
Charlie (36:54) There is. Yeah. Great chat.
Well, I would rather prefer not to, be honest, because it means I’ve had a really bad time. But I’ll, I’ll do it for you. Thanks.
Josh Bressers (37:04) Okay.
I love it. All right, awesome. Thanks, man.