Josh discusses the TARmageddon vulnerability with Alex Zenla, CTO of Edera. In this episode, we explore the discovery of the TARmageddon vulnerability. It’s especially interesting because it’s Rust, but also involves multiple end of life crates. Alex shares the story of how Edera managed to figure all this out (it was not simple). Hard problems are still hard, but there’s a lot of lessons in this one.
Episode Links
This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.
Episode Transcript
Josh Bressers (00:00) Today, open source security is talking to Alex Zendla, the CTO of Edera Edera is a fricking cool company and we’ll, we’ll touch on that momentarily, but the actual reason I asked you to come here today is because your crew found what I’ll say is one of the cooler name vulnerabilities, TARmageddon, right? Very well done there, but I’ll just kind of let you start out. Alex, tell us who you are. Tell us about Edera a little bit. Cause as cool as that is, and I, I,
Alex Zenla (00:17) Yes. Yes.
Josh Bressers (00:25) So I’m gonna try hard not to dive into what Edera is up to right now. I’ll be back for that. But just let’s start with who you are and kind of what’s going on.
Alex Zenla (00:33) Yeah. So, uh, hi, I’m Alex Zenla CTO of Edera. Um, I have spent my life in open source. I started very, very young. Um, I think my first open source contribution was probably like around 11 or 12 years old. Um, I have been working full time in tech since 14. So I’ve, I’ve been around, I think I did the math. I’m like 11 and 0.25 years or something.
Josh Bressers (00:51) Thanks.
Alex Zenla (01:02) ⁓ full-time work, which is a little scary because I’m only 26. But ⁓ I grew up in open source quite literally. If you’ve seen any content from me, you might know that I started because of Minecraft ⁓ and a little bit because of Chromebooks and Chrome OS. So I kind of got into like district development early on and my passion has always been around open source and then I kind of transitioned that into
technology when I got a random email to get hired at a company when I was 14. ⁓ And as the CTO of Edera, I co-founded the company Edera and we build container runtime solutions and container security solutions, as well as GPU virtualization technologies and ⁓ basically a full stack security solution for running workloads secure by default. ⁓ And I…
Josh Bressers (01:39) Nice.
Alex Zenla (02:02) ecstatic about it. think I’m technically about a year and seven months in or something like that. ⁓ But we had no money at the start. So, you know, I exclude some of those months from when I wasn’t getting paid. But ⁓ but yeah, it’s ⁓ been a wonder and and ⁓ open source is important to me. So I’m super excited to chat about this wonderful bug called TARmageddon that you’re about to introduce. And yeah.
Josh Bressers (02:15) Thanks.
Yeah, it’s really cool. Okay, before we get to
TARmageddon though, I’m going to back up a little bit. And I think you undersold Edera a little bit because you’re talking about container security, but really what your crew is doing. And you can correct me if I get some of this wrong, because I’m obviously not an expert. But when we think of containers, it’s that very, you know, a shared kernel, all of the containers while they’re running in their own namespaces, they’re functionally running with, you know, the same privilege level we’ll say in processor.
Alex Zenla (02:35) Okay.
Josh Bressers (02:57) And your crew is using like virtualization technology to segment container images, which is, it’s a really cool idea. I know it’s been kind of tried before, but it’s always been done badly and it’s been poorly implemented. And your crew is doing this in a way that actually makes it usable instead of, you know, a bag of parts someone throws on the table and it’s like, good luck nerds, know, figure this out.
Alex Zenla (03:15) Yes.
Yeah. yeah. I use the project, not a product thing a lot because it’s really difficult to consume some of the existing solutions. But ⁓ Edera was built out of a frustration I had while working at Google. ⁓ when I worked at Google on the Internet of Things, which is Edge Systems, I was very honored to be able to work on basically every building that Google owns. They deploy like a hardware device.
inside ⁓ the building in the electrical room. So it’s used and connected directly to the HVAC systems and all the various kind of industrial generators and such that they need to control. ⁓ And we had this problem where we needed to run multiple things from different vendors on a single piece of hardware. And how do you do that securely? Well, containers would be the obvious option, but
Docker and such and the standard kind of container runtime is not really a security boundary, which is not a unique line at all that people use. But it’s really important for people to understand like containers are not a concept in the Linux kernel, like at all. It is a hodgepodge of insanity that makes a container feel like a container. And… ⁓
⁓ At Edera, we have an open source project called Styrolite that’s kind of equivalent to bubble wrap, but written in Rust. ⁓ And you can go take a look at what it takes to run a container. And it’s honestly horrifying. ⁓ And so Edera, ⁓ one of the key things I realized is we need to do this with like a hard security boundary. ⁓ Being at Google, I looked at GVisor, which was a common project that
Josh Bressers (04:50) Yeah. Yeah. Yeah.
Alex Zenla (05:07) Probably a few people will know that already. ⁓ And it basically wraps an executable and emulates the Linux kernel in user space. But this is not very performant. ⁓ And it also has a whole bunch of usability things. Like you don’t get the full Linux kernel. There’s some like, you know, lot of the performance degradation actually comes from it not being the legit Linux kernel. ⁓ And ⁓ you…
a whole host of other problems. And then you have kata containers, which kind of pioneered with an Intel open source project to pioneer using KVM to isolate ⁓ containers. But the fact is that A, GVisor not being performant and B, kata requires hardware virtualization and not every machine you run on has hardware virtualization. ⁓ If you get an EC2 instance, unless you’re paying big bucks for the dot metal instances,
You don’t have hardware virtualization. And we wanted to build something different. So ⁓ I and my free time, ⁓ free time being relative, while at Google, started working on this project called Krata which turned into ⁓ effectively a rewrite of parts of the Zen hypervisor and Rust so that ⁓ Zen is a hypervisor that some people may know. ⁓
Josh Bressers (06:15) Right.
Alex Zenla (06:32) It kind of was the first to pioneer standard hypervisor techniques and ⁓ existed prior to hardware virtualization existing. So Edera’s technology is all about isolating containers into actual security boundaries using this concept that we call zones. ⁓ Anyway, I can go very deep into this, ⁓ I think it’s… Yeah, yeah, definitely I’ll come back to it. Yes.
Josh Bressers (06:53) We’ll have you come back, because this is awesome. I love it.
Oh man, want to jump into this. We’re not gonna, we’re not gonna today. I want to talk about TARmageddon TARmageddon is more timely and the name is awesome. So tell us what TARmageddon is, Alex.
Alex Zenla (07:02) Yes. Yes. Yes.
Well, to tell you about it, I feel like I have to tell you how I found it. ⁓ we actually in our publications about this have not talked about how we found it very much. So this is somewhat of an exclusive. I think there’s maybe a comment out there from me explaining it. ⁓ So we’re working on GPU virtualization technology ⁓ late into the night. ⁓
Josh Bressers (07:16) good, yes!
Hmm
Alex Zenla (07:40) of mine, ⁓ Stephen, on the Edera team were working, I think it was literally like 1 a.m. and we were working together on pulling an NVIDIA container image to test out the performance of our GPU virtualization and it failed. And we’re like, what is going on here? And then I started investigating and I’m like, wait, it’s supposed to be extracting like the OCI like
combination tar. Basically what gets output when you do Docker image save. If you’ve ever run that command, that’s the input that should have been being processed. But when it extracted it, ⁓ our product was extracting that container image. ⁓ It started extracting NVIDIA dynamic libraries and stuff. But that tar file should have only contained other tar files.
And immediately I was like, what is happening? And I honestly didn’t believe my eyes at first because I was like so confused. ⁓ And then it occurred to me, the other tar files inside of this tar file are being extracted, but we only requested to extract the root of the tar. So something has gone majorly wrong.
And then it clicked, ⁓ no, this is a bad day. So ⁓ let’s talk about ⁓ the bug a little bit and what it is. So there’s this library out there that everyone uses for asynchronous TAR parsing and Rust ⁓ called tokio-tar ⁓
And tokio-tar is a project that is actually a fork of another project called Async TAR, which is also kind of a fork slash like, I guess rework of another library called TAR in Rust. And that was the library that we’re using to extract this ⁓ TAR file that we’re processing for pulling down this container image. ⁓
What we determined was going on after pulling our hair out going like, why is this failing? ⁓ And I’ll get into the failure mode in a moment. We realized that, it’s reading the tar file wrong. And for those that don’t know, the tar format is extremely old. ⁓ If I’m not mistaken, it originates from Unix itself. ⁓
Josh Bressers (10:22) Yes.
Alex Zenla (10:30) And ⁓ it has various iterations. One thing that’s important to know is that the original tar format is not a binary format per se. It is an ASCII encoded format. So you can basically run cat on it and it doesn’t screw up your terminal. That’s how I usually describe ⁓ whether or not it’s a binary file or not. ⁓ But that was the original format. And then on top of that,
All these other formats got built on top of it. the branches are so complicated and the timing is so weird, it would be impossible for me to… I’d have to draw a diagram to show you what the formats ⁓ forks are. But basically there’s a few different formats. One of them is the GNU format. ⁓ One of them is called ustar ⁓ And then the other is ⁓ kind of PAX or the POSIX.
kind of tar format. ⁓ And what was happening is that this specific Nvidia container image we were pulling happened to have very large files. And the tar format can only support four gigabytes, I believe, ⁓ of a single file. And if it needs to ⁓ have a file larger than that, then it has to represent it using the PAX format.
Well, the library we were using did not properly handle the PAX format. ⁓ Specifically, the PAX size field ⁓ was not being read properly. And the way that PAX works is if the PAX field is filled out, then the original size is set to zero on that file. And what happened is…
Josh Bressers (12:21) Right.
Alex Zenla (12:24) when that file that was large was being processed by this library we were consuming, it was not advancing in a pass the file content when scanning the file because the size was zero and it didn’t know about this PAX size. And then it started looking for the next tar header and the next tar header was inside of the file that it was processing itself. And so it started extracting that file as a ⁓
as part of that root tar. ⁓ And this is terrifying for a whole bunch of reasons. ⁓ One of them is that you can craft a tar file to be processed by this library that specifically looks like it’s okay when validated using the tar command, but when extracted by this library behaves entirely differently.
And you can even do things because tar files are not like a zip file. When you have a zip file, it has a central directory header. And that basically tells it all of the files in that zip file. With a tar file, it just basically scans over the tar file and looks for an entry. And then if it finds that entry, then it can extract that file. It doesn’t have a central place to look it up.
Josh Bressers (13:49) Because it’s a tape,
right?
Alex Zenla (13:50) Yes, it’s a tape archive.
Yeah, I forgot to mention this, but it’s literally called tar because it’s a tape archive. It’s not supposed to be fully scanned like a zip file. And I’m sure I don’t need to tell this audience this, but tar files are like the most common archive format these days for Linux and Unix systems. ⁓ All your binary and source code are generally packaged in tar files. ⁓ So ⁓ this bug.
⁓ effectively made it possible to craft a tar file that when extracted could extract a file ahead of the other file that looks valid and put a malicious file in there. So you could extract a tar file with the tar command and it looks okay and maybe even the go tar like any tools that might monitor container images ⁓ can go look at that. ⁓
But ⁓ when processing the ⁓ tar with the Rust library that’s vulnerable, ⁓ it would then ⁓ extract the malicious file that you maybe embed in there. ⁓ So it’s called a differential parsing bug. ⁓ And this was horrifying. ⁓ Interestingly enough, our product was not vulnerable because our
because we bailed early because we detected something was wrong. so we immediately went, but there’s a lot of things using this library. We need to go investigate it. ⁓ But ⁓ we can talk about now the complicated nature of tracking down this bug and disclosure perhaps next.
Josh Bressers (15:39) Yes,
and that’s the thing that really attracted me to this particular bug because if it was just the TARmageddon bug by itself, it’d be like, wow, that’s a weird bug. But this is an end of life piece of software, which you, the dial, like you ⁓ turned it past 11 even, because in terms of difficulty, like I have done coordinated disclosure many times in the past, but never for an end of life piece of software. So I’m excited to hear this story. I mean, I feel bad for you because I know how much work it is, but like.
Alex Zenla (15:50) Yes.
Yes.
Josh Bressers (16:08) I’m also really interested.
Alex Zenla (16:10) It was very stressful. had, ⁓ it was stressful enough that we pulled together a small group inside of Edera to handle this directly because we didn’t want to share the stress across the company. Like we, it was that complicated. So ⁓ let’s talk about the lineage and how, and what’s vulnerable because I think this is very critical to the story. ⁓ So, ⁓
Josh Bressers (16:32) Yes.
Alex Zenla (16:36) ⁓ A person named Alex, if I’m not mistaken, who I’ve interacted with in the past in the Rust community, ⁓ wrote this library called tarrs or just tar. ⁓ And it is the tar parsing library that everyone uses for synchronous operations in Rust. There’s kind of one canonical tar library for synchronous operations. ⁓ Rust has this problem though, which I can go very deep into that
⁓ If you’re using the asynchronous operations in Rust, you kind of have almost your own entire world of libraries you have to interact with. And someone forked the tar library into async tar. And async tar is a library that uses the async standard library, which is actually no longer really in use, ⁓ and implements the tar
⁓ RS library as an async library. ⁓ This one is relatively maintained but ⁓ it has less users. ⁓ Then that library was forked from tokio-tar or to tokio-tar ⁓ by another user and this library is commonly used. It owns the package name on crates.io tokio-tar
And this uses the tokio standard library, which is an async standard library for Rust. ⁓ This library was unmaintained. And so we forked it to krata-tokio-tar which is a tokio-tar that was specifically for Edera, specifically for the Krata open source project.
was not really intended to be maintained per se. It was mostly for us to make modifications to it because of certain requirements that we had. And ⁓ then our library, because it was not really intended to be maintained as like a proper, I would say open source library for us to maintain for others. It was really intended for us internally, ⁓ was forked by Astral to make astral-tokio-tar
and astral-tokio-tar is maintained and is open for external use. ⁓ And this bug that we found affected everything from AsyncTar all the way to astral-tokio-tar. Now, what do you do with this? Because Rust has a problem that a majority of the libraries that get written
are not maintained by large open like large open source efforts. They’re usually like a person doing it as a hobby, which there which is not uncommon in other ecosystems. But I would say Rust is more I’d say immature in that regard that there’s no coalescing across like efforts on a single library to do X or Y. It’s kind of just there’s probably like 20 different
error handling libraries for Rust that have ridiculous amounts of users, right? Like there’s not so much coalescing around. And I think primarily that’s due to lack of, you know, ⁓ corporate ⁓ assist sponsorship and assistance. ⁓ But ⁓ this means that other people downstream users of these libraries are using any number of these different forks.
like we’re using crowded tokio-tar cause it’s internal. ⁓ UV, the package manager for Python ⁓ used astral-tokio-tar ⁓ There were some test containers things that use tokio-tar and ⁓ among other kind of like common binary installation utilities for cargo and things like that. ⁓ And the list of downstream impacted projects is like insanely large. ⁓
but they’re each across all these different forks. So how do you handle this? And to be honest, it was nearly impossible to do it right. We had to make a lot of choices.
Josh Bressers (20:59) Yeah, yeah, right, right.
Yes, I can imagine. Even if these were all well-maintained, this would be difficult to coordinate. And the fact that, what, three of them are dead? I don’t even know how I’d start doing that, to be honest.
Alex Zenla (21:09) Yes.
Yes.
Right. It’s difficult. We got in contact with a majority of the people, but not everyone. We also went a step further and talked and brought ⁓ downstream users into the disclosure group because one thing that we realized is how we did this matter to us. And we wanted to have the least amount of negative impact possible that we could reasonably muster. ⁓
And so we brought in people who depended on the library, especially if they depended on the library that was not being maintained. We had a number of people who did not respond to our disclosure, which it’s 2025 and you should have a security.md or equivalent in your open source library, especially if you are a company. It’s…
I don’t want to go too deep, but I will say that there are various companies who did not respond to us that should have had the resources to respond to a security vulnerability like this. The one that did respond very well was Astral. I want to send a huge thank you to William over at Astral, who I realized you actually had on a podcast before. yes, yes. Yes, yes.
Josh Bressers (22:32) William Woodruff? he’s at Astral, okay. I didn’t know where he worked
actually, but William is like the Morlock of the internet. He is everywhere. Everywhere I look, it’s like William Woodruff is there doing work. It’s ridiculous.
Alex Zenla (22:41) Yes.
Yes,
yes, he was incredible and very good at helping us out. ⁓ And I would say ⁓ the most important thing ⁓ lesson that I have here is take security vulnerability seriously. If you are someone who is distributing software. ⁓ One thing that I find very critical is there are people who
release libraries or binaries that are intended to install other software. And that kind of heightens the requirement, I think, of your security ⁓ responsibility, I would say. And thankfully, many of the people who had libraries that did consume tar files that were doing that ⁓ didn’t respond to us. But ⁓ I’d have to look again. But I think our disclosure list was something like 20 people, some of which were just
Josh Bressers (23:42) Holy
Alex Zenla (23:43) Gmail accounts that we just blind emailed and I’ll never forget the day that and my colleague Anne who helped do most of the coordination here as kind of the the the the lead on this ⁓ She sent out so many security at edera.dev emails that it that my inbox was just flooding because we were contacting so many people ⁓ But
I think the lesson that I most want people to take actually is ⁓ it’s okay to ⁓ fork libraries. Like there’s nothing wrong with that. ⁓ But we as an ecosystem and Rust in particular could do a lot better about coalescing around libraries and that includes Edera. ⁓ And as one of the takeaways that we had is that we
archived our fork and started using the Astral fork as the canonical fork so that we could be good citizens as well. ⁓ And I think the rest ecosystem needs more of that.
Josh Bressers (24:56) I I think you could make this argument. So this is part of the discussion I was hoping we would eventually get to is there’s a lot of open source. I mean, this is one of my side projects is I ingest a bunch of open source data and I look at pretty graphs and things like that. And right now there’s like 11 million open source projects, I think, that are being tracked across the major ecosystems. And then if you break that out into versions, you end up in hundreds of millions.
Alex Zenla (25:20) Yes.
Josh Bressers (25:24) right? Because everything has multiple versions and things like that. like, this is one of the challenges, I think, and I don’t have a good answer for this, but it’s like, there are libraries that do many things. There are a lot of libraries that are functionally dead, which, I mean, you, this is what you were literally dealing with is you have this like weird chain of libraries. There are things using the dead libraries that are also dead, right? There are instances where the like developer is literally dead.
Because like we’re humans and there’s all these like weird problems and intricate issues. And I think it’s easy to say we should coalesce on libraries, right? But I mean, the reason we have this ridiculous diversity is because someone didn’t like some other library for some reason. It could be a million things, right? I mean, there’s goodness knows in this universe we live in of open source, it could be like, I don’t like the logo. I’m going to fork it versus, they won’t add my patch.
Alex Zenla (26:19) Right. Right.
Josh Bressers (26:21) And I think like
this is one of the challenges we’re going to see happening more and more often is there are an enormous number of dead projects. And like, what do we do as security people when we, when we have to deal with it? It’s like, you’re the first example I know of that has actually had to do this at scale.
Alex Zenla (26:39) Yes, will. So I had a little bit of a laugh when you said that we’re human because I think one of the takeaways that I’ve always had in open source is pretty much everything that happens in the open source world is just people problems. I could name a thousand different instances, but I’ll allide those for now. I think, you know,
Josh Bressers (26:55) Yes. Yes.
Alex Zenla (27:07) We collectively as an open source world and ecosystem struggle with the people problem very heavily. And there’s a whole bunch of reasons. mean, there’s probably nerds aren’t naturally great people.
Josh Bressers (27:23) I’d say many of us got involved at open source because there weren’t
a lot of other people to talk to.
Alex Zenla (27:29) Exactly.
mean, same me me as well. And you know, I retreated into open source in some way from the world. And, know, I think ⁓ that is ⁓ a huge problem. open source and forks and everything tends to come from two different things. One is ⁓ not working like not wanting to communicate with the ⁓ maintainer.
because honestly there is sort of a bit of a boundary or like a barrier of communication, I think, when it comes to using someone’s open source library, particularly if they are a hobbyist and you don’t want to come off as, know, especially if you’re working for a company. Like I deal with this a lot where, you know, we’re a company that’s VC backed and it looks scary to people when I want to go and like get involved in their open source project that they’re a hobbyist, right?
And ⁓ I try to be as good of a citizen as I possibly can. And that includes sponsorship. That includes, you know, contributing. ⁓ But we collectively, I think, have a problem where we we are willing to kind of write our own thing. ⁓ And I have a blog post coming out soon about ⁓ the topic of ⁓ why building an island is not ideal. You have to build bridges. ⁓
And ⁓ this happens a lot in open source where people build a really cool thing in an island and then don’t build the bridge in order to consume it. ⁓ So it’s a collective problem that I would be lying if I said I had any solutions for. But I think, you know, my motto personally has always been if you’re a good person and do the right thing and the open source community, then at least ⁓ that can get you somewhere. But when it comes to this fork situation,
I mean, there’s a number of problems. One is, you know, people naturally have situations in life where they may not respond for a certain amount of time. And generally, especially if you’re a company, you know, it’s kind of a do now kind of culture, right? You know, you’re not expected to be blocked on someone who doesn’t work for a company that you’re a part of, you know? So I think that becomes a real challenge.
I would assume that there’s a certain number of forks that occur out in the open source ecosystem where that is the case. And then I think there’s a certain amount where, you know, there’s just, I would say drive by consumption of a library. Like what happened to us with the krata-tokio-tar is we didn’t really intend for that library to be consumed by others, but we didn’t necessarily make it as explicit as we should have had, have had. And that complicated.
⁓ things because we were getting pull requests and we weren’t responding because we weren’t making super clear that like this isn’t intended for other people this is for us. ⁓ So that ⁓ is another component is like people consuming your library that you aren’t intending on signaling that like I’m going to support this library. ⁓ I had a discussion with ⁓ Marina Moore
⁓ our research scientist is at Edera who is ⁓ part of the CNCF tag security. And one of the things that I’m realizing is ⁓ there needs to be better indication of how someone intends to support a project. ⁓ So if I am, you know, if I have a GitHub repository with some code in it, that doesn’t mean that I’m
you know, here to support, you know, other people’s contributions or that I have the time to do so. ⁓ And some way of indicating that would be super great, I think, in improving the situation. ⁓ And because there’s a lot of maintainers out there, co-founder of Videra Ariadne Connell, who famously writes things that you are using right now that you don’t even know probably like package conf and
Josh Bressers (31:47) Yes.
Alex Zenla (31:50) libU context and worked on the APK and if you use any of chain guard stuff, she wrote all the technology behind that. ⁓
Josh Bressers (31:58) For anyone listening, like the XKCD
comic that’s like the one open source developer in Nebraska, that was Ariadne. Like legit.
Alex Zenla (32:05) Yeah, literally,
literally it is her. And I’m very lucky to be able to work with her and have co-founded Edera with her. ⁓ she just, she’s constantly working outside of work hours on open source projects. Recently, Wayback, if you’re familiar with Linux, she built Wayback as kind of a, ⁓ I like the joke that she likes to build things out of spite. Like she’s like, this thing sucks. So I’m gonna build something out of spite.
That’s basically what happened Wayback. And, and she creates these really cool things, but she’s taking on that burden and she’s willing to do that. And she signaled that from her past contributions to the world, ⁓ she arguably created a multi-billion dollar industry with what chain guards doing. and, and I think that is an incredible thing, but not every developer is her like,
Josh Bressers (32:55) Yes, for sure. Yes.
Alex Zenla (33:04) ⁓ there have been times in my life where I’ve just like, have multiple open source repositories. You can go find that aren’t maintained because I have a job to do and you know, I’m a CTO and I’m constantly doing things and I don’t have time to necessarily maintain everything. And that I’ve ever written. ⁓ so that’s a huge contributing factor. I think is like, we have no collective language to talk about how we care about and support our own code.
And I think that the assumption that if it’s on GitHub, then that means that someone maintains that and holds the burden on that is a bad way to go. And I would argue that probably that contributed to the fork situation with TARmageddon
Josh Bressers (33:51) Yes, yes, in this, mean, I feel like everything you described is spot on because a lot of, we’ll say consumers, and consumers is like normal people, companies, whatever, view open source as like this magical fountain of free stuff. And that is true to a degree, but yes, there are developers. Like I have literally hundreds and hundreds of things on GitHub that like I wrote a thing and I was like, well, that was fun. And that’s it, I’m done forever.
Alex Zenla (34:20) Right.
Josh Bressers (34:20) Right? I’m never going to look at this
Alex Zenla (34:21) Yeah.
Josh Bressers (34:21) again. If you send me a bug report, like I’m probably, I’m never even going to see it because I’ve turned off all notifications on that repo. And that’s okay. There’s nothing wrong with that, but you’re right. We don’t have any way to denote this. And I mean, this is something I’m seeing coming up more and more often, just from like a professional perspective, as I talk to like customers and coworkers and you know, folks like you is there’s this attention now to saying we need like these open source insights. We need scorecards for this open source. I need to know what’s good and bad.
And I will say every scorecard that exists today is hilariously broken for countless reasons. And so please just don’t use any of them. But it is a legitimate question. like if I look at a project and it hasn’t been touched in seven years, like that is a dead project, right? Like, but there is a ton of infrastructure running on projects that haven’t been touched in seven years.
Alex Zenla (35:04) Right. Yeah.
Yes, yes. Well, you know, I think the intuition of a lot of people is then they go fork that project and maintain it and things like that. And that can be good. ⁓ But I can tell you that there are in crates.io, the ⁓ cargo registry for Rust, there are many libraries that get pulled into the average dependency tree that haven’t been updated in five years.
Josh Bressers (35:38) Yep, yep, I know.
Alex Zenla (35:40) It’s slightly terrifying, ⁓ but yep. Yep. Yes, yeah.
Josh Bressers (35:43) Okay, I want to stop you for a moment and clarify this isn’t a rust problem. Every ecosystem
has this problem. Rust just feels easier to identify for whatever reason. I don’t know why that is, but yes.
Alex Zenla (35:55) I think I know why. I think it’s because there are a lot of common libraries that do get pulled in that are more visible. I think ⁓ Rust is more in your face. Rust has the amount of dependencies that get pulled in to where you can recognize most of them. And where like NPM is so insane that
Josh Bressers (36:14) That’s probably fair.
my goodness,
yes.
Alex Zenla (36:19) It’s just like, I pulled in 500 dependencies, like good luck ever identifying any of these. there’s nothing wrong with a library, like especially some libraries are truly feature complete and do not need to be updated. Like to be very clear, there’s a number of those in Rust. ⁓ And so project age isn’t necessarily an indicator of quality or anything like that, especially in Rust where there are some libraries that are like,
Josh Bressers (36:24) That’s true, yes.
Alex Zenla (36:47) not security sensitive or just very simple libraries. But it is a recurring problem in every ecosystem. I’m just more aware of it because of Rust. we did some, ⁓ when we first built Edera, one of our big concerns was we’re gonna build this really complex thing. And how do we trust the Rust ecosystem? Like, do we need to?
trust the provenance of, or are we able to trust the provenance of everything that goes in? So I built like a little tool called Verify Dep, which is not open source, but it was kind of just a tool for me to analyze our dependencies. And I found that I think it was like 5 % of all of our dependencies ⁓ were built off of a tag that was not pushed to GitHub.
and some did not have GitHub repositories attached. Some were hosted on GitHub, had Git repository information that was out of date. ⁓ And you know, this is a problem everywhere, but it’s just more analyzable, I think, with Rust.
Josh Bressers (37:43) Wow.
Yeah, yeah, for sure, for sure. Cause I can tell you now, I have done some of this analysis and other ecosystems and it is, everywhere you look, it’s just burning tires. It doesn’t matter what direction you look in, which is, now here’s the question I want to end this conversation on, cause we’re kind of running ourselves out of time and this is a really cool bug. and so it’s easy to say, like all this stuff is broken, yet society functions, right? Like I’m going,
Alex Zenla (38:06) Yes.
Josh Bressers (38:26) to pay my bills tomorrow with a web browser from my home. Like, I’m always torn over this. As a security person, I look at some of this stuff and I’m like, this is ridiculous. But then also at the same time, I’m like, yet society is working. So like, don’t know how to draw that line. I just, this is like that whole like you have two wolves inside of you. Well, mine are arguing about this. And like, I don’t know what to do.
Alex Zenla (38:51) Right. ⁓
Yeah, recently we had a discussion internally at Edera about confidential computing, which if you know anything about it is basically just like hardware encrypted enclaves and these things. And we were joking about how like there’s people who really care about confidential computing, but then the majority of the security is like, you know, you’re just deploying a really large, you know, Debian eight image with out of date everything into this encrypted environment. So.
That’s how I feel about security in the world today in a lot of ways. ⁓ One of the reasons why we built Edera is because we were frustrated with the state of security and we also felt that like the burden should not be large to solving these problems. the security should be easy, basically, not a very complex, you know, there’s…
This whole CVE ecosystem is one of those things where if I try to explain it to my dad that knows nothing, he’s going to be like, you are absolutely nuts. And he doesn’t care. He’s logging into his bank to check his balance or whatever, and everything’s just fine, despite that being built on a mass of insecurity, as he mentioned. There is no great solution, and I would be lying if I didn’t.
Josh Bressers (40:02) Yes.
Alex Zenla (40:21) say that I’m part of the security ecosystem and I think both of us are, you know, we’re part of the industry. ⁓ So it matters to us because it is the problem that we solve. ⁓ And I will say like open source is one of those things where you have to give a lot of leeway because it is just people, you know, doing their best honestly, or sometimes just playing around. I mean, I can’t tell you
Josh Bressers (40:27) Yes, yes.
Alex Zenla (40:50) I used to mod Minecraft and the insecurity around the Minecraft modding systems are insane. like Discord token stealing is probably super common in there still. Like the nature of the world is that it’s insecure. ⁓ But I care about the problem because I don’t want other people to care. Like I don’t want my father to have to care about the security, but I also want him to know.
Josh Bressers (40:59) Unbelievable, yes.
Alex Zenla (41:20) that because we care about this problem like you and I, that they don’t have to worry as much. ⁓ So it’s somewhat of me wanting to take on that burden that I care about security, but it is a complicated topic that there is no great answer for. ⁓ But I hope that as an industry, we legitimately care about it as much as we can.
because someone has to. But I think we can’t give and shovel problems to other people, developers or users. We have to shovel solutions out to people. ⁓ And we are very bad about it as an industry. There are multiple products you can go buy today that will tell you nine million different problems that you have in your security at your company or whatever. ⁓ But I want to be…
Josh Bressers (42:01) Yes.
Alex Zenla (42:19) and build the things that don’t do that. I want to solve problems for people ⁓ and not have people think about.
Josh Bressers (42:30) Yes, yes, I love that ending. And I think you bring up a very good point that I forget about sometimes, and I think is very related to this, is right now in a lot of security things, especially all this open source madness we just talked about, it’s a lot of heroic feats that are keeping the lights on. And by building new technologies, things like the work you are doing right now, this is creating basically like systemic solutions to systemic problems instead of needing
heroic efforts every time. And that is really what we need. So yes. All right. I feel a little better. All right, Alex, this has been an absolute treat. I’m really excited to have you back to talk about Edera. Like it was, I took everything in me not to dive in that hole, but this was awesome. Yeah. Good, good. But this was great. I mean, thank you for the work like TARmageddon Great name, great research. Well done. But yeah, it’s been fun. Thanks.
Alex Zenla (43:06) Hahaha!
Yes. I definitely can.
Awesome, thank you so much, I appreciate it. Thanks for having me.