Josh chats with Paul Asadoorian about a tool he wrote called fettle and a recent report Paul published on CVEs. Fettle is a tool to help update and manage Linux systems. The big sell on this one is checking if your firmware is out of date. We then talk about a report Paul created that doesn’t obsess over CVEs, but rather the vendor updates. It makes more sense to worry about advisories as those are actionable, where CVEs often are not. It’s a great chat and Paul is a legend in the industry.

This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.

Episode Transcript

Josh Bressers (00:00) Today open source security talking to Paul Asadoorian a principal security researcher at Eclypsium Welcome back to the show, Paul.

Paul (00:06) Josh, thank you so much for having me. It’s always wonderful. I don’t I don’t feel like I it’s a chore or something I like I I have to do. Like you and I just kinda hang out and talk. Like it’s our time to

Josh Bressers (00:16) Yeah, yeah.

Paul (00:17) to catch up, which is great.

Josh Bressers (00:19) Yep, yep. Actually, so it’s funny you say that because I just a a couple episodes before this one, I had a chat with Patrick Garrity who’s also, you know, just a friend and stuff. And it’s

Paul (00:27) Yeah, yeah. I love Patrick. Yeah.

Josh Bressers (00:28) like such a difference when it’s like someone you really know and you just get to hang out for a while and and have a chat versus like when there’s someone I’ve never talked to before and I have to have them as a guest and it’s like I have no idea what’s gonna happen, you know.

Paul (00:37) Yeah, it’s a little nerve wracking. Right, right.

Josh Bressers (00:40) So this is yeah, it’s a treat. It’s a treat. So

I you’re here for a couple of reasons. the first I’ll let you talk about is you have a tool you’ve been working on, which I jokingly told Paul before I hit record. he has a podcast called Paul Security Weekly, which if you never heard, you should go listen to. It’s a great show. But you have a tool called fettle and I jokingly said you need to rename it to Paul’s Fettle Weekly, because I think you you talk about it a lot right now.

Paul (01:00) Yes. I mean,

Josh Bressers (01:03) But tell us a little bit about Fettle, because it’s kind of a cool tool and

Paul (01:05) Sure.

Josh Bressers (01:06) and I do like it.

Paul (01:07) Yeah, I think it’s it’s important to start with like the history of it. So, you know, I started doing systems administration on Unix and Linux systems in the like mid to late nineties and then I was like a full time Solaris Unix admin and you know, in those roles you would develop scripts to help you up keep things updated to keep your systems in in good order. And

Josh Bressers (01:31) Yep, yep.

Paul (01:32) as I progressed through my career

in administr you know, systems administration and cybersecurity, I would always have scripts, develop scripts to keep things up to date. Fast forward to when I eventually switched to Arch based distribution for my desktop, I was constantly doing update and in the beginning I would do an update and things would go wrong. And then I’d be like, you know, I should really like clean the caches before I do an update and, you know, I should really like do this other thing before I do an update and so I started

putting that in a script. So every time I update I’m like, yeah, go sync with the mirrors, go clean the cache and just that little bit of hygiene, like, hey, you know, my updates are going way more smoothly. then I fed the shell script to Claude. I was like, Claude, go make this better. which it did great ‘cause it was based on my, you know, original work. It understands the mission. Now it has some background. And then I was like, you know and and I started using Topgrade, which is a great project by the way. Topgrade’s awesome. But I wanted a little more

security features and keeping your system in good working order. features and which is why I called it fettle actually, ‘cause I just want good hygiene, ‘cause good hygiene speaks to I think a more secure system. And so

Josh Bressers (02:47) Yeah, yeah.

Paul (02:48) then I converted it to a native Python app. I try not to have any dependencies. and I used Claude and I just started adding features, right? Gave it lots of direction and I just let’s just keep adding features like and then I started adding

features today that I think are pretty amazing that give me great visibility into my system from both a security and a supply chain perspective. So it tells you like, hey, you know you’ve got packages installed and your distribution hasn’t put out these security fixes yet. So it tells you what those are. So I can go, well maybe I want to remove that package. Maybe I want to do some other compensating control based on that. it tells you all about orphans and alien packages.

which happened naturally on all ecosystems. then I started, you know, it has an HTML report. I am working on a web interface, but I put that on hold to add support for more platforms. So today it supports Arch Linux, in Ubuntu and Debian based distributions. I’m adding Red Hat Enterprise Linux, Fedora, Alma Linux, Rocky Linux, in a bunch of Ubuntu variants. So right now it’s building a test lab so that I can start to get all of these features

working, which is more than just updating your system. Again, it’s keeping in good working order. So it runs some commands that I did based on a cheat sheet that I wrote on how to query your hardware and firmware and get some

Josh Bressers (04:12) Yeah, yeah.

Paul (04:12) telemetry. So it incorporates that as well. And the other thing that’s really cool is it does a hardening on it. So it tells you how your binaries are compiled and which security flags they were or were not compiled with on your system. Again, not to be a vulnerability scanner, but to give you the option to go

Well that binary was compiled with like no security flags. Maybe I really wanna move that, run it in a VM or not have it on my system anymore. So it’s all about keeping your system in good working order, which that hygiene I think we’ve talked about since the dawn of cybersecurity, that having good

Josh Bressers (04:44) Yes.

Paul (04:45) hygiene speaks to having resilient systems that are inherently more secure.

Josh Bressers (04:51) Yeah, and I love the firmware angle here

Paul (04:54) Yeah.

Josh Bressers (04:55) because I feel like I have never heard of anything that can tell me when my firmware is like comically out of date. So that’s pretty

Paul (05:00) Yes.

Josh Bressers (05:01) cool.

Paul (05:01) Yeah, so it’ll it’ll use native Linux commands, third party tools, and FW upd from Richard what’s his lat yeah, Richard Hughes.

Josh Bressers (05:10) Hughes, hughsie Yeah.

Paul (05:12) it uses that so you have that installed in your system, it’ll use that as well.

Josh Bressers (05:17) Nice. Yeah, it’s just it’s a fun tool. Like I I also I told Paul this before we we hit record, like I need to find a machine that’s like really out of date and really crunchy. ‘Cause like I really I wanna see it like light up, you know, ‘cause my s I

Paul (05:30) Yes.

Josh Bressers (05:31) I I try to keep my stuff mostly in order. So when I ran this I’m like, Well this is unexciting. Like, you know.

Paul (05:34) Yes. Well, that’s good. It should

be unexciting, right? So I am building like virtual VMs that are vulnerable. and also for larger upgrades, I’ve used this feature now three or four times. So there’s like a big warning label, like this isn’t fully tested, but I have been testing it. And so what it does is it uses Inksy to collect your hardware and driver and software information from your Linux system. It then

looks at the update that you’re about to apply, it collects that data. And right now it only supports clawed anthropic models. and it sends that data to a model of your choosing. I’m using Sonnet five, I believe. It’s about thirty or sixty cents per query and it takes that data and it comes back with a recommendation. Like are you gonna have a good time or are you gonna have a bad time applying this update? Here’s some things you can do before and after the upgrade to make it go smoothly and to double check that everything

went smoothly. I’ve used that four three or four times for big updates, Josh, and it has worked so well. I’m like super happy with that feature.

Josh Bressers (06:41) Well and it should also be noted that like you run Arch, you know, by the way, and and

Paul (06:45) Yes, by the way.

Josh Bressers (06:47) and like upgrading Arch isn’t quite like upgrading, you Debian or Ubuntu or Fedora or something, where often in those you just run, you know, apt get upgrade, DNF upgrade, whatever, and it usually works, whereas Arch sometimes needs a little bit of, you know, poking and prodding along the way.

Paul (07:00) Care and feeding.

Right, right.

Josh Bressers (07:03) Which is cool. Yeah. Yeah. It’s it’s fun. I I love this kind of stuff.

Paul (07:06) There’s a lot more fe a lot

more features there. You can find it on my GitHub P link in the show notes.

Josh Bressers (07:10) Link in the show notes.

Paul (07:12) thank you, Josh. Yes.

Josh Bressers (07:14) Awesome. All right. So the the actual reason I I wanted Paul to come talk to us is you have a I don’t know if it’s a report or a website or what you’re gonna call it. Okay. Okay, because yeah, it it’s it’s from your day job, Eclypsium

Paul (07:24) It’s both. Yeah, it’s monthly reports and a website that you can query the data. Yeah. Yes.

Josh Bressers (07:30) and it’s it’s the website’s called what? Infratrust pulse and you put out the July twenty twenty six. This’ll this show will definitely come out in August, but the July Great

Paul (07:37) Yeah. Yeah. Well yeah, we’ve already started talking about what August is. So the pulse

is the monthly report. Info infra dash trust dot org is the site where anyone can go to for free, no registration, no payment, nothing. You can just go interact and query the vulnerability data that we’ve made available. then we’ll talk about the the context behind that. And then every month we look at a thirty day snapshot of that data and provide you some insights into like what was

most exploited, what you should be concerned about the most, what’s interesting in terms of trends, in terms of what was vulnerable that month.

Josh Bressers (08:13) Yeah, yeah. Now while this is I mean vaguely open source ish related, but but I know you you have a lot of like, you know, commercial hardware on here. ‘Cause I mean that’s what Eclypsium does is you focus on, you know, a lot of embedded type devices, you know, firewalls, things like that. And this this list is I think it’s almost all firewalls, isn’t it? On your July?

Paul (08:32) Yeah, so

we tried to like Patch Tuesday, Microsoft Patch Tuesdays for Microsoft. there was some journalists that do Chipmaker Patch Tuesday, and talk about

Josh Bressers (08:42) Mm.

Paul (08:42) AMD Intel, Qualcomm and Nvidia, and there’s a couple of other maybe initiatives in that. But what was missing was what about all of the Dell Lenovo H P type firmware updates and what about all of the network edge devices? You know, Palo Alto,

Josh Bressers (08:58) Yeah, yeah.

Paul (08:59) Cisco, all of all of those, Arista

they released regular advisories as well and we wanted to give people a a snapshot into those. And you may ask like, well, what’s the difference between like a chipmaker and your Dell Lenovo HP? And here’s part of my frustration and why I I’m excited about this project is because when AMD or Intel as an example issues a security advisory, there is a pretty high degree of probability that you won’t be able to do anything about it. Yes, there’s a

a microcode patch, there’s a firmware update for this chip. However, you can’t just go on your Dell PC and apply that fix. You have to wait for Intel or AMD to send that fix to Dell. Dell has to test it, incorporate it, and then Dell has to issue that firmware update that you can install on your model because each of these OEMs customizes the hardware and firmware that

Josh Bressers (09:56) Yes.

Paul (09:57) you run as your your s PC server or laptop.

Josh Bressers (10:01) Yeah, yeah. And there’s there’s like one thing in this report that like really triggered me, and I love it, is you have in your fact you have why count advisories instead of CVEs. And this is the kind of the way you’re approaching this isn’t, you know, here’s a laundry list of CVEs that affect these things. You’re treating it as like these are the updates to the things you have that you need to apply. And

Paul (10:24) Yes.

Josh Bressers (10:25) I feel like

I feel like this is a big deal right now because and and this is what I really wanted to talk with you about is like if we look at the industry today, the number of CVEs is completely out of control. Like no one can even pretend they can keep track of this stuff. And there’s like this obsession around this data now where like it’s mostly just talking about how big it’s become. And like the amount of actionable guidance, I feel like is non existent.

Right, where there’s just like this story about how ever there’s tons of vulnerabilities and we’re all doomed versus, you know, what you’re trying to do, which is like here are some actual things you can do. And I feel like that is such a big deal right now.

Paul (11:05) Yeah, I wanted actionable results and you know the team at Eclypsium who’s amazing by the way. I mean this initiative involved everyone from founders and CEOs, to engineers, and researchers. We’re all involved in this project. And one of the things that we set out to do was give people those actionable results. The you need to be concerned with the patch and what you can patch, not the fact that there’s a vulnerability.

Because those

Josh Bressers (11:34) Yes.

Paul (11:35) are two different things. And so what’s interesting was I had developed a tool that I’ve talked about on the show on Paul Security Weekly many times. it has not been released, it is just an internal tool, and I’ve been sharing what I’ve learned by developing this tool that analyzes public CVE vulnerability and exploit data. And I attempted to do this with my internal tool. and I was like, Well, tell me about all the vulnerabilities that were released for Cisco.

for whatever vendor. I’m not picking up vendor but whatever vendor. And I go, our data’s incomplete. Like we don’t we don’t know, like what if Cisco released an advisory and didn’t publish a CVE. If you query the public CVE or vulnerability data, you would never find that. It’s not in the NVD database. It’s not it’s not anywhere. And there’s a multiple situation also the time window, right?

You may get a CVE issued by Intel. Lenovo may not issue an update for sixty days. Because the supply chain is complex and it takes time to re and I was observing that. I’m like, well Lenovo released a CVE this month, but it’s not in this month’s public CVE database snapshot ‘cause the CVE is ninety days old. so I’m like,

Josh Bressers (12:52) Yep, yep.

Paul (12:53) we really need the vendor advisories. Turns out we have a team at Eclypsium that is

focused on and has been since the inception of the company, collecting things. They’re like the hoarders of cybersecurity. And so they hoard vulnerability data. They hoard firmware and patch information. They hoard even virtual images and we collect all of this data and we go through all of the data and we feed it into our product. And so I I reached this juncture where I’m like

You know, really we need to scrape every single vendor, advisory, website,

Josh Bressers (13:32) Yes.

Paul (13:32) and curate that data. I’m like, I am not going to vibe code that ‘cause that’s gonna be a disaster. And I’m like, Well it’s a good thing we have a team of five people that have been working for years and have collected this data and th and it drives our product, right? So we actually, as Eclypsium set out to collect this data. And so that was the foundation for InfraTrust And it was a it’s a unique

view and perspective on the data because it tells you in the last thirty days what were the vendor advisories that are associated with a patch that you can actually do something about. And I think that’s the real the crux of the issue and the important thing for defenders today. because as a defender I’m like, yeah, you can go subs try and subscribe to all these advisories. It’s a huge

Josh Bressers (14:16) man.

Paul (14:17) it’s a huge pain in the butt. The data’s not normalized. The data’s in a million different formats

We did have CSAF, which was a standard in an attempt to standardize vulnerability data from it you know,

Josh Bressers (14:27) Yeah.

Paul (14:27) adoption on that is virtually non existent. some of the vendors you need accounts for in order to collect this data. The I’m not saying too much now, but there are instances where vendors will detect scraping and crawling and not just block you, but feed you back bogus data.

Josh Bressers (14:49) wow.

Paul (14:49) So

your crawler has to be intelligent enough to know when it’s getting bogus data. and so I mean we’ve had a team in Eclypsium largely the first thing was UEFI. Like we want to know about all the UEFI updates, we want to put it all in the database, and then when a customer uses our product, we want to tell you, hey, your UEFI is out of date. By the way, the latest version is this and in certain cases we’re like you can just use our tool to install it. In order to do that, we needed that back end database and we’ve been building

Josh Bressers (15:17) Sure.

Paul (15:18) upon it.

You know, since inception, so

Josh Bressers (15:21) Yeah, man. I mean, the data’s crap. My favorite, like you mentioned CSAF, Paul, is

Paul (15:25) it’s awful. Yes.

Josh Bressers (15:27) there are a number of vendors that release CSAF advisories and they are all different enough you would swear it’s not the same format. And it’s like this is

Paul (15:37) Same.

Josh Bressers (15:40) the the dumbest thing. It’s so frustrating.

Paul (15:42) Yes. And you may say, Well,

you know, you can choose an LLM and sort it all out which we do. However, you can’t just throw start throwing prompts in an LLM one, cost is gonna be a huge factor. and two, it may get you anywhere between eighty to ninety eight percent accuracy. And so how do you reconcile you know, the rest? You you need

Josh Bressers (16:04) Yes. Yeah.

Paul (16:06) to have a system with checks and balances, so

Josh Bressers (16:09) Yes, yes, for sure. And I mean the thing I found is especially when dealing with like vulnerabilities and versions, the versions are often s very similar. And the LLMs, they’re not picky enough to care. They’re like, that’s like ninety percent of the same numbers and letters. It’s probably the same. And it’s like, that’s not the same at all.

Paul (16:27) That’s what happened. And

we’ve all I think many of us have observed it doing with CVEs and just going,

Josh Bressers (16:32) Yeah, yeah.

Paul (16:33) Well, that CVE close enough so here’s your answer because it wants to give you an answer. So you

Josh Bressers (16:35) Right. Yes.

Paul (16:37) have to like really tune it to go, don’t ever do that when your data has to be accurate like that. Yeah.

Josh Bressers (16:43) Yes, yes.

Convincing them to tell you I don’t know is like the hard it’s like dealing with a child. Like just shut up. Anyway.

Paul (16:50) There was other

things, you know, i in you talked about versions and affected products. That’s super hard. when you have a vulnerability that we all know about and we want to determine this is like this should be easier, but we want to determine which products are affected. Not every vendor fills out the CPE information. You can pick on CPE all you want. And I I have my gripes with it too, but every even CNAs

who are creating their own CVE records are not putting in the CPE information. And what’s interesting, they put in the CWE information, but not the CPE. And I’m like, but you own all the products that are affected by this vulnerability. You’re a CNA. I think it’s harder to derive CWE than it would be for you to go, this vulnerability lives in these products in these versions. Let me just go fill out the CPE.

And so that’s like super frustrating.

Josh Bressers (17:49) Yeah, yeah.

Paul (17:49) I think the one of the answers for the CVE data is crap is well we’ll appoint CNAs and they can create better records and have more accurate data. But that’s not happening. So my my like plea with the CNAs of the world is to please take the time. Fill out all the fields. I know they’re not all required. Maybe that needs to change too, but take the time to craft these records and give us as much metadata as possible because that helps that helps us.

In a big way.

Josh Bressers (18:18) Yeah,

yeah. I mean I totally blame the CVE program for this. There’s no reason they don’t make these fields mandatory.

Paul (18:24) I agree.

Josh Bressers (18:25) I mean, but getting the CVE program to do anything is yeah, good luck on that one. But

Paul (18:30) Takes a lot of momentum, right?

Josh Bressers (18:32) it’s they’re stuck in the mud, Paul.

Like

Paul (18:35) Yeah.

And there’s politics involv I mean you can’t escape the politics angle from it. And I don’t you know wanna start throwing stones at people. I you know, I working with people in this area as you do as well, Josh, I think everyone does have good intentions. but we it’s almost like many times we don’t get the desired results that are based off those intentions. We don’t get the rubber meets the road actually affecting change. But we have good at least we have

for the most part have good intentions. There are some backwards incentives, but I think we we have tools and analytics that can help work through those incentives. And that that’s the example. You know this.

Josh Bressers (19:16) You’re

you’re more optimistic than I am, but but that’s good. That’s

Paul (19:18) Yeah, I try and be optimistic. But I the

other case where that happens is when the CNA scores the vulnerability and NVD scores the vulnerability and the CNA

Josh Bressers (19:26) Amen.

Paul (19:27) is a a vendor with a commercial product, what happens? In almost every

Josh Bressers (19:31) Yeah.

Paul (19:32) case I’ve seen, Josh, you know the answer to this already, right? NVD scores it higher than the vendor. And some will say, Well, the vendor has better insights into the severity, but the vendor also has an incentive to downplay the severity of that vulnerability ‘cause they’re like, well, I don’t

Josh Bressers (19:46) Yep. Yep.

Paul (19:47) want a whole ton of

critical vulnerabilities in my product. People lose face in it faith in it and they won’t buy it. Whereas NVD doesn’t have that, you know, biased incentives. They’re just scoring it the w calling it like they see And so in my tool I present I present both and I also lean towards the higher score. I think me as a defender, I wanna know the highest score. Right? If someone thought this was this was more severe, I want the most severe score and that’s kind of how I reca reconcile it. But I also

provide in like below the fold, I can see who scored it and what those scores were.

Josh Bressers (20:23) Yeah, yeah. Well then I mean I’ll tell you now, I work with a lot of regulated industry and especially in the US federal space, NVD score is all they care about. They don’t care what the vendor says in any way. Which

Paul (20:33) Right. Right.

Josh Bressers (20:34) sometimes is good and sometimes is bad. It it makes

Paul (20:36) It’s sometimes it’s bad

‘cause sometimes the vendor is correct, right? Sometimes there’s a case where the vendor goes, But this vulnerability maybe is in an open source component in my product and by the way, the way we’ve implemented it, it’s not vulnerable to that. So our CVE

Josh Bressers (20:50) Yes. Yes.

Paul (20:51) score is zero. Like we’re not even we’re not even issuing it’s it’s not there. and it th those cases I get.

Josh Bressers (20:57) Yeah. So okay, so on CVE. On CVSS, you have a thing in your report here that I also really like where you you you mentioned, I’m just gonna read this part because I think it’ll it’s easier than me to try to explain it. You have the CVSS scores indicate Cisco criticals this month, but the remote and unauthenticated vulnerabilities total twenty-six.

it says, and those are not the same set you would get by sorting by score. A CVSS of 7.5 that an attacker can reach across the internet with no credentials is a bigger problem for you than a 9.8 that needs local administrator rights. And I like that because I think we often obsess with CVSS severities, you know, in the vulnerability

Paul (21:36) With the score.

Josh Bressers (21:36) management industry. And you’re right, like it’s very context dependent because yes, something on the local or on the public internet.

is a totally different story than something on, you know, a private network or need log in or whatever. And so I think we do a really bad job of taking context into account. So I like that you’re doing that. That’s cool.

Paul (21:55) Yeah, I think the the access vector A V component to the CVSS score is very important.

Josh Bressers (22:02) Yeah.

Paul (22:03) Is it is it network or is it local? And then the next qualifying criteria, independent of score, is is it an authentication bypass? Is it remote code execution that has that network component? And so I don’t care what the score is, if someone can reach out and touch it over the network

an exploit a vulnerability to gain access to my system. That’s what I care about. And authentication bypass is one that I think might score a little lower because you’re not executing code. However, you’re giving the attacker control of the device. I think Fortibleed is the best example of that. If I can gain control of a Fortinet device, I can turn into a passive network sniffer. I don’t need malware. I don’t need a remote code execu execution exploit. I just

need access to the device. And so if a lower score gives me network access to bypass authentication and interact with the device, then that’s as an attacker that might be all I need. I’m gonna live off the land.

Josh Bressers (23:04) Yep, yep. I know I I have a an old red team or friend who told me every time he goes into any sort of engagement, he would figure out what software they had. And then he said I’d go look at for the low vulnerabilities because I knew they wouldn’t be patched. And then he’d you you know use those to jump his way into whatever he needed. And I’m like, that makes me sad and I get it.

Paul (23:23) Yeah, and you can’t ignore right

you can’t ignore the lows and I think this also impacts the exploited in the wild. And I think people get all excited, well if it’s exploited in the wild, I have to patch it. I’m like, Yeah, but so you had copy fail on a Dell device that was in the the pulse report. Should you drop everything and patch that when this month there were network reachable authentication bypass vulnerabilities? Priority wise for me, you should focus on those. And I know.

Josh Bressers (23:52) Yes.

Paul (23:52) You should still fix it. I’m not saying don’t fix it. Local privilege escalation is still s you know, an attack surface worth defending, but there are other more pressing issues that are not tangled up in CVSS scores that are tangled up in how the attacker interacts with your system. So I was like, don’t focus on the Dell one. I mean get to it eventually, but it’s local privilege

Josh Bressers (24:15) Yeah, yeah.

Paul (24:15) escalation. We see a new local privilege escalation for Windows and Linux probably every week.

And so I

Josh Bressers (24:22) There’s probably a

ton of So I asked Greg KH this one time. It was right it was right around copy fail. And I basically said, Greg, like you’re releasing literally hundreds of CVEs every Linux kernel r release. I said, How many of these do you think could actually be used for like local privilege escalation? And he said, Almost all of them. It’s

Paul (24:42) Yeah.

Josh Bressers (24:43) just we’re not looking.

Paul (24:44) Right.

Josh Bressers (24:46) Which is

Paul (24:47) Which is why you need to keep

your systems in good order, right? I it’s one of the reasons why I created Fettle was following these vulnerabilities,

Josh Bressers (24:53) Yeah, yeah.

Paul (24:54) then looking on my own systems and being horrified was something I was getting tired of, right? And I wanna

Josh Bressers (24:59) Yeah. Yeah.

Paul (25:00) go I wanna go, can I apply this patch? And well w if I if I can apply the patch, I wanna know that I’m vulnerable. Sometimes it’s sometimes it’s hard, right? Sometimes vuln scanners miss thing. There’s a lot of conditions there.

And so in copy fail is a great example. I’m very concerned that if you’re just comparing versions, that you’re gonna miss it. ‘Cause the actual vulnerability is in in a kernel module. And so unless you’re like reversing the kernel module and detecting the actual vulnerability, that version comparison can give you a false negative or a false positive. And so

Josh Bressers (25:36) Yes.

Paul (25:37) that’s that’s another one of my soapboxes.

Josh Bressers (25:40) No, no, for sure. So okay. So here’s kind of the the Josh and Paul make crap up point of the show, let’s say, of like, well, what what do we do about this? I mean, so like I think the the the path you’re taking with this sort of report, I I really like, but you’re also like that’s tightly constrained, right? You’re focusing on, you know, very like IoT style devices. It’s not

a huge, huge collection of the things we care about. And I think anyone in this universe is just like completely drowning at this point because we can’t keep up with all the vulnerabilities. I mean, it’s it’s almost totally out of control. We have tons of regulatory bodies telling us you have to patch every critical and important in five days or whatever it is. I mean, I know what is it? Is it FedRAMP or no the the the DOD DISA someone in the federal government just said, you have to patch all criticals in like three days, right? Which is like that’s just

You’re you’re wishing. Like there’s

Paul (26:34) Yeah.

Josh Bressers (26:35) that’s a wish, right? But I feel like and and I I don’t have an answer for this, Paul, but it’s just like w what can we even do? You know what I mean? I mean, other than like we’re just whip harder seems to be the current attitude we’re seeing in industry. And I I don’t I feel like that’s ridiculous and we’re just burning people out super fast.

Paul (26:55) Yeah, I think there’s a couple of answers there, Josh, that I’ve been noodling on. Technologies that I really like are are AI and automation for defenders. It’s why I built that feature into fettle It was kinda like, what if I were to use what I’ve learned about security and AI and apply that to defensive posture? Because I think the friction is I wanna be on the latest version, I wanna apply all the patches, but there’s operational risk.

So how do I reduce that operational risk? Well, you have to collect all of that data and model it out. and from that model determine a path that gets you to the latest patch quickly, efficiently, and safely, right? Not safe security wise, but safe operational risk. We’ve all been the admin that’s blown something up and caused an outage and no one wants to be that person. So we using the newer technology like AI to help us get to that patch more quickly.

Josh Bressers (27:52) Okay,

so I have a question about that for you then, ‘cause I legitimately don’t know the answer to it is like Arch has a wiki, like it’s a legend, and they document like known problems with potentially upgrading. is

Paul (28:06) Yes.

Josh Bressers (28:07) there anyone else doing anything like that? Like I don’t I don’t know of any if there are.

Paul (28:11) a hundred percent. And so it’s a great point, Josh, and it’s kind of a a a hole in my theory, right? Is that someone has to apply the patch and describe their experiences good or bad and make

Josh Bressers (28:25) Yeah. Yeah.

Paul (28:26) that public to have the data so that you can create a better model based on other people’s experiences. So basically someone’s gotta be first, right? And Ar Arch

Josh Bressers (28:35) Got patient zero.

Paul (28:36) Arch does do a great job.

Manjaro specifically creates a stable update thread on their forum. And in there they give advice like, hey in our testing here’s what we noticed and then anyone who has an a an account, a user, can put their experiences. And they have even a voting system. And that’s great data to consume and transpose onto your configuration and go, you know what, you have an NVIDIA card and this person with that NVIDIA card applied the update

and their graphics subsystem blew up. So and here’s the fix for it, right? so having that data is great. We do you’re right, we don’t have that necessarily for a lot of other platforms. and that that’s one thing that would make an AI assisted, you know, upgrade path tools much more effective is having having that data.

Josh Bressers (29:32) Yeah, yeah, for sure, for sure. All right, what else you got for me?

Paul (29:38) let’s see.

Yeah. Where did you want to go? I feel like there was more stuff on infrastructure trust. Like h what what do we do? the other part of what we

Josh Bressers (29:43) No, I I was asking like how what what do we do, right? Just ‘cause we’re drowning.

Paul (29:48) do, yeah, sorry. I got I got distracted with that. The other thing

Josh Bressers (29:52) It’s all good.

Paul (29:52) is not just keeping your systems in good order, but increasing your detection and prevention, right? The thing that we’ve always said is that system harden well, two things, system hardening and active detection. Right? If I can’t patch it, I need some visibility to

consume threat indicators to be able to detect indicators of threats. obviously there’s a huge industry built around this in Windows. Linux there are tools. I I still think it’s not as mature on the Linux. Like there’s lots of Windows is Windows for the most part. Linux

Josh Bressers (30:31) Yeah, yeah, yeah.

Paul (30:32) is a million different variations, so doing threat detection is super hard because

People can like roll their own, customize it in a million different ways. So when I go look for a threat, and I this is my day job, right? I go, I wanna look for a threat in malicious systemd files. Well now I have to understand how every Linux distribution and every user

Josh Bressers (30:53) Yep, yep.

Paul (30:55) implements systemd whether or not they implement it, how they implement it,

Josh Bressers (30:58) Yes.

Paul (30:58) in order to develop a detection that is somewhat useful to go, Well on this system I gotta check it this way and on that system I gotta check it that way, right?

So I think that’s where Linux has a disadvantage. It’s even worse when we get to IoT and network edged

Josh Bressers (31:12) yeah.

Paul (31:12) enterprise devices. That’s the closed ecosystem that again part of my day job, right? We’re trying to break into and go, How do we give really good threat detection capabilities to enterprises that have this enterprise gear? Right? And then it it runs the gamut

Josh Bressers (31:29) Yeah, yeah.

Paul (31:30) within that. Some systems are based on more open Linux platforms, some systems are very much more

closed down. So but you ha you have

Josh Bressers (31:39) Yeah.

Paul (31:40) to do it ‘cause you’re not gonna be able to move quickly towards a patch. You’re not gonna be able to harden as much as you like to, because if you need the feature, you need the feature. So you’re gonna have an attack surface. How do I monitor that attack surface to make sure? I think with this onslaught of vulnerabilities and patches, Josh from AI that we’re seeing today, it’s gonna force our hand that we’re gonna have to rely on better detections because we’re just not gonna be able to patch everything.

It’s just not gonna be possible.

Josh Bressers (32:07) Yeah, yeah. I mean so that’s

You’re probably right. And it’s funny because I feel like there are like various groups of people. You have the people that are claiming you have to patch everything immediately, right? Like no questions asked. Like patch it now. you’re too late already. And I often feel like the people that say you have to patch everything and keep all of your open source as up to date as humanly possible. Like that feels like you’ve never run any sort of infrastructure is like my opinion there.

And then we talk about things like threat detection and you know more like like the traditional infosec stuff. And me I I’m I’m always torn on this one because like it is a really hard job to do and it’s like you’re going to fail, right? And I guess your goal is to maybe fail less bad than if you weren’t there at all. It’s like it’s such a hard thing. So I don’t

Paul (33:04) Yeah.

Josh Bressers (33:04) man.

Paul (33:05) I think some of the y you know, patching only gets you s so much of the way there and what we’ve observed in the data from multiple sources is that there was a zero day exploit before a patch came out. And if the

Josh Bressers (33:18) Yeah, yeah.

Paul (33:19) patch comes out, there’s an exploit for it pretty much immediately. and therefore, you know, your patching strategy is never going to be able to keep up in in a lot of circumstances.

So I think that, you know, you have to rely on reachability. how reachable is the system, how reachable is the vulnerability, and you have to do really good threat detection and there’s a lot of work I think to be done to enable that, especially on these proprietary IOT network edge platforms, where it’s Linux underneath, and you’ve gotta defend it. And you don’t have the visibility or controls to defend it.

Josh Bressers (33:57) I know.

I know what? You know what you just made me realize, Paul? This is a people problem, which I mean they it always is. But like we have app sec people not talking to InfoSec people and they’re both trying to solve problems using the only levers they have instead of saying, let’s talk about this and come up with solutions where we both have levers and we will pull them together and solve things. Because I think

Historically, the the AppSec types just didn’t care or have to care about

Paul (34:32) Yeah.

Josh Bressers (34:32) about InfoSec. And I feel like that is that is colliding today for sure.

Paul (34:36) And that’s a huge

access vector, right? And so we’ve talked about this on the show and it’s a great it’s a great point because you can have let’s say a closed system, a device that’s a firewall or router, let’s say, and you develop firmware, even if it’s Linux, like f firmware I’m using kinda loosely there, let’s it’s based

Josh Bressers (34:54) Yeah, yeah, yeah.

Paul (34:55) on Linux, you know all the binaries, all the configurations, what they should be, you can lock that in. You can say, Yes, this binary should exist on the system

It should always ha have this hash and it should always do this thing and it’s locked in, it should never change, and put really tight parameters around your device. The problem is the web application. Attacker comes along and they

Josh Bressers (35:19) Yeah.

Paul (35:19) exploit OS command injection in the application. That binary didn’t change. The signature is still good, that binary is running. Now it’s just running commands in memory and

Josh Bressers (35:30) Yeah, yeah.

Paul (35:31) executing commands in the operating system. So even though you have this tightly locked down validated platform that can only run certain binaries so I can’t drop malware, I don’t need to drop malware. I can I can I can just do OS I can do an authentication bi my favorite chain, right? Authentication bypass to internal OS command injection and now I control the device.

Josh Bressers (35:53) Yep, yep. And they all have Python, so you could do literally anything you want.

Paul (35:56) Well, they’re all and

they’re all b and they’re all different languages and more and more we’re seeing more languages being included on these devices. We’re seeing Python, we’re seeing Lua, we’re seeing PHP and we’re seeing NPM and all of the associated supply chain and module and library dependencies that come along with those. We’re seeing that in this ecosystem and I think actually in one of my demos I live off the land in NPM. Like I just I don’t need a

Josh Bressers (36:21) Nice.

Paul (36:22) I don’t need a shell and anything else. I have NPM.

I can execute operating system commands, I can do what I can write applications, I can execute JavaScript code, that’s all I need as an attacker. So going back to the solution to that, right, is really hardening and writing code that’s resilient on the web application front. So many of the devices that we’ve seen from all the major network manufacturers, the weaknesses in the web application.

Josh Bressers (36:49) Yeah, yeah. Well, and I guess hilariously, while they might be able to lock down the like firmware running on device, you can just point an LLM at a web app and be like, Go to town on this thing and it’ll find a bunch of weird problems. Like that that’s that’s like that’s the other challenge here, man. It’s like love or hate LLMs are actually pretty good at finding vulnerabilities. So like that’s a huge part

Paul (37:10) Yes, they are.

Josh Bressers (37:11) of this mess.

Paul (37:12) They are. And and d some of these models are available locally now. I tested the Antares model from Cisco

Josh Bressers (37:18) nice.

Paul (37:19) locally. It was just okay. It wasn’t great. but I and I only ran it once against one one code base. But you know, in finding vulnerabilities is becoming easier, especially when you can have the web interface from whatever device, then you have the firmware for that device and you

pull apart the firmware and you grab the web application code from it and then you give the LLM here’s the live, here’s the, you know, dynamic testing, here’s static testing. You can do both now. and with access to the source code

Josh Bressers (37:51) Yeah, yeah.

Paul (37:52) and access to the live interface, man, L LLMs have a can have a much better time. Provided

Josh Bressers (37:58) Yep, yep.

Paul (37:59) you guide it and tune it using the right models accordingly.

Josh Bressers (38:02) Right, right. And and I’ll I’ll put a plug in. you have an Eclypsium podcast called Below the Surface, where you guys talk

Paul (38:07) Yes.

Josh Bressers (38:08) about like all this kind of stuff all the time. It’s a really fun show. I I do enjoy listening to that one. nice.

Paul (38:10) Yes, thank you. Yeah, we’re recording later today.

and we talk about you know, Vlad is one of our engineers and researchers,

Josh Bressers (38:17) Yep, yep.

Paul (38:18) and Chase is in our marketing team and he uses and they’re both very technical e the marketing person is also very technical, Chase. And we talk about how we use AI a lot as a topic of conversation. Yeah.

Josh Bressers (38:28) Yeah,

yeah. It’s pretty cool. All right, Paul. It’s been a treat, man. I dig it.

Paul (38:32) Josh,

thank you so much for the opportunity to come on the show.

Josh Bressers (38:35) Yeah, yeah, it’s always fun. Until next time, my friend.

Paul (38:38) Thanks, Josh.