Josh talks to Keith Hoodlet about a research project he co-authored about the success rate of AI generated vulnerability patches. Keith has a very scientific approach to this paper and does a great job of breaking down all the data and outcomes. There isn’t a lot of AI research that takes this approach, this paper is a breath of fresh air. Keith has some pretty scary details during the discussion, but he manages to give us hope at the end.

This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.

Episode Transcript

Josh (00:00) Today,

open source security is talking to Keith Hoodlett, the head of Off by One Labs at OnePassword. I asked Keith to come here because he’s one of the authors on a paper. I’ll let him say the name of it because it’s very long and I can’t remember it. But it is, I would say, taking a very reasonable and practical view of kind of AI patching, which is a super hip topic right now. And I love people who take like a very systemic scientific approach to this stuff instead of just like screaming.

You know, we’re all doomed or it’s going to save us all. So, Keith, welcome to the show, man.

Keith Hoodlet (00:34) Thanks, Josh. as I said, you know, when we were in pre-recording, longtime follower, first time caller, really glad to be here

Josh (00:39) Awesome.

Keith Hoodlet (00:40) and and connect with you. so the paper you’re referencing, we named it Fix Like Artifacts with Embedded Defects or Flawed, for those following along with the acronym. So gosh, I mean I never I never know at these things. Like it’s sort of like English is hard and

Josh (00:48) Nice. Is it an acronym or a backronym? Did did did you did you come

up with flawed before you came up with the the title or was it just like, if we change this one word?

Keith Hoodlet (01:02) so so in this case

it was a little I will, you know, a little sneak behind the curtain here for a second.

As we were working on it, we were like, you know, how should we reference what these patches actually are? And one of the members of our team said, Well, they’re sort of like fixed like artifacts, right? They they look like fixes, but they’re not actually fixes. And I was like, wait a minute, I think we can work with this. And

Josh (01:21) Nice.

Keith Hoodlet (01:22) and it suddenly became the acronym that that turned into the paper. I don’t think we started with that. We had some other like special project name for it, and then it became flawed.

Josh (01:29) Sure, sure. Nice.

Awesome. All right. So tell us what you’ve done, Keith.

Keith Hoodlet (01:34) Yeah, so one of the things that we did starting back on May the twentieth is we asked ourselves the question that I think most defenders were starting to consider, which is Mythos is now a problem that we potentially have to deal with. There’s going to be an increasing volume of vulnerabilities being disclosed, both in open source projects and in closed source projects. And so the the simpler question was: how effective are large language models at

Vulnerabilities, like actual vulnerabilities. And there’s been a lot of research when you look at academic papers that often will do these things in retrospect. So they’ll look back on vulnerabilities that were six or twelve or eighteen months old, or they’ll look at benchmarks that are of a similar age. And one of the things that is pretty well understood from working with large language models is that those

lengths of time usually means that content ends up in the training data set. And

Josh (02:33) Yes.

Keith Hoodlet (02:33) so solutions end up being in the training data set. And so therefore you have an artificial sense of success rates with this problem.

Josh (02:43) Yes. Yes. In fact, I had this come up one time as I was doing some testing. It was one of the Qwen models. I do a bunch of local stuff for fun.

Keith Hoodlet (02:50) Nice.

Josh (02:50) And I was looking for confusing code to see like ca how hard is it to trick these, right? Asking what’s going on. So I I looked up the obfuscated C programming competition, which you know us geezers are well aware of, where you

Keith Hoodlet (03:01) Yeah.

Josh (03:02) j write C code that looks like it does one thing and it does a very different thing. And I remember I pasted one into just was just Qwen, right? So we’re not even dealing with a super advanced, you know, frontier style model.

And it was like, this is an entry from this programming competition this year. I’m like, well crap, this doesn’t work. Like, you know what this is. But yeah, yeah.

Keith Hoodlet (03:18) Yeah. Yeah, yeah.

Josh (03:20) And and in fact, that’s funny you say that as well, because you cover this in your paper. But there’s also the problem of you did tests on open source software. And so what stops a model from just going to look upstream to say, well, how did they fix this bug, right?

Keith Hoodlet (03:32) Yeah, yeah, that’s exactly it. So so part of what we did when we selected six vulnerabilities, we just established high level criteria. First, it had to be impactful enough that it would drive attention from defenders, meaning that the sort of thing that you want to fix quickly, which generally would mean you you might reach for a large language model to help you address that problem fast. so the average CVSS score for the six vulnerabilities that we looked at was a CVSS of nine point one.

Or it’s like nine point eight, so I’m rounding up a little bit here. But that said, the second criteria was it had to be novel, meaning that it was less than two months old. so I think the oldest one was something like March the twelfth or maybe it was March the twentieth, and then the newest one was like May the twelfth. I might be getting those those numbers wrong slightly, but

Josh (04:19) Yeah, close enough.

Keith Hoodlet (04:20) yeah, it’s in the ballpark of what we were trying to accomplish. And that was because we didn’t want to have the memorization problem

Of a model sort of remembering how to solve this problem and then just going and doing that you know solution. And so those were the the first two big criteria. The third one, which is surprisingly very important, and I’ll come to this perhaps later in our conversation, but we’ve begun to see a trend where when you are looking at a patch that touches more than one coding file, so excluding tests, right? Like set those aside and look at just the code that actually operates the program. When you

Increase that beyond a single coding file, the rate of patch success goes down substantially. And so initially though, we didn’t know that, right? We we came into it with the assumption that these are open source projects, there’s going to be some level of memorization here, but also the the model should be familiar with these projects in the same way that perhaps a maintainer would be familiar with a project. And and so granted, novel, but

That said, new, complex, high impact. And the point that you just called out about memorization, one of the things we did in our study is we also looked for what we we called cheating. And it’s actually a formal term used in large language model benchmarking and testing, which was when it goes and looks for the answer upstream rather than actually doing the the benchmark test that it’s designed to test for.

So we generated over 6,480 patches in this data set, so a little over a thousand for each vulnerability. And 400 of those were found to be exposed in some way to the actual upstream patch, either via Git operations or web search or what have you. And so those got removed from our final reporting because we actually care about how successful it is at really patching, not how good it is at finding the patch somewhere else.

Josh (06:20) Right. And we should specify for anyone who doesn’t quite grasp what what Keith just said. I’ll try to summarize and you can tell me if I get this right. Is you took I’ll I’ll pick on I think copy fail was one of the one of the the big ones you you covered, right? The Linux kernel, it was a local privilege escalation. It was a huge deal when it came out. And you gave it the Linux kernel git repo up to the point I think you said in your paper one commit before.

The

fix, right? So, like right before

Keith Hoodlet (06:49) Yes.

Josh (06:50) the fix landed, you gave it the git repo, you explained the vulnerability, you gave it like reproducers and things like this, and then you basically said fix this. And sometimes, well, 400 times, well, less than that, because right, that’s all the issues, but some number of times it basically looked at the upstream Linux kernel git repo and was like, here’s the fix for this. And I assume it just took it and used it as it I hope it did. I mean

Keith Hoodlet (07:15) Yeah, yeah, yeah. So th those are

so so to use the sort of parlance from the paper, we we looked at five scenario outcomes, and the

Josh (07:22) Yes.

Keith Hoodlet (07:23) first scenario outcome is what we would call a successful patch that does not alter the application’s behavior. Meaning and I’ll explain perhaps that with scenario two, but those patches were looked at the upstream canonical patch would be categorized as a scenario one patch because it was like real.

accurate to the the real patch that it should be benchmarking against that we benchmarked against within the the actual paper itself. And so it was the accepted patch is is effectively what we were looking for. To talk about the other. So that’s like best case scenario. the second

Josh (07:58) Yeah, yeah. That’s that’s what we

want, right? Like that’s that’s the success, yes.

Keith Hoodlet (08:04) Yeah, that is that is, I mean, that’s the ideal outcome, right? Is is you get a patch that fixes the vulnerability, but it doesn’t change the application’s behavior. Because scenario two is a patch that fixes the vulnerability, but it does change the application’s behavior. So a really good example of this is when you turn an allow list into a deny list, for example, when you change

Josh (08:22) Right, right.

Keith Hoodlet (08:23) that logic, the application’s behavior has changed. It may still function, it may still pass all of your tests, but some use cases in the wild are likely to now break.

we’ve also seen some like file local parsing. And so you may have parsing logic that you’re using for like JSON strings or email addresses or something else where it will re-implement all of that code locally in a way that might break like plus aliasing on your email address, for example. It’s like this is not a well-formed email. Well, it was and it is, and up until that point it was accepted in

Josh (08:55) Right.

Keith Hoodlet (08:55) the project and suddenly it’s not because of this this change. So those are like the successful patch outcomes of the research.

Then we dive into like where things go off the rails a bit here. And so first, scenario three is an unsuccessful patch. tongue-in-cheek, I say that this is like rearranging deck chairs on the Titanic. the code may have changed, but the vulnerability is still exactly where it was, and it operates in still exactly the same way from an exploitation standpoint. So you haven’t fixed anything. scenario four is you fix the vulnerability.

By adding a new vulnerability.

Josh (09:36) Yes.

Keith Hoodlet (09:36) This was like this was one that I thought was both sort of like sad and humorous all at once. It was like a sad trombone moment because like it fixes the vulnerability, but it adds a use after free. It’s just like that’s not really what you want to see. So a new exploit

Josh (09:51) Right, right.

Keith Hoodlet (09:53) where the code used to live. And then scenario five, the the worst case outcome, was it does not successfully fix the original vulnerability, and it adds a new vulnerability.

So that’s like, you

Josh (10:04) Nice.

Keith Hoodlet (10:05) know, d double or nothing on the vulns so to give you a breakdown of percentages for your listeners, scenario one, successful patch outcomes without altering applications behavior across six thousand and eighty patches in this data set was only twenty-six percent.

Josh (10:22) Yeah, yeah.

Keith Hoodlet (10:23) And to give you just to give you the rest of the numbers, scenario two, twenty point one percent. So that means more than half

53.9% of the vulnerabilities that had patches generated for them either did not fix the vulnerability originally, added a new vulnerability, or both. which is is

Josh (10:43) Right, right. And like that’s the headline, right?

Keith Hoodlet (10:46) Yeah, yeah. And and the other side of this too, and like this is something that I think we we go into in a lot more depth in the research paper. And we do mention it and we call it out with I think bold text on the blog itself. But for scenario one and scenario two.

More than a third of those patches, so around like 9% of scenario one and around 6% of scenario two, were what we would consider fragile. And fragile in this case was a terminology we coined to represent patches that patch specifically to the vulnerability that or the POC I should say, but

The vulnerability still exists underneath. And so if you subtly change your proof of concept exploit, it meant that you could very likely now exploit that vulnerability. In some cases, it’s like it’s still fragile because the vulnerable code is still there, but the exploitation path has been now obfuscated in a way that it’s not currently reachable, but in a future coding change, it could become reachable again. So what’s interesting to me is shortly after we published this research.

There was a piece of research that came out around a vulnerability in Microsoft’s Defender code. and what was interesting about that vulnerability is Microsoft has very publicly said that they are using AI to generate patches and using AI in general for their code base. And the Shield Break zero day POC that came out on August the 12th, so less than a week later, was de found in Microsoft Defender as an

incomplete patch that led to system level access for an attacker. And so for me, it was like very real world validation that our research was very likely beginning to show itself in real world scenarios in major code bases because that is effectively either a scenario three patch. It’s you know same vuln slightly changed or you know

still exists, but it’s probably more like a fragile scenario one or scenario two patch where they patch specifically to the original vulnerability, but a subtle change to the POC and now you have same vulnerability, slightly different.

Josh (13:07) Yeah,

yeah. And the th that one, so I remember reading that and that that speaks to me. So I’ve been doing this like open source security thing for a long friggin’ time. And I’ll never forget it was very humbling where in my early days I would send reports to upstreams and I’d often send a patch and be like, this fixes the problem. And I think zero of those patches were accepted because I was often fixing like where the crash was happening when the problem was it was like a missing integer check, you know, three functions up.

Keith Hoodlet (13:38) Mm-hmm.

Josh (13:38) But I didn’t

know that. Of course, the developers knew right away. And I think this is this feels also very like LLM, where the LLM doesn’t necessarily have like a holistic view of what’s going on. It’s going to look at the crash or it’s going to look at the reproducer and say, this is where the problem is. Let’s add a check here. Let’s do this one thing here. When that might not be the right thing to be doing at all.

Keith Hoodlet (13:57) Yeah, and that’s that’s something that we talk about in the research paper as a challenge of large language models, which is the attention mechanism problem, right? Like attention mechanisms are how a large language model functions. And so we saw very explicit instances where code that was vulnerable was patched, but the same character for character code in an adjacent function was

was entirely like left to exist as it was. And so a robust patch, in my opinion as a human, you know, going back to my my days of leading a devsec ops team at the world’s largest laboratory sciences manufacturer,

Josh (14:34) Nice.

Keith Hoodlet (14:35) is you look for variants of a vulnerability as soon as you’re fixing the first one. You you look for same or similar code snippets, you look for same or similar code behavior.

And you then go and patch sort of all of the different places where those mistakes have been made. Sometimes that’s centralizing it in a library. Sometimes that’s just making sure you’re updating the same coding function in different places. And so that was a sort of interesting situation from the research itself where it’s like, okay, it’s it’s patching very explicitly to what it has been given, but it’s not thinking about like

all up application behavior and how the the impacts are perhaps somewhere else in the code base, which which I think happened with the Google Chrome or excuse me, Google CLI for for Gemini, where that was a in the research paper a call out where we talked about how there was like a an env file or a dot m file that people are used to having that was an introduction point for exploitation.

And that was entirely looked over. Like it was never touched by the large language models for actually patching that problem state. And again, you know, very interesting observation of attention mechanisms with proof of concept patching, but not necessarily thinking about what is the real underlying problem and where do we actually need to patch it.

Josh (16:01) So

let me ask you about that actually, because in your paper you talk about what, prompt richness. You have like an equation

Keith Hoodlet (16:06) Mm.

Josh (16:07) where depending upon what kind of information you send, did was and I don’t remember if I’ve read this or not, so apologies if it’s in here, but did you try asking the LLM to maybe take a more broad view on this, or did you always keep it very portholed onto just the vulnerability?

Keith Hoodlet (16:25) So we had a couple of different modes and a couple of different like prompt types that we looked at. So there were three modes and then there were three prompt or nine prompt types, so 27 total combinations that you could make from that. And one of the modes, or at least a couple of the modes, had the opportunity to go broader, such as AI agent discovery was one of the the actual prompt types that would allow the agent to determine like

How is it going to say that it has succeeded or not? And so it sort of gave it that freedom to move. but then in also the the modes themselves, we had an exploratory mode. So let me give you a quick breakdown of modes and and prompt types for your listeners. So we had a one-shot mode, which was just like just go patch it. You have like one attempt, like that’s the simple prompt response sort of effect.

We had an iterative mode, which was really meant to replicate some of the things we’ve seen around like Ralph loops, for example, where it’s just continue to work through the problem until you eventually solve it. And then we had an exploratory mode, which was allowing it to be a little bit more freeform in the way that it actually explores the bug and determines how the fix should take place. So that mode I think would cover what you’re positing here, Josh. But then in terms of prompt richness, though, I think this really factors in on the prompt type. So

Default was effectively what I would call hindsight or 2020, where you actually have all of the information. So you have like the the bug includes the CWE information, the stated vulnerability itself, the exploit call path, the entry point, the vulnerability primitive, exact APIs and the fixed direction that it needs to take. So it’s the sort of thing that you would have if you had already fixed it or if you were a

deeply knowledgeable maintainer about that specific area of the code and you understood the vulnerability and how it needed to be fixed. we also had static analysis tool output. So again it’s the concept of you got a you know SAS output and now you pump it into a large language model to generate the patch. We had two app sec engineer personas or types. The first was prescriptive on it had on how to fix the vulnerability and it gave correct information.

Like it it’s basically the engineer understands what the problem is and how it likely should be fixed and then steers the LLM in that direction. Because it also happens in the real world, we had an appsec persona that was prescriptive in the wrong direction. It’s

Josh (18:54) Yeah, I love that.

Keith Hoodlet (18:56) the the you know, this happens all the time in companies where you have a junior engineer or you have someone who’s new to the company, or you have someone who’s never worked in that area of the code base asked to, you know, solve that problem because they’re on call. And so they steer the LLM in the wrong direction.

We also gave things like bug bounty. Again, external person who’s now providing you a finding. So they may have some limited set of information that they can prove based on application behavior, but they don’t have code. so again, limited set of information. Embargo was interesting because it was meant to replicate situations where you’re a major open source project like Google Chrome as example, where you’re trying to patch a vulnerability, but you’re doing so in a way that’s intended to be subtle.

Because you don’t want threat actors to recognize that that’s an exploitable vulnerability. then the last couple or last few were a proof of concept. So just gave it the crash dump or the POC and said, like, hey, go fix this. we also give a dev self-discovery where it’s like a developer’s walking through the code, they see something and say, This feels wrong to me. I don’t know if it’s vulnerable. Can you please just like take a look and then go fix it? And then finally that AI agent discovery. So three modes, nine prompt types.

So that’s 27 combinations. And we ran 20 patches per what we would call a run. And a run is a combination of a mode and patch type. So 270 patches per model. So and then both GPT 5.5 with trusted access for cyber and then Claude opus 4.8 with cyber verification program guardrails. So

you ended up with twenty patches per per run and then you ended up with you know you have the multiple models that are running.

Josh (20:50) Right. Right. Okay. This is this is it’s no, this

Keith Hoodlet (20:54) So anyway circling back.

Josh (20:57) is great. I mean, i it it’s fascinating, right? Just because there’s like an infinite number of ways to use these tools. And I mean, I’m sure this is one of the complaints that a lot of the AI fans are gonna have is that you were you were holding it wrong, right? Your prompts weren’t right or you were doing this incorrectly. And I mean you do you have a graph in here, prompt richness.

Where

you show, like the better the prompt, the better the output, which I don’t think would shock anyone necessarily, right? Like that feels kind of reasonable, but but this still feels like I mean, here’s here’s the thing in my head, and and you can redirect me if if if maybe you have opinions on this or not,

There are a ton of efforts going on right now where the idea is like we’re gonna use LLMs to like patch all of open source. There’s a bazillion vulnerability reports, and we obviously can’t fix them all. So we’re just gonna point the LLM at them and tell it, like, let it rip. And I was hesitant when a lot of these were announced to think like this feels sketchy. And you have receipts now to say this is almost certainly sketchy, or maybe not sketchy, but.

It is going to be harder than you think, right? This isn’t like push button get patch. This is definitely like the devil is in the details on this. And so I I want these projects to succeed because goodness knows we we need to fix like the open source vulnerabilities that are being reported without burning people out left and right. But I’m also concerned now, like, are the patches we’re creating going to actually create more work than the just the vulnerability report by itself?

Keith Hoodlet (22:33) Yeah, the interesting thing, so there’s a couple of there’s a lot of different tables in here, admittedly. But if you actually look on

Josh (22:37) Yeah, yeah.

Keith Hoodlet (22:37) the top of page twenty-six, we have a nice little graph that talks a little bit about prompt richness and the introduction rate of net new vulnerabilities.

Josh (22:42) Yes. Yes.

Keith Hoodlet (22:44) And so on the far right of that graft is a prompt richness score of 21. And what that effectively equates to is the default prompt richness, which is you have perfect information or perfect hindsight,

Josh (22:55) Right, right.

Keith Hoodlet (22:56) which has around you know, a high-level 80% fixed success rate, right? So

If you can give the LLM perfect information and everything is completely accurate and the fixed direction, all of it is is truly spot on, you’ve got a four in five success rate for that. The interesting thing that you factor in is when we started to look at, and I don’t know that we have it in a table, but we certainly discuss it in the paper, is the moment you bring in incorrect information of any kind, we saw something like

You know, just correct information statistically averaged out for everything except the default, around like sixty-five percent. but you add in incorrect information, your success rate drops to fifteen percent.

Josh (23:42) Yeah, yeah.

Keith Hoodlet (23:43) And and so you you have a real problem here, I think, which is the human experts that you need can only scale so much. And

Josh (23:51) Yes.

Keith Hoodlet (23:52) the code base, depending on its size, is likely to be very complex and you’re going to need

probably multiple people who have expertise in different areas of the large code base that you’re working in in order to create that successful prompt to actually give all of the accurate information without any inaccurate information. And so it I mean, overall, yeah, we do see that prompt richness does seem to have a correlation with increased or improved rates of success. but when you look at what’s interesting,

Where you’re at like prompt richness 12 or 13. and I’d have to go back to the table to look at like what that correlates to overall. But your new vulnerability rate also jumps

Josh (24:36) Yes.

Keith Hoodlet (24:36) as you start to have correct prompt richness too. So that was a a somewhat interesting thing that we noticed as well that seems sort of concerning. It’s like with the cure you might get a little poison and you don’t know whether or not it’s poison until you actually go and review it.

Josh (24:55) Yeah, th this graph is super it’s a bathtub curve where like crappy prompts introduce vulnerabilities, but good prompts introduce vulnerabilities. And I remember looking at this, I’m like, what? How is what?

Keith Hoodlet (25:04) Yeah. Yeah.

It’s it’s it’s kinda wild, right? It’s kinda wild. And the the thing that we noticed as humans as part of this process is we did manual review of ten percent of the patches generated per vulnerability. So that ends up being something like five hundred and forty or f excuse me, fifty four patches, because it was two hundred and seventy total patches. time or yeah, fi fifty four vulnerab patches per

randomly selected, right? So trying to give as as much as much depth as we can. And humans reviewing subtly different patches to the same problem. my team described it as mind melting. Like

Josh (25:45) Yes.

Keith Hoodlet (25:46) like that is, I think, one of the real challenges that I I foresee happening in this space is you can generate a bunch of patches and you’re going to have some percentage of them being successful patches, but

knowing that they’re successful patches without altering the application’s behavior requires human review, which is something that Anthropic commented on and you know gave us a statement for the blog post that we published around, which is you need a human reviewer as the last step. So now you’re gonna have a problem of like fatigue for that person that has to review a bunch of patches

Josh (26:20) Yes. Yes.

Keith Hoodlet (26:22) that all look very similar, but some of the some are good and some are not good.

Josh (26:29) Yes, yes. It I mean this is this is the concern, right? Is the cost of finding vulnerabilities. I mean, even two years ago was quite high. The cost of finding vulnerabilities is quite low now, but the cost of fixing vulnerabilities has remained the same. Maybe it’s higher. I I mean it’s hard to say, you know, one way or the other. But that is that is the thing I think about constantly about things like this, where we can’t just throw like more AI at it and and solve our problems. You know, we

We can find more bugs, we can find more vulnerabilities, but we still we need a competent person somewhere in this chain, and competent people don’t scale at all. You know, as you’ve mentioned many times and is mentioned in this paper many times. Like that is what keeps me up at night, is like this problem. And and and unfortunately, the failure rate for people is they just burn out and stop doing anything. And now we have an abandoned open source project potentially that’s widely used, and that is the worst case scenario.

Keith Hoodlet (27:27) Yeah, yeah, you know, it’s interesting. I was talking with the executive director at the high performance computing cluster and organization at University of Alabama.

And they ran some Monte Carlo simulations, you know, according to our research and the percentages that we saw. And they said that effectively, and they’re still working on this, so I’m giving a little bit of a preview, but as

Josh (27:49) Face.

Keith Hoodlet (27:49) things stand right now, you know, without the additional data points that are forthcoming on additional reviews of open or public data sets around this very same problem, you end up with a a runaway vulnerability creation problem.

Just from the patching process, let alone like

Yeah. So so if you effectively you say, okay, you have a a hundred vulnerabilities, you now run AI to patch those hundred vulnerabilities. If you look at the percentages as they exist today, like a hundred vulnerabilities becomes 120 something vulnerabilities becomes 160 vulnerabilities. It’s like it’s it

Josh (28:26) my goodness.

Keith Hoodlet (28:28) you end up in a runaway state where suddenly you have more vulns than you can ever patch, no matter how much compute you throw at the problem.

Josh (28:34) This

like I I got a shiver, which doesn’t happen to me very often when you said that, but that’s not something I’d even considered as a possibility. But yeah, it’s it’s essentially a feedback mechanism where we now have more problems than we did when we started trying to fix the problems we have. my goodness. my goodness, that’s crazy.

Keith Hoodlet (28:53) Yeah. So that is

that is a real potential outcome, right? And so I I will say, you know, like now that I’ve given everyone a moment to freak out about this,

Josh (29:02) I’m gonna be freaking out about this for like months, Keith. This is my goodness.

Keith Hoodlet (29:06) well, good news, Josh, we’re working on seeing what we can do to overcome this, right? So Off by One Labs is not only we’re not just here to throw stones at glass houses, right? We’re also trying to figure out how can we like make more resilient glass. and so

Josh (29:20) Right.

Keith Hoodlet (29:20) or make it bulletproof.

Right. And so we are looking at additional things that we can try to see if we can improve the outcomes, you know, and just move the needle, right? So people often talk about like, what if you throw it into more loops? What if you throw it back into a vulnerability discovery loop and then cre recreate the patch again? Like, can you do that 10 times and you now get a 99% true successful patch? or or and the question is, I don’t know what the outcomes are going to be, but we’re going to try. And

So because of that, I also suspect though, and the case study section of the research paper gives a really good example of this, is there are very likely to be some vulnerabilities that no matter how many times you generate patches for it, they will always create vulnerable patches. And the case study section of the research paper that covers free Nginx vulnerability shows exactly that happening.

and that was a vulnerability that was disclosed as part of the Project Daybreak and Patch the Planet initiative from OpenAI and Trail of Bits, where that vulnerability, the initial one that was presented by the Trail of Bits engineer, was a scenario five patch. It introduced a new vulnerability and did not solve the vulnerability that was being reported. Maintainers basically said, no, we don’t want that patch, created their own. They created a scenario four patch where they introduced a new vulnerability that patched the old vulnerability.

And so we disclosed this to them, they fixed it, but we then ran that through ChatGPT 5.5 with Trusted Access for Cyber, which is what Project Daybreak was purportedly using. And all 270 patches that we generated for that ended up as either the patch that the maintainers generated in terms of its abstract syntax tree and functionality, or the patch that Trail of Bits created, or just like a true like net new vulnerability patch.

There was never a successful patch of any kind, scenario one or scenario two. And so that’s really concerning. And it tells you, you know, if you’re an open source project, if you’re a business, one of the things you should be doing is taking the tooling that we released publicly. it’s off by one labs on GitHub, dashes between each word. Yeah,

Josh (31:32) Put a link in the show notes.

Keith Hoodlet (31:33) yeah. Put it in the show notes. and take some old vulnerabilities in your code base that you know you have successful patches for, like.

Tried and true successful patches. And try using different AI models to patch that same problem with checkouts just before the successful patch came in. And then for yourself, see how it compares, right? Because you are going to find, and it’s in our data set from our research, some will have very high success rates, and that’ll vary by model or prompt type or mode or what have you. And others will have no success at all.

And we’ve seen that with vulnerabilities of companies that here at OnePass were we’ve acquired and we went back twelve or eighteen months and looked at vulnerabilities we’ve patched for them. And

Josh (32:11) sure.

Keith Hoodlet (32:11) we saw that. We s we saw basically the same outcomes occur. So right now I think that that’s the best that open source projects can probably do is stress test this against old vulnerabilities that they know they’ve successfully patched to determine success rates for their code base and the problems that they’re solving.

Josh (32:27) Right, right. man. This is so much crazier than I thought it was, Keith. Like,

Keith Hoodlet (32:32) Ha ha.

Josh (32:33) I don’t know if you’ve ruined my day or my month.

Keith Hoodlet (32:38) Well, my hope is Josh that the next time we speak, which which maybe sometime sooner than later, we’re getting ready to publish a follow-up piece of research on this, which is a little bit worrisome. But all that is to say, we’re also doing the work to see what we can what we can find out about how to actually solve this problem or if we can solve this problem with known methods and processes today.

Josh (32:59) Yeah. Yeah. I I

mean I like that. I I think, all right, so let’s let’s land this plane. And I think that’s

Keith Hoodlet (33:05) Sure.

Josh (33:06) that’s I I think that’s the message. Because you’ve just literally scared me half to death here with with the feedback loop of vulnerabilities. It’s just like my brain doesn’t know what to do with that at the moment. But as you’ve said, you’re trying to look into this. And and I do think right now, especially in security, it’s easy to be all doom and gloom and say we’re doomed. There’s nothing we can do. This is like untenable.

just go live in a cave or whatever. But but like you’re doing work to try to try to solve this. I I I know a handful of other people that are trying to do some work in this space. And I I think I don’t know if I’d say we’re necessarily making progress, but I think we’re starting to understand the problem. And I think that’s a huge part of this. Like this paper d it it it defines the problem very, very well versus I think there are a bunch of like AI fanboys that are just like just

Mo AI harder, everybody, it’ll be fine. And it’s like you are not helping in any way when you do this. So give me hope, Keith. Give me the hope I need to end the day.

Keith Hoodlet (34:05) Yeah, I think the the real benefit is of all of this is that for you and your listeners, there are people out here who both know how to use these tools and use them capably, like my team, for example, at Off by One Labs, that also hold the right amount of skepticism to be able to ask hard questions like this and then bring the answers

Josh (34:25) Yeah, yeah.

Keith Hoodlet (34:26) publicly. So as you people will see in the blog, as we’ll see in the research paper, you can

You can count on Off by One Labs continuing to publish all of our receipts. You’re going to get blog posts to the extent that we have time and availability, you’ll get full and extensive research papers. But we’re committed to also releasing the full data sets as we already have with the tooling from from like all of these six, you 6,480 patches we generated are all available to go and use with our tool to go and inspect for yourselves, right? So just know that there are people like Off by One Labs who are committed to.

working in the open to making this a repeatable process for people and trying to find real answers. And and if there aren’t real answers, also making sure people know about real problems so that they can hopefully avoid them for themselves.

Josh (35:13) Yeah, I mean this is th this feels very scientific to me because like anyone who’s kind of been in the world of science, you have to show your work, obviously. You can’t just publish a paper and be like, just trust me, bro, it’s fine. But then also part of the reason for that is there are people who will try to prove you wrong. Like that is part of the process. So I I love this. I think this is great. And I hope we see more organizations doing some work like this, because I think it’s hugely important and we desperately need it because goodness knows this universe is just

filled with garbage at the moment and it’s really hard to know what’s real and what’s not. So awesome. All right, Keith, thank you.

Keith Hoodlet (35:47) Thank you so much, Josh.

Josh (35:48) This has been a ma well, you scared me to death, but apart from

Keith Hoodlet (35:51) Ha ha ha.

Josh (35:52) that, this has been phenomenal. And I just want to thank you for the work and thank you for everything. And yeah, I can’t wait to chat again, man.

Keith Hoodlet (35:58) It’s our pleasure, Josh. And thank you so much for having me and looking forward to chatting again soon.

Josh (36:01) Awesome.