Josh chat with Roi Nisimi about some research he’s done into missing backport security fixes. Roi calls these Backport Gaps. The idea is looking for security fixes on the main branch of a project but not in a released version. This is the sort of thing that was very hard in the past, but is very easy to do now. Roi has some marvelous insights into how things used to work, how they work now, and what the future could look like. It’s one of those new and complicated security problems nobody really knows how to solve.

This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.

Episode Transcript

Josh (00:00) Today, open source security is talking to Roi Nisimi

The principal security researcher at Orca Security, Roi reached out to me and had a I will say an interesting observation about unv no, what’s the word vulnerability gaps? I I’ll let you explain because I forget the the terminology you used, but the it is brilliant and unexpected and totally expected at the same time. So Roi, welcome to the show.

Roi (00:25) Thank you so much. So yeah, I just I felt like it’s it’s a good idea to try to get to more people and educate more people about this subject because the more I drilled into it, the more I figured out how bad the situation is. So just to start out and create a high-level explanation of what are we talking about, I’ve been managed to find backporting gaps in a lot of well-known open source projects.

Which in simple words means maintainers fixed a security issue, they fixed a security vulnerability in the main branch of their GitHub repository, but they never actually added or cherry-picked the same commit to the latest release, which creates a vulnerability in the latest release, but not just a vulnerability, something that can be easily found by attackers because it’s public.

inside the Git history. So that’s just a high level explanation of the research and why I wanted to actually, you know, spread the word and get to more people and educate more people about it.

Josh (01:38) Yeah, yeah. I I love this topic. So I let’s break it down into maybe a couple simple pieces because I think there are many reasons this can happen. And it could be they’re fixing security issues and they don’t even necessarily know it’s a security issue. They might be fixing security issues and they forget, right? Because I mean, goodness knows we’re all humans and we often forget things.

Or they could think they’re fixing a security issue like in secret, we’ll put secret in scare quotes, and then they’re gonna backport it like some other day at at some point, right?

Roi (02:13) Yeah, so this can happen from a bunch of different reasons. And from what I’ve been seeing, they know it’s a security issue. They know they’re fixing a security vulnerability.

Josh (02:23) Okay.

Roi (02:24) And most often the commit actually refers to a GitHub security advisory issue. GitHub

security advisory ID. So that that was actually what helped me find the most severe, most valuable findings because it was was really easy to actually see in the commit

The referral to the GitHub advisory security advisory ID. So the reason they missed it is their maintenance maturity, which means some maintainers adopted more complicated, more complex automations inside their project, and some didn’t. Now backporting is

in and of itself a manual practice because only the maintainer know what releases, what versions are still being supported. Now people can create automations, but usually automation requires a decent or high skill level and usually people neglect it. If we’re talking about a project with not a lot of maintainers, then most of them are focused are focused on features and delivering value to to customers.

Not on not on the engineering part, not on the DevOps info part. So a lot of times this is being neglected, or it they created the automation, they added a label, a backport label that you can actually assign to the PR, but they never use it. Or when they use it, it doesn’t work as it should

Josh (03:57) Mm.

Roi (03:57) work. But this issue is is quite complex, and the evidence is we are seeing so many popular.

high-end projects that are suffering from it from this issue and we’re not talking about you know easy bugs we’re talking about serious vulnerabilities that can actually create damage to to customers and now I’ll I’ll hand it over to you I just want I just wanted to say that

The way I got to this finding, the way I got to this research is quite interesting, but we can get to this later, like how it all begins.

Josh (04:38) N no, I would love to hear that actually, because this is

Roi (04:40) Yeah.

Josh (04:41) so when you when you reach out to me, I read your blog post, I I think what you’re talking about is adjacent to what I had in mind, but this is equally fascinating and also one of the, I guess, uniquely open source problems. So yeah, let’s just start at the beginning. Like, how’d you get here? This is I I love stories like this. So like I can’t wait.

Roi (04:59) It’s an amazing story. And I actually I just I said that right now because I was like, okay, I I I should have started with the story because it’s a really good story. But a moment before the a moment before the story, I just read a LinkedIn post and I agree, I think it’s the most problematic open source issue we have right now in 2026, in the AI era of attackers finding against defenders. And I read a LinkedIn post, and in the LinkedIn post the the creator

Of the post, he said attackers can now exploit CVEs before defenders know there’s even a CVE to defend against, which totally makes sense because attackers can, and that’s what I’m doing right now. It’s crazy. Like I have a Slack channel, okay? I named it Vulnerabilities While We Sleep, and I just scanned the entire internet for backport gaps, and I feel bad to share the results, but the results.

are in the thousands. So I’m just gonna put it right here. The issue is real and it’s a serious issue. Now how did I how did I get to this

To this research, to this idea. It’s quite funny. So, you know, I’m the I’m the security savvy friend in my friends group. and whenever people have security issues, they reach out to me. So I had a friend who used an open source platform, which I would not mention the name of the platform, but it’s a platform that you connect to your DB and it shows you statistics about your database, and you can have great graphics.

And UI, and it’s all fun and everything. And he hosted it online, and someone hacked into this platform, and through this platform, the hacker has been able to actually run SQL queries on his database. So I was like, ‘Yeah, this happened. I stopped it. I you know, I I I closed the like I removed the public access, and then he gave me the name of the project. I was like, okay, I wonder if it was.

a one day that no one knows about or it’s a zero day or I just wanted to understand what what’s going on. So I I tried it I tried to actually research the project and I I I’m not sure what the hacker actually did over there but I found something different and what I found is they have they had so many vulnerabilities that they did not backport it.

So I’m not sure what the hacker actually used in his case, but this actually opened the door for me to try and look for the same pattern in different projects. So I tr I started my research with AI on this project, and I found three, maybe four vulnerabilities that are not backported And I was like

Wow, that’s crazy. Like I I was n I didn’t think that this can actually be real in a 100 K stars project. It felt surreal when I actually saw that and I was like, okay, maybe I can run this at scale and try to find the similar pattern in other projects. Because you know, I’m I’m a hacker, I’m not a builder in my day to day. I’m not

actually maintaining open source project. I mean I have open source I’ve developed a few stuff but I was I’m not really into the tiny little details of what it means to maintain an open source project and this is definitely one of them and when I saw that I actually decided to run it at scale and we know the results already so that is my story and that is how everything you know

developed until I’ve been able to find, you know, vulnerabilities in a project like NeoVim, for example.

Josh (09:06) Yeah, yeah. And you have a blog post actually. I’ll put a link in the show notes. It’s it’s titled what fixed on main, vulnerable in release hunting backport gaps at scale. I do like the term backport gaps. I think that’s that’s a nice term. And then you talk about you’ve got

Roi (09:18) Yeah.

Josh (09:18) a link, or you describe Neovim, I I think towards the bottom. and and that one I think what? It has a GHSA and a CVE So it was fixed on main but not backported to the current release, yes?

Roi (09:34) Exactly. CVE,

Josh (09:35) Okay, okay.

Roi (09:36) CVSS eight point eight, and it’s not part of the latest release. Was not part of the latest release

Josh (09:43) I mean

Roi (09:44) before I before I reported this.

Josh (09:47) It is it now? Did they did they include it?

Roi (09:51) So this this is something th that’s a different topic, like

being able to actually communicate the stuff to the maintainers, they did right now.

Josh (10:00) Yes.

Roi (10:01) But I I had to actually, you know, convince them to do so. I had to actually push them to do so. And I wrote a LinkedIn post about it that as I started to find more discoveries and I started to talk with more maintainers, I’ve noticed the same pattern that they they’re not willing to actually

release a new version even though their users are being exposed. Like they observe the cost of releasing a new version as as more than actually having their users expose. Now

I think that if you have the right automations in spot in place, releasing a new patch version should not be something that costs a lot. It should be something that is, you know, you should you should press a button and you have the patch version released. and if it’s if it’s something that puts your users at risk, I think this is something all of them should actually do. So many of the maintainers, I I was like, I told them, okay, when are you going to release the new

Version, they were like, we’ll see, you know, maybe this month, maybe next month. But I was like, But it’s a vulnerability, like people can actually hack into forget the vector, it can be pre-auth, it can be post-auth, you know, forget the vulnerability itself. Users are at risk. Why won’t you just release a new version? But they they have a lot of things going in their mind, and you know,

How does it look like? What’s the cost? So this is something that I’ve noticed as well.

Josh (11:44) I mean, I’m sensitive to this one just because I mean, this is a topic that comes up on the show constantly, is all of open source is being buried right now in security vulnerability reports. These projects are grossly understaffed, they’re underfunded, they have like l fewer resources than they could possibly need, often by magnitudes. And so I’m I I get it, right? I under I I think there’s two sides to this coin and I’m sensitive to them both where

I recognize like obviously when there’s c public security vulnerabilities in something that are bad, like yes, I think there’s often an expectation, like please fix this. But also I recognize like open source projects are super under-resourced. So like it’s a very delicate balance for sure. And I want to bring up something you mentioned just a minute ago, which is the difficulty in contacting the projects. So this is something I do in my day job where I

We I I work with the team that we do coordinated vulnerability disclosure. We reach out to lots of open source projects and try to coordinate, you know, vulnerabilities with them. And it is it is very difficult to find any sort of contact on a lot of open source projects, which I’m sure you’re obviously starting to figure out. Yeah.

Roi (12:53) So if there is no GitHub advisory security GitHub security advisory in the project, it’s almost impossible. It’s nearly it’s near impossible. I tried a lot of times to find the guys on LinkedIn, never picked up. you know, email never worked.

If they do not have a security advisory, or you know what? I had a lot of projects that they have bug crowd and hacker one. It’s it takes decades for this to actually get a response. I think if a project has a GitHub security advisory, usually these projects are the the best at communication in terms of security. All the other options

to the research researcher perspective from the reporter perspective is a complete nightmare.

Josh (13:49) Yeah. I’m totally sensitive to this. I I get it a hundred percent. Yeah, yeah. I mean now there is there is, however, the we’ll say the chaos gremlin inside of me says it’s already public, who cares? Just like open an issue pointing it out as as a a bug, right? Because often most of these projects do have issue trackers, you know, like GitHub issues or whatever that you could just open something in and it’s public and but it’s already public, so like what’s the difference?

Roi (14:15) the difference is you know, you can have two people looking at the same thing and only one of them actually sees the thing. So if I see the thing

Josh (14:24) Mm.

Roi (14:25) and I publish and I publish it to anyone, everyone sees the thing. And if amongst this everyone we have a bad guy who wants to hack into people’s data and steal their information, then you know I did wrong. So that’s that’s that’s the funny part about it and I think it comes down to skill set.

Because nowadays everyone can do everything, right? Like you have AI, so you can do everything. Like I’m a security vulnerability researcher. Everyone can research vulnerabilities. Everyone can be me right now.

True, but not true, because everyone can actually close the gap that I’ve been accumulated for the past twelve years, but none of them can see the things the way I see them. None of them can actually understand patterns the way I understand them. So it all comes down to, you know.

How do I use it? And if I know how to use it correctly, then I can find the way to create damage. And if I share this with all the community by creating a GitHub issue, then it’s bad news. If it’s only rested in their you know commit history, and I know how to use AI in the right way to actually find loopholes, then only I can find it. But once I publish it to everyone, then everyone can do it. So

That’s that’s one of the reasons why you know I would not do it and to f to be frankly honest, I have a lot of projects which I I didn’t find a way to actually report this. So it’s still in my backlog.

Josh (16:06) Sure, sure. Okay. I agree with everything you said. I do think there is a there is a gap between people running AI, you know, like the dog meme.

versus security researchers who know what they’re doing a hundred percent. But I do think in the context of what you’re talking about, this is low hanging fruit just to find this stuff. Not necessarily exploiting it, right? But just finding it, I do think the LLMs are really good at this because this is something I have multiple friends and and contacts at at some are companies, some are just randos doing rando things that are looking for similar information. And it it goes beyond just what you’re talking about, ‘cause you’re talking about things fixed on main

But there’s also people looking like back in time saying, I found an old commit. This is clearly a security vulnerability. It was never tagged as as a security vulnerability. I’m going to point this out to the world. And like that is shockingly easy to do right now, which is just terrifying, I think.

Roi (17:04) It it is. It is. and still you need you need to know how to steer AI to actually do it in an efficient way. But to be completely honest, I agree because nowadays if you just and I had a conversation with a friend about it, a very high skilled researcher, he just had a very

Impressive project that finds vulnerability for him. I would not share the details, but he showed me his methodology, and everything was breaking down to a lot of things, and then there was this one component who actually finds the vulnerability, and I was like, Okay, how did you do that? And he was, I just asked Claude to find the vulnerability. Just and I told him, What was the prompt? You know, and he was the prompt was go ahead and find the vulnerability. And obviously, like

Josh (17:57) my goodness.

Roi (17:59) obviously.

stuff to actually being able to do this, but it was not context to the LLM. It was just, you know, creating a baseline. And at the very end his strategy and he was like, you know, but I’ll improve it in time, but it worked found like so many vulnerabilities that right now I don’t have to. So his prompt was just Claude go ahead and find a vulnerability. So I I agree. You know to create a a a scaled solution as I did I think

you should be more skilled but to actually go and if you want to hunt a vulnerabil a single vulnerability in a single project using the same technique you can probably do this with a single very very novice prompt

Josh (18:49) Yes, yes. And I mean something I’m seeing in the industry right now, especially like the reports we take in at work, is it is very common to get multiple vulnerability reports in parallel, because obviously a lot of people are using the same LLMs, s often the same skills. And you know, unsurprisingly, they’re finding the same things. Which is also one of the things I wonder about, and I’m curious on your thoughts. Is so you did this work, you found this data. How like surely there are

Nefarious organizations doing the same thing.

Roi (19:21) So it all depends on w what are you looking at because I think a lot of people, a lot of companies, organizations, researchers, they’re using AI to find vulnerabilities. And I did the same. I just choose I decided to choose a single path of how to find them. So right now in my head there are

Three paths that I think are very legitimate. One of them is what we’re talking about right now, which is the backporting gap. And it’s a very unique and lucrative way to look at commit history. And hackers have been looking at commit history throughout the years to find a lot of things, mostly secrets in recent years, but now you can actually you can actually understand behaviors and understand and label diffs. So backporting is

Number one. Number two is you can look at all the CVEs, right? That a project had. You could look at their patches and you can try to find a bypass. That’s a different strategy. That’s a different way to steer AI to find vulnerabilities for you. And the third thing is

You can look at a project, you can look at all the CVEs and you can start to build this pattern of where developers are making mistakes and tell AI to actually find similar vulnerabilities to the ones that are already being published. So that’s a third way to steer AI to actually find things for you in a more efficient, more effective way. And the fourth thing is you know to just tell them to

Go and find something, whatever it may find find. So everyone is looking, everyone is using AI to find vulnerabilities, not all of them are using it or steering it the same way, and not all of them are focused on the same projects. Now I I I can allow myself, I can afford myself to scan the entire open source

entire open source scene because I’m well funded. I would put it this way. and I

Josh (21:43) nice

Roi (21:43) don’t mind. So other people, you know, they can focus only on a few, they can focus only on a few strategies. But again, it depends on how do you use AI, how do you steer AI, and on what targets.

Josh (21:58) Yeah, yeah, for sure. And and I see both of those, right? Where I see reports coming in that it’s clear someone pointed their tooling at one project and there’s other times you get like fifty reports from the same person. That’s just like a total random selection of things. And it’s like, I don’t even know how you came up with this list, but whatever. Whatever.

Roi (22:18) Yeah.

Josh (22:18) Okay, man. All right, all right, Roi. I wanna we’re gonna have to start landing this plane. So I’m I’m curious what your thoughts are on kind of what’s next. Like what

Where do you think this sort of thing is going to go? What like what are you working on? Like what’s your next goals in this project? Like what should we be paying attention to?

Roi (22:38) So as I mentioned before, I have an automation in place and I’m getting no notifications all the time and and the the aforementioned Slack channel.

And it’s it’s pretty crazy because I told the joke that okay I I I can you know I have no work anymore. I can just look at the channel and I can

Josh (22:59) Yeah.

Roi (22:59) find a vulnerability, new vulnerability every day. Now you can debate whether people would consider it a vulnerability or not because it’s always a duplicate, but that’s that’s a different topic. you know, we are being faced with a crazy time. We have a new vulnerability every time.

every day and it’s open source. It’s always open source. And again I had I had a post because I’ve been in this in this game for twelve years now and when I started to be really you know like really skilled we we had in in my little community in my little group we had this this saying that if it’s open source it’s more secured

Because there are a lot of eyeballs on open source. So probably someone already found the vulnerability that you’re trying to find. If it’s closed source, no one is giving attention, probably you can find the vulnerability because no one is looking at it. And it’s not true anymore. And I think if you’re asking where are we going, we are going to see endless, endless vulnerabilities in open source.

Endless and it’s gonna be really difficult for defenders to actually know what to do because it’s really difficult to use AI for the defensive. I mean you to be

You know, let’s take a maintainer as an example. Okay. So the maintainer he focuses on delivering value, on creating the best features. And as you said

The open source community, they ha they’re they’re not well funded, not many maintainers, not many people are working on this project, so they cannot actually allow themselves to also run AI harness that will find all the security bugs and all the different strategies that I I just mentioned. But all the other hackers, researchers, they are they are doing it. So the maintainers are not even in the game of finding these loopholes that attackers can now do at scale with AI.

So

We are about to see more. That’s what I can say. And I think we would have to create a shift in open source security because right now, if only the bad guys or the researchers right are in the game and no one helps the maintainers, no one helps open source to be in this game as well, of creating the right harness to utilize AI

Josh (25:46) Yes.

Roi (25:47) to find the loopholes, then

open source the maintainers, the project owners will always be on the losing side. And that’s what we’re seeing right now and we’ll keep seeing it if if we’re not if a shift would not happen.

Josh (26:03) That’s a lot to unpack, man. And and what I will say is I don’t know if open source project will ever want to necessarily like run the the the security tests. I mean, that’s obviously up to them because some some projects wanna, you know, like I wanna make a library that makes cat memes. I don’t care about the security crap. And I get that. I’m totally sensitive to it. But there’s definitely projects that would want to do this. I mean there’s

Tons of projects that like fuzz, right? They’ve got like AFL running in their CI pipeline. So they’re fuzzing every commit and things like that. And so this is kind of maybe a natural progression is incorporating some of this tooling. But additionally, just, you know, accepting the reports can be like like a huge deal. And the thing I don’t know the answer to yet, and this is what no one does, and I would love if you had any thoughts on this, is finding vulnerabilities used to be really hard, right? It had to be someone very skilled like yourself, and it took

you know, days, weeks, or months to do that. Now we have tons of people finding vulnerabilities in like five minutes while they sleep and and whatever. But fixing vulnerabilities is still really, really expensive, right? So the scales are completely lopsided. They used to be balanced, but they’re not balanced anymore. And I don’t know how we can add more weight to the other side because we can never take the weight off this side, right? Like that, that, that it’s done. This is the way it is. So we have to find a way to help the fixing side. And I I don’t know what that is.

Roi (27:26) me neither. And

Josh (27:29) That’s a fine answer.

Roi (27:31) Yeah, that’s an answer. You know, AI became really good at finding the ver finding vulnerabilities and it’s really good at at writing code. and I you know, I I never try to actually use AI to find the right way to patch it because I was I was never on the other side of working on a patch.

Josh (27:54) Yeah. It’s not it’s not great.

Roi (27:56) maybe may may maybe I’ll try it and see how it goes.

But

You know, at the very end it’s it’s way more it’s always it’s it’s more comfortable to be on the attacking side because you don’t care about the consequences. You don’t care about whether

Josh (28:12) Yeah. Yeah.

Roi (28:14) it it per works perfectly good or not. You know, there there are no consequences. But when you use AI to create a patch, there are tons of consequences. Because you can create a patch that will create

Josh (28:24) Yes.

Roi (28:24) a vulnerability at some at some it’s a different place that you’d not think about. So I agree with what you said. The cost is still high.

And you know, you cannot just ask AI to find a patch and you know lift your hands and say, I trust you, good job. As an attacker, you can do that because what can you cannot nothing there’s nothing to lose. But and you know, that’s exactly the thing because what we are seeing right now, and I’m seeing it with the work I’m doing right now, is the following. And

I hope people will pay attention and I hope researchers will not. But I’m seeing a lot of patches that are easily bypassed because they are written by AI.

Josh (29:15) Yeah. Yes. Yes.

Roi (29:16) And it it re it reveals it reveals the story.

Josh (29:20) A hundred percent. I’ve had Daniel Stenberg on this show many times from Curl, and that is one of the things he’s mentioned many, many times is often he’ll get patches with his security reports, and the patches are almost always incorrect because they’re fixing the problem in the wrong place. Whereas obviously he and his team know the code and they know where they actually need to put the fix, which is yeah, it’s tough, man. But okay. Okay, Roi.

Roi (29:45) Usually usually

b back back in the days also that’s important to say, back in the days when the researcher reported the vulnerability, he actually knew what he’s talking about. So he could work with the team to patch it correctly.

Josh (29:57) Yes.

Roi (29:59) And right now that’s not the case because

Josh (30:00) That yes.

Roi (30:02) the researcher has no idea what he he what he found. He just he can just deliver the information and the security team can just need to just deal with it.

Josh (30:12) Yes. Yes. my goodness. You’ve given me so much to think about. Man, this is I love this. This has been awesome. All right, Roi.

Roi (30:19) Yeah.

Josh (30:20) I’m gonna call it. This has been an absolute pleasure. Like this research is great. I can’t wait to see where it goes from here. I’ll be certainly paying attention. I’ll have to have you come back at some point when you find, you know, a million unpatched vulnerabilities or some horrifying number for sure. But this has been a treat. I mean, thank you so much for the time.

Roi (30:33) Yeah. Still still counting.

Sure, same. Thank you, Josh.

Josh (30:39) Yeah.