Join us for a conversation with Foxboron (Morten Linderud) and Anthraxx (Levente Polyak), members of the Arch Linux security team. We talk about the difficulties of maintaining a Linux distribution, the challenges of handling CVEs, and the dedication of volunteers who keep the open-source community working (and how overworked those volunteers are). We explain what makes Arch a little different, how they approach their security process, and what sort of help they would love to see in the future.

This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.

Episode Transcript

Josh Bressers (00:00) Today, open source security is talking to Morten AKA Foxboron and Levente AKA Anthraxx

They’re Arch Linux developers and members of the Arch Linux security team. So Morten and Levente I’m ecstatic to have you here because Arch Linux, feel like is, if there could be a UNESCO heritage Linux distribution, it’s Arch. So I just, why don’t you two start out? I’ll start with Morten cause he’s the farthest left on my screen, but you know, give us a little intro, kind of tell us what Arch is maybe and we’ll go from there.

Morten Linderud (00:21) but but but but

⁓ so Arch Linux is a Linux distribution that was founded in 2002 by Judd Vinnett, I believe. He’s a Canadian software developer. ⁓ It’s sort of, I think he was annoyed by crux, so he wanted something else a bit more ⁓ usable. So he made up, so he did, I think he did Linux from scratch, bootstrapped the distribution from that and then wrote Pacman, which is the Pacman package manager.

And these days it’s sort of, it’s sort of this sort of one of these sort of sort of unique Linux distros where we don’t really do a lot of things out of the box. It’s more of sort of choose your own adventure as to pick and choose what software you want, what desktop environment you want, how you want everything to run and go and how you want to set it up. There’s not a lot of hand holding. These days we do have an installer which sort of does, can set up most of it up for you. But it’s sort of just sort of been this sort of

I wouldn’t say it’s an archaic thing from days old when you had to do things yourself, put strap distribution and stuff, but it’s been working out for 23 years now, which is sort of a long project, but none of us has been here from the start at all. I didn’t know what Linux was in 2002.

But beyond that, think the most famous made before is the ArchWiki, which is the resource documentation for the distribution that a lot of people use. So that’s the short, interesting intro.

Josh Bressers (02:07) And Levente what about you?

Levente Polyak (02:10) So I’m Levente, I’m an open source software engineer and I joined Arch in 2015, which is 11 years now. Oh man, like time really flies. I didn’t even notice.

Morten Linderud (02:20) You

Josh Bressers (02:26) Nice, nice. Awesome. Okay. So, and you two are members of the Arch Linux security team. And I asked you to come here because obviously this is the open source security podcast. And I feel like there is a lot of value in, first of all, just kind of understanding what does it mean to be part of the security team for, for somewhere like Arch. But then I also want to just talk about the fact that like one of the things I’ve noticed, and I’m sure many, listeners have as well.

is whenever I search for a question about some Linux thing I have going on, I would say 80 % of the time I land on the Arch Wiki, which is amazing. And one other thing is your security vulnerability data is amazing that your team publishes. So let’s just start with kind of the Arch Linux security team and what does that mean? And like, how big is it? What do you do? What’s going on there? I’ll let you two can thumb wrestle for.

for who gets to start.

Morten Linderud (03:24) So, Leventha started this, so let’s start the team so you can begin I think.

Josh Bressers (03:28) Nice.

Levente Polyak (03:31) Yeah, maybe I can give a small rundown. So today we are seven members in the security team, but it didn’t used to be like that. So when we started the efforts, that was actually even before I was officially part of Arch Linux. So in 2014, there was this crazy idea how about actually running a security team for the distro. I know crazy idea.

Morten Linderud (03:32) Hahaha

Josh Bressers (03:47) Nice.

Morten Linderud (03:58) Hahaha

Levente Polyak (04:00) ⁓ Before that, there were a little bit of efforts here and there, but it was really very unstructured. There was no really a team behind it or anything like that. And then some ideas started to circulate also by using the Wiki actually. So just a bunch of tables trying to keep track of all the information. ⁓ And out of that, basically we did like a small ⁓

kick off brainstorm some ideas and then in 2014 we were founding the Arch Linux security team which is now like the official team in Arch Linux and one year later I joined as a package maintainer and that was kind of interesting so I was already part of the team before being part of the team so yeah.

Morten Linderud (04:51) Ha ha ha.

Josh Bressers (04:52) guys.

Morten Linderud (04:54) Ugh.

Josh Bressers (04:56) Awesome, awesome. All right, Morten, I’ll ask you the question then. What does like the day to day look like when you’re a member of a Linux distro security team?

Morten Linderud (05:05) So I think at this point it’s important to sort of prefix that none of us are paid to do this work. It’s all volunteer based. I think it’s extremely important. And I also think it’s important to point out that even though there is a team and we do deal with security things, it’s important to point out that the team is a bit burned out because of the CVE ecosystem itself is sort of not necessarily in a good place. And sort of

Josh Bressers (05:10) Yes, thank you.

Morten Linderud (05:32) consuming the firehose that is CVE data is extremely difficult, especially if you don’t have the resources to sort of dig through it on day-to-day basis. So at some level, it’s sort of important to point out that even though there are activity, it’s not that much activity that used to be. It dropped off, especially after the liberalization of the CNA setup. So these days, most of deal with sort

what’s sort of high severity things that need to be dealt with. then sort of low severity, medium severity stuff is, mean, some packages, I act on it, but we’re necessarily going to do advice on it because people are burned out. But like day to day, it’s mostly, mean, being there for the team is important. Having people that sort of understand CVEs and can handle patching and sort of understand how to communicate with things. A lot of this is at some level also sort of

open source intelligence based. we do chat with members from other Linux distro security teams, exchange information, somebody going, this looks strange. Is this a CVE and what else I’ve seen and stuff? then some people do trade patches. So a lot of the work previously was sort of track time patches. Linux kernel patches is always interesting because people don’t know which patch relates to which CVE and how it gets fixed.

Levente Polyak (06:39) you

Josh Bressers (06:51) Yeah.

Morten Linderud (06:55) Chrome stuff is interesting to deal with. They do a new release and do like 20, 30 CVEs and you don’t know which patch belongs to which CVEs. Since we’re also just dealing with those sort of ⁓ monthly, weekly releases and then sort of figuring out what was ⁓ the important stuff and what should be acted on. Most of it is through, sort of, OSS security. You have the distro embargo list, but you also sort of have IRC channels with ⁓ us and other security team members from the

from things like SUSE even to Debian, Red Hat, Alpine, NixOS, Gentoo So sort of exchanging information on a need-to-need basis. So it’s mostly just staying on top of what is important and what’s less important.

Josh Bressers (07:40) Which is a huge job. I remember I was doing this work, it was probably a decade ago at Red Hat, and it was already ridiculous at that point. I will say, it always pleased me that all the security teams, whether they were competitors or not, or their management liked each other or not, all of the Linux security teams got along quite well, which is, I’m glad to hear that’s still a thing.

Morten Linderud (07:45) Yes.

Yeah.

Yeah. And like, it’s, then like, that does a lot of fun things, like if you’re not sure if the CVSS score from MITRE is correct, then you can least go check Red Hat at the most. They’re a bit more correct. And this previously, previously now, I’m not sure how true is this, but previous years, previously, right out to the model of the enrichment stuff around CVEs So if you didn’t have all of the context for the different CVEs entries, you could just.

look at the Red Hat security database and usually add a little bit more information a bit earlier before it was upstream to the NVD ⁓ database. So it’s quite nice.

Levente Polyak (08:38) Yeah, I also enjoy it very much. So sometimes it can feel like there is a competition between distros for the user base. But I never had the feeling that really also bleeds into like the security team. It was always a very pleasant interaction with all the different security team members from the different distros as well. It’s very helpful like ⁓ trying to help coordinate what kind of patches you need to backport if there is no release.

Morten Linderud (08:40) Yeah.

Levente Polyak (09:07) ⁓ which of the software configurations may be affected or not. So I had ⁓ quite a lot of chats with different people out there in the community and it felt always very welcome. it’s more like we try to keep everyone safe across all these drills.

Morten Linderud (09:25) It’s quite nice.

Josh Bressers (09:26) for sure. Okay, you just said the word backport. What’s a backport?

Levente Polyak (09:31) Right. in a perfect… I mean, it is because it’s also some most of the time a little bit of a sad story. In a perfect world, we wouldn’t actually need to do that at all. A backport actually is… ⁓ software, just try to describe it a little bit more high level so everyone listening may understand.

Morten Linderud (09:32) Ha ha ha ha!

Josh Bressers (09:33) That’s a trick question, everyone.

Levente Polyak (09:54) So when you write software, it’s tracked somewhere like nowadays mostly in Git and you have individual patches which are commits for contributing to the project. And backporting essentially means if you have an old release because development just keeps going and they’re not cutting a new release which contains the security fixes, then you need to do something to still not be exposed to that security vulnerability.

And then you need to like find out which of the patches are relevant to fix this security hole. And then you are basically applying this patch to the latest release. And that’s called backporting patches on top of old releases.

Morten Linderud (10:39) Thanks.

Josh Bressers (10:40) And

it’s super common in the Linux distribution.

Morten Linderud (10:44) Yes.

Levente Polyak (10:44) Yeah,

it is and that’s kind of the unfortunate part because that creates so much work across all the different distros and in a perfect world when there is like any kind of security issue, then in my opinion, Upstream should just like roll out a new release and call it a day instead of like replicating all the efforts across all the different distros and yeah, that’s the unfortunate part.

Morten Linderud (10:47) You

But the end result we do get is sometimes when there’s embargoes and stuff like distros have like a patch and they go like, oh, this is not going to apply on top of our super older list. So here’s a backport reverse of this patch and they will just distribute it because all this problem needed, which is a bit tedious sometimes because then you end up with like three or four different patches or the same secure disclosure and people need a different set of patches to fix it.

Josh Bressers (11:35) yes, yes.

Morten Linderud (11:35) But it’s

also sort of a collaborative effort though, which is sort of nice ⁓ between these shows. people have, for some people it’s easy, for some people it’s harder to do. think Arch sort of stays a bit more ahead on package versions usually. So we sort of avoid a lot of the trickier backboards where there’s the CVE for major version four, but you have package major version two, so you had to backport it like two entire major versions back behind.

We usually avoid those issues to large degree, so we just have the backport like one minor version or one patch release version, which is not that hard usually. You just apply schema

Josh Bressers (12:16) Now that you’ve said that, you’re going to have something terrible show up, like tomorrow. That’s going to be one minor version.

Morten Linderud (12:22) ⁓

Well, maybe? I mean, it’s Tuesday tomorrow, isn’t it? ⁓

Josh Bressers (12:27) ⁓ we’ll see, we’ll see. that’s, ⁓ anyway. Okay, so

I think one of the questions that is going to pop up is if you are a modern developer, you’re probably doing NPM or Python or Rust or whatever. And I think the common method now is just to run the latest version of something. So this concept of backports probably feels foreign to a number of developers. Like, why even do this? Why not just?

I mean, Arch, like you said, is a pretty modern operating system. Why can’t you just keep running current on all of this open source?

Morten Linderud (13:04) Is this a question? It’s a statement. I mean, some people can’t run latest. Some people need stability in their life. So they would preferably use the latest LTS release Ubuntu and get their enterprise patches on top because it gives you some stability or some foundation to build a platform that can run for a year without you having to do larger refactors,

Josh Bressers (13:06) I mean, no, it’s meant to be, I’m setting you up for an easy answer here, because I know what the answer is, but not everyone does.

Morten Linderud (13:35) larger iterations of the platform and doing software releases. While on Arch breakages happen, I mean, not regularly, but at some level you have to deal with new versions when you update and you have to see on top of that. You can’t wait two years and then sort of expect things to work because there’s going be a lot of software changes you need to deal with. Doing the stable release stuff sort of gives you a bit more stability on that front.

Josh Bressers (14:00) And it’s probably worth pointing out that a Linux distribution has a very delicate web of dependencies, we’ll call it. So if something just depends on an older version of another thing, upgrading that to latest is not an option, right?

Levente Polyak (14:18) yeah, sometimes that’s also like a lot of work when we get into the area of package ⁓ maintainership. Of sometimes you need to chase upstream to make a version compatible. And sometimes it also feels a little bit tedious to convince them like, why can’t you just run the old version? Because like not every software ecosystem is designed in a way that you can run multiple versions of the same library or whatever.

Morten Linderud (14:29) ⁓

Levente Polyak (14:48) then sometimes that’s not so easily possible and you just have your space for the library and ⁓ you need to link against it and that’s just latest. And then sometimes you need to come up with also supporting older library versions and…

But that’s a lot of burden also for the package maintainers trying to figure that out. So the easiest is just to have whatever works and convince upstream to use latest. That’s not always easy.

Josh Bressers (15:24) Well,

and these are volunteers, right? This isn’t like someone being paid to do this horrible work of, you know, backporting or upgrading or any of this.

Levente Polyak (15:34) Yeah.

I mean, that’s the hard part also in Arch Linux. So all of us are volunteers, which means we have basically only intrinsic motivation. So either it sparks some kind of joy and that’s why we do it or just not getting done. So there is no extrinsic motivating factors like, okay, I get my paycheck anyway. So whatever, I’m going to spend eight hours now.

Morten Linderud (15:47) Hahaha

Levente Polyak (16:03) ⁓ entering some CVE data into inform or whatever.

Morten Linderud (16:04) Okay

Josh Bressers (16:07) Yes.

Morten Linderud (16:08) Like

last week I worked eight hours, came back, came back. I like, yeah, no, I need to fix the Go compiler. I tried to a new version of the Go compiler. The test doesn’t work anymore because there’s a memory leak thing deep in some sanitation code thing happening somewhere in the compiler. think it’s related to the G-Libc and or a new version of GCC. It’s seven in the evening. Do I want…

do something else or do I want to sort of stare at the Go compiler and the glibc C code for three or four more hours? It’s, it’s so sometimes it’s, it’s, it’s a bit hard to find motivation. The answer to that question is that I disabled the test. So I’m not running the tests and I built and pushed the package because what else are you going to do? There was, there was a bug file in June. it’s.

Josh Bressers (16:47) Right.

Someone will file a bug, it’s fine.

Morten Linderud (17:07) I’m not push more you can do. ⁓

Josh Bressers (17:08) Nice.

I mean,

again, like take it out of your paycheck, right? It’s fine.

Morten Linderud (17:16) Yeah, yeah, I wish, Yeah.

Levente Polyak (17:16) Exactly.

Josh Bressers (17:20) ⁓ No, I mean,

that’s a big deal. I cannot stress enough how much of modern society is held in the backs of unpaid volunteers. And so, like, thank you both for the effort, because Arch is awesome. And I know how much work this is. Like, I was paid to do it, and it was terrible.

Morten Linderud (17:35) Thank you.

Buh.

Josh Bressers (17:42) Okay, so I do

Levente Polyak (17:42) Yeah.

Josh Bressers (17:43) have a logistical question for you. So you’re talking about backporting security issues or upgrading packages to fix security issues or whatever. And so I don’t know the answer to this question at all, but I’m curious, like, what is the dance between the security team and the actual package maintainer? Does the security team do all the work? Do you work with the package maintainer? Like, what does that look like?

Levente Polyak (17:46) Yes, sure.

Morten Linderud (18:03) It’s all flat, like, several of the security members are packages stuff. So we mostly just work as sort of a…

It’s more of a supportive role, really. If there is a huge issue, it’s coordinated with the packagers because they know the upstream better than us. They know the source code. Well, they might even know the maintainers. the logistics of, again, it depends. Is this an embargo? Is this a public disclosure thing? Or is this a vulnerable version? Sometimes we just deal with it because it says fine. Sometimes it requires a bit more coordination because they know something we don’t. And other times it’s literally

Josh Bressers (18:21) Okay.

Morten Linderud (18:45) Hello, this is a world-breaking security embargo thing. We are updating your package. You can’t know why. It depends a bit on the severity, depends a bit on the disclosure model of it, and it depends largely on how the upstream works, how to do the patch release, how to fix the patch first, or is it a package update that just has to be done on the same day?

Josh Bressers (19:12) Okay, I mean that makes sense.

Levente Polyak (19:14) Yeah, for like kind of simple things, ⁓ mostly you maybe just ping the package maintainers, that’s fine. Like also, like Fox said, you want to do some kind of coordination if it’s like a high severity or really a very critical issue. But we also, like most of the team used to not be package maintainers themselves. So it just kind of bleeds into like…

If you do that work for long enough, then instead of annoying other people to do the final thing, like rolling the actual package, you can maybe just offer more help and do it yourself. And I guess that’s also the reason why it’s just a matter of time, but mostly everyone on the security team is nowadays the package maintainer as well.

Morten Linderud (19:55) Yeah.

Josh Bressers (20:08) This does not surprise me at all. I know how this works. it’s like any project, it sucks you in, right? It’s like, fine, I guess I’ll do it. And then before you know it, you own that thing. And it’s like, no, what have I done?

Morten Linderud (20:17) Yeah.

Or you just do like, oh, we can just do this. And people are like, yeah, sure. Go do it. And then you’re like, oh, wait, wait a second. Am I maintaining now? And they go like, yeah. That’s not what they call you now. And then suddenly 10 years passed and you’re like, oh wait, I’m still stuck with this. Yeah.

Josh Bressers (20:30) Yup, yup. That’ll teach ya.

Yes,

yes for sure for sure, okay Okay, ⁓ it’s funny cuz it’s true

Morten Linderud (20:47) Yes, and like I’ve been there snap so many times people going like I want this to work and you’re like, oh, but you can just do this and they were like, yeah, and I go like, yeah, wait, that’s not a lot of effort. And then suddenly you’re written it and then suddenly had to deal with it.

Josh Bressers (20:59) Yep,

yep, yep, exactly, exactly. Okay, so I want to also ask you about the data you collect and the things you do. Cause you mentioned at the beginning about focusing on kind of what’s important, which I think makes sense because one of the, I think difficulties in the Linux distribution is understanding that, you know, there is a million pieces of work to do and you have enough time to do a hundred.

So you have to figure out what are those hundred things we’re gonna do and those other things we’re not going to do. So talk to me about deciding what’s important and what’s not important and kind of what that means in this context.

Levente Polyak (21:40) I guess so. What it mostly first boils down to is that we first need to know what we want to assess, right? So we need to have like the actual data, the information to make any judgment about priorities. And there are already kind of starts to the pain point. So basically we concluded we are all intrinsically motivated volunteers in ARCH. And ⁓

Morten Linderud (22:07) Thank

Levente Polyak (22:09) The

issue is like it absolutely does not spark joy entering the CVE data for hours and hours. And where we started, so in 2014, it was just a very crappy wiki page. It was literally a table and people were just very motivated. This thing now exists. So that was their intrinsic motivation to put in all that work. But that kind of burned out. And now we get like a little bit back to what Morten said at the beginning.

that there is also a little bit of struggle we hope to be able to overcome it. But just let’s explore more like how did we actually get there. So then in 2016, we developed our own security tracker, which is basically the platform you can see when you visit security.archlinux.org. And this one is like not only presenting the data a little bit in a nice way,

But it also helps the security team to maintain the information and the severity. Of course, we have behind the log-in form a little bit more ease to insert the data, but also to track the data, to triage the data and decide what we really want to act on now ⁓ to have this kind of prioritization, Josh you basically asked for.

But this also brings us again to a little bit of a pain point, ⁓ why the team is a little bit burned out right now. And that also has a little bit to do with the influx of CVEs over the past couple of years. And that just does not spark joy at all. And we need to find some kind of way to eliminate that busy work again. So some kind of data scraping.

But that’s basically like ⁓ what you can imagine we need to do before being able to assess.

Morten Linderud (24:09) Yeah. So, so previously it was all hand inserted. So if I’m a CVE, we inserted the blog, the descriptions. And now we realized that this doesn’t scale anymore. So we need some way to sort of parse the CVE data stream programmatically and then sort of have like a queue in our dashboard thing. So you can sort of just literally dump how to do most of the work yourself. The issue is that that work, that that piece of work is hard. And it was very hard at least in 17 and

18 before and the NVD published or proper JSON schema dumps of their data. Previously it was mostly XML zip files, I think they just published on websites. There was no other way to sort of consume the CVE data. That’s terrible. this is better, but there’s still some nobody that has volunteered to sort of do this work. And it is still a lot of work sort of figuring out how we’re supposed to do all of that considering the amount of CVEs that get published each day.

Josh Bressers (25:06) And for everyone listening, back in like 2016, 2017, that time span, you were talking about less than 10,000 CVEs a year basically. Now we’re gonna break 40,000. I think we broke 40,000 last year. We were close if we weren’t. Like it’s completely ridiculous. And what makes it even worse is NVD, as much as we love to hate on them,

Morten Linderud (25:15) Yes.

UGH

Josh Bressers (25:28) They were adding CPE information to CVEs prior to, was it 2023? Probably 20, yeah 2023 is when things kind of started falling apart. And now less than half of the CVEs have the version and package information NVD used to add, which now means your team is trying to un, now you’ve got, you know, literally like five times the number of vulnerabilities, no CPE information from NVD, and yet your security team is basically the same size.

Morten Linderud (25:48) and

Josh Bressers (25:59) So it is, that is ridiculous. I mean, if I worked at a company and they did this to me, I’d be like, I’m going to quit if you don’t hire five people right now.

Morten Linderud (26:06) hahaha

⁓ If it was that simple in open source. ⁓

Josh Bressers (26:10) I know, right? But, but I mean, look, this is, this

is such an important point though. Like this, this is one of the reasons I wanted to have you two here is I don’t think some of these struggles and difficulties are understood. I often hear people say things like, the developers will handle it or, let’s add, you know, let’s make developers use two factor authentication or add, you know, Sigstore to their project or start using Zizmor or whatever. There’s all this stuff. And it’s like,

These people already are at the end of their rope. Like you cannot tell them to do more things. Cause the fact that your team is holding this together as it is, is a miracle, quite frankly. Like the fact that you guys are still keeping this all together is like, it blows my mind just cause I know how big and horrible this all is.

Levente Polyak (27:01) Yeah, mean, Josh, you’re exactly pinpointing the problem here, but in my opinion, it even goes a little bit further than that. So at this point, I also want to make a warm shout out to all the open source developers out there, because I think like people not just maybe not understand like what it means on our side as a security team member to do all this work, but there is also

so many open source developers out there doing it voluntarily for free. And they’re also burning out with all the pressure and people like demanding things from them. And it’s not like they’re paying customers, they’re providing service free of charge to like the whole world. And yet there are sometimes this very awkward demands and that creates so much pressure on all the different volunteers, developers, projects and security team members as well.

So we just need more people, more help basically tackling this from all sides, from the security team side, but also for all the project. Just go out there, help them get things done. And that’s how we can keep this all rolling.

Morten Linderud (28:15) I think I know what the element wrote and I can’t help but think of this one hop take from Massive One I read yet, I think for every open source maintainer there is, there is two burned out open source maintainers ⁓ arguing about the sustainability of open source. And I think that tracks us.

Levente Polyak (28:33) yeah.

Josh Bressers (28:33) I have no doubt.

I believe it, 100%. Okay, so let’s talk about what Levente just said, about what helping. If someone out there listening does want to come help the Arch Linux security team, what does that look like?

Morten Linderud (28:50) So, mean, that’s a sort of composite answer because at some level it’s sort of being active in our IRC or Matrix channel. I think it’s on Matrix and sort of helping sort of watching out for important stuff. But at the same time, the team is burned out, so there’s not necessarily lot of people acting out it. What we really do need is somebody who’s invested in sort of improving how we consume CVEs So it’s easier for us to actually act.

on the data stream that is CVE so you can actually do automate more of it. Because ⁓ the act of sort of being aware of a CVE and then linking it somewhere and then expecting someone to deal with it is sort of, we already have several people doing that. We don’t need more reminders. What do you need is sort of people helping us sort of improve our tracker and sort of figuring out how we can realistically consume this information flow.

Because if you don’t solve that, can sort of just add people to a problem, which is sort of people helping us punch in CVE data. But at that point, we’re just sort of providing burnout as a service and that’s not sustainable. So it’s sort of the, we sort of need help with tracker, frankly. And then when that’s done, we can maybe try help on board more people that’s interested in helping us tracking CVEs and patches. But it’s still sort of this pipeline of

consuming CVEs how do you act on them, and also sort of engaging the package maintainers because the package maintainers out of this is not better. And that’s also important to realize. So it’s…

Josh Bressers (30:24) Yeah, yeah, for sure. And I feel really bad

that I laughed at your joke, Burnout, as a service, but yeah.

Morten Linderud (30:31) Well,

I mean, that’s open source in a nutshell it’s a lot of hot takes now, it’s sort of interesting. You sort of have to figure out how you can make things as enjoyable as it can be so people don’t burn out. But it’s hard when you’re sort of dealing with this. Lately, mean, like security companies are now also consuming the data you publish either voluntarily or involuntarily. And that sort of adds to the expectations of like a service that’s sort of supposed to be available.

Levente Polyak (30:33) ⁓

Morten Linderud (31:00) I know Alpine has had a bunch of issues with people consuming their CVE data stream improperly and not doing it in the right way. And that sort of just adds more pressure to the security team. it’s sort of, I’ll just say, the more sort of VC funded security companies you get, you sort of introduce more problems into this sort of ecosystem as well. So it’s sort of, yeah, giving more help to the tracker and them.

we can solve maybe other problems as well.

Levente Polyak (31:33) Yeah, I agree that this would certainly be the best priority. we would surely also be… ⁓ Fox really hit the spot there. So I also don’t want to trick people into getting a burnout, but we could still… So one part we actually haven’t talked about yet is the advisory part.

Morten Linderud (31:45) ⁓

Levente Polyak (31:58) So one thing is getting the software patched and getting an update rolled out to our repositories and our mirrors. But that doesn’t mean necessarily that it already arrived on the end user systems. So obviously the users need to update their systems to receive the ⁓ security fixes.

There are bunch of stories along the way when you’re a package maintainer, when you realize there are so many people who are updating their systems differently than you would do it. So I personally update it sometimes multiple times a day. I just want to really get all the juicy stuff in as quickly as possible. But then you realize there are people out there who update their system maybe

Morten Linderud (32:31) Hahaha

Levente Polyak (32:46) Once a month is already a lot, maybe even less, maybe once every half a year. And that’s where security advisories really become very important. Because if you’re not getting the message out there that it is worth upgrading, then probably you’re going to delay it if it’s not like very important for you. And that’s the part where we are currently

because of the lack of volunteers, they’re also struggling a bit. So right now we are essentially only sending advisories for really the very, very critical things. Mostly those things were also on some kind of embargo list and coordination upfront. So really the very nasty stuff. But it would certainly be very helpful if we can also revive that a little bit and get more news to the people to allow themselves to judge when they want to upgrade their systems.

Morten Linderud (33:47) So somebody on IRC was like, how do you guys host your IRC bouncer setups? And I was like, I put this box up a decade ago. And I was like, wait, when was the last time I updated it? So actually I went to check yesterday and it’s been up for 1,400 days. And I’m not quite sure why it’s running kernel 4.19 yet. I think that was released in 2019. So if you’re curious about how security context people are dealing with this, I should probably…

Josh Bressers (34:02) Excellent.

Nice.

Morten Linderud (34:14) go update that box.

Josh Bressers (34:18) I am completely sympathetic to this problem. It’s okay. It’s alright. It’s an IRC bouncer It’s probably fine. I mean, it’s probably fine. Just keep telling yourself it’s fine. It’ll be okay.

Morten Linderud (34:19) Uhhhh…

Yeah, yeah,

yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah,

Josh Bressers (34:30) See, there you go, there you go. Just hack the uptime binary

to not show you the uptime, it’s fine. Yeah, yeah, subtract five years from this number.

Morten Linderud (34:36) Just null out the four digits, yes. Yeah. Yes,

yes, yes, that’s, yeah. ⁓

Levente Polyak (34:44) We probably all see this

picture with the burning background and like this small slogan, everything is fine.

Josh Bressers (34:52) Yup, yup,

Morten Linderud (34:52) Yeah,

yep. Yep. I have the…

Josh Bressers (34:54) exactly. ⁓ No, I get it though, right? You have a limited amount of time and resources, right? And you’re going to do the things that matter the most, especially as security people, that’s what we do. Like I get it. Yeah, yeah, anyway, okay. We’re gonna wrap this one up. I have one last question for the two of you, because I don’t know the answer to this either. So I see the memes everywhere that I run Arch Linux, by the way.

Do you guys love those or hate those?

Levente Polyak (35:27) Should I go first? You wanna go first?

Josh Bressers (35:28) I

Morten Linderud (35:29) I can explain the origin of

that meme and why I hate it, because it’s terrible. So everybody here that’s listening on this that’s above the glory age of 25 probably knows what 4chan is and what the technology board on 4chan is. So before ARCH became popular in the 2010-2011 era, it was Gentoo that was the meme distro

Josh Bressers (35:32) Sure, yeah, yeah, for sure.

Levente Polyak (35:33) Yeah, please, please go ahead.

Morten Linderud (35:55) What happened was that sort of at 2010, 2011 Arch gained popularity. I’m not sure why. But then people sort of inserted that they saw that this way. And by the way, I’m using Arch and then sort of people want to troll. So they do sort of the Oh I used by Arch, by the way, meme So that sort of originated. The issue is that, or the issue is that during 2012, 2013, a lot of the 4chan internet culture that’s sort of playing in the internet today,

Josh Bressers (36:13) ⁓

Morten Linderud (36:25) jumped over to become mainstream internet culture through Reddit. So all of those IT tech memes that we’re seeing today just froze in time in 2013 and never progressed. If the meme culture jumped in 2010, we would have the same jokes about Gentoo instead. that’s just… The internet culture just froze. just sliced off 4chan into 2012, 2013, and it just continued.

⁓ pretending like this is our culture now. I personally don’t like it. I’m so tired of it. I cringe every time I hear the meme internally.

Josh Bressers (37:05) Fair enough.

Levente Polyak (37:08) can certainly understand that, but I have quite the opposite feeling about it. So I do love those kind of things because it keeps us at a good spirit actually.

Josh Bressers (37:16) Awesome.

Morten Linderud (37:19) I mean, you have the arch cat ears behind you, I think.

Don’t you? No? No.

Levente Polyak (37:25) ⁓ No, no, no, not right now. ⁓

That would be an amazing reveal, but no, unfortunately not. But I find it a little bit playful, because you’re not always just taking you very serious, but also have this little bit of fun to all the work that’s behind the curtain. And as long as if…

Morten Linderud (37:34) aww

Levente Polyak (37:52) it doesn’t get fully out of control, then I think it’s a very healthy thing also for the community. And it keeps us gluing together and you joke about it. And I think that’s like ⁓ just a very entertaining thing. But I agree, like it got stuck in time. So we use this kind of meme for like a lot of years now and probably more years to come. we also as developers still pick it up like

When we do a presentation, I’ve seen it so often that our own people say like, I use Arch by the way. But overall, I do think like a fair amount of memes is like very healthy for the overall community and keeping up the spirit. There’s also this Uwu kind of thing. So we now have also Arch Linux Uwu stickers and

Morten Linderud (38:41) ⁓

Josh Bressers (38:46) Awesome.

Levente Polyak (38:46) we try

to find the next thing for our memes so yeah I do enjoy them.

Josh Bressers (38:53) All right, for everyone listening to the audio version of this, it is clear from Morten’s face that he just died inside a bit during Levente’s explanation.

Morten Linderud (39:00) I’ve…

That’s… He can enjoy this. It’s fine. It’s fine. It’s fine. It’s fine. He can have his joy.

Josh Bressers (39:06) my goodness. This, okay. This, this,

I love it. I want to thank you both. This conversation has been absolutely fantastic and I truly appreciate your time. So Martin and Levente, thank you so much and thank you for everything you do with Arch Linux. It’s amazing.

Morten Linderud (39:12) Uhhhh

Thank you for having us.

Levente Polyak (39:25) Yeah, exactly. Thank you so much for being on your podcast.

Josh Bressers (39:28) Awesome. Awesome. All right, until next time, gentlemen.