Josh talks to Kat Cosgrove about a how companies should be treating open source more like their critical infrastructure than free stuff. Kat has a ton of knowledge about how the interactions between companies and open source communities can work well, or not work at all. Kat’s time on the Kubernetes Release Team. We touch on how a project like Kubernetes is super successful, while another, Ingress NGINX, was not. It’s a super insightful discussion with a ton of lessons and advice for everyone.

This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.

Episode Transcript

Josh Bressers (00:00) Today, open source security is talking to Kat Cosgrove, the head of developer advocacy at Minimus and a member of the Kubernetes Steering Committee. Kat wrote a blog post recently and so I asked them to come on and I’m just gonna read the title quick and then I’ll take it from here. It’s When Projects Fail, Why Companies Should Treat Open Source’s Infrastructure. So Kat, welcome to the show. I’m very excited. Take it away.

Kat Cosgrove (00:22) Super excited to be here. Thank you so much for having me on. This is one of my favorite things to get on a soapbox about. But the general gist of it is that ⁓ for a long, long time, not maintaining upstream open source software was considered something that you did largely for the brownie points ⁓ to feel morally good about giving back to this tool that you probably make money off of. It’s part of your paid product that you make money off of.

And so you support it either with maintainers or with money ⁓ for the brownie points. But that is no longer the only reason to do this. ⁓ Now not maintaining things that you rely on is a pretty significant security risk. ⁓ The most dramatic example of this is the shutdown of Ingress NGINX, which is a piece of Kubernetes.

that was in close to 50 % of cloud native environments. That’s pretty significant considering how widely used Kubernetes is. And we had to shut it down on, I feel like we gave pretty good runway on this. There was like two years of maintainers begging for help. And then we gave six months notice of the shutdown because it became an unmaintainable security nightmare. ⁓ It was not possible for us to keep up with

the CVE reports ongoing. It was being maintained by only two people who were not being paid to do that full time. They were working in their spare time to maintain something that again, 50 % of cloud native environments were relying on. And because of the fundamental way in which Ingress NGINX works, it is as Tabitha Sable, a member of the Security Response Committee said, a never ending CVE pinata. It just like constant.

It’s not possible to keep up with it. And because we’ve had to shut it down, anybody who didn’t pay attention and didn’t actively choose to migrate off of it on the Gateway API or like literally anything else is now at risk because it’s going to keep working. Like the last build of it works, but we can’t guarantee that it’ll stay CVE free forever. You know, I would bet money that somebody’s. yeah, like I would bet.

Josh Bressers (02:43) I guarantee there’s a bunch of them now.

Kat Cosgrove (02:49) money that somebody is sitting on a nasty RCE and just waiting for a good opportunity to drop it and everybody who didn’t switch is vulnerable and that was an entirely avoidable situation. If that 50 % of people of companies had bothered to help maintain it, we would not be here. And that problem is only gonna get worse.

Josh Bressers (03:12) Yeah. Okay.

And so here’s the real reason I wanted you on is kind of, I feel like there’s an intersection of questions that we can use this as kind of the jumping point off of. So first of all, you know, you’re part of the Kubernetes universe and that’s part of the CNCF, which is a behemoth foundation. And so you’ve got people that are like, didn’t the CNCF just start paying people to take care of this? So there’s that angle, right? And then there’s

Another piece of like, okay, so this didn’t work, but like Kubernetes has attracted any absurd amount of contribution and coordination and cooperation from like the greater world at large. like, how do we square this circle? How do we, how do we say like, okay, so why does Kubernetes work? Why are people volunteering for that? Why aren’t people volunteering for this other thing? And also why isn’t the CNCF just picking up the pieces here and saying, okay, we’ll take care of this.

Kat Cosgrove (04:09) So on the question of the CNCF, ⁓ I think a lot of companies mistakenly believe that when they join an open source foundation, the membership dues that they pay to the CNCF or the LF or the OpenSSF or whoever ⁓ goes towards buying maintainers and it doesn’t. That’s not what that money is used for.

Josh Bressers (04:30) Yeah.

Kat Cosgrove (04:34) ⁓ One of the things that money is used for is for paying for our infrastructure. The Kubernetes project requires infrastructure of its own. ⁓ We also get resources in the form of token donations from Microsoft or Google or whatever for AWS or GCP resources, ⁓ Fastly, DigitalOcean, all of them donate by giving us free credits.

The money you’re paying the CNCF is not there as an emergency fund if a project needs resources in the form of engineering. Sometimes, ⁓ it gets us access to things like we have access to the CNCF’s PR engine. When the Kubernetes project needs to blast something out really wide, we can use the CNCF’s PR team for that. ⁓ It gives…

⁓ smaller projects that are incubating in sandbox ⁓ resources and structure and guidance on legal issues, marketing issues, technical writing issues, that kind of stuff. That is what your money is paying for. It is not paying for the CNCF to come in from on high with hired, salaried engineers to save a project that cannot save itself.

Whether or not that’s moral and correct is an entirely different argument, but the reality is that that is not what your dues are paying for. So we had another. Yeah.

Josh Bressers (06:03) I want to clarify as well, there are like

a literal infinite number of projects that you could justify spending huge buckets of money on. So that’s like, it’s very difficult to even draw that line if you wanted to.

Kat Cosgrove (06:08) Yes, time.

Yeah, and sometimes like money is not what we need. There was another CNCF project called External Secrets Operator. Still exists, like it hasn’t gone away. It still exists, people use it. It’s a sandbox project still, I think, but has applied for incubating status. And they pretty famously last year posted on Reddit threatening, like their own words were blackmailing our users because they weren’t getting maintainers. ⁓ And that wasn’t what they needed. They did not need

money, they needed bodies. So a thing that you have to also be doing as a company who wants to give back to open source, whether it’s for the brownie points or for the security consideration is commit bodies to it. Like you don’t need to hire a whole engineering team that’s like dedicated to contributing to upstream Kubernetes or whatever. But there needs to be some amount of like actual engineering work because usually what we’re lacking is

maintainers, reviewers, approvers, not just contributors. Contributors are also always a thing that we need, even the Kubernetes project, but we need people that are around long term and then can eventually be trusted with elevated permissions to help review and approve PRs. So that’s the CNCF money versus people angle. ⁓ With the why didn’t Ingress NGINX get maintainers and like why do other projects struggle with this? But

Kubernetes is the second largest open source project in the world by contributors. ⁓ It’s not flashy. And I hate this. ⁓ I really, really, really truly hate this. ⁓ It’s because it’s not flashy. Ingress NGINX is a critical component for a lot of clusters, but it isn’t flashy to say that you employ a maintainer of Ingress NGINX. It is flashy to say that you employ a Kubernetes maintainer. ⁓

Josh Bressers (07:52) Yeah.

Kat Cosgrove (08:11) I see a lot of this because I am, in addition to being on the steering committee, I own the Kubernetes release team sub project. And the Kubernetes release team is, I’m not gonna say it’s the easiest way to get involved in Kubernetes and become an org member, but it is a way that requires like the least amount of upfront effort. It just requires a lot of luck because we have an open application for it three times a year. And it requires a team of about 20 people to cut a Kubernetes release.

over the course of the cycle. Yeah, it’s very structured. But we have an open application process. And so we get like 200 or 300 applications every time. Yeah, for like 20 positions. And we have to be acutely aware of the fact that a lot of those people applying are doing it because it looks good on their resume. They’re not going to stick around. And we want the release team to be like an on-ramp to contributing elsewhere in Kubernetes.

Josh Bressers (08:41) Wow.

Sure.

Sure.

Kat Cosgrove (09:10) the majority of people who go in through the release team are not going to do that. They want to be a Kubernetes org member, because it looks good on their GitHub profile, and they want to be able to put Kubernetes version 1.36 release signal shadow on their LinkedIn. So we have to be aware of that. I don’t know how to fix that problem, aside from scaring people. Ingress NGINX died because we couldn’t get people to contribute it.

it and people wouldn’t contribute to it because it’s not as flashy as Kubernetes. And that sucks, you know?

Josh Bressers (09:47) Yeah. mean, look, this is one of those difficult problems. think we see this in open source all the time where there are components of open source. even, like one of my favorites is like vulnerability IDs, right? Like CVEs. That’s a world I’m unfortunately just become just a part of my life. And like, it is like infrastructure, right? It’s like building a road in the middle of Nebraska, right?

No one’s going to be like, ⁓ let’s go build a road in Nebraska. Sign me up. Like let’s write a press release. Like, no, it’s boring, but it needs to be done and someone needs to pay for it. And unfortunately, we don’t know how to pay for that yet. In fact, a couple episodes before this, I’ve got Vlad from open source pledge. talked to, and like, this is one of the topics is like, how do we get some of these companies to actually pay for some of this stuff, especially for the things that aren’t necessarily understood.

Kat Cosgrove (10:20) Yep. Yep.

Josh Bressers (10:39) Because as you’re well aware, I mean, you you work for Minimus, so you’ve got like, you know, turtles on top of turtles in terms of dependencies. And like, there are many organizations that are running stuff they don’t even know is in there, right? Because they’ve, yes.

Kat Cosgrove (10:46) yeah.

that’s how the Equifax breach happened.

They didn’t know they were still running Struts 2 Or at least not the vulnerable version of it. Yeah, they had no idea. And that’s terrifying. Like I said this, ⁓ did a talk about like ⁓ open source maintainers beefing with each other and how bad that is for the ecosystem a couple of years ago. And afterwards I gave ⁓ an interview with Jeremy Rickard to the register and we…

Josh Bressers (10:59) They were running all the strutses.

Kat Cosgrove (11:22) came down on the only way any of this changes is if something really big falls over. Because people don’t believe that this is a problem. ⁓ I had hoped that Ingress NGINX falling over would be the thing, but we had, and when I say we, mean the steering committee and the security response committee, we had so much trouble getting people to listen. Even using the CNCF PR engine to push out requests to the media, we got so

far fewer requests than we expected. We couldn’t get people to write articles about it, even though we released what we considered to be a pretty inflammatory statement about it, which is rare. I don’t think the steering committee and the SRC have ever released a statement together, and we released a scary one, and people would not listen. So I think, unfortunately, it’s going to take somebody getting pwned catastrophically because they stayed on Ingress NGINX for anybody to listen to this as a real issue.

And I don’t want that. I don’t want that for the company. I don’t want that for whatever junior employee is forced to take the fall in public. I don’t want that for the users of those products, but holy fuck, man. Like ⁓ what else are we supposed to do? Like we begged.

Josh Bressers (12:35) I wonder,

I wonder if you’re even correct because I look at things like remember how log for shell was going to change everything and almost nothing changed. then XZ was going to change everything and then almost nothing changed. And like when I remember seeing the announcement from your folks about Ingress NGINX and I’m like, I want to say this is going to work, but I don’t think it is. And I was unfortunately correct.

Kat Cosgrove (12:44) ⁓ yeah. Nothing changed.

Mmm.

It,

you were right, it did not, didn’t really do much of anything. And how bad does it have to be? Like how expensive does the risk have to be for a company to decide, okay, yeah, maybe it’s worth having engineering spend like a few hours a month contributing upstream to this? Like surely that’s cheaper than like the financial and reputational hit.

of a massive widespread breach, especially through something like Ingress NGINX. It’s a fucking Ingress controller. Like, the amount of access it can grant to the rest of your environment is terrifying. All of it, the whole thing. It just doesn’t seem like something to play around with to me, but…

Josh Bressers (13:40) of it.

Kat Cosgrove (13:50) You know, people who are paid far more than I am have decided that it is worth the gamble.

Josh Bressers (13:58) So I,

okay. So I don’t want to, I know we could spend like hours just punching down on this and I get it and it is bad, but so, so I’d be, let’s turn it into maybe a conversation about, you know, kind of what we can do. So here’s what I wonder sometimes about these sorts of conversations and these sorts of problems is I think the security industry to date has done a very bad job of ever communicating a lot of this information.

Kat Cosgrove (14:04) We could rabbit hole on it,

Yeah. Yeah.

Josh Bressers (14:28) in a coherent, insane manner. Because historically it’s been, you’ll be sorry. And like they never were. So who cares? Right. And I’m curious if you, you, you have thoughts on how we, mean, I know you did communicate this obviously in a way that was less successful than we had hoped, but I’m curious your thoughts on this. Like what can we do different? We aren’t doing today, maybe.

Kat Cosgrove (14:36) Yeah.

So I can come at this with two different pieces of experience. ⁓ I have not always worked in security. Working in security is actually ⁓ relatively new to me. Historically, I’ve just worked in DevOps, and that involves having an awareness of security concerns and security tooling, but we don’t communicate with people the same way security teams do. ⁓

I also was the person responsible for handling the massive PR and comms fallout of Kubernetes’ Docker shim deprecation. So I have personal experience with not communicating things very well and personal experience dealing with security teams as an outsider. And ⁓ there’s not really a way to say this that ⁓ doesn’t sound mean. So I like so apologize to the security professionals reading, listening to this, but ⁓ the way a lot of security teams communicate

Josh Bressers (15:27) ⁓

Bring it.

Kat Cosgrove (15:48) with people, whether it’s like their own internal teams or whether it’s communications externally is wildly condescending. You, yeah, it’s you talk down to people. You make people feel stupid for not having like an intimate knowledge of the security implications of whatever the fuck. And that is not how you get people to move or act.

Josh Bressers (15:57) Yes.

Yes.

Kat Cosgrove (16:18) That is how you get people to dig their heels in because you’ve insulted them. And I think usually you don’t realize it when you do this, right? Like I used to talk down to people in a way and I definitely had to like learn how to communicate better. I took a PR class, honestly, like full disclosure, I paid somebody to teach me how to not sound like an asshole, which I recommend doing if you’re in a leadership position, you 1000 % need to do that.

Josh Bressers (16:21) Yes.

Nice.

Kat Cosgrove (16:47) But ⁓ yeah, you gotta watch the condescension because it makes people not want to work with you. ⁓ But also you have to be really careful to understand how much your audience knows about what you’re talking about. You overestimating how much they know is just as dangerous as underestimating how much they know. Either way,

can look condescending depending on the way you’re speaking to them. ⁓ And either way is also not really going to ⁓ succeed in getting people to understand the weight of what you’re trying to communicate to them. In the case of the Kubernetes Docker shim deprecation, ⁓ we massively overestimated how much the average person understood about the relationship between Kubernetes and Docker. And it turned into a shit show.

that resulted in ⁓ major media outlets, like The Guardian, wanting to talk to us, which was terrifying, right? This was years ago. But we screwed up. So we over-rotated on ⁓ communication. We now communicate maybe more than we actually have to, but we try to speak ⁓ different languages when we’re speaking to different people.

Josh Bressers (17:50) guys.

Kat Cosgrove (18:11) So the statement we put out about Ingress NGINX from ⁓ steering in the SRC was intentionally written in a way that it should be understandable to anyone working in the Kubernetes ecosystem, regardless of how much experience they have. It was very explicit about like, this is why it’s dangerous and this is how. But for people who are more technical than that and need more detail than that to like really truly understand how

concerned they need to be, we had secondary and tertiary articles that went into significantly more technical detail. So you can’t just use this like one fits all approach when trying to communicate about something. ⁓ It’s really important that you target for your audience the way you speak to them and be real careful to not sound like a dick. like,

Security y’all y’all gotta work on that one like

Josh Bressers (19:12) Yes, I agree. No, no, not cooler. Definitely not. Like for sure not. I mean, I have, I could speak for hours on how terrible I used to be. I mean, I tell people I talk to like, I will spend the rest of my life atoning for my sins of the past of just being a terrible, horrible person to everyone all the time. And I mean, I remember talking to other security people when I worked at Red Hat.

Kat Cosgrove (19:13) You are cooler than everybody else, for sure, but you don’t have to be an asshole to…

Mm-hmm.

yeah.

Josh Bressers (19:40) thinking we were smarter than the kernel developers working on the kernel. And in hindsight, I’m like, why would we even think that? Like what on earth was wrong with us?

Kat Cosgrove (19:48) There’s

something about security that leads people to feel a sense of intellectual superiority, and I don’t really get why.

Josh Bressers (19:51) I, yeah, yeah, we thought we were awesome.

It was terrible. I mean, in hindsight, it was like the most toxic, horrible like environment and we fed off each other. But okay. So everything you just said, I think makes perfect sense. And there’s the, the difficulty of kind of the unexcitingness of a lot of this open source, but it’s also kind of critical. so, you know, building on top of everything you were just talking about having, you know, different ways to talk about things.

So I think kind of the next piece of this then becomes, so it’s going to be up to some of the technical leaders to understand what are the pieces of our infrastructure that we’ll say are critical. And like, obviously, know, NGINX know, Ingress NGINX was 100 % a critical piece of infrastructure for no doubt a ton of people. And so now the challenge becomes we have to be able to take, you know, say like, this is, this could be a problem. And we all see this, right?

There are dependencies in our infrastructure that might have unfixed CVEs where we then go to GitHub. And for two years, people have been begging the developer to fix the CVE. And now we have to convey to leadership, like why it’s important we get involved here. But we can’t say go read this GitHub issue because our leadership is going to be like, what the hell am I looking at? Like, I don’t know what any of this means, you know?

Kat Cosgrove (21:07) No.

Yeah, no, they don’t get it.

No. I think that’s why it’s important to have ⁓ people in technical leadership that can bridge the gap, that can speak engineering, but can also speak executive, because they are thinking about things from two extremely different angles. In executive, you have to present the security risks, the risks to ⁓ your reputation, the ROI risks of…

not doing this. They have very different concerns than like you or I would. And that can be a difficult thing to juggle, right? Because a lot of engineers do not want to speak executive or learn to speak executive. It’s like exhausting.

Josh Bressers (21:54) Yeah. look, there’s only, there’s

only two messages. How will this make us more money or how will this save us money? That’s it. Like one of those two things, hopefully both.

Kat Cosgrove (22:01) Yeah. Yeah, that’s it.

Yeah, you got to understand that you got to like be OK accepting that like engineers are beholden to the whims of sales numbers to write like you’re not better than your CRO because you don’t care about sales numbers, right? ⁓ Yeah, and then then boy, you care you care. I got laid off last year. It sucked. Can’t recommend that experience.

Josh Bressers (22:26) You don’t care until you get laid off, right?

Kat Cosgrove (22:37) at all. yeah, it’s another it’s partially another communication issue. For sure, like somebody has to be aware of what your dependencies are. Somebody has to be able to argue like, hey, contributing back to these dependencies saves us money in the long run. Like the engineering hours are cheaper than the potential downfall of this thing falling over. And yelling at

maintainers on an issue because there’s a CVE you want them to fix? That ain’t the way, baby, because it’s open source. PRs are welcome. You want it fixed? Fix it. ⁓ We are not under a legal or moral obligation to fix anything, to respond to any issue, to even shut down a project gracefully. We didn’t have to do that. We could have just turned it off. There was actually a discussion ⁓ at KubeCon Atlanta about

Josh Bressers (23:07) I know.

Kat Cosgrove (23:33) an emergency shutdown at KubeCon Atlanta. The decision to extend it for six months until KubeCon Amsterdam happened frantically over the course of about 12 hours at the maintainer summit. We were going to shut it down the next day. ⁓ So, and we would have been well within our rights to do that. So ⁓ another thing for leadership to keep in mind is that we don’t actually owe you anything. You’re not.

our employers, you’re not signing our paychecks. We’re doing this as a favor for you, actually. So, you know, some help would be nice.

Josh Bressers (24:11) Yeah,

mean, open source owes you nothing. That’s the lesson. And additionally, like, so, I mean, this is one of the challenges. Their maintainers often have this like very altruistic view of the world, right? And they feel bad. They feel like they’re letting people down. And so they will like ruin their weekends and skip important events to help other people sometimes, which I understand the altruism of that, but also like, don’t do that.

Kat Cosgrove (24:14) Not a thing, yeah.

Yeah, we do.

for sure.

Josh Bressers (24:41) You know.

Kat Cosgrove (24:42) I wish we did it less. ⁓ I think it’s dangerous. It’s a burnout factory. We already have like a massive problem with maintainer burnout and doing stuff like that just makes it worse. ⁓ In the Kubernetes release team sub project, we have a written down and well established policy of we will delay this release before we overwork this team. If getting the release out on the target date requires us making the release team work after hours or work on weekends even once.

Josh Bressers (25:03) Nice.

Kat Cosgrove (25:12) I’ll delay the release. don’t care. AWS, GCP, Azure, Linode, whoever else, they can all wait. You waited our pleasure. I’m not going to burn out my team to get it to you a week early. Our release dates are aspirational, always. ⁓ And I wish that was more common in open source.

Josh Bressers (25:23) Yeah.

Kat Cosgrove (25:34) It’s not even really common throughout the Kubernetes project. It’s just that we rule SIG release with an iron fist and the iron fist is employed making sure that our members don’t burn out. But we don’t control what other SIGs do. they do what they want.

Josh Bressers (25:52) Yeah, or well, and even beyond SIGs,

right? Like there’s just lots of open source projects that are literally one person. I mean,

Kat Cosgrove (25:58) Yeah.

Yeah, and that’s terrifying, by the way. Like, if you’re relying on something that’s maintained by one person, you need to either be using something else because that’s not a healthy project, or you need to be devoting full-time engineers to that project, like, stat, because that’s hugely dangerous. Let’s hear it. Okay.

Josh Bressers (26:17) So I have some bad news for you. I have a lot of data on this and

the number of single maintainer projects is an alarming graph. And it is no matter how you slice and dice the numbers, because I have a graph where I’ll show like all of the single project maintainers. It doesn’t matter what ecosystem you pick, the graphs always look the same, where the single project is like literally 80%. And then it’s, know, kind of a Pareto curve down. And then people say, that’s…

all of the projects, if you look at only the popular ones, it’s different. And then you look at the popular ones and the graph looks exactly the same, exact same shape. It’s just obviously smaller numbers. Then you’re like, that’s just NPM. Python’s totally different. Literally. ⁓ I have, I have tried to find a way to not see the same graph and it’s always the same. So if you are using open source, I 100 % guarantee you have an enormous number of single maintainer projects holding up your scaffolding.

Kat Cosgrove (26:54) Cool.

no, Python’s loud, yeah.

And that sucks.

That is awful too, because it’s that fucking XKCD comic, again, of just a tower of one by ones. and that’s not good, that’s not sustainable. ⁓ That one person is crispy, I guarantee it. They’re so crispy, and that’s a time bomb.

Josh Bressers (27:18) except it’s one strip that’s two miles high.

Yeah. Yeah.

yes.

Yes. Yes.

Well, and what makes it even worse now is so I know a lot of people who maintain like single person things, because they just, that’s what they do. And thanks to XZ and Jia Tan, now they are, they’re skeptical of every PR they get of their project. And they’re like, can I trust this person? What’s going on? I’ve had people contact me more than once. like, can you look at this PR? Does this look weird? And more than once I’ve been like, actually, this does look a little weird. Like, I don’t think I’d take this, you know?

Kat Cosgrove (27:42) We gotta do better.

Mmm.

Yeah, that’s scary. That makes the problem so much worse over time. A lot of open source is like, it’s a trust-based thing. We’re a community, right? Most of us within an ecosystem are real life friends who go get beers together and do not talk about computers. And violating that trust feels like, it feels so gross.

Josh Bressers (28:15) Yeah.

Yes.

Yeah. Yeah.

Kat Cosgrove (28:37) If somebody wants to help, want to be able to trust that, know, trust but verify, right? But to immediately go, I have to look at every single PR under the guise of like, is this a state sponsored attacker trying to worm their way into my good graces so that they get admin rights over the repo so that they can push a poisoned release that nobody will catch for nine months.

Josh Bressers (28:53) I know, right?

Yeah. Yeah.

Kat Cosgrove (29:07) Sucks.

Josh Bressers (29:08) I mean, you tell me with the release SIG, like how often have you looked at the application and been like, this has got to be a North Korean.

Kat Cosgrove (29:12) Mm-hmm.

You know, I’ve never looked at one and been like, this has got to be a North Korean because we don’t give ⁓ elevated rights to new members of the release team. They get org membership and like they get, they get the access. It depends on which sub team they’re on, but they might get the ability to like add milestone labels to something or be able to LGTM a PR, but they don’t get any kind of administrative rights, even like the release lead.

Josh Bressers (29:27) okay.

Kat Cosgrove (29:45) has some elevated rights for a pretty brief period of time for the duration of a cycle, so like 15 weeks, but by the time somebody is the release lead, we’ve worked with them for a year and a half, two years, on Zoom calls, seeing their face, seeing their work, that kind of thing. ⁓ So very rarely is it, I think this person might be a nation state actor, but.

We do get some really stupid applications. There’s like a trap question in there that is, ⁓ have you read the handbook for your role? The answer is yes, okay? Put yes, even if you haven’t. Lie to me, right? Like if you select no, you immediately get rejected and people still do it. We get like two or three per cycle of people that say no. Just lie to me, dude, lie. I have no way to check.

Josh Bressers (30:26) Obviously, yeah.

Well, you just ruined your secret now.

Kat Cosgrove (30:48) I’ve said that on like four or five different podcasts and I keep getting people just saying no. I don’t know. Whatever. Yeah, it worked. It works. But no, we don’t. I’m going to think about that more closely. I’ll open the application again in like two and a half weeks. And I’m going to think about that real closely, but I don’t think I will find any that look like

Josh Bressers (30:53) Excellent. I whatever, whatever. I guess it’s good. It filtered them out, right?

Kat Cosgrove (31:17) This is a North Korean.

Josh Bressers (31:18) Do

you work for a nation state? Yes, no.

Kat Cosgrove (31:21) You know what?

Yeah, I’ll just add that to the application questionnaire. Why not? Yeah. Do you work for the government of North Korea?

Josh Bressers (31:26) Excellent, Awesome. Then

wait till you get yeses.

Kat Cosgrove (31:34) my god, I mean, one of my asshole friends will fill it out and put yes just to fuck with me for sure.

Josh Bressers (31:37) Yeah, it’s true. There you go. There you go.

Okay. Okay. So I want to start landing this plane Kat And so we’ve, I feel like this has been really doom and gloom and I think it’s just kind of the nature of the beast at the moment. I do think there are some difficulties that we don’t necessarily know how to solve yet. And there are people doing a lot of work on this, right? Like CNCF does a lot of work. There’s like the Sovereign Tech agency.

Kat Cosgrove (31:45) Okay.

Josh Bressers (32:03) out of Germany, there’s like the Alpha Omega project and the Linux font and the what, OpenSSF. I don’t know exactly how they’re connected. And I just talked to Mike a little while ago.

Kat Cosgrove (32:09) The OpenSSF

and the CNCF are both directed funds of the Linux Foundation.

Josh Bressers (32:14) Right. But Alpha Omega is also weird because it’s sort of connected to the OpenSSF and sort of connected to Linux foundation. And I don’t know exactly how to draw that line, but whatever. doesn’t matter. There’s a ton of people kind of doing some work. There’s like the open source pledge. I just talked to Vlad a little while ago. So, so I guess those feel big and probably out of the grasp of us normal folks where I guess I shouldn’t say that you can go get involved, but I’m like, let’s talk to.

Kat Cosgrove (32:16) Yeah, they’re weird.

Josh Bressers (32:40) the someone working at a company that’s not like overflowing with resources, but they are using open source. Like what, what are the things you think they should kind of, are the things we can start doing that will have like a real impact on kind of our universe right now?

Kat Cosgrove (32:56) So you don’t need your employer’s permission to contribute to open source, usually. Some companies will really lock you down, but you don’t, especially not in your free time on your own machine. So if you want to do this for the love of the game and not because you’re being paid to do it, you can just go do that. You do not need your boss’s permission to do it. And if you are scared to contribute to open source for the first time,

Josh Bressers (33:01) True.

Kat Cosgrove (33:23) That’s totally understandable. I was too, just about everybody I know was. You are opening a PR and explicitly asking to be judged by a bunch of weird nerds who are like top-down experts in this like very niche thing that you are just now trying to enter. You don’t understand the cultural expectations, the community expectations. You don’t understand the style guide. You don’t understand how to talk to these people and they’re all strangers.

Josh Bressers (33:25) Yeah.

Yes.

Kat Cosgrove (33:52) That is in fact scary. ⁓ However, we are all well, not all of us. Most of us are really nice and you can ask for help. I say most of us because every project has a couple of people who are just like kind of mean, but you keep them around and it’s not great. And I wish we didn’t, but we do. ⁓ But generally we’re really, we’re nice and we’re helpful and we want you here. We want you to, we want this one PR from you, but we would also like more if you have a menia. ⁓

Josh Bressers (33:59) That’s true.

Yeah, yeah.

Kat Cosgrove (34:21) usually open source projects will have some kind of like Slack workspace or a discord or a discuss or a forum or God help us some of them IRC and you can just show up there and ask to be pointed in a direction. ⁓ For the Kubernetes project, I am more than happy to point you in a direction. If you just message me on Slack, I do it all the time. It is part of my job and I would love to do that for you.

⁓ If you would like to convince your boss ⁓ to allow you to do this on company time, you probably need your manager’s support on this one because you need somebody with the necessary rapport with like upper level leadership to ⁓ make that, hey, this will save us money or this will make us money argument for you. But that is a conversation that is worth having. ⁓ On a much less doom and gloom note also,

I don’t think that this is permanent. ⁓ And I don’t think that like this whole situation with maintainer burnout and projects falling over is just an inevitable result of the way open source is. I think that this is a growing pain situation for a long, long, long, long time. The norm for software was closed source proprietary and open source stuff was

Josh Bressers (35:37) Yes.

Kat Cosgrove (35:49) for ⁓ weird FOSS hippies, right? And my dad is a retired software engineer and he did not believe that open source software was safe or stable or profitable or a thing that would last until the Red Hat sale. And then he believed it. My dad is 72 for context, for listeners, and has been an engineer for his entire adult life.

Josh Bressers (35:51) Yes.

⁓

Kat Cosgrove (36:19) This whole open source being the norm, open source being a good business decision, open source components being a part of 70 to 90 % of all software built, that’s new. That is absolutely in like the total history of computer science, it’s new and we are still figuring it out. So I don’t think that this is forever. This is growing pains. ⁓ We have suddenly…

Josh Bressers (36:33) Yeah. Yeah.

Kat Cosgrove (36:48) overtaken the world and nobody really knows how to respond to it because we’re still being run by a bunch of enterprises ⁓ who have leadership who think the way that my dad did and they haven’t really figured out how to work with us yet. So I think this will get better. ⁓ We do have to tough it out, but ⁓ it will change eventually because we are the norm now and we were not before.

Josh Bressers (37:16) I agree with everything you just said. That is my suspicion as well. And I have a, I don’t have any numbers in this, but I would bet you if you went to senior leadership in nearly every software company and asked them what percentage of their code is open source, they’d probably tell you like 10%. That is my suspicion because.

Kat Cosgrove (37:32) Yes, Gartner. So I talked to somebody from Gartner, I think, that did a study on this and it was ⁓ something like 13 % of software is open source and they expect it to be like 23 % within the next few years. But ⁓ total pieces of software containing any open source component, according to the Linux Foundation, is anywhere between 70 and 90%.

Josh Bressers (37:58) Yeah. Oh, that’s funny. So Gartner is saying

13 % and Linux foundation is saying 90. Obviously they both have their own angles here, but there’s no way it’s 13. No.

Kat Cosgrove (38:06) Yeah, they both have their own angles here. There’s no way it’s 13.

There’s not a chance. it’s, truth is somewhere in the middle there, I suspect. And the open source will probably rise as we move off of like legacy applications that have been like lifted and shifted into containers. But, know, oh no. And you know, my dad.

Josh Bressers (38:28) They never die and they never will.

Kat Cosgrove (38:34) gonna periodically remind me that I could be making so much more money if I just would learn Pascal.

Josh Bressers (38:40) or COBOL like either or, you know? No, I know. ⁓ man. Yup. I get it. I totally get it. But all right, Kat, this has been a lot of fun. I really appreciate the time. This is great. And you’ve, you’ve restored some of my hopes. So I always appreciate that.

Kat Cosgrove (38:41) So, or COBOL. Yeah, yeah.

thank you.

Good, I’m glad.

Josh Bressers (39:00) Awesome, thank you so much.

Kat Cosgrove (39:03) Thank you.