Josh talks to Michael Wisner about a talk he gave at FOSDEM as well as his work on Alpha Omega at the Linux Foudnation. Michael is approaching open source security in a way that nobody has ever tried before. What if we could fund some really big, really hard projects? It’s not cheap or easy, but he’s getting it done. We spend a lot of the time discussing package registries, which are a huge topic. Michael is doing some amazing work helping package registries which is the first step in a very long journey.

This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.

Episode Transcript

Josh Bressers (00:00) Today, open source security is talking to Michael Winser the co-founder of Alpha Omega. Michael’s a person I’ve been trying to get on the show for a long time. So Michael, welcome. I am so excited.

Michael Winser (00:12) It’s really great to be here. We’ve been crossing paths for a long time and I’m really excited about what we’re gonna talk about today and just to be on your show, it’s great place.

Josh Bressers (00:20) For sure. So our topics today are obviously Alpha Omega is a big one, because I think that project is amazing. But then you also gave a talk at FOSDEM about package registries. That is one of the best talks I’ve seen in a long time. So we’ll get to that. But let’s just start with Alpha Omega. Tell us about Alpha Omega, because I love this project so much.

Michael Winser (00:31) All right, that’s very kind of you.

So ⁓ Alpha Omega started in 2022-ish. ⁓ I was at Google, and I had joined Google’s GOST team, the Google Open Source Security team. I think I just did LCD display there. ⁓ And ⁓ was sort of heading product for that and looking at a whole bunch of different interesting problems there. And

Somewhere along the way, someone said, you should go talk to this guy, Michael Scovetta at Microsoft. And Michael had written a paper titled Alpha Omega that was just thinking about and theorizing about an approach to sort of how we can secure open source. And I’ll get back to the meeting in a second.

At the same time, we were sort of figuring out what was going on. And I don’t need to remember, but sort of in the early days of OpenSSF, which was pre-COVID, and then so there was sort of an interregnum, right? At the time, both Microsoft and Google had put, you know, when they led the creation of the foundation, they’re like, this is going to cost money. And in the eternal wisdom of never let a good crisis go to waste or OPEX that you can get, do it now.

Both organizations put $2.5 million into the LF and I said, we’re gonna figure this out. But because of that gap in the time, the money actually just sat there for a while. And then Michael and I got together and I read his paper, I liked what he had to say. Hilariously, I didn’t think the name Alpha Omega was that good at the time. I’m like, we need a better name than this. And we never did get a better name than this. And it turned out it’s working out pretty well. ⁓ Which is really a proof that names don’t matter, what you do matters.

And so this was at a time when, you know, there was an awful lot of energy happening around different initiatives and OpenSSF and stuff like that. But Michael and I were just focused on what can we do? How can we, you know, actually use money towards making some of these things happen? And nobody had done this, right? There’s a lot of sponsorship things and, you know, collectives and whatnots. ⁓ A colleague of ours,

once described this as you really mostly you’re giving them money to thank them for work that they’ve already done rather than driving change forward, right? ⁓ And that was actually a pretty important epiphany along the way. And so we started out just thinking, well, what can we do to spend some money and change security in some way? And when the paper was originally written, Alpha sort of represented

Josh Bressers (03:09) Yeah.

Michael Winser (03:29) the 100 or a thousand, some number of the most critical open source projects, right? ⁓ I’m not a fan of the phrase critical open source project anymore. And I’ll explain in a sec. And then Omega was the long tail or at least the fairly fat torso of hundreds of thousands of other projects. What we’ve learned along the way is that various attempts to create the list of critical open source projects

really turned into either everything or favoritisms or whatever. But also what’s really interesting was that the projects that were obviously right there didn’t need our help anywhere near as much as a bunch of other things. And that was because the most successful projects, pick one, we’ll Kubernetes, there’s an entire team of people working on security there. Us showing up and saying, hey, we’re from Alpha Omega, we’re here, let me make you secure, would have…

Josh Bressers (04:24) Yeah.

Michael Winser (04:28) been a pretty entertaining conversation. ⁓ And so over time, Alpha has come to represent points of leverage where we can do something with a, and we use the word catalyst a lot. We can put something in a place. We can do something in a small way that will catalyze change long beyond that and across entire ecosystems and communities of software. And Omega,

is looking for solutions at scale, where we could pour money into compute somewhere or give everybody a free this or that or whatever and cause a whole bunch of things to happen, right? We also try to various things along the way and the scale part is really hard. I wanna be super clear, like that is, and we’ll come back that in a bit, but like at the end of day, there is a…

Josh Bressers (05:19) Yeah.

Michael Winser (05:28) human-shaped graph that determines the velocity of anything you can do in open source. anyway, bold ambitions, we start doing things. And the first points of leverage we identified were go into a large ecosystem where it’s currently everybody’s job or nobody’s job to worry about security. And

Josh Bressers (05:38) Yes.

Michael Winser (05:57) make it possible for it to be someone’s job. And it actually started with Node ⁓ and Security Engineer and Residents, ⁓ and then with Python. ⁓ And I’m gonna sing the praises of lots of people here. I will forget some of the names because I’m nervous of remembering the names right now. ⁓ So Seth Larson was one of the first security engineers and residents

Josh Bressers (06:12) Yep.

Michael Winser (06:29) Python and immediately started having outsized impact. It wasn’t like nobody at Python was worrying about security, but all of a sudden you had somebody who came from the community, cared about security, now is in a position of, I don’t want to use the word authority, but leadership is I think the word here, and could take on certain challenges and then also come back and say, I think we should go this way or what do you all think?

great, we’re going this way. And even just asking the right questions at the right times. ⁓ And we got pretty happy with that model. And so we’ve actually done a fair amount of that ⁓ across a bunch of different organizations. ⁓ And the impact from Seth and others has also gone beyond their ecosystems. And so this has been, I think, probably the one of the most surprising and

honestly durable impacts of Alpha Omega is that we’re creating a quiet little community of security focused people in open source. And the cross pollination is amazing. We host a public meeting once a month, bunch of people show up there, but we also host a quarterly meeting that we try to in person once a year with essentially our grant recipients, the people we work with directly. ⁓

And it’s everything from directors to engineers to ⁓ business development people, the whole mix. And we talk about all the challenges we’re facing. It’s intentionally unstructured sort of half-day set of conversations. And everyone’s like, this is the best thing ever. You’ve got people from different roles, different foundations know each other. And it’s playing out in all kinds of interesting ways.

Josh Bressers (08:19) Yeah, for sure. mean, I know Seth has been on the show more than once and obviously he’s sung the praises of Alpha Omega. But also if you look at the work he does, I feel like it’s a lot of stuff that would never get funded if he wasn’t being paid just to do security work, right? Because unlike the pecking order of tasks, this is probably not on the top of the PSF’s like things we need to do to survive, right?

Michael Winser (08:37) that is correct.

It’s in any software project. So one of the lessons I’ve learned is it doesn’t matter whether it’s a commercial product or an open source project, right, or a business or whatever you want, right? People start with growth. Growth wins over everything else, right? Whether you’re seeking money or seeking GitHub stars or seeking creds when you show up at a conference or t-shirts, it doesn’t matter, right? Growth is, in our society, has trumped almost everything else.

Right? It is what it is, right? ⁓ We don’t get to choose really, unfortunately. So.

Josh Bressers (09:15) Capitalism, yo.

That’s right.

Michael Winser (09:26) growth, correctness, security, security is definitely lower down. Another key point, right, ⁓ I’m gonna pick on left pad all day in this conversation because the sheer entertainment like that, right? But like somebody builds a thing, they’re good at that. That’s what they wanna work on, right? And so all the nuances of inserting white space into inside strings, right? Of which they’re probably more than.

Josh Bressers (09:41) Nice.

Michael Winser (09:55) I would know, right, ⁓ they become the speciality of that person. Security has nothing to do with that, right? Yes, security, that has a lot to do with security the other way around, right? Like it’s messy, but like being an expert on white space insertions of the strings in no way makes you want to be, let alone competent to be an expert on the 7,000 things that you have to be good at in order to secure your project, right?

And from a tooling perspective, a industry, everything about our industry right now, like we are, one of the things I like to say is like, for those of us old enough to remember, this is a Y2K like problem, but without the same clarity of problem, solution or date. We literally don’t have the tools, we’re still making them up. The date is yesterday or further, right? So we’re still dealing, right? And it’s…

It’s fascinating too, like when you extend from, ⁓ so these conversations, used to have these conversations with enterprise CTOs when I was at Google.

And one of the things I learned is that, you I’ve got joke for you, first of what do you call a, you know, like collective noun, a pot of whales, a herd of cattle? What’s the collective noun for a ciso?

Josh Bressers (11:21) that’s a good one. ⁓

Michael Winser (11:22) It’s a compromise.

Compromise of CISOs. Obviously because they’re always having to make compromises about what they prioritize and sooner or later their systems will be compromised. ⁓ So when you talk to a CISO about something like software supply chain security, right? It falls into this analytical framework that almost everybody uses, right? What is the likelihood, the risk of this problem hitting me?

Josh Bressers (11:35) Yeah, yeah.

Michael Winser (11:53) what will be the blast radius, what are the severity of the impact when it does happen, and then what can I do about it? The actionability. And unfortunately, the actionability tends to be pretty poor for most organizations, and there’s other fires that they have to fight, so it’s not the top thing to do, right? Like, you should fix all your supply chain stuff. Great, how? Right, and enterprise, enterprise exists to be good at doing one thing and to hire…

sub-vendors to do the things they don’t want to be good at doing. There’s a lot of vendors claiming to be really good at making your supply chain really secure, but it’s by no means a you just pay for this and you’re done.

Josh Bressers (12:36) Right, right. so fundamentally, I guess that’s kind of what Alpha Omega wants to do, right? Is solve that, because really, I mean, expecting to buy something to make the open source you’re using secure feels sillier than just saying, let’s just make the open source secure, right?

Michael Winser (12:54) Yeah, to say the least. ⁓

What we are doing intentionally is normalizing.

everything about making open source more secure, whether it’s tooling or, ⁓ you know, just having somebody whose job it is, even prioritizing it. ⁓ So I think it’s worth explaining, like our sort of four pronged strategy at this point, because it actually will connect the next part of the conversation as well. So our investments are sort of loosely categorized into four buckets. Bucket A.

is staffing roles, security engineering and residence roles, where we can pay for someone, we’re paying their salary so that they have a job that otherwise would not exist. Nobody would pay for Seth if we didn’t do that. ⁓

Those ones are big commitments to us, right? First of all, humans are expensive. ⁓ Second of all, ⁓ the last thing we would wanna do is sort of pull the rug out from a Seth or anybody else for that equivalent, right? ⁓ After one year, that would actually be worse than not doing it in the first place. But our funding is year to year. ⁓ So, you know, we’re just as, you know,

tied to the OpEx war as anybody else is. And so we actually keep a reserve of a year for our staffing engagements, specifically so that we can give them a full year’s heads up if we don’t think we can keep funding that role. When we started, we really wanted the organizations to wean themselves of our funding within like two or three years. ⁓ Unfortunately, that did not time it well with the economy and various other things. ⁓

But what we are seeing is some organizations are having more luck than others at making security a headline item in their budgets. And I am encouraging all of them to start reporting out. Like your financial report should include X percent of this foundation’s budget goes towards security work. And it’s okay if that X percent is currently funded by Alpha Omega. I want it to be normalized in people’s expectations of when they are coming into a foundation

as a supporter in some way that some of that money has to be going towards security. So that’s A. Category B, it’s package registries, largely, there’s a few sort of smaller things around there, which is sort of infrastructure related, or again, these sort of like language things that are super high points of leverage, right? And I think of package registries as the app stores of software development.

Josh Bressers (15:47) Yes.

Michael Winser (15:48) Right. And yet they don’t have anywhere near the protections that even, even the paltry protections that we get from our app stores today, they don’t have anywhere near that. Right. But talk about leveraged impact, right? We can make changes there that will change behavior upstream with publishers and downstream with consumers. And also they represent points of security risk. If you do a threat model on the whole picture, gosh, if I could take over PyPI, I don’t need to do anything else. I’ve got everything I want.

Josh Bressers (16:18) Yep. Yep.

Michael Winser (16:18) Right? you

know, making sure that infrastructure is secure is high. ⁓ Category C, honestly, we should have put this one first ⁓ because it’s how we start a lot of our engagements, audits. We love audits. ⁓ We partner with a variety of organizations, OSTIF in particular, we’re huge fans. And what we’ve learned is you learn everything you need to know about an organization’s readiness to start their journey of security.

by doing an audit. And we’ve had organizations come to us say, we want funding for this thing here, we wanna build this new thing here. We’re like, great, when was the last time you did an audit? And they’re like, no, no, we don’t need an audit, we need to go build this great new thing here. We’re like, no, really? When was the last time you did an audit? And along the way we’ve been learning and full shout out to the Eclipse Foundation, they started developing this concept called the rapid security review. And so an audit,

Pretty expensive thing, hire expensive security engineers, come and do like white box, black box, pen testing, all this stuff, Costs, know, easily $100,000 for the average project. Rapid security review is like, let’s go off and do a bunch of things automatically, poke around and look at things, and then let’s have a form that we ask them to fill out, would ask us some basic questions, and then let’s have a little interview chat conversation, and then after that, we’re gonna make one suggestion, like,

If you could just go do this one thing, that would make your world a lot more secure, right? And all of a sudden, you learn everything we wanted to learn, and you’d have the same impact of starting their journey, or them making it clear that they’re not ready to start that journey. And you can learn that in calendar days, and now with automation and AI, we’re approaching ⁓ the prospect of doing 90 % of this automatically in hours.

Josh Bressers (18:16) for like $5, right?

Michael Winser (18:18) Exactly, right? ⁓ And we’re well connected on the token front. We’re going to be able to get a lot of that fairly well. And then category D is really just experiments and innovation, where we honestly embrace the fact that this is so new, none of us know what the hell we’re doing. ⁓ Dave Nalley from AWS, they’re also a sponsor now or a member, ⁓ said that you guys aren’t failing enough. You need to try harder to fail.

And I think it’s good advice. Like if you’re not doing some experiments that are pushing your boundaries of what you know the right things are and like that, then you probably just take an easy way out and not moving the space forward.

Josh Bressers (19:00) Well, I mean, I think part of your problem as well, Michael, is just like you can’t fall off the floor, right?

Michael Winser (19:04) Well, I think, you know, when we first did Alpha Omega, you know, we had no idea, right? And we had some spectacular screw ups and we’ve learned, think, ⁓ some interesting lessons along the way. ⁓ I gave a talk at Vulncon last year that I tried to capture these things. And these lessons, like every good engineering principle are…

the result of screw ups that caused us to realize we need to be a little bit more precise about that, right? And so we prioritize shovel ready projects. Like we want to give money where it’s going to go immediately into action, not sit in some other foundation budget waiting for them to find the person who can then come up with a plan to hire the person to then come up with a plan to do the work. Like, no, no, no, no. Like everything needs to be right there. The money goes in. And when that’s the case, we can decide very quickly. we’re biased for quick action and impact. ⁓

Josh Bressers (19:57) Right on.

Michael Winser (20:01) ruthlessly focus on creating a public good. We always want to make sure that the work we’re doing is towards a public good. And it is complicated because not all the actors that we need to work with are non-profit foundations. It doesn’t make them good or bad. It just, you know, it makes things complicated. But if the outcome where we could pay trail of bits to go off and do work, as long as that work is open source and is a public good, we’re pretty comfortable with that trade as appropriate.

Josh Bressers (20:29) Yeah, yeah, that seems fair. Okay, so I want to switch over to your FOSDEM talk, which I think is related to all of this. So the title of your talk was The Terrible Economics of Package Registries and How to Fix Them. And this is related to the, I chatted with Brian Fox, actually, in the context of where this episode will come out. It would have been last week. Well, it was last week, literally, for me as well. And I talked to Brian many months ago about that letter.

Michael Winser (20:31) Yeah. Okay, let’s do it. Yeah.

Okay, yep.

Josh Bressers (20:57) that the OpenSSF released about package registry. So explain to us, just set us up for what this all means and why we should care.

Michael Winser (21:05) So the origin story was actually last year, I know 2024. In 2024, because we were now funding and working with so many package entities, registries and foundations that manage them and so forth, and seeing what resources were available and also getting a sense of like the work ahead of them. There was a summit in DC that we sort of talked about.

the securing software repos guidance and that work ahead too. And it became more more clear to me that we were looking at two lines crossing each other the wrong way, right? One line up and to the right, growth of package registries, just more packages, more downloads, more uploads, more malware, more everything, right? And at the same time, right? The number of entities that were funding package registries specifically was approaching zero.

Josh Bressers (21:55) Yes.

Michael Winser (22:03) and the funding that, you know, might’ve been in times of largesse enough to kind of bridge that gap was not really there. And then also a lot of this was subsidized and hidden subsidies are always the worst ones, right? With cloud credits, Fastly has been a tremendous contributor to open source through this. and, ⁓ Hannah Aubrey, when she was there was amazing because she moved everybody to five year contracts.

Josh Bressers (22:24) Yes.

Michael Winser (22:33) because otherwise it’d be hand to mouth each year and can’t even imagine the stress. When you’re serving petabytes of information and you’re getting it for free and you don’t have it guaranteed for next year, you have a really interesting problem. So it started talking about that and we had an Alpha Omega board meeting at the member summit in November 24 and I brought it up at that meeting and we all agreed we need to do something about that.

And so 2025 was really an effort to start building towards that. Talking with Brian, the letter that came out, Brian took the lead on that, but it was very much in line with what we were trying to do. We’re very happy to be part of that. And we’ve been trying to like, it’s, as someone who used to work at Google and large corporations and moving to open source, that speed of human change is very different. Like, you know, if this was a problem inside a company, we decided we wanted to solve it. We’d just go reorg things and hire people and.

Josh Bressers (23:30) Yeah.

Michael Winser (23:31) spend zillions of dollars, still fail, but try really hard and fast to get there. But with open source, there’s just a lot of talking to go on. You’ve got to find the right people, the right place, and there’s a lot of cultural changes to work through. ⁓ But I’m comfortable saying that in the course of last year, we organized and had a lot of meetings about the topic. And I think that everyone’s frame of reference has shifted significantly now to where

there’s a sense of inevitability to this is something we have to do something about. Which wasn’t the case in 2024.

Josh Bressers (24:05) Yes, so I would say from what I’ve observed, I think the package registries and a lot of the really like deeply embedded open source nerds would agree with that statement. But I feel like a lot of normal people still have like absolutely no idea that anything weird is going on in the package registries.

Michael Winser (24:23) And honestly, nor should they, they just, you know, like it gets back to like, they’re not security experts and they’re not infrastructure package registry experts, right? You you, you’re working on a thing. Let’s whether you were at an insurance company, NPM installing foo or a maintainer of an open source package. By the way, one of the beautiful ironies of the XZ utils mess was observing that open source developers.

Josh Bressers (24:32) Okay, yeah, that’s fair.

Michael Winser (24:51) are just as bad as corporation type developers in terms of how they treat their upstream. It’s magic, it came down from the sky on the back of a unicorn with rainbows coming out of every pore. Not my problem, this is great, this is fine.

Josh Bressers (25:07) That’s the story of people, right? If I can make something someone else’s problem, I’m 100 % going to do that and I’m going to ignore it.

Michael Winser (25:08) And if I can issue,

I create an issue or even better a PR with a vague pile of work that says you go do the work and I’ll be the beneficiary of it. Why wouldn’t I do that? Which also accounts for a lot of the really weird feature creep in upstream open source because people keep, please do that. sure, more popularity, more GitHub stars I’ll do this. No, you idiot, do less.

Josh Bressers (25:33) Yeah, yeah. OK, but I want to talk about this this talk specifically, right? Because I think you you touch on him. It’s not very long for anyone who wants to give it a go. I’ll put a link in the show notes. And obviously you have a very lengthy Q &A at the end, which I think is great. And and you kind of the thing I love is you you stress the fact that like Fastly is the like unseen hero.

Michael Winser (25:38) Okay.

The whole point.

Josh Bressers (25:57) of a lot of these package registries, which I know like even I didn’t know this until someone told me about it maybe a year or two ago. And I’m like, wait, what? That what? Which is just mind boggling. But then you also stress.

Michael Winser (26:06) Unseen hero,

single point of failure, you decide.

Josh Bressers (26:10) Exactly. But, but as you point out in this talk, the bandwidth isn’t really the problem. Even if Fastly wasn’t donating it, we understand bandwidth. And you made the argument that there’s a bunch of other things we really need to focus on. For example, like all the security stuff you just mentioned, right? Like those are the things that are actually really hard to do and really hard to fund. And we don’t really know how to solve them, right? But that’s not what we talk about. We obsess over the bandwidth of these registries every time there’s a discussion.

Michael Winser (26:38) So yeah, like bandwidth is obvious, right? But just as Canada discovered when they started taxing cigarettes and getting a lot of revenue to get people to stop smoking, right? At some point you become addicted to the revenue and people stop smoking. And the first thing that’ll happen is if you start charging for bandwidth at any level, people will start caching which is great. We want you to start caching just as we wanted people to stop smoking. But the problem is all we’ll have done then is changed how

Josh Bressers (26:51) Right.

Michael Winser (27:08) much it doesn’t cost fastly for us to not like, you know, whatever CDN costs don’t change. And the gap that we’re looking at, which is who is currently paying for security engineering work on these things to drive out a list of features that’s as long as your arm and then some, right? And again, there’s like all of the package registries in their own historically tortured ways have achieved some sort of balance in terms of resources, capabilities and growth.

except the growth is driving, putting huge pressure on everything they do and they are, you know, treading water in terms of work they can do above that. And so, yeah, monetizing bandwidth might create a blip. It’s not a bad idea. And the other day, like it’s also not the only metric to look at in terms of what drives cost. know, the, you know, the number of packages being uploaded.

There are cases, ⁓ Brian Fox was talking about how some people are using Maven Central as what I can only think of as a file system of just uploading weird stuff, right? Then all of them have that. And then other people like, well, we build this stuff, you know, 3000 times a day for whatever reasons, we’ll just push 3000 times a day for whatever reasons, right? Doesn’t cost me anything, so why would I not do that, right? And there are lots of unmeasured, unfelt costs that work their way through the whole system. And then also, by the way, if you’re operating,

Josh Bressers (28:14) Yes. ⁓

Michael Winser (28:37) If you look in aggregate across all of them, like the 10 trillion downloads a year, right? I don’t know how many uploads, it doesn’t really matter. The numbers are big enough. Even if the cost per thing, cost per download is really small, this colleague of mine, Paul, made a really eloquent point. If you take a small number and multiply it by a million, it’s probably not a small number, but it’s not a big number, right? If it’s really small, a million times that, it’s like, okay.

Like a millionth of an inch, I multiply by a million, I’ve got an inch. Okay, fine, right? If you multiply by a billion, it’s starting to get big, right? If you multiply by a trillion, it’s big, right? People don’t get big numbers. Like honestly, the one thing that unifies all humans is that they think they understand big numbers and that they don’t, right? Exactly. And so 10 trillion aggregated across all the package registries is a lot of traffic. It’s more than Google does a year.

Josh Bressers (29:16) Yeah.

One, two, three, more than three, like, right? ⁓

Michael Winser (29:35) Right now it’s different traffic. It’s not apples to oranges, things like that. But, you know, at Google, for example, nobody would operate a service with less than 10 people on the team. On call, incident response, development work, you know, all the things, right? These, these registries are being run by an aggregate set of people across all of them with like maybe 20 people and a lot of them are volunteers. Right? And volunteerism is great.

Josh Bressers (30:00) Yeah. Yep.

Michael Winser (30:05) Right up until it’s not.

Josh Bressers (30:07) And look, most of these registries are critical infrastructure for like, honestly, like all of society at this point. I mean, if one of the major package registries, like literally just up and went down for a week, an enormous amount of society would come to a crushing halt.

Michael Winser (30:22) This actually was one of the things that helped out and no, we’re not going to do this on purpose. ⁓ OpenVSX was a single maintainer abandonware project that was ultimately being used by all of the VS code clones as their package manager, including the up and coming AI universe. ⁓ And then they had a bad week. They had a total failure on a storage system that didn’t recover well.

old adage, nobody needs backup, they need recovery. ⁓ And it was a sufficiently big brownout that a whole bunch of companies whose lifehood, livelihood depended upon the wellbeing of OpenVSX realized that it would be chump change for them to like pay attention to this. ⁓ And so, you know, we’re not gonna go off and cause a brownout at PyPI or anything like that. That’s not an anyone’s playbook. ⁓ But it…

People don’t know what they are, this is you said earlier on, it’s transparent. They don’t know about this. Like it’d be pretty entertaining if NPM, every time you installed it, showed you how many human bits of time and energy and like that were consumed in the production of this package. you know, people ignored, people ignore explicit security warnings not to use this package. So they’re not gonna pay attention to anything other than that, right?

Josh Bressers (31:38) you

Right, right. But okay. So, like, what do we do? Right? And this is one of the things I will say, like even watching your talk, I feel like the, well, I guess you do mention at one point, like we actually don’t know how to solve this, but I feel like you probably have some suspicions is my guess.

Michael Winser (31:57) We

have opinions and I think, to be honest, I didn’t want to sort of, this is a work in progress and I don’t want ⁓ to impinge upon the work that’s happening while we’re talking about it. My goal at the talk unabashedly was to get people to break into bandwidth jail and pull them out, learn that it’s more complicated than that. Well, actually, higher it a bit, understand that holy shit, there’s this infrastructure.

Wait, wait, and it’s costing money, right? Like if you walk away from the talk with that, great. If you then go, we should charge your bandwidth and you realize, oh wait, it’s more complicated than that, And you know, collectively there’s been an awful lot of thought about how do you create a product? How do you create something that you can create revenue for? Because at end of the day, there’s only a certain set of sources of funding, right? You can have donations, you can have grants from governments.

You can have grants from Alpha Omegas and similar things, or you can have transactional revenue.

And I actually think you need all of the above. You want to have a diverse set of income sources so that the ebbs and flows of one thing or the other don’t cause your organizational process, because these things are too small to absorb significant changes in budget from year to year. ⁓ But when I say transactional revenue, I really mean like all the transactions around packages, whether it’s uploads or namespace verifications or like that.

everything you can imagine, malware detection, all the things that we need to do, all the behind the scenes engineering work to make that thing work and keep working at scale, right? And then yes, the downloads and the installs and the metadata checks and all that, you know, there’s a lot of stuff happening on those queries, right? At the other day, if your revenue is not at least linear in some way to that, right? Who was it? Titus, I think it was Titus Winters.

talks about like, know, every process you can imagine is either sub linear or super linear, right? And at big enough scale that starts to matter a lot. right. So getting it to a place where the constants even are no longer crazy or the exponents are in fact getting them down to like small exponents, that would be good. Right now we are nowhere near that, right? We have literally flat and declining revenue and

every indication of geometric, if not exponential, growth across the space.

Josh Bressers (34:37) Yeah, the growth is bananas. I mean, I know this is something I’ve looked at a lot and every year it just keeps getting crazier. And I keep thinking, maybe this will be the year it stops growing. And like, nope, certainly not.

Michael Winser (34:41) Yep.

Well, so there’s an intrinsic, like, nobody deletes packages. And in fact, there are reasons not to do that. It gets complicated again. But like, so there’s just more stuff, like sedimentary stuff that just piles up. And it’s not like it’s free. It’s not like they’ve built a sharding system of you only get to query PyPI from 1999, or, you know, that’s not how that works, right? So it’s, so.

And look, there’s no escaping that the growth of LLMs and AI-based solutions, right, represent a huge load on these systems, right? AIs are querying these things and learning from them, ⁓ somewhat frustratingly also getting it wrong and hallucinating package names across language ecosystems, which ended up creating this concept called slop squatting that Seth coined. It’s brilliant, right?

Josh Bressers (35:28) Yes.

So good.

Michael Winser (35:50) What that means now is that the 404 feed from a package registry is a zero day waiting to happen. It’s a malware vector, so it’s now secure. ⁓ So getting back to what can we do about this, as a erstwhile product manager, you have to have a product that you can sell. You have to have customers and you have to know who your customers are. It doesn’t mean we’re going to

Like one of the easy things to say is, let’s just do the iTunes model. We’ll charge a buck for every package, right? And that just, it falls apart instantly. First of all, these packages are free. And as I made clear in the talk, like every one of these obvious answers has an obvious workaround that will not only not produce the revenue to sustain these package registers that we need, but will actually create a balkanization of solutions that makes it even worse. Remember, I don’t actually specifically care about the package registries.

I care about them as critical infrastructure and points of leverage to help make everybody more secure. If there was a path where our packet registries went away and didn’t have that problem and we were somehow more secure because of that, and I could see where everybody gets more secure and nobody installs a crappy version of log4j that would be great. But first of all, that’s not happening. And second of all, the packet registries are a viable point of information and metadata that is really crucial for security.

Right? And trust, right? They are brokering trust between the consumer and the publisher without having to have everybody have pairwise trust relationships. But what that means is consumers are implicitly trusting the package registry the same way that we trust our app stores to give us the thing that we think it is and for it to be reasonably okay, right? Whether they’re conscious or not, they are making that assumption.

Josh Bressers (37:20) Yeah. Yeah.

my goodness, this is so complicated. So, okay, I’m going to put a link up.

Michael Winser (37:44) It is really messy.

I want to go back to the product point, right? So we’re giving stuff away for free right now. Bandwidth, packages, namespaces, every feature you can imagine, right? The only way we can make money out of this, right? So unless all the governments of the world decide, we were kidding, here’s a pile of money, which is not going to happen. Like a friend’s assistant say, it would literally take an act of Congress. Like not figuratively, but literally, right?

That’s not gonna happen.

If it doesn’t, we don’t get to that. We have to get to some sort of product based revenue. And the only way you get there is by creating some scarcity where today we have abundance, unlimited access today. Right. So there’s going to be a journey of figuring out what that is. Agreeing on what it is. think that’s going to be significantly easier for us as an industry to move forward here. If all the package registries are on board and saying the same thing.

because you know how it would play out, right? If one package registered, hey, you know that free stuff we were kidding? Everybody pays a buck per package or you have to sign up and pay a buck per year or whatever, right? There’s a million of you, that’s a million bucks, it’s great. People would like, first of all, there would be the fork the next day of somebody’s alternative thing and second of all, they’d be really pissed off and all this mess, right? So that, ⁓ but, you know, just to put the math there for you, right? So let’s say that there are,

Josh Bressers (39:04) Yes.

Michael Winser (39:14) 10 major package registries. There are definitely more than that, but in terms of volume and cost and things like that, we’ll say 10 instead of averages out to more than that. And let’s say that their current budgets should be 5 million or not, and will be 10 million because they need to be. That’s an interesting number, so that’s $100 million. If every company that had revenue over a billion dollars paid just $10,000 a year,

That’s $100 million. So the hard part is getting from here to there. If it’s normal for, it’s the cost of doing business to have, we’re building software. People pay more than that by huge margins for SaaS products that are integral to their lives in thousands of ways. If this was normalized in some way, we would be fine.

Josh Bressers (39:48) Yeah. Yeah.

Michael Winser (40:17) And I’m not proposing a giant Alpha Omega board of registries that would take all the money. On the contrary, I want those monies to flow where the growth is on those ecosystems. That’s why the transactional aspect of this matters. But in aggregate, it’s $10,000 a year from the billion dollar companies and nobody else should have to pay anything.

Josh Bressers (40:35) Well, now that’s just for the registries though, right? There’s still open source developers. There’s ⁓ many other things upstream of registries that if registries are getting paid, they’re going to be like, hold on a second, right?

Michael Winser (40:39) Yep.

So this is where there’s the difference between, and I think I talk about this in my talk, Infrastructure versus source code. And so the per unit cost of source code, open source code, trends towards zero as it gets more users, right? The per unit cost of a registry or any kind of infrastructure, including ecosystems or deps.dev or NVD, right? Or OSV, all these things, right?

Josh Bressers (41:08) Yes.

Michael Winser (41:19) That trends towards infinity as it gets more users. So there’s a fundamental difference here. I’m not for a second belittling the huge gap of problems we have and especially on the security front like that, but I also pretty clear about my mission. I’m here to make open source more secure, not to solve all the economic problems or wanna be economic problems of every open source project. I’d love to stop working at the insurance company doing work that they pay me for and get to be full-time maintainer of

meaningfully not huge project over here, right? Like, it sounds great.

It’s real, that space is, and it actually makes securing things hard too, because when people are doing this stuff as nights and weekends work, you can’t pay for more nights and weekends. And so, you know, even if I had all the money in the world and I could pay people to go and become security experts and be security experts, right?

Josh Bressers (42:12) Yes, that’s right.

Michael Winser (42:24) unless I can give them a full-time job that they don’t have to worry about going away and has all the benefits and perks of their current job, right? Like, so like the business of being a full-time open source maintainer is not what it’s cracked up to be, obviously. And being a part-time one, well, that definitely sucks because it’s nights and weekends, right? And so ⁓ there are some complicated economics and human factors at work here, that’s for sure.

Josh Bressers (42:48) Yes,

a hundred percent. And I had a show, talked to Thomas de Pierre, the I am not a supplier guy about all this. And yeah, I mean, that’s one of the things he said is like, unless I’m making, you know, as much money as I make today in my normal job, I can’t like sort of work on open source, right? Like it’s exactly what you said, which is, yeah. And this is multiplied by a million people, you know, that’s, it’s wild.

Michael Winser (42:53) Yeah.

⁓ yeah.

It is.

And there are, it’s an uneven, unevenly distributed space too. There are some projects that have significant resourcing. One of the Alpha Omega initiatives is something we call beach cleaning. And so in the wake of the XZ thing and observing that, you know, open source maintainers were not paying attention to their upstream, I started thinking about, well,

what would call resources to flow to upstream. And at the same time, we had been doing some efforts on the Omega side where we were working with some vendors and building out some automated tooling to go off and automatically find, fix, submit, and then not so automatically push PRs through on fixing a bunch of stuff. And we got to the point where we could probably do the math on how much it would cost to…

do a lot of stuff, right? And unfortunately, that’s how it came out to describing it, doing a lot of stuff. It was like sending a boat out to the Pacific Garbage Patch, and the boat comes back and it’s filled to the brim with plastic. What size boat is it? Doesn’t matter because out there there’s the Pacific Garbage Patch, which is gigatons of plastic. And so nobody can do anything differently and nothing has changed except that we need to send, we need a lot more boats, right? It’s like the old joke about genies and you don’t get to wish for more wishes, wish for more genies, right? But we don’t.

Josh Bressers (44:26) Yeah, yeah.

Michael Winser (44:34) We don’t have that. And, and so, ⁓ I’m sorry.

Josh Bressers (44:36) it’s so true. Okay, Michael, I’m gonna call this one. I feel like I could talk to you for hours, but I won’t. Not

this time, no, this is great. I’ll text you back, but.

Michael Winser (44:44) So

we should do something about beach cleaning because it’s a whole topic. Okay.

Josh Bressers (44:47) That sounds fascinating. And yes,

I would absolutely love to just talk about that next time. But for now, anyone listening, what should they do next? What can people do to learn more or get involved or kind of the next thing you want them to go do.

Michael Winser (44:53) It’s a great talk. Yep.

So if you are a maintainer or operator of a package registry, you should show up in the Alpha Omega Slack channel or come to one of our public meetings or ping me directly, LinkedIn or all the various medium and say hello and we’ll get you into the process at the level. If you are a…

If you are an individual open source maintainer, contributor, consumer, typical normal, know, Mary or Joe working on stuff, right? ⁓

Honestly, there’s not much you can or should do. ⁓ If you aren’t, as a project level, maintaining a cache of all of your dependencies pinned and managing that cache explicitly, maybe you should. I’ll remind you all of leftpad, colors.js, faker.js. There are three risks, correctness, integrity, and availability. Make sure that you don’t depend upon the internet to build your software. And then if you are a business,

whose livelihood depends upon building stuff all the time and is consuming metric tons of open source packages on regular basis. Call me up. Tell me how much you think it’s worth to you. Again, I would entertain that conversation, but we are gonna come to you with that conversation as well. So, yeah. Perfect. Yeah.

Josh Bressers (46:29) nice. And I’ll put links to a bunch of this stuff in the show notes for anyone interested. But

Michael, holy cow, what a fire hose. This has been an amazing discussion. I truly appreciate it. And I cannot wait to have you back to talk about just all of this. Right.

Michael Winser (46:37) You

⁓

In the date and the time I’ll be there. Thank you very much for having me. ⁓ for, you I’m honestly very grateful for Alpha Omega to be, it’s been those sort of, this is my retirement, by the way. I’m not really full time, but it’s been, it’s been the best thing ever. So yeah, it’s not as, as they go, it’s pretty good, right? Yeah.

Josh Bressers (46:58) Pretty epic, man. That’s an epic retirement.

Fantastic. All right. Thank you so much, Michael.

Michael Winser (47:05) Thank you.