Josh talks to Richard Hughes about the world of firmware. We cover how Richard’s journey from developing the ColorHug led to the creation of the Linux Vendor Firmware Service (LVFS), changing how firmware updates are managed for nearly every Linux user. Updating firmware has always been dicey, and on Linux it used to be impossible. Richard helps us understand how this all works and how we can all help out.

This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.

Episode Transcript

Josh Bressers (00:00) Today, open source security is talking to Richard Hughes. He’s a Red Hat LVFS maintainer and Richard, I’ve so many questions and so much to say to you. Richard and I worked together when I was at Red Hat like a long, long time ago. And so just welcome to the show. Why don’t you tell us a little bit about who you are and what you’re doing and we’ll go from there.

Richard Hughes (00:18) Sure, hi, I’m Richard Hughes. I’ve worked for Red Hat, like you say, for like, don’t know, like 18 years now. ⁓ I ⁓ started at Red Hat and started working on power management stuff, color management stuff, ⁓ and more recently, sort of firmware management stuff. ⁓ In 2011, I built a color sensor, which was kind of inconsequential, ⁓ but it meant that it required firmware. And so I built a firmware updater for this little device.

And then we made it into a generic thing, which could then apply UFI firmware as well. And that was, I don’t know, what, like 2015 or something. And since then we’ve just grown and grown and grown and supported more and more hardware. Then the LVFS got born, which was the website that vendors upload firmware to, and we use for downloading metadata and archives and stuff. And now we have 140 OEMs and ODMs using this service. And we’ve deployed like…

135 million firmwares

Josh Bressers (01:20) Wow. Wow.

Okay, so I did not know that the color hug was the inception of this, which…

Richard Hughes (01:27) Yes, the colour hug was this tiny little colour sensor and all it did was measure the colour and put it on the screen. Open software, open hardware, and I needed to design one for this colour subsystem that was building. so I built a firmware update like all the other companies do. They did like a little flask app to deploy the firmware securely, did the signing, I wasn’t using MD5s and all the other stuff that people were using at the time. And I thought this is crap.

really crap, fact that I have to re-implement all of this for every single device, for every single vendor. Wouldn’t it be good if we could do something shared? That could have like share between different vendors. And about the same time Peter Jones was working on lib for FW update, which is like a way of deploying capsules for UEFI hardware. And he said, well, wouldn’t it be good if we could use your system to do like capsules as well? I said, okay, I’m sure it’s easy enough to make you narrow.

and I can do a really nice web front end so we can deploy the stuff with high quality metadata. And that’s kind of the nexus of where it all came from. And then with fwupd we had this high quality tool which we could use to deploy loads of different types of firmware. And Dell said, yeah, we want to use that for IoT stuff, and then we said we want to use it ThinkPad. But we didn’t really have any center of the shared metadata that we could use for actually deploying onto devices.

which is where LVFS came from and then that’s how the ecosystem kind of exploded and that was my kind of like one day a week job exploded into my five day or six, seven day job.

Josh Bressers (03:07) Nice, nice. Okay, holy cow. That’s cool. Okay, there’s a lot to unpack in everything you just said right there. I need to figure out where to start with everything you just said. I wanna start actually, no, I know where I’m gonna start. I’m gonna start with a story and it’s the day I met you in person for the first time.

Richard Hughes (03:10) Yeah.

What did I do?

Josh Bressers (03:25) And you changed

my life. And I think about this conversation at least once a week. Yes, you probably have no memory of this, but so we met and I can’t remember who introduced us anymore, but you started asking me questions about. automatically applying updates in fedora, right? Where normally back then you would have to run, you know, is yum, you know, yum update.

Richard Hughes (03:30) my god, go on, what did I do? No, I don’t.

Yeah.

Josh Bressers (03:50) and then answer yes to a bunch of questions usually, because new keys and everything. And you were like, right, and you were like, we should just install updates automatically. And I was like, that’s the dumbest thing I’ve ever heard, right? Which at the time I was a terrible security person where the, it was, the answer was no to every question, right? That was just the way security used to work.

Richard Hughes (03:54) Yep.

So weirdly that kind of ties into the fwupd story as well, the firmware.

Josh Bressers (04:17) Yes,

exactly. I think it is exactly part of this. And so obviously you were right and I was wrong where I’m very willing to admit that and installing updates automatically, generally speaking is the right thing to do, especially for like consumer devices. Now we can kind of fast forward. If you would have asked me back then, should we install firmware updates automatically? I would have been like, we have to fire this guy cause he’s an idiot, right? Like

Richard Hughes (04:24) Hahaha

Yeah.

Josh Bressers (04:44) But now, like this is part of the story, right? Is you’ve built the ability for Linux users to install firmware updates basically automatically. I use this all the time, right? I have hardware that your software is updating my firmware. And I remember back in the day, if you were a Linux user, you basically didn’t update firmware because there was no way to do it.

Richard Hughes (05:03) Yeah,

so if you did updates, when I started this, the only way of updating firmware in Linux was literally finding a like a free DOS, like 1.4 megabyte disk and putting the firmware on that and then using some esoteric. It was horrible, absolutely horrible. yeah, so going back to what you said about like the updating stuff. So I built package kit and then all the GUI tools.

And then we ran the stats and we found that actually like 95 % of people were updating their system by clicking a button. Not by typing on the console or getting an error system administrator to do it, but literally just looking at a application, clicking on a button, waiting for progress bar to do this and then restarting. And that was a real wake up call. So when I did firmware updates, I said, okay, well this stuff, we have to make this stuff look beautiful as well. like, you like, don’t assume that people speak English.

to provide the release notes in many different languages, provide pictures for people that don’t know how to hold left, enter and start at the same time. So literally providing line art rather than a long English description. And because we made it beautiful, I mean, integrated it literally into KDE and GNOME and all the other desktop environments, 99 % of the updates that go out are from someone clicking a button rather than from someone using the command using their terminal.

Josh Bressers (06:30) That’s amazing. That’s so cool. And I mean, look, I use it all the time. use, I mean, I’ve been running Fedora with GNOME on my computers for like forever. And yeah, more often than not, I just open software manager. don’t even know the name of it. I just typed software into the search box and then click that icon.

Richard Hughes (06:48) So now

it’s kind of gone even further. So people are not only like installing firmware on their machines using all this stuff I’ve built. They’re actually choosing hardware because they need that ability. And so they’re going to like fwupd.org/ and saying, okay, what model is supported? Is my ⁓ framework mainboard supported? Is my ⁓ Dell XPS 13 supported, et cetera. And they’re choosing to buy the machines based on whether it’s supported or not. And then we’ve kind of gone one step even further.

So now we have this this Prodcer thing where we have this FWF-friendly firmware. So if you’re someone like Woodstron, Foxconn, or one of the people actually building the hardware, you can go to this list and find companies that are friendly towards fwupd and LVFS and who’ve done this process already, and then choose components based on how easy it’s going to be for them to provide updates via LVFS. And so we basically nailed supply chain because we’re encouraging vendors to build stuff that’s Linux friendly.

so that we can make it easy for people to deploy updates on Linux. It’s absolutely mad. There’s two vendors of notes that aren’t using the LVFS in some way, and that’s Apple and Broadcom. But I think almost without exception, they’re the only two.

Josh Bressers (08:05) Well, Apple, understand. Apple, I mean, but wow, that this is like the dream we used to talk about over beers, you know, 20 years ago. Amazing.

Richard Hughes (08:16) Yeah.

It’s kind of like, it’s what we were aiming for. it’s like being in the right place at the right time, having the experience of all the previous projects and stuff, that’s a huge part. But a lot of it was 10 years of obsessing over hex dumps awful lot where you get to midnight and you’re staring at some hex editor because the NVMe update isn’t working correctly. there’s a lot of…

Josh Bressers (08:43) Right.

Okay. So let’s, let’s start with that then is so when we talk about firmware updating, like what does that actually mean? I suspect there’s a lot of people that have heard the term firmware updates, but it’s not necessarily like they don’t know what’s going on.

Richard Hughes (09:01) So firmware, the way I kind of describe firmware is just really hard to update software. So there’s nothing magic about it, there’s nothing special that makes firmware special as opposed to software. So good example, this mouse, this mouse is a mouse, it’s a peripheral, but inside this thing you have a system chip, does the mouse stuff. You’ve then got a radio chip by a different vendor, which the main chip communicates with.

And on the other end you’ve got a receiver with a receiver chip and then with a main processor chip. And so you’ve actually got four bits of firmware just for moving the mouse. So on a ThinkPad there’s 26 individually like, updatable components on a ThinkPad. Everything from like the trackpad, the little red dot in the middle, ⁓ the interposer between the webcam and the mainboard. Like all of these things have really hard to relate firmware.

and they all do it in these really weird esoteric ways. So it might be like using some legacy serial protocol or it might be using USB ⁓ like HIDs, like get set type reports. ⁓

And this kind of segues into another thing. fwupd tries to make everything very generic. So there’s a joke in the software, firmware community, if call it that, where Flash ROM tries to treat everything like an SPI chip, and fwupd tries to treat everything like a USB device. And that’s kindly how we model everything. Everything’s a device where you can do stuff too. So we enumerate all the stuff on the system, whether that’s a UEFI firmware, a mouse, a web cam

whatever ⁓ and then we allow to install blobs of stuff on those devices. Now you get from the blob to the actual device varies loads between different vendors but it’s really like although the vendors make a big deal about their intellectual property and they’re like totally bespoke update protocols they’re using they’re all basically the same so one of the cute tricks that I do when we’re treating when you’re a quick talk to a vendor and they’re like

you need to join the LVFS because your OEM says you have to. And they’re like, no, we can’t tell you what our update protocol is because it’s some proprietary secret. I’ve got this great party trick I do with them. And I say, look, can I guess your update protocol? And they’re like, no, it’s like 10 years worth of research and it’s some bespoke, beautiful update protocol. I’m like, hey, let me try. You’ve got a command that turns the device into bootloader mode. You’ve got a device that erases a block, a command that writes a block, a command that reads a block.

and then another command that turns it back to runtime mode. And they’re like, ⁓ how did you know our upload protocol? And I was like, cause they’re literally all the same. Sometimes you’re going over different transports or you’re encapsulating it different ways or you’ve got different headers and footers or hashes or signatures or whatever. But it’s basically the same. And so if you say to all these companies, look, if you just reuse all the primitives that we’ve created in fwupd for, if you’re a HID device, just derive from FooHID device, you don’t have to re-implement all of the, like the device goes away, device comes back.

you don’t have to re-implement all of the get device, set device, all this kind of stuff. You just implement the bits that are unique to your protocol, and that means you have an update plugin, which is like 16 kilobytes in size.

Josh Bressers (12:25) That’s awesome. I love that so much. And it doesn’t surprise me at all.

Richard Hughes (12:26) Yeah. Also,

it’s weird how Chrome OS happens. So now fwupd is a hard requirement for Chrome OS for the work from Chromebook certification. It’s not for the reasons, it’s not for like ideological kind of like open source purity or anything. It’s space. there, like a quarter of their EMMC storage in a Chrome OS machine was firmware updaters. Like they had like 30 different copies of G-Lib C linked in to like…

I know like Samsung SSD updater and Logitech peripheral updater and all these different updaters have these multi-megabyte updaters and when you’ve got one image that has to do all of the hardware and you might have a thousand different updaters actually the fact that each one’s three megabytes is a big deal and so when I said I can install any NVMe firmware on any of that vendor’s device as long as it supports the end-of-the-update protocol in 16 kilobytes

Google were like, this is a no-brainer. This is literally mixed total sense. Because we could cut gigabytes of their install size just using one update protocol.

Josh Bressers (13:39) That’s amazing. And I love the fact that without like the open source universe, this almost certainly wouldn’t exist.

Richard Hughes (13:46) Yeah, absolutely. It’s the best part of open source. it’s another good example is like, we have a vendor that comes to you and says, look, I’ve got this doc and I’d like to have Linux support because I need it. Other customers that are using RHEL or Ubuntu or whatever, they say, look, how do we get our doc supported? And I’m like, what does your doc consist of? they’re like, it’s Synaptic MSC, it’s Goshen Ridge, Intel Thunderbolt, and it’s real tech audio. And I’m like, okay, well, give me the vids and pids and it’s magically work.

And then but how does the software how do you build the software and how do we contribute the code and I was like No, but all the code is written because someone else or the other vendors has already contributed that code All we need to do is add the vids and pids which means it matches your hardware and then we can all share a code base And it just blows their minds because they haven’t had to pay for it Like it’s like how much does this cost and I was like nothing like it’s literally like it’s like six lines in a file And it just blows their mind

because everyone is in the closed source world out for millions and millions and millions just for software development and then hosting this stuff. And then they know it’s literally free.

Josh Bressers (14:56) Okay, well, sort of, sort of free. So let’s talk about that. So you have LVFS. I don’t know the name of it. Explain the organization you’re a part of. I know the Linux Foundation helps sponsor, Red Hat helps sponsor, but then you also have the hardware vendors like joining for fees to help all this along.

Richard Hughes (15:15) So that’s a good story. So when we first started the LVFS, they were actually the website where vendors upload firmware and use down-limiter data. But it was being used, I don’t know, I think it was like five million people had used at that point. And it was literally on an HP microserver underneath my stairs. And it was being used by Bank of America and three-digit US agencies. yeah.

It all worked beautifully and the way we’ve designed it means that it does scale just from one central node. You only need a few people to use a central node and then everything else scales out on a CDN. But it wasn’t a resilient system. I was talking to the CISA in the US and things about this and there was some strong encouragement that someone should help maintain the infrastructure behind

make it microservice, make it fully resilient, make it multi data center, all this kind of stuff. And I don’t have any of that knowledge, or any of the money to do that. Like I was running the whole LVFS like on like $30 a month or something. really, it’s nothing, right? It’s a CDN cost. And so the suggestion was use the Linux Foundation. The Linux Foundation said, yep, this makes total sense for us because it’s a critical infrastructure project which is used by Linux users.

increases limits adoption, makes total sense for us to kind of pay for this and shepherd this and provide some of like the legal insulation because until then it was like Richard Hughes London rather than kind of any LLC or anything legal stuff. was totally naive. So they’ve made like a little like a self-contained LLC and hosted it for all these years, et cetera, et cetera. And it’s growing slightly exponentially. Now when the costs, now the costs are only like, they’re, don’t know what they’re.

30,000 a year for the entire cost of everything. And they’re kind of happy with this, but not happy with this going up exponentially. The problem now is that if I get hit by a bus, the whole ecosystem falls apart. And so lots of people are very anxious. Lots of big governments and lots of big companies and things are very anxious that there shouldn’t be just one random guy working in his shed in London.

this should be a team of people that are actually like with 24 hour support and like training and like compliance stuff and all this kind of stuff. And the bitter pill is that soft and squishy people are really expensive. And so we said the only real way that you could justify this is go to the biggest companies who are using the LVFS and say, will you sponsor us? And if you sponsor us, we can work with you on these custom features you want. can.

embed LVFS stuff into your dashboard, we can allow you to update from build pipelines and all this kind of random stuff that they’ve wanted for years. And I was going to the biggest vendors on the LVFS are like Google, Lenovo, Dell, HP, etc. And I’m saying to them, if you sponsor us, then we can provide this resilience that you really want. And with one or two exceptions, we’re asking for large amounts of money and they’re like, yep, this is a no-brainer, makes total sense to us.

where do I sign? So it’s fantastic. the only, like the only, kind of, I’m desperately kind of a pro, like open hardware, and I never want there to be a fee to distribute the kind of updates I was developing for Colorhug. So if you’re distributing like, I don’t know, 50 million downloads a year, and you’re on 500,000 different laptops, you can afford to sponsor an open source project.

If you’re open source hub, if you’re open hardware vendor and you’re selling, I don’t know, 500 units, it should always completely be free

Josh Bressers (19:05) Yeah, that makes sense. That makes, I mean, yeah, exactly. So explain to me then the, I guess the process of hosting vendor firmware, because like, what does that interaction look like? What is the vendor doing? What are you doing? What’s happening for me as the end user then? Because from my perspective, this is all kind of magic, but obviously there’s a lot of things happening.

Richard Hughes (19:28) So from an end user point of view, it should be a case of something pops up on the desktop, you click install and it magically works. To make that happen is really hard. traditionally what the BIOS vendors will do, at least like System Firmware is the example, they’ll create a UEFI capsule, which is like a EFI binary blob, and then upload it to an FTP site somewhere. And then someone clicks on it, sticks it on a USB disk, sticks it on a computer, and then that’s it. Right? Now that doesn’t work. So what I ask them to do is take that

Deliverable stick it in an archive like a zip file or cab archive or whatever with some high quality metadata So like tell us what CVEs it fixes tell us the translated release notes Tell us whether this is a security vulnerability or a low priority enhancement update Tell us what type of hardware this is like tell us all the information we need to be able to include this in like a metadata catalog so that gets uploaded by Usually not the OEM. So it’s usually if you if you’re someone like

but you might be someone like Foxconn or Whistron or one of the people actually making the hardware for the OEM like Dell or Lenovo It’s very unusual for the OEM to upload now. And also it’s building lots of build pipelines. If you’re Dell, you don’t manually upload the firmware. You just build it in your CI system and it gets automatically uploaded to the LVFS It’s a completely automatic system now. From there it gets put into like an embargo remote which you can arrange to be moved to stable and

testing a stable on predefined times and days or you can do it manually. When it actually hits the LVFS we run loads of tests on it. So we’re doing even dumb tests. So a really good example is we scan for the words do not trust, do not ship, which is how we found the Intel CVE up a river regarding the ⁓ default example signature. And that was an 18 month embargo period on that CVE.

just from looking for the words do not trust, do not ship. So that’s the most dumb test. What we also do is we’re working with Benali. We upload it to their ⁓ binary scan platform as well, which gives the vendors early access to like, hey, you’ve messed up this. We do static analysis on some of the P binaries. We check the microcode versions, make sure there’s not accidental microcode downgrade. So there’s a whole barrage of tests we do on the firmware. And if all those tests pass, it can then move to the testing remote, which is available to about, I know.

people who could have opt into this manually. Once it’s been through like usually internal QA and it’s been verified what happens is the internal QA person will download the update onto the actual physical hardware, reboot, apply the update, reboot again and then when it comes back up they will sign a like an attestation which basically says this firmware update has gone from this old version to this new version.

which gets automatically uploaded to the LVFS so that we can then show the end user this update has been tested by, and then you list the OEMs or the ODMs. So you have this kind of like, provenance, that’s the wrong word, but kind of like you have this, you can’t just upload something that hasn’t been tested. Because for a long time it was a wild west where you could just upload stuff and oops, it didn’t work, we tried to deploy it on 1,000 PCs, it didn’t work. But now if it fails to deploy more than 1 % of the time, we just.

de-deploy it back into embargo because we can’t… updating firmware is really scary and for the user to… for a machine to go black and have to actually reset it or something it might as well be a brick so it has to be one of those things that works 100 % of the time and so even as an end user like you’ve done firmware replace yourself like sometimes if you do like F.M.G.R. get-devices it’ll say this went successfully would you like to upload this to the LVFS?

And it’s not, you don’t sign it saying this is from Josh. I work at so and so. It’s just an anonymous report. But if you have like a million downloads go out and you have 900,000 reports that come back and say this worked really well and seven that went back said this didn’t work for me, then you’ve got fairly high confidence there’s actually a good update and should be continued to sort of go out to end users. So kind of, it closes the loop. So rather than an FTP solution where you just fire and forget, it’s just kind of like, it’s like a

ends to end so the vendor knows, okay well I know this is going out 10,000 times a day in this nice kind of, it’s usually a nice decay curve where it goes out like this and it knows whether, and we even have like a rule engine so if it fails we can sort of work out why. So it could be, okay well this is failing, this is failing at 5 % of the time, let’s dig into the failures and it could be like, okay well this is, only people with SELinux turned on and it’s failing for.

And so then can okay, right, well let’s do a known issue which redirects people to a Wiki page to say, you need to update this fwupd before you ⁓ update if you have SELinux turned on. So we have all this clever, cleverness to make things just work. And there’s a huge amount of infrastructure just to make it click and go.

Josh Bressers (24:37) Wow, that’s way more intense than I thought it was. But I mean, you raise a good point though. Firmware updates are scary. I just had to update the firmware on my motherboard the other day and I was like, I wonder if I’ll be buying a new motherboard in an hour

Richard Hughes (24:50) Yeah, and of course, if I provided an update through LVFS, is the vendor’s responsibility to upload it and test it, et cetera, but ideologically it’s me that’s providing it. And if that bricks your PC, not only will you never trust it ever again, you’ll tell all your friends and people on your podcast and stuff that do not do this because it bricked my machine. And especially if it’s out, if it’s not in a longer in support, that’s even worse because you can’t get it repaired. So it’s one of those high, very, very critical things you don’t want to get wrong.

We have got it wrong. There’s been reports where we’ve broken people’s systems and I’ve always been given the OEMs a really hard time and that’s why we’ve brought in all this reporting and this self-demotion and all this other stuff that means that it fails fast.

Josh Bressers (25:37) Yeah, yeah. And that’s no doubt super important. mean, I’ve break more than one computer with firmware updates and it’s.

Richard Hughes (25:42) Yeah.

Why do you think vendors keep sending me hardwares? Because I keep bricking them. You get a firmware update protocol that’s meant to be unbrickable and I’m like, I’ve done it. Correct.

Josh Bressers (25:49) I mean, that…

Challenge accepted.

mean, so that’s an interesting idea here, right? Is so you’re being sent hardware and then you’re working on updating the hardware and just out of curiosity, like how often are you breaking this stuff? Is it very consistently all the time or just occasionally now or what’s like, how does that look?

Richard Hughes (26:14) It’s occasionally.

So the bad old days 10 years ago, what would tend to happen is you’d have like a system firmware would have like an A bank and then you’d update the A bank and if it was wrong, your system shouldn’t boot. And then it slowly got better. So you’d have like an AB where B was smaller than A. B would be like a recovery partition. So it’d be just enough to turn on the laptop or just enough to turn on the PC and to flash the actual firmware. it’s, you had some GUI.

And now as flash costs go down, they’re more and more vendors just doing AB. it’s just, again, with Google Chrome OS has helped, Google Chrome OS has said, look, this is such a scary thing. If we roll it out to billions of people, we only want to roll it out where there is an AB capability. And so it pushes the vendors to spend that extra 12 cents on their bill of materials. It is insane. Just 12 cents on their bill of materials to make them have full AB firmware, which means that

not only do you, it’s better than not being able to brick it because you can deploy the firmware while you’re using it and then you only have to reboot to go into that new system.

Josh Bressers (27:20) Yeah, yeah, for sure. It’s, it’s a different world. A hundred percent. Okay. And, and so I want to tie the, I think that the security story in this, which we’ve not gotten to yet, but I want to cover it before we end is there have been like a ton of vulnerabilities in UEFI firmwares lately. And I feel like probably 10, at least for sure, 20 years ago.

No one updated their BIOS unless something was catastrophically broken. And now we’re seeing like BIOS updates happen regularly for like vulnerabilities fundamentally, right? And from the security angle, this is a good thing. We’re happy about this. And I feel like without, without something like LVFS that just wouldn’t have happened at all.

Richard Hughes (28:06) think the change between BIOS to UEFI, it sort of changed the assumption of what Firmware was doing. So BIOS was just enough to get the OS loaded and UEFI provides this full pre-OS environment with networking and access to all of your devices and all the graphical stuff that people need for various different things, TPMs, all that kind of good stuff. But in doing so, it massively increases the attack surface.

And I agree, without something like Windows Update and ⁓ LVFS, without the two systems, it would be a disaster. Because your system firmware isn’t just what turns your machine on when you press the power button. It’s in most cases your root of trust. If you don’t trust your firmware, you can’t trust your kernel. So there’s various different vulnerabilities where because the system has never had a firmware update, it’s critically vulnerable to some like…

So good example is CSME. So CSME is like your Intel ME firmware. in this Intel ME firmware, you have a remote desktop capability. So as long as you have the LAN connection plugged in with power, you can remotely turn on a system, flash its firmware, and then turn it off again. And so if you don’t have your CSME up to date, then it’s game over. Like you might as well have no OS security.

So one of things that came out of the security angle of this was HSI. So about 2015 I think it was, we started saying, look, fwupd can enumerate all of this hardware on your computer. It knows your system firmware, your system BIOS, all the different TPMs and all this kind of stuff. It enumerates all the different devices. Maybe it could check those devices to see if they’re set up correctly. And we started doing things like…

checking your CSME was not critically vulnerable and checking that you’re doing like a TPM replay, so you have your TPM event log and you replay it, it should have exactly the same hash as your TPM presents. And so we rolled this out to like millions of people and then had loads of bugs that came back and said, look, this says that the TPM is broken. It turns out all the Infineon TPMs were broken, like 10 % of the cases.

and we loads of real Red Hat customers come back and say, look, your system’s saying that we’ve spent like $10,000 on this machine, and it’s saying that the CSME is critically vulnerable. And it was, right? So it was a super useful thing. And by making it visible, so as soon as you put it into like, so we made the output of FWMGR security, you can just do dash dash JSON and then pipe that into whatever management framework you’re using. So if you’re using something like,

Red Hat Insights or something like that, you can like pipe hundreds of thousands of machines data into this thing and have a dashboard that shows how many of them have a pending firmware update, how many of them have critically insecure CSMA, and you can, as soon as you make this like apparent to people, then it becomes important to them. And I think sometimes the metadata from the LVFS that tells you what updates are available is sometimes more valuable than the mechanism to deploy them because you know what security posture your company’s in.

Josh Bressers (31:28) And I didn’t even think of the angle of a broken TPM or something like that because that that happens all the time Where you have a device that for whatever reason has a broken chip or the key didn’t get written correctly or whatever

Richard Hughes (31:45) it’s way worse than that. So like MSI motherboards, if you have a gaming motherboard from MSI that’s more than about five years old, it will be stuck in what’s called manufacturing mode because they never actually clicked the button which makes it go into production mode. And the problem with manufacturing mode is it means you can flash your own platform key, which essentially means that your root of trust is gone. Anyone can even just flash their platform key on your system, which means that you have no longer.

all of your system and then they can turn off manufacturing mode. So it’s worse than having no security, it’s like letting someone else lock you out of your own system. but until we started doing this there was no interest in MSI to fix this and we rolled this out, this detection for even manufacturing modes and all of a sudden there were a lot of people going back to MSI to say, well hang on my system is getting this HSI score of zero because I’m in manufacturing mode, what the hell’s manufacturing mode and why is it turned on?

And just recently MSI have reached out and said actually we’ve changed all of our systems. One of our like going out the factory checks is now is the system in manufacturing mode, which is brilliant. It just it’s it’s a continual battle of like carrot and stick because you don’t want to keep bashing the manufacturers with you’ve done this wrong, you’ve done this wrong, you’ve done this wrong, you’ve done this wrong. You have to be able to give them like a carrot. And so like with the SBOM stuff that we’ve done, if they provide an SBOM with their firmware, they get a little green checkmark.

next to their firmware on the LVFS, which is the carrot of, yes, this passes some compliance checks that you can sell to the US government.

Josh Bressers (33:22) Wow, that’s really, man, I bet you have stories you could tell.

Richard Hughes (33:26) Maybe not here, but yeah. Yeah, I think so. Oh wow. So I got like one that’s probably broadcastable is I got a call up from GCHQ, which is the UK Secret Services essentially, basically to talk about how this stuff works. And I was like, yeah, that’s like, this is not a scenario that I imagined five years ago.

Josh Bressers (33:29) I gotta buy you a couple beers next time we’re in the same place.

man, that’s awesome. Wow.

Richard Hughes (33:56) What happened? We’ll have to wait for later.

Josh Bressers (34:01) Okay, I could bug you a ton and I would love to. Maybe another day. I don’t want to drag this on too much. So Richard, land this plane for us. Tell us anyone interested in learning more or maybe looking to get involved or kind of if I’m a user, I’m a hardware vendor, whatever, tell us kind of next steps after this discussion.

Richard Hughes (34:21) Okay, that’s a great segue. So if you’re a user, we always need people to help us test things. Even if for translations, some of you translate fwupd into, I think it’s like 35 different languages. So that’s a hugely valuable thing you can do for other people. If you understand Rust C Python code, that’d be great. It helps with some reviews and helps with some low hanging fruit in the fwupd GitHub issue tracker. If you’re a vendor,

and you want to get involved in this. The easiest thing to do is email me because like I said before, it’s highly likely that someone in your organisation has already gone through all the legal approvals and everything to make this really easy. So if you’re someone sitting there in ASUS or ASA or any of the other OEMs thinking we need to do this for our next model, talk to me and I can refer you to them and then we can set up ⁓ groups on the LVFS. And if you’re a consumer, like if you’re looking to buy stuff and you want like,

Firmware is such an important part of your whole security stack. It’s literally at the bottom of the pyramid. With that updated, up-to-date firmware, you have no security. So buy hardware that’s supported by LVFS and fwupd Literally go to the website and say, okay, well, I want to buy a ThinkPad. Which ThinkPad should I buy? Or I want to buy a Dell workstation, which one should I buy? And buy the ones that are supported, because the only way…

The only way you can get manufacturers to do anything is do one of two things. One is stop them selling to someone they’ve sold to before. So if you say, look, you can’t sell to the DoD until you give us an SBOM. And the other is allow them to sell to a market they’ve never sold to before. Allow them to sell to people that wouldn’t normally buy their products. there’s lots of like a USP could be works in RHEL that’s a really good, like if you’re an industrial PC maker, that’s a really nice bullet point on the slide somewhere.

which costs you almost no money. So that kind of, I think that’s most people, ⁓ but if this stuff excites you as much as it does me, like, send me an email, I’m happy to chat.

Josh Bressers (36:29) Fantastic. what a, this has been awesome. Richard, thank you so much. I love it. This is so good.

Richard Hughes (36:36) No worries, thank you for having me.