Josh chats with Lance and Anton from Caution about verifiable computing. We start out discussing all the challenges in bootstrapping software that almost nobody thinks about anymore. We then make our way into discussing the art of the possible. There are some great things happening in this space today and plenty of solutions we can use right now. There’s a ton to learn and it’s never been easier to trust your software.

This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.

Episode Transcript

Josh (00:00) Today, open source security is talking to Lance and Anton, the founders of Caution, the secure hosting platform. Lance and Anton are here to talk about verifiable computing today, which I’m really excited about because this is one of those spaces that my knowledge is probably a decade old.

And

so when they reached out to me, I’m like, this is a perfect opportunity to take the lazy way out of not having to do a lot of research and talking to experts. So Lance and Anton, welcome to the show. So thank you so much for being here.

Lance Vick (00:30) Glad to be here.

Anton Livaja (00:30) Yeah,

it’s a pleasure. Absolutely.

Josh (00:33) All right, so let’s do some intros. Lance, I’m gonna start with you ‘cause you’re on the top of my screen at the moment. But kinda tell us who you are, tell us what you’re doing, tell us while you’re here, and then we’ll move on to Anton.

Lance Vick (00:43) yeah. uh c couple decades of security research and sysadmin um that have made me uh very distrustful and paranoid of everything, and that has resulted in building a whole lot of things, uh and largely with with this guy Anton.

Anton Livaja (01:04) Yeah. Uh similar background, security engineer, not for as long as Lance, but I met him uh about seven or eight years ago. He made me a lot more paranoid about everything. And we’ve been trying to figure out ways to improve the baseline of security uh for the internet at large. Kind of uh one of our driving forces is like how do we improve the privacy, security, and freedom of as many people as we can. And one way that we found to do this is to um create open source tooling.

That by default uh allows us to to build more secure systems. So better building blocks for for the internet. And yeah, we had uh Distrust, our security consulting firm, and now we’re working on Caution, the hosting platform.

Josh (01:44) Nice, nice. Okay. So when you say kind of trust in computing, there’s I I would say my typical guest, when we talk about trust, it’s very like software focused about can you trust the open source you’re running or can you trust the vulnerabilities aren’t in your code or whatever. But you guys operate on a different level. So let’s like set us up. What do we need to know for the beginning of this conversation for this whole verifiable computing universe you’re going to lead us into?

Lance Vick (02:13) Yeah, there’s a lot. Um I think uh there’s been a lot of building blocks that have been happening towards verifiable compute over um, you know, across the whole open source community. Um there’s been some exciting work, like the Stage Zero project, uh which uh wanted to really attack the the problem of like Ken Thompson’s trusting trust, which is just again, like how do you trust that there’s not some weird backdoor and a compiler that backdoors everything else? And

Uh, this all kind of stems from this idea of just because somebody gave you some binaries and said those binaries came from the source code doesn’t mean that’s true. And so being able to prove that’s true, especially in the world of all these supply chain attacks that are in the news all the time, um, we have to be able to compile software deterministically and get the same output every time, which you would expect is the default if somebody wasn’t close to this. But unfortunately,

You bu build the same Firefox uh binary and you’re gonna get a different result pretty much every time on most Linux distributions, which is kind of scary because then it means there’s a single person somewhere that built the golden binary that the whole world is using. And that is way too much trust in a single person. So that’s sort of like the problem space that we kind of started in of just like, how do we even trust that this source code is actually what even got shipped? Um and so some of the ways that we

you know across the community the bootstrappable builds Project did a lot of work here in trying to act on a talk uh I by uh by Carl Dong at breaking Bitcoin, which also like a way to break Bitcoin is also just like if the software stack that built it is broken, then everything’s broken. So we don’t even need talk about Bitcoin. We need to talk about the software stack. Anyway, these pieces actually all started to come together over the last like three years.

to where we actually could start with like 186 bytes of machine code uh and build ourselves all the way up to like modern compilers and modern distro in a deterministic way. there weren’t any distros that were doing this 100%, although Geeks was the first to get uh a major portion of their OS built this way. Uh and so we started Linux distro called StageX, where uh we were able to build everything.

Um from zero, the entire distro, and get a deterministic output. And now that we can say you build the entire distro from zero, I build it from zero, and suddenly we get the exact same result every single binary identical every time. So we can be confident that this source code that we’re looking at is what is in the output binaries. And that was kind of a foundation that we had to build before this whole kind of verifiable compute story could kind of even start.

Josh (04:54) Holy cow, you took

Lance Vick (04:56) Ha ha ha.

Josh (04:57) us on the roller coaster, didn’t you?

Lance Vick (05:00) Yeah, that was a lot.

Anton Livaja (05:00) Yeah, I’m supposed to be number one.

Lance Vick (05:02) And that’s the that’s the short version, I promise.

Josh (05:04) Yeah.

Anton Livaja (05:04) He’s

good at that. Yeah. Yeah. There’s there are a lot of pieces

Josh (05:05) It it it’s complicated.

Anton Livaja (05:08) here that have to come together. It is very complicated, but truly what he described is the foundation. It’s funny. Like you build a program from source, you haven’t changed the source, but each time the resulting binary isn’t bit for bit identical. And this is a problem. If you want to be able to tie source code to the binary, it needs to be identical. Otherwise, how do you know that this came from this? Um yeah.

Lance Vick (05:30) Yeah, imagine like everybody put, you know, numbers to a calculator and got a slightly different result every time. That’s kind of been the state of software distribution for ah pretty much its entire history.

Josh (05:42) Yeah, I mean that it’s funny you say that because I I think a lot of security people have been very aware of this for a very long time and even just non-security people, maybe people who just exist in the we’ll say the build universe. But y you’re right. And and I wanna start out, because you mentioned Ken Thompson’s trusting trust, and then you also mentioned like the the golden build, right, so to speak.

Where where you have like one person who’s built the thing. And and I wanna start on that because I don’t think we think about this enough. And I think Ken Thompson’s trusting trust is a good example. So I’ll let one of you two tell the story because I’m sure you’ll do a better job than I will of just like, what was trusting trust? And like, was it real?

Anton Livaja (06:29) Yeah. Yeah, it’s actually very real. And just recently there was some security research done in the Nix um ecosystem that proved that it’s legitimate and and like a practical attack that can be done, where you can have a backdoor hidden in one binary that propagates to a bunch of packages within a distribution. But the short of it is even if you have source code that’s secure and reviewed and you trust it, if the compiler itself is malicious during the built

Build time, it can inject malicious behavior. And so then the question is, how do I make sure the compiler itself can actually be trusted so that this doesn’t happen? And that’s where the bootstrapping and reproducibility efforts came in, where you know, naively you might say, well, I’ll just take the source code for the compiler, review it, and build a compiler from that, and then I’m done. But wait a minute, I need a compiler to do that. So that’s a chicken and egg problem, right? So instead, what we do is bootstrapping, which is essentially how do we make

the the process of building a compiler transparent enough that we’d know that there’s no malicious behavior that was injected at some point. And that’s where Lance mentioned this a hundred and he said 186, it’s actually 181 bytes uh of a seed, ah that

Lance Vick (07:40) Appreciate that.

Anton Livaja (07:41) that that you start with and then you iteratively build up essentially the way to think about it is like you start with a super rudimentary uh software that’s kind of like a compiler, and then you build a slightly more full-featured compiler with some

helper tools and you keep doing that and it’s hundreds of steps until eventually you get to a modern GCC uh tool chain and then you have full provenance, you can see yeah these are all the steps, nothing fishy has happened here. You do so deterministically and now you can build the universe. You have a compiler that’s trusted that if you have some trusted source code, you know that reliably it won’t add some unexpected behavior into the resulting binary. And so now we’ve solved trusting trust.

Josh (08:26) So, okay. This part I don’t understand at all, and I’m very curious about because, like, historically speaking, for anyone who’s never done this before, and you guys correct me if I get any of this wrong, ‘cause it’s been it’s been a minute. But when you would generally build a compiler, like this is GCC, I built GCC in the past for reasons, and you’d have to first build the bootstrap compiler, and the bootstraps compiler is to build like another compiler, and then that compiler builds the real compiler, right?

And you’re talking about, I guess, functionally that process, except you’re starting from like nothing and then bootstrapping your way into a compiler. Am I and and so this is the part I’m curious. Like, how do you like how do you even start from nothing? I mean, like you’re welcome to get as technical as you want here because this is intriguing to me, and I’ll just ask dumb questions if I don’t know what you’re talking about.

Lance Vick (09:17) There. Uh yeah, so step one, this is what’s um this project called Stage Zero uh kind of came up with, which is you start with this tiny, tiny um uh bit of machine code, um, but is intentionally uh written in a very commented and structured way uh to be human authorable, human readable. So you’re literally just writing hex characters, machine code, and this is meant to be a human auditable tiny binary.

That if run will host itself.

Josh (09:49) Okay, hold on, hold

on. I want to interrupt for one second. You’re saying you’re not writing like assembly. You’re literally writing a binary with like hexadecimal characters inside of it.

Lance Vick (10:01) Yes. so it’s called hex zero and we only want to

Josh (10:03) That is crazy.

Lance Vick (10:04) do that in one place because it’s a pain. It is it is not something you want to maintain. and so so you start from this this this auditable uh seed and the idea there with that seed is uh that it should be simple enough of a spec that you could even implement it yourself probably more easily than in a different language than you might be able to write it uh in hex zero.

Um and there’s multiple alternative implementations. You can even approximate it with like sed and awk one-liner. Um and so uh but the cool part is the output of this is you get a binary that you can then feed its own you know commented annotated code, and then back out will come the same exact executable with the same hash. So it’s self-hosting and then that can run as a little bit more complicated version of itself until you get up to an actual basic assembler. Uh and then from a basic assembler, we can go through

a very long series of steps, uh incrementally ratcheting our way through almost a simulated computer history uh until we get up to you know scheme and then you know ultimately like a basic C compiler tiny CC. um So then we get like all of these early uh very uh painful steps to get to like tiny CC and from tiny CC is actually a very big unlock because although it’s a very very minimal C compiler

It’s sufficient enough to actually bootstrap Golang directly, um, which is pretty cool because Golang has a uh a really rich standard library and you can do lots of things with it. Um not a big surprise that it’s easy to bootstrap Go, given that one of the authors is Ken Thompson. Um and so he really wanted that to be bootstrappable. In fact, it’s the only language that actually has like bootstrap ability as like 100% built in as as a as a maintained goal. Um

Anton Livaja (11:48) And determinism, yeah, it it’s great for that.

Lance Vick (11:51) We actually, when we built when we bootstrap all the way up to Golang, our hashes of our resulting binaries of our compiler match the ones that Google produced with their very extensive Bazel tool chain. so that’s pretty cool. Um but other than Go, unfortunately we have to go through a much longer track to work our way up through um a non-C version, uh early non-C version of GCC, like GCC four.

And then step our way through the history of GCC releases, uh skipping a few, thankfully, uh, until you get up to like modern CCC. Uh and then from there, this is all X86 by the way. Um, from there we can finally build an X8664 cross-compiler stack and then pivot to building like a modern LLVM toolchain. And then the modern LLVM toolchain thankfully can build the rest of the universe.

Josh (12:40) That is truly mad scientist sort of stuff. Wow.

Anton Livaja (12:43) Yeah. That’s great.

And just to zoom out again for a second, the alternative is like you grab a compiler that someone else built that’s a completely opaque binary and are like, Great, I’ll just use that to compile everything else. But Ken Thompson says that’s a bad idea.

Lance Vick (12:58) Yeah, that’s the

Josh (12:58) Well in

Lance Vick (12:59) trust that I don’t think people think about so much. Like when you just grab like like a Docker container, it’s just like from Rust, and then you put in your Rust code and you build, what you may not realize is that’s just that’s actually a Rust compiler binary uh that was built by a single member of the Alpine team somewhere. Uh

Josh (13:17) Yes.

Lance Vick (13:18) and that person could do literally anything to your resulting binary that very often is used for very sensitive use cases.

Anton Livaja (13:24) Right. A

person or a single built server or whatever whatever. It’s a single single

Lance Vick (13:28) Single point of failure.

Anton Livaja (13:29) place where it’s where it’s built. Yeah.

Josh (13:32) And and I mean that’s how we’ve been using software for forever. Where I mean, I’m I’m running Linux on the computer I’m talking to you. I got it from Fedora. And and actually I mean, so one of the fun I used to work at Red Hat a long time ago. And one of the funny things was like when you built the next version of, you know, Red Hat Enterprise Linux or Fedora or whatever it was you were building, you had to like

progress through the packages in the same order. So like when those there were those rebuilds, like what CentOS and and there’s Alma Linux and things like that. And and that was one of their challenges, because we wouldn’t tell them, but they had to figure out like what was the order everything was built in that we actually get the same results, which sounds loo loony. But then you still that that obviously doesn’t at all solve the problem you two are talking about, which is if there’s a problem anywhere in your tool chain, you it’s over, right? You like there’s nothing you can do about that.

Lance Vick (14:25) Yeah, or even in a tool chain that built the tool chain that built the tool chain, because we’re talking about hereditary

Josh (14:28) Right. Yes. Yes.

Lance Vick (14:30) vulnerabilities uh as well.

Josh (14:33) Yes, yes, for sure. And and even I mean, okay, so here’s I guess my first question I have. And this is I’m I it’s going to sound a little maybe adversarial, but I’m not intending it to be, is like every

Lance Vick (14:43) We love those.

Josh (14:44) I shouldn’t say every, but most modern compilers are like so gigantic now, we can’t possibly like actually audit that code. Right. So if the if there’s a problem in the source code of the compiler, you guys are still gonna build it in, right?

Anton Livaja (14:58) Yeah, that’s correct. That’s correct. And and these are definitely like separate problems, but the way we think about it is like

Josh (15:03) Sure. Okay,

yeah.

Anton Livaja (15:05) If we made it easy to go and eliminate this whole class of attacks, why wouldn’t you? And historically it hasn’t been easy. And so one of the motivating factors for us has been like, let’s make this easy enough that we can just like forget about these types of attacks or at least like significantly mitigate them. So we can focus on other uh types of attacks. And that’s kind of a part of our philosophy is like surface area reduction and like trying to get as close as you can reasonably to eliminating certain types of attacks so that

We don’t have to really worry about those uh as much. And and a lot of these are to be fair, like, you know, black swan events, but some of the biggest hacks uh in history, like solar winds have happened because of

Closely related problems where there’s a lack of reprop

Josh (15:47) Yes.

Anton Livaja (15:48) reproducibility, where I can’t actually verify that this software originated from this source code. And in that case, it’s actually a code base that could and should have been properly reviewed. And you should have been able to draw a line from this source code to what was actually delivered in this case to Fortune 500 and governments and all sorts of uh v very uh high-risk organizations. Um

And so that that’s not the only example of this, but there have been some major cases, you know, Xcode Ghost, the Nix ex example that happened recently. That was more like a research kind of academic uh thing to just proof of concept this type of attack is viable. Um but the point being like why not make it so that these attacks don’t work.

Lance Vick (16:31) One

thing I would also

Josh (16:31) Yeah, yeah.

Lance Vick (16:32) add to that would be that like the xz attack comes to mind, which is like, hey, that’s that’s kind of what you’re talking about. Like the source code feels like it was compromised and distros were just gonna take it. Um but it’s actually uh there’s there’s an extra bit of nuance to that story that I think is often glossed over, which is that the source code that was actually reviewed in the repo was fine. Um

Josh (16:52) Yes.

Lance Vick (16:53) it was actually

the bundle that was uploaded by one of the maintainers that already had had some extra things that were added in that was actually compromised. And so one of the things that that we do is we always want to grab exactly the source code from version control that was actually reviewed and then faithfully build that. So that way so in the case of XZ, we would not have been impacted by in our StageX distro because we’re just grabbing um we’re we’re gonna go ahead and run an autoconf and all those those steps that are done for us, um, the the code generation steps on our own.

Um, to to reduce those risks. So that way there’s high visibility and high accountability. So if somebody wants to hide a vulnerability, they will have to do it in the public source code that everybody gets to see.

Josh (17:34) Right, right. And so okay, let me unpack that and make sure I I grasp what you’re saying. So in the case of X Z, what was happening is there was a blob of crap that was disguised as test data. And obviously when you have a compression slash decompression algorithm, like all of your test data is blogs of blobs of crap. Like it’s just how it works.

And then in one of the blobs of stuff was actually the the backdoor shell code. And that was being inserted into the binary at build time, but it was also it was the the build system was like pulling this down and putting it in, right? This wasn’t in the source code. And you’re saying, and and keep me i the here’s where I’m not sure if I’m completely grasping what you’re you’re saying, Lance, was you’re saying what what you guys are suggesting you do is like take the take the source code from source management control.

And basically like hermetically build it in a way that you can then obviously compare the output of what you’ve done in the hermetic environment versus the output of what the thing is we got from our RPM build system or Debian build system or whatever.

Lance Vick (18:41) Yeah, there there’s a half step in the middle there that I think is is the one that uh is really a bad habit in the industry as a whole, which is uh or just the kind of open source software development, is that they you tend to have here’s the source code repo. And then to make things convenient for people, um, we will go ahead and pre generate, like like literally do a bunch of computer generated code, like autoconf

Josh (19:02) Ah, yes.

Lance Vick (19:03) stuff, and then add all that generated code into a bundle and then say, here you go, it’s already generated, it’s much easier to build now.

the configure scripts and everything are all ready for you. the problem is is that in that all of that extra generated code, nobody reviews that. And that is where the extra stuff gets to hide. It is way better for Linux distros I believe Linux distros should all be doing all of that code generation themselves with their trusted tool chain um and stu and just grab directly from the reviewed source code and not anything else that wasn’t reviewed by humans.

Anton Livaja (19:34) Right. Yeah. And like it’s it’s fair to say, but like you still need to need to review source code and yes, but it’s a lot harder to review a binary. I mean you can do you you can go and inspect a binary, you can try to decompile it. There there are methods to do this, but it’s a lot easier and simpler to review source code and more reliable. It’s a lot easier to hide things inside of binaries or compiled code. So

Lance Vick (19:57) You also asked a great question,

Josh (19:58) Yes.

Lance Vick (19:59) which is uh about like what if LLVM itself is is compromised? And I actually you know if there was there’s that’s a lot of code. Um LLVM is pretty cool though, in that unlike other compiler tool chains, it actually compiles down to uh LLVM intermediate language. Um and there is human readable versions

Josh (20:16) Yes.

Lance Vick (20:17) of this. Um and uh and something I’ve been experimenting with a fair bit, and we we hope to experiment with more in the future, is

there are actually tool chains to add mathematical proofs to your LVM IR. Uh in which case you could go ahead and have all the benefits of a modern language, run it through LLVM, get your LLVM R uh IR output, and then heavily analyze that LVM IR uh output to make sure that it functionally executes the system calls it’s supposed to run, run setcomp on it, run other tools on it, and make sure that it actually um has the output on a running system state that you want.

And then you even get into other cool tool chains like Risk Zero that actually lets you prove the impact that particular binary has on the memory of a given system. So I think there’s actually a couple other tactics that there’s very early research on that look really promising to take this even much further.

Josh (21:09) That’s really cool. And that’s um what I I know for for anyone in in let me let me I feel like I’m re-explaining a lot to you, Lance, because you’re definitely a bit over my head on some of this, but this is fascinating as well. Is so what you’re talking about is like there’s the concept of like mathematically proving software, right? Where you’re like you prove beyond a reasonable doubt that it does exactly what you think it does. And like a bunch of TLS libraries have done this, for example, where or I think Amazon has a bunch.

Where they’ve like literally done like whatever the magic math is. I don’t t understand it at all. But they’ve like proven I forget the fancy term for that, but right, it’s like formal

Anton Livaja (21:44) Sure. Formal verification, yeah.

Josh (21:46) verification. Thank you, Anton. Yeah, because I talked to uh, I think it was a guys from pycryptography a couple of months ago. And this was like one of the things they were talking about is having the ability to like swap around their back ends so you can use these formally verified backends versus

‘Cause like open SSL is often not formally verified, but some of the other ones are. So that is fascinating. So you’re saying if I’m if I’m understanding you correctly, you’re saying take LLVM’s output and use that as your formal verification process versus trying to formally verify C. Is that what I’m understanding?

Lance Vick (22:19) Yeah, be be able to

prove yeah, yeah. Be able to prove not just with tests that like, we put in two plus two and we got four in this add function, b to be able to prove that if I put in any two numbers, I should get the sum of those two numbers and there isn’t some like extra special if statement that does something special on certain inputs.

Josh (22:37) Right, right. Wow. That’s cool. Nice. And and okay. go on Anton.

Anton Livaja (22:41) Yeah, so I

I was just gonna say all of this is actually just the very beginning of how we get to verifiable compute. We just spent laying the foundation. Uh

Lance Vick (22:50) Ha ha ha.

Josh (22:51) Yeah.

Anton Livaja (22:52) to get to the fun part

Josh (22:55) No,

no, I l let’s do that, right? ‘Cause I we could rat hole in this space for hours, I

Anton Livaja (22:59) Oh, easily.

Josh (23:00) think, and I don’t want to. So yeah, I mean i like take that Anton and and take us where we need to go with.

Anton Livaja (23:06) Yeah, right. So this was this was very important context for where we go next. And it’s a it’s uh yeah, necessary prerequisite for doing verifiable compute. And what verifiable compute is is actually it’s probably best to talk about confidential compute.

Which is essentially a feature that exists in a lot of the modern chipsets. You can find it in the latest AMD chipsets. Intel, Nvidia, Blackwell, and Hopper have this capability. And the easiest way to explain it is it’s hardware back cryptography. There’s a key baked into the hardware, and this key can be used to do some cool things. You can encrypt memory.

Very useful for isolation and protecting your workload from its surroundings. And you can also do what’s called cryptographic remote attestation, which essentially allows you to prove, cryptographically prove what’s running in a workload. ah And the way this ties into what we just talked about is with cryptographic remote attestation.

If your code is reproducible and deterministic, you can in real time prove exactly what is running on a server, which is something we can’t do today. Usually we deploy some software, we take a bunch of steps, we see the steps, but once it goes into a server, we just hope that it’s actually the code we intended to deploy. And there’s no practical way to reliably actually have proof of what’s in here right now. And that’s what actually uh this technology uh provides to us. When we can actually fully prove what’s in there, we call that.

verifiable compute because there’s a lot of use cases where someone will take confidential compute, throw some software into this memory encrypted workload and they call it a day That’s not verifiable compute. It’s good. It gives you some protections. But verifiable compute goes beyond that.

And it says it’s not enough just to encrypt the workload. We also need to be able to prove exactly what code is running in here, because then we can be certain about its behavior, right? If I can read the source code here and say, hey, this system doesn’t exfiltrate my data, it doesn’t modify the responses, then I can be confident it’ll actually behave in a manner that it’s supposed to.

Josh (25:08) Okay,

here’s my question about this, because this is where I So to give you guys some context, is my knowledge of all of this basically cuts off about the time AMD SEV first came out, which was like AMD’s encrypted memory. And I I I was at Red Hat at that time, like Secure Boot was just catching on. We added Secure Boot to Linux, which that I could tell stories for days about that whole process. But that’s kind of where

Where my knowledge kicks out. So kind of Anton, what you’re talking about where we’re doing verification, does that catch things like, for example, like if I have a a a security exploit, the exploit modifies memory? Does do we catch things like that? Or is it just strictly like the thing I’ve read off disk and loaded into RAM, I can verify is the thing that, you know, Anton gave me to run?

Anton Livaja (25:55) Yeah, so it’s it’s about yeah, it’s not about source code level uh issue prevention. It’s about yeah, proving what code you actually deployed into workload. And if there are exploits present in

Josh (26:05) Okay. Okay.

Anton Livaja (26:06) the code, that that’s not what this solves. Although there are ways to minimize what you can actually do uh through designing the actual operating system that runs in the workload, the you can you can reduce uh the effectiveness of of a wider rate of attacks. But on a simple, high level of what

Cryptographic remote at the station does this, it doesn’t it doesn’t solve source code level issues. Uh it just allows you to prove what you actually deployed. Yeah.

Josh (26:27) Sure, sure. Well, no, no, okay,

hold on. That this is complicated. I don’t I’m not using the right words, so I apologize. But like attackers modify memory, right? Where you have it, you have your binary, your binary gets loaded into memory, your binary starts to run, and an attacker comes along and somehow modifies that memory. Does the remote attestation that we have today have the ability to know that that memory was modified for the running process, or is that outside of the scope of this?

Lance Vick (26:56) Almost. Uh I think that’s

Josh (26:58) Okay, okay.

Lance Vick (26:58) uh uh so what we can prove reasonably well is that the TPM that’s on a motherboard and the CPU um both uh one loaded X off you know binary you know image off the disk and another one loaded in a RAM and they both agree. so at least we

Josh (27:14) Okay.

Lance Vick (27:14) know our starting point very confidently. Um however on Intel systems they have this TXT extension technology and there is

A formally verified hypervisor called Nova that we’re very heavily investing in recently, uh which um and s it has this interesting way of as VMs change can actually give you a live snapshot of the RAM now as opposed to just when it’s booted,

Josh (27:38) nice.

Lance Vick (27:39) which is really exciting. Um don’t currently have a path to do that on AMD yet, but there is a a compelling path on Intel that we will absolutely ship when we have a good implementation.

Anton Livaja (27:48) Yeah. So so most solutions to answer your question can’t because it’s you you verify the code at boot. but as Lance just uh explained, there is a path to actually doing ongoing kind of in real time uh attestation or or proofs. So

Lance Vick (28:02) That’s the next level.

Anton Livaja (28:03) that yeah, so so we’re working towards that. Um so yeah, not not quite there yet. So that’s a really great question and something we definitely care about a lot. but you can get pretty far.

uh by building systems or software that, you know, i i is hard to exploit in this manner.

Lance Vick (28:22) Yeah. And if you know what

Josh (28:23) Yeah, yeah.

Lance Vick (28:23) came off the disk is like a the only thing that came off disk, the image was just a very minimal Linux kernel or even like a really hardened uh like Astarinas, you know, tiny Rust kernel, and then you’re one binary and it’s all unprivileged and that’s all there is, um, then you can at least get pretty high confidence of the boundaries that that that piece of software is running in and what it can do, which which goes a long way uh for you having confidence about what’s it ran.

Josh (28:48) For sure, for sure. Okay, okay. I’m gonna I’m gonna lob you guys uh an easy one here. So people are gonna say, like, why would I need this? Right? Like, what’s the point of of being like this paranoid? Right? No one’s going to attack me. Like I got my software from Microsoft. It came on a CD, it’s fine.

Lance Vick (29:12) I

I love this. Uh so a c a couple extremes. Uh one is if you’re like a crypto custodian, um maybe named uh something like Binance, and uh and you have

Josh (29:20) my goodness. For okay, for anyone who doesn’t know, like the whole Binance

debacle of them losing like six trillion dollars or whatever it is is like it it just started yesterday at the time we’re recording this.

Anton Livaja (29:31) Yeah, yeah,

this is fresh off the press.

Lance Vick (29:33) Yeah.

But but let’s say you’re you’re a crypto custodian, you have you know billions of dollars of value uh sitting in like a single secret. Um you don’t want to have engineers have access to that secret. You want to put it in a place to where multiple people can verify the state of the computer, but nobody actually gets to see the memory of that computer uh and actually gets to see the key. So a

Group of people can coordinate and say, all right, I deployed the server, everybody else verifies that it’s a server we all expect, it’s running the code that we all expect. Great, we can now all work together as a quorum and submit shares of a key. So no single person gets to hold the really dangerous key. So any cases where you don’t want to trust a single person, this is really, really valuable. But on the other extreme, if you just want to say, hey, I want to prove to my customers maybe I’m a VPN provider and I want to prove to them that I’m not logging, not a privacy policy, but like

You can literally look at the image that’s terminating your connection and see that no ability to log even exists in the image. ah And so if if you just want to prove to end users the state of a system, this is also a high value, or even just to prove to yourself that my server is still in the state that I deployed it and has not been tampered with.

Anton Livaja (30:41) Yeah. I I also want to go beyond that. It’s it’s you know, it’s great for high risk use cases and traditionally that’s where confidential compute has been utilized the most. but really what we’re trying to do is make it so easy to use that there’s no reason not to. Be and and so th this like making it as easy to deploy into verifiable compute as it is to like deploy to like fly over cell, while eliminating, you know, or like

significantly mitigating entire classes of attacks, you know. Like say we can say conservatively, like 30% of attacks basically stop working. That’s a pretty cool proposition, you know? And and traditionally like the the barrier to this technology has been the complexity. It’s a fairly new technology. It’s tricky to work with. Most people see it as this like really arcane, insane thing. I don’t need this level of security. But if I tell you, hey, you can keep your development like DX like workloads you’re used to, but you get it.

a ton of security benefits, then that kind of changes the the you know the decision tree here. It’s like, why wouldn’t I do that? And that’s basically what we’re trying to do. Um you can use this to to sandbox agents more effectively. You can use this to vault data that you don’t want agents to get to. You can use it to do things like provably private AI inference. Amazing. I can use you know uh LLMs and not you know pipe all of my uh personal data to to a third party

in approvable way, like awesome. And so that that’s like um that’s a part of the I guess education and the narrative shift that we’re trying to influence here. And it’s it’s essential to the adoption of this technology that it’s actually extremely easy to use. And we’re we’re obsessed about like exactly that part.

Lance Vick (32:26) We w this is what comes after TLS. We want this to be the standard for the internet. This is just something that like no most nobody did it and now like why would you not? Like if and if you’re not, it should be suspicious.

Josh (32:36) Yes, yes. You know what this reminds me of actually is like how all of the Apple devices got their secure enclaves to hold like passwords and secrets and things like that. I remember when it came out, a lot of people were like, why would I need that? And now, you know, ten years later, it’s like no one would dream of buying a device without a secure enclave on it.

Anton Livaja (32:52) Yeah. It’s a great

Lance Vick (32:54) It’s

Anton Livaja (32:55) example.

Lance Vick (32:55) standard.

Anton Livaja (32:55) It’s kind of once people understand how this works, they’re like, holy crap, yeah, like why aren’t we using this everywhere? And the answer is because it historically it’s been complicated to work with. That it’s it’s what’s

Josh (33:04) Yes, yes.

Anton Livaja (33:05) a it’s a legitimate uh concern and reason. Yeah.

Lance Vick (33:06) You had to rewrite your whole application to use it.

Anton Livaja (33:09) And so it’s like, can we take

Josh (33:10) Yes.

Anton Livaja (33:11) a containerized app, a dockerized app, and just like deploy it in this manner without people having to jump through hoops? And like the answer is yes, you can totally do it.

Lance Vick (33:17) Unmodified software off

the shelf. The s the services that we already use, Nginx, etcetera, you could just run them in an enclave today.

Josh (33:25) Yeah, yeah. And and there’s definitely especially right now, like anyone who isn’t paranoid every time you pull a Docker container or install a a Python package or an NPM pack or anything. I mean, I’m just like I literally have in the last couple of months I’ve set up a a like an infrastructure in its own private network. I call it the plague ship in my basement ‘cause I’m like

I’m afraid to install any of this stuff. Like I just I’m

Anton Livaja (33:48) Yeah. uh

Josh (33:51) like, I’m running Docker containers in here. Like that I I don’t want to do it inside the real network, you know? Like it’s terrifying. And like you guys are talking about a world where it’s not that this stuff all completely goes away, but it definitely

Anton Livaja (34:02) No.

Josh (34:03) it it takes away a a branch of the tree of vulnerabilities, right?

Anton Livaja (34:09) Absolutely.

And like that’s all you can do. Like nothing is a silver bullet, but it’s like, wouldn’t it be great if I don’t have to worry about these classes of attacks anymore? And then we can focus on things like source code level, which is where most people are focusing

Josh (34:20) Yes.

Anton Livaja (34:20) right now. Not as many are focing focusing on like secure by default infrastructure. And that’s that’s why we’re spending energy on this because that’s a like our contribution where we’re like we can move the needle on this. And uh yeah, it feels good ‘cause ‘cause it’s it’s really coming together and it’s working nicely.

Lance Vick (34:36) We would love if the worst problem left was just we need more eyeballs on source code. that that would actually be a great place to be.

Josh (34:44) Yes, yes, I know, right? And i it’s funny you say that because I think a decade ago, if you’d come to me and said, We think we can solve the verifiable computing problem before we can solve like the source code, the bugs and source code problem, I’d be like, No F in way. Like

Lance Vick (35:03) Yeah.

Anton Livaja (35:03) Right. Yeah. It’s it’s insanely cool and complicated, but but really it’s becoming very not complicated. And even the like the earlier conversation we had about like bootstrapping and reproducibility.

What came out of it is uh a drop-in replacement for Debian or Alpine, StageX the Linux distro, that just has all these supply chain security benefits. And and the end user doesn’t really have to think about it, but they always have the option to

Lance Vick (35:27) It’s container native.

Anton Livaja (35:28) go and exactly they they have the option to go and reproduce the software and verify when and if they need to. And that’s why a lot of companies that are like very high risk gravitate to StageX But same thing about verifiable compute. It needs to be most people shouldn’t have to think about it at all. They should just get the security benefits.

At the end of the day.

Josh (35:47) Well and look, those are the only security features that ever get adoption, right? The security features that like work transparency transparently are like the they’re the only ones we ever see that work, right? Because expecting

Anton Livaja (35:58) Right.

Josh (35:59) any users to have any friction, like you’re done. It it’s I mean, how many of us turn off SE Linux on our Linux systems, right? Like all of us do. Why?

Lance Vick (36:06) Right.

Josh (36:07) ‘Cause it’s a pain in the butt. Okay. So

Lance Vick (36:10) This is why we chose Docker

Anton Livaja (36:10) Right.

Lance Vick (36:11) as the primitive. So that way as you mentioned earlier using Docker or rather using OCI’s native containers, everything’s just OCI native all the way, you’re using Alpine or whatever today, just change one line at the top of your file and you get all these supply chain benefits.

Josh (36:22) That’s really cool. Okay, so let’s let’s let’s land this plane. What’s next? What should what do we want anyone listening who’s interested to do and and kind of like like fill us with some hope and optimism on the way out?

Lance Vick (36:37) Hope

and optimism is Anton’s department.

Josh (36:39) Ha ha ha.

Anton Livaja (36:41) Well, look, I mean we’ve like as mentioned before, all of our tooling is fully open source, not just like open uh core. Um and so if any Yeah.

Josh (36:50) And I’ll put a link in the show notes for anyone

interested.

Anton Livaja (36:54) Yeah, we built a lot of open source tooling, starting with StageX but like caution itself, like the platform. And we built a lot of tooling and and kind of modules that are part of that to make all of this work. So if anyone wants to help out and contribute or just get involved, like that that’s great. Uh and I think the positive note is like we’ve spent a a very long time working on this problem. Like it took us

Two or it’s close to three years to get StageX up and running. And then working with confidential compute for the last like decade to really figure out how all this should work. And now it’s finally coming together at a very good time when AI is blowing up the internet. And so we’re we’re able to deliver at least something, one building block that’s a part of the story that can help us figure this out. It’s not a complete solution. We still have a lot of work to do. Uh, but it feels good to be able to like actually have something that that is effective and usable.

Uh and there’s still a lot of really cool, you know, research and work that we need to do here and related technologies like post quantum is a whole wrench that we could spend another, you know, several hours talking about. Um, but but it’s cool and it’s cool to see the community coming together. You know, you have people going out and sending swarms to go and patch vulnerabilities uh and you know, uh wherever they can find them. And and so there’s the good guys and gals are doing a great job, and it’s really exciting to see all this kind of transpire.

Josh (38:15) Awesome. All right, Lance, I’ll give you the last word here.

Lance Vick (38:19) Uh yeah, um I don’t know. I think anything that you’re running in production, uh, you should be able to prove to your users what you’re running. So that way the privacy policies aren’t something somebody has to read. They can just prove what software is running. Let’s make that the standard.

Anton Livaja (38:31) Or or to yourself.

Prove it to yourself. You know, maybe make sure that you know what you’re running, not just your users.

Josh (38:38) Awesome. Awesome. All right. Well Lance and Anton, I’ve learned a ton. Like I’m super glad you guys came. This was amazing. It was fun. This is great.

Anton Livaja (38:46) Yeah, thanks for having us. Yeah, this was great.

Josh (38:48) And you’ll have to come back at some point when you tell us about whatever new and crazy thing is going on. So this has been awesome. Thank you so much.

Anton Livaja (38:56) Absolutely.

Lance Vick (38:56) Thank you.

Anton Livaja (38:56) It’s been a pleasure. Thanks for having us.