Josh talk to Dirkjan and Joe about Rustls (pronounced rustles), a Rust-based TLS library. Dirkjan and Joe are developers on Rustls. We talk about the history that got us to this point. The many many challenges in writing a TLS library (Rust or not). We also chat about some of what’s to come. Rustls has an OpenSSL compatibility layer which makes is a really interesting project.

This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.

Episode Transcript

Josh Bressers (00:00) Today, open source securities joined by Dirkjan professional Rust maintainer and Joe Birr-Pixton the author and maintainer of rustls I’ve asked you two to come talk about rustls which is a Rust TLS implementation. And I’m extremely excited to talk about this because I love this project, but Dirkjan, let’s start with you. Give us a little intro, kind of who you are, what’s going on. And then we’ll move on to Joe and then we’ll talk about rustls.

Dirkjan (00:25) Yes, I’m Dirk Jan. I have been writing a lot of Rust code over the past, I don’t know, seven, eight years. And a few years ago, I became a maintainer for rustls and I maintain a lot of Rust projects actually. And these days I’m full-time Rust maintainer working on a bunch of projects, including rustls.

Josh Bressers (00:48) Nice.

And Joe?

Joe (@ctz) (00:52) Yeah, I’m Joe. I started the rustls project in 2016. I’ve worked at a bunch of tech companies since then, but a couple of years ago I moved to working on rustls full-time, funded by a bunch of grants and generous sponsors.

Josh Bressers (01:16) Holy cow, 10 years pretty soon. That’s, wow.

Joe (@ctz) (01:21) Yeah,

the first commit will be 10 years next May.

Josh Bressers (01:25) Wow. I didn’t realize it was that old. That’s real. I mean, congratulations. That’s wow. A decade. That’s impressive. OK, so let’s just start out. I’ll Joe, if you’re the creator, I’m going to give you the floor on this one. Like, what is rustls? Like, just give us the the elevator pitch of what we’re even talking about today.

Joe (@ctz) (01:32) Yeah.

you a of layered answer, so the top level of detail it’s like a TLS library written in Rust, so it’s the memory safe subset of Rust, so there’s no like unsafe in the TLS part of the project. ⁓ And then if you go the next sort of layer down detail,

There some other goals rather than just writing a thing in Rust. ⁓ And those are like trying to be quite ⁓ careful about which parts of standard TLS we implement. And we sort of offer to people and almost sort of promote. ⁓ And having good defaults as a sort of follow on from that.

people who aren’t necessarily experts in TLS can pick it up and do something good and secure with it. And that’s kind of learning from some other TLS libraries which are more toolbox sort of toolkit based where they just give you all the bits and you kind of have to work out which of those bits are good and which are the ones you don’t want to use because they are, you know,

was standardized in 1998 and everyone thinks they’re a terrible idea.

Josh Bressers (03:09) Oh my goodness, it’s so true. mean, that’s very OpenSSL, right? Where like, here’s a bag of parts, good luck nerd, and that’s it. And it’s hard to use and it breaks a lot. I mean, okay, so you said you started rustls in 2016 and Heartbleed was what, like 2014-ish, somewhere right around there? So was Heartbleed like the inception of this project?

Joe (@ctz) (03:29) Yeah, 2014.

well, do you want the full long answer about the whole inception of this thing? Because it is related to Heartbleed. So yeah, there was Heartbleed and at that time I was working for a company which had a big installed base of… ⁓ It was a mobile ⁓ app, SDK, which embedded OpenSSL So we were caught quite badly, like a lot of people were, by that.

Josh Bressers (03:39) I would love it.

Joe (@ctz) (04:02) ⁓ But actually that’s not even the reason. ⁓ But after heart bleed…

people started taking OpenSSL maintenance a lot more seriously and there was this thing called the IBB, the Internet Bug Bounty that was I think funded initially by the Linux Foundation ⁓ and that got me interested in my spare time ⁓ like looking for bugs in OpenSSL and I guess this was sort of April 2015 so

I basically just started fuzzing bits of OpenSSL ⁓ and I was specifically fuzzing parts that I thought were kind of juicy in the sense that they probably hadn’t been fuzz before and they were like a target-rich ⁓ security surface. Like the specific example was…

explicit elliptic curve encodings in certificates And these are interesting because basically no one uses them. But at the time they were supported by OpenSSL with that in like default configuration. Anyway, so I started fuzzing this and I found a bug. this is not like a world ending bug, not even a memory safety problem in fact. It was just a

sort of reachable infinite loop. So you’d give OpenSSL certificate in the context of like a TLS handshake and it would just sit there forever burning up all your CPU on one core.

So I went back and found out where this bug came from and this is where like my

this thought came in my head is like this has got to be better than this. So it’s what happened in 2002. There were two parallel branches and on one branch, this is of OpenSSL on one branch someone found and fixed this bug.

and although the branch, someone completely rewrote the code that the bug was in ⁓ and improved the performance of it. Now the second thing happened to block comment at so if zero kind of pre-processed comment, the thing with the fix in, so these got merged together, so you ended up having the fix in commented out code, no tests.

that would show the existence or otherwise of the bug. And then a very fast version which actually still had the bug. That’s a bit of a shame and that situation continued until I found it for a second time in 2015. So this is quite a long story, you might want to chop it down, but the long and short of it is I reported this bug, I got sort of…

Josh Bressers (06:49) No, this is great.

Joe (@ctz) (06:57) of European holiday money from IBB for it, which I was very grateful for at the time. And I submitted a report and a patch and a test suite edition to OpenSSL Because in my opinion the original problem was just lack of tests.

It’s not that someone made the performance better. It’s not that someone commented out some code. It’s not that someone fixed the bug. It’s just like the combination of all those things are addressed by just having a test when you fix a bug. And anyway, so that bug got fixed. It had a CVE 2015 something, some numbers. But the thing that

really annoyed me and really made me think, God there’s got to be some…

ability to improve an OpenSSL was that it took two years for OpenSSL to merge the test suite. ⁓ So they fixed the bug and then took two years to… ⁓

Josh Bressers (08:04) Wow.

Joe (@ctz) (08:14) the ultimate test for it and that was just like, nah. So, ⁓

Josh Bressers (08:21) Yeah, that’s a very what radicalized you moment there, but I mean, it should be said. So I just talked to some OpenSSL folks a couple of episodes ago and the OpenSSL of today is not the OpenSSL of 2015. So we should kind of clarify some of that as well. Like we’re probably going to crap on OpenSSL a lot here, but it’s okay.

Joe (@ctz) (08:37) Yeah, yeah, yeah.

Yeah,

yeah. You’re absolutely right. And I want to underline that. That OpenSSL was in chaos, I would say. And I think maybe they’d probably agree with me on that. And the OpenSSL of today is a completely different ⁓ project.

Josh Bressers (09:02) yeah, for sure. All right, Dirkjan I’ve got to ask, what got you into this crazy library? Because not many people wake up in the morning and think, I’m going to work on a TLS library today.

Dirkjan (09:07) You

So I quit my job in 2018 and I woke up and thought I’m going to make a QUIC implementation. And it was kind of like that. I was planning to do a talk at the Rust Fest Paris like six months later, and I was going to make a QUIC implementation, the topic of the talk. And so I thought in six months I might be able to implement a decent chunk of that.

And then I spent quite some time looking at the TLS parts of QUIC, which are, mean, in the OpenSSL community, there has been some debate as well. I gather about the QUIC TLS API is quite different from what TCP TLS needs. And so that needed some like sort of invasive changes. And so I worked with Joe.

In, I guess it’s so this was in 2018, uh, to make, uh, to build QUIC support in rustls. Uh, and this was even before 1.3 was standardized, I think. So it was like, had provisional 1.3 support and then build QUIC around that. Uh, and that’s, so that’s how I got involved initially. And then a few days later, so the Porssimo project wanted to work, have some work on rustls.

I was working on making it more like to avoid panics. And so to basically rewrite a lot of the APIs to be statically safe from panicking rather than having like runtime characteristics that preempted panicking. And that’s, think when I became a maintainer eventually, when Joe got sick of like reviewing all my stuff or I was hoping I would review some of the other stuff.

⁓ And so we got to working together.

Josh Bressers (11:14) Awesome. Awesome. Okay. So let’s talk about Prossimo and the Rust foundation now, because the last time I talked to Josh Aas who’s one of the rustls he’s what the project manager, I think his title is. don’t, maybe it was, is it still? Okay. Okay. So in, Josh Aas is part of the what internet security research group ISRG, which is the entity behind let’s encrypt is like their giant claim to fame, but Prossimo is their memory safety project. And the last time I talked about rustls

rustls was, and this is, you corrected me before you record, rustls wasn’t, like, Prossimo doesn’t host projects. rustls was funding, wait, did I get that right? Yeah, Prossimo was funding rustls, and now rustls has kind of moved itself into the Rust Foundation, which I don’t totally understand what that means, and you’re going to explain it to us, so I’m excited to learn.

Dirkjan (11:52) trust me, was funding.

Joe (@ctz) (12:04) So basically we had some conversations with potential other funders and the feedback there was basically like, yeah we’d love to fund you but…

basically just need to improve your governance and what I take that to mean is two things. One is, sure the project is good now, but is it good in five years? And the other thing is, if I get hit by a bus or whatever, then is this project sustainable?

in the future and that goes for the other maintainers as well. ⁓ So it’s really like maturing the project from being ⁓ a project which one person started to a project which like two or three people have worked on regularly to a project that can have some long-term stability and ⁓ yeah so

That is one of the things that the Rust Foundation is helping us with. There is a ⁓ financial and fundraising aspect to it as well, which is that they are a US non-profit.

as is ISRG to be clear, but ⁓ this kind of allows us to drive the fundraising.

and that just kind of makes the whole product a little bit more self-contained.

Josh Bressers (13:37) I gotcha, okay, okay. So this is obviously a good thing for the project that you’re getting all of those annoying administrative bits handled by the Rust Foundation for you versus trying to do it yourself, which is 100 % the right decision. Because yeah, governance on open source projects is very painful. And it’s a shocking amount of work. Like most people don’t realize how much effort it actually is to like run an open source project until you have to run an open source project.

Joe (@ctz) (13:50) Exactly.

Josh Bressers (14:04) It’s like, holy crap, this is way harder than just like accepting pull requests and pushing the release button occasionally. So, okay, okay, that makes sense. I love it. I love it. All right. So the other thing I wanted to ask you about is I was obviously doing some research into rustls and the last time I talked about rustls with Josh Aas and one of my questions was kind of like, are you binary compatible with OpenSSL, right? The intent being like, I’m going to rip out OpenSSL and drop in rustls. And he was like, no, it doesn’t do that at all. In my understanding from reading your website is you have

I suspect it’s not complete because OpenSSL is like just a burning dumpster fire of options, but you have the ability to let rustls function in place of OpenSSL in some instances, yes?

Joe (@ctz) (14:45) Yeah, so we’ve got what I would describe as like a compatibility layer,

is like a clean room-ish, from what OpenSSL documentation mostly, implementation of the ⁓ OpenSSL, libssl half of the library. if you ⁓ for background OpenSSL kind of ships two libraries, libssl which contains the whole protocol thing, and libcrypto which contains all the cryptography.

⁓ Most applications just depend on both of these, ⁓ almost as if they are one, but some applications will have no interests at all in TLS, and so we’ll just use libcrypto. This part of the project is like a drop-in replacement ⁓ for the libssl part of aim to solve like…

a small slice of the problem of memory safety in network comms, which is like, I just want to fix my memory safety problems in this small ⁓ part of my stack, which deals with TLS and everything below that and everything above that is still, you know, remains the same. So it’s quite an incremental approach to solving the problem.

And yeah, you said it’s incomplete. It definitely is incomplete. But it’s like there’s a long tail of OpenSSL API surface which people use but is not widely used by applications like Nginx, curl, all these things. ⁓ And so we’ve just kind of taken the most popular

quarter, something like that, of the API service, implemented those so that Nginx curl and stuff ⁓ works. I would describe this as like alpha quality. So it’s like, go and try it, go and see how your systems work with it. You can just drop it in. You can do A-B testing if you want, ⁓ because you can…

choose to use it basically by fiddling around with your dynamic link of whatever application you’re

Josh Bressers (17:25) I mean, this is one of the things that makes Rust awesome, right? Is you can build C libraries basically, right? Just binaries that kind of work in that regard. So, okay, okay. Man, I could ask so many questions. All right, Dirkjan I’m going to pick on you a little bit, I think, for this next question. And so when we’re talking about memory safety in a TLS library, and I warned you I was going to ask this question before you record.

But so rustls isn’t like a pure Rust implementation, right? There are bits that are C code underneath, the actual crypto functions are still like C and assembly at the very bottom of the stack, yes?

Dirkjan (18:09) This is true. Yeah, I mean, okay.

Josh Bressers (18:11) Now, hold hold on. I’ve got one more follow-up to edit

that. Now you also mentioned when I, I said this at the very beginning before we hit record is that you don’t, you’re not using unsafe anywhere in rustls. So like explain what unsafe is and then explain how that all fits together and using the, the, you know, C and assembly way at the bottom. And if my question makes no sense, you can tell me I’m dumb. That’s fine.

Dirkjan (18:36) No, it definitely makes sense, but there is a lot of different avenues to go. think so. The first part is today Rust is not a good language to write low level cryptography code in because compilers are smart and they might undo some of the guarantees that you want when you’re writing low level crypto code. For example, like for timing attacks, you want constant time algorithms.

Josh Bressers (18:39) Yeah, yeah.

Yeah.

Dirkjan (19:01) And so if you write those in Rust, the compiler might be too smart and get rid of your constant time checking. So writing that code, you probably want to do it in assembly, which gets to the next thing, which is that a lot of the underlying cryptographic code that’s rustls users typically rely on is written in assembly. And we don’t have any unsafe code in

the rustls protocol implementation, but we depend on other crates that do inline assembly or other kinds of assembly and probably also contain some unsafe code to do FFI or whatever. And then I think the third part that we might want to talk about is that Joe has actually been working on Graviola, a new project that

Josh Bressers (19:47) Okay, gotcha.

Dirkjan (20:01) implements or that takes a bunch of formally verified assembly from AWS and wraps it in a very thin layer of Rust API so that there is like a minimal amount of C or unsafe potential for unsafety, let’s say in general. And so that is, think where we see a future for decreasing the amount of unsafe code included in projects that use rustls.

Josh Bressers (20:30) That’s pretty cool. And it should be added as well that in the security world, like the security problems we see in TLS libraries, I can’t think of any in recent memory that were because of the actual cryptography, because as you mentioned, they are verified, right? This is not just random code versus everything above that is where many of the bugs are. And we should also add, and I’ll let one of you two, you can argue over who. ⁓

Just because Rust is memory safe doesn’t mean a TLS library is free of bugs, right? You’re just free of a certain class of bugs, a very dangerous class of bugs in the TLS world, but there are still the potential for other bugs, right?

Dirkjan (21:11) That’s right.

Joe (@ctz) (21:11) Yeah,

it’s definitely about eliminating bug classes rather than producing perfect software. ⁓ Because you can produce perfect software, it is possible, but it will take you like decades and decades of person effort to do that. actually, when you look at the cost of doing that, people don’t value software.

that level. So there have been projects in the past which have produced a complete, formally verified TLS library and those projects kind of don’t exist I don’t think anymore. It was a project coming out of Research called Project Everest.

Quite a lot of the parts are still used, know, like a lot of their cryptography went into Firefox. coming back to the previous question about assembly, and C I would say

We’ve got quite like a strong interface that we introduced a of years ago now through which rustls gets all its cryptography. ⁓

It is correct to say, like if you take rustls today and accept all the defaults, you’re going to be running on top of a cryptography library which has lots of C, mostly assembler in it.

but that is not like a fundamental, so you can choose not to do that. You can go with rust based thing. I don’t believe there’s one which is completely rust based and zero unsafe. I think you have to throw away quite a lot of performance for that and I don’t think that’s very palatable to most people. ⁓ But ⁓ yeah, there’s a certain amount of choice there. ⁓

So we have defaults, which is a library which we think is pretty well maintained, has it like a professional ⁓ maintainer team ⁓ at AWS. But if you don’t want to that default, then there is actually quite a wide variety of other options with a different level of…

how much unsafe code, how fast they are, all these sorts of different sort parameters that you’re probably going to want to think about.

Josh Bressers (23:46) And we should add that Dirkjan suggested the Rust compiler might introduce bugs with timing attacks or whatever with the cryptography. But even just the, these are complicated algorithms. If someone re-implemented one of these algorithms that have been formally verified in C and assembler and are widely used and they re-implemented in Rust, they are 100 % going to make mistakes and introduce bugs. Like there is…

Absolutely no way we can just kind of do that drop in it. This is hard stuff. And so I also completely understand that rustls focusing on the TLS layer versus a cryptography layer, a hundred percent makes sense. And is those are two different skill sets that don’t necessarily cross into each other, which I think is something we forget sometimes when we’re talking about, you know, a TLS library, for example.

Okay. I’ve got two more questions before we wrap this one up. Question number one is your mascot is awesome. Cause it’s a crab wearing some bling. How did that come to be? I’m curious.

Joe (@ctz) (24:46) So the Ferris Clipart is ⁓ public domain, think it’s CC0. It ⁓ was originally drawn by someone who lived with one of the core team back in the early Mozilla days or something. yeah, so I basically drew that in ⁓ Fireworks MX.

Josh Bressers (25:01) No.

awesome.

Joe (@ctz) (25:12) which was this terrible macromedia thing that they used to sell for website design and chopping up images. I don’t actually have a copy of that anymore, so this logo is frozen in time, I just have a really, really, really high resolution thing of it.

Josh Bressers (25:32) awesome.

don’t wanna knock your work, but I don’t think turning this into an SVG would take all that long. I’m just saying.

Joe (@ctz) (25:39) Yeah, probably not

Josh Bressers (25:40) Okay. I love it. It’s such a good, it’s such a good logo. I love it so much. All right. And here’s my last question and I’ll give you each kind of a turn and I’ll start with you, Dirkjan is what’s next, right? I feel like, you know, rustls is almost 10 years old. It’s definitely becoming a functionally useful library, which is awesome. We get all the benefits of Rust, but I’m sure you have some plans of like, where do you want to see this project go in the next, you know, one year, five year, 10 year kind of what, what, what do you see?

What excites you?

Dirkjan (26:12) Yeah. So one thing that we’re working on pretty intensely right now is to work on the next.

API incompatible revision, cause in rustls or in the Rust ecosystem, we have these Semver compatible versions. And so we try not to do that. Change the API too often, but, ⁓ if we want to make changes that make it safer and better, then obviously we sometimes have to. we’re working on rustls 0.24 right now, which is really meant to be a stepping stone to 1.0, right? A point where we say the API is pretty great.

And we’re going to keep this stable for a very long time so people can build on top of it without having churn. And so that’s where we found a whole bunch of ways to improve the API. One thing I recently worked on is like identities for servers and clients and how we make that more modular so that ⁓ for TLS 1.3 there’s this raw key extension ⁓ standardized, which allows

⁓ project to not use certificate change but just to use keys directly just used ⁓ primarily by P2P systems. We’re working doing more work on making the crypto provider so there’s this layer between rustls and the cryptography level to make that more modular and in a way be less ⁓ particular about which

Provider, we make the default and ⁓ kind of pull it out so that we encourage sort of variety or picking your own, what’s best for your project. And beyond that, I think there’s a bunch of work to do on making the likes of what TLS is in a browser, right? ⁓

The browsers implement a whole bunch of extra stuff that on Unix implementations or on Unix platforms you typically don’t get. Certificate verification lists are basically not there. On Windows and on Apple systems you can get the platform verified to do some of that stuff. So one thing we would like to do

is to make sure that we have a better level of safety or security on these Unix systems that don’t have a platform verifier. And so if we can provide some of that in safe rust, that seems like an asset. And then I think also we’ll continue work on the OpenSSL compatibility layer to make sure that, you know, people eventually can just plug in rustls for their.

their TLS needs even in software it’s not easy to port to Rust directly.

Josh Bressers (29:14) Awesome. Yeah, that’s exciting. All right, Joe, what about you? What’s next? What are you excited for?

Joe (@ctz) (29:20) I’m to steal a phrase from the Rust project itself, which is like stability without stagnation, if you’ve heard that phrase. So the past, I think it’s almost coming up to two years, we’ve been on the 0.23 release line of rustls. And we’ve shipped a lot of features while not making any breaking changes. ⁓

almost two years ⁓ and I think that that’s been quite a nice ⁓ test run should we call it for the longer term of the project so like sure so so maybe next summer or something we pull the 1.0 chain and say this is this is like a stable API now

But doing that in a way which still allows us to ship new features and track like ⁓ new cryptography and ⁓ new TLS features. I mean they don’t come often but usually when they come they are

sometimes things that we really want to support like post quantum key exchange which is something we’ve done over the past year and we did all that in quite a good way I think without doing a breaking change so that is something that I hope to continue on the 1.0 API.

Josh Bressers (31:03) It’s always terrifying when you have to like cut the 1.0 API because you know, like this is it, right? If what did we screw up? We’re going to find out. It’s, one of those things like as soon as you hit send, you like see every spelling mistake in your email type thing. So I get it. It’s terrifying, but that’s exciting.

Joe (@ctz) (31:18) Yeah, especially

when you haven’t had to do that for like 10 years. It’s like we’ve been almost completely down to the straight, we can just go, yeah, whatever, you know. So that’s part of growing up, I think.

Josh Bressers (31:24) Yeah.

Right, right.

Dirkjan (31:29) That’s.

That’s right. It’s really good that we have had this almost two year run of being on 0.23 where we’ve been actually compatible and we have identified like all of the pain points of things that are causing trouble when we want to evolve the API. And so we can figure out how can we make the future API better in a way that allows us more flexibility in evolving it.

Josh Bressers (31:57) Awesome, awesome. Holy cow, this has been a ton of fun. I want to thank you both. Like I’ve learned a whole bunch. This is an awesome project. I can’t wait to see where it goes in the future. So just thank you so much for the time. I truly appreciate it.

Dirkjan (32:09) Thanks. It’s been fun.

Joe (@ctz) (32:11) Thanks a lot.