Security is really about Risk vs Reward

Every now and then the conversation erupts about what is security really? There’s the old saying that the only secure computer is one that’s off (or fill in your favorite quote here, there are hundreds). But the thing is, security isn’t the binary concept: you can be secure, or insecure. That’s not how anything works. Everything is a sliding scale, you are never secure, you are never insecure. You’re somewhere in the middle. Rather than bumble around about your risk though, you need to understand what’s going on and plan for the risk.

So this brings us to the idea of risk and reward. Rather than just thinking about security, you have to think about how everything fits together. It doesn’t matter if your infrastructure is super secure if nobody can do their jobs. As we’ve all seen over and over, if security gets in the way, security loses. Every. Single. Time.

I think about this a lot, and I’ve come up with a graph that I think can explain this nicely.

Don’t think in the context of secure or insecure. Think in the context of how much risk do I have? Once you understand what your risks are, you can decide if the level of risk you’re taking on can be justified by what the result of that risk will be. This of course holds true for nearly all decisions, not just security, but we’ll just focus on security.

The above graph puts things into 4 groups. If you have a high level of risk with minimal reward (the Why box), you’re making a bad decision. Anything you have in that “Why” box probably needs to go away ASAP, you will regret it someday.

Additionally, if your sustaining operations are of high risk, you’re probably doing something wrong. Risk is hard and drains an organization, you should be conducting your day to day operations in a manner than poses a low risk as the day to day is generally not where the high reward is.

The place you want to be is in the “Innovation” or “No Brainer” boxes. Accepting a high level of risk isn’t always a bad thing, assuming that risk comes with significant rewards. You can imagine a situation where you are deploying a new and untested technology, but the benefits to conducting business could change everything, or perhaps using a new, untested vendor for the first time.

We have to be careful with risk. Risk can be crippling if you don’t understand and manage it. It can also destroy everything you’ve done if you let it get out of hand. Many of us find ourselves in situations where all risk is seen as bad. Risk isn’t always bad, risk is never zero. It’s up to everyone to determine what their acceptable level of risk is. Never forget though, that sometimes we need to bump up our level of risk to get to the next level of reward. Just make sure you can bring that risk back under control once you start seeing the outcomes.

What do you think? Let me know: @joshbressers

Ransomware is scary, but not for the reasons you think it is

If you’ve been paying any attention for the past few weeks, you know what ransomware is. It’s a pretty massive pain for anyone who gets it, and in some cases, it was a matter of life and death.

It’s easy to understand what makes this stuff scary, but there’s another angle most haven’t caught on to yet, and it’s not a pleasant train of thought.

Firstly, let’s consider a few thing.

  1. Getting rid of malware is expensive
  2. Recovering from a compromise is even more expensive
  3. Ransomware has a clear and speedy ROI
  4. Normal people don’t have a ton of important data
So let’s start with #1 and #2. If you are compromised in some way, even if it’s just some malware, it’s going to cost a lot to clean up the mess. Probably magnitudes more than the current ransom. It’s cheaper to pay than to clean up the mess. This will remain true as there isn’t an incentive for the authors to price themselves out of business. The ransomware universe is econ 101. If you’re an economics PhD student and you want to look impressive, write your thesis about this stuff; you’ll probably win some sort of award. We’ll get back to the economics of this shortly.
If we think about #3 it’s pretty obvious. You write some malware, it literally pays you money. This means there is going to be more and more of this showing up on the market. Regular old malware can’t compete with this. Ransomware has a business model, a really good one, except for that whole being illegal and really unethical part. Non ransomware doesn’t have such an impressive business model. This is a turning point in the malware industry.
To date most of the ransomware seems to have been targeted at normal people. The price was a bit too high I thought, $400 is probably more than the average person will or can pay. The last few we’ve heard about hit hospitals though, and they charged a higher fee. This is basic economics. A hospital has more money than a person, and the data and infrastructure means the difference between life and death. Paying the fee will cost less than hiring a security professional. And when you’re in the business of keeping people alive, you’ll pay that fee if it means getting back to whatever it is you do.
If the ransomware knows where it is and what sort of data it has, the price can fluctuate based on economics. Some businesses can afford a few days of downtime, some can’t. The more critical the data and system is to your business, the more you’ll be willing to pay. Of course there is a ceiling on this, if the cost of hiring some security folks is less than the cost of paying the ransom, anyone with a clue is going to pay the expert to clean up the mess. This is the next logical step in the evolution of this business model.
If we keep thinking about this and bring the ransomware to its logical conclusion, the future versions are going to request a constant ongoing payment. Not a one time get out of jail free event. Why charge them once when you can charge them over and over again? Most modern infrastructures are complex enough it will be hard to impossible to remove an extremely clever bit of malware. It’s going to be time for the good guys to step it up here, more thoughts on that some other day though.
There is even a silly angle that’s fun to ponder. We could imagine ransomware that attacks other known malware. If the ransomware is getting a constant ongoing payment, it would be bad if anything else could remove it, from legitimate software to other ransomware. While I don’t think antivirus and ransomware will ever converge on the same point, it’s still fun to think about.
What do you think? Let me know: @joshbressers

I’m going to do something really cool in 3 weeks! … Probably.

If you pay attention to the security news, there is something coming called Badlock. It just set off a treasure hunt for security flaws in Samba. Rather than link to the web site (I’d rather not support this sort of behavior), let’s think about this as reasonable people.

I can imagine three possible outcomes to the events that have been set in motion.

  1. On April 12 a truly impressive security flaw will be disclosed. We will all be impressed.
  2. Someone will figure this out before April 12, they have no incentive to act responsibly and will publish what the know right away, better to be first than to be right!
  3. Whatever happens on April 12 won’t be nearly as interesting or exciting as we’ve been led to believe. The world will say a collective ‘meh’ and we’ll go back to looking at pictures of cats.
Numbers 1 and 2 rely on the flaw being quite serious. If it is serious, I suspect there is a far greater chance of #2 happening than #1. As an industry we should hope for #3, we don’t need more terrible flaws.

The really crazy thing to think about is if the issue isn’t actually serious, it probably won’t be found. Everyone is looking for a giant problem. They’re going to pass up minor issues (if you do find these, please report them, it’s still useful work). The prize is a pot of gold we’ve been told, not some proverbial the journey is the reward nonsense.

The thing everyone always should remember in a situation like this is there are a lot of really smart people on the planet. If you think of something clever or discover something new, there are huge odds someone else did too. 3 weeks almost guarantees someone else can figure out whatever it is you found. It’s especially interesting in this case since we have a name “Badlock” so we know it probably involves locking. We know it affects Samba and Windows. And we know who it was found by so we can look at which bits of Samba they’ve been working on lately. That’s a lot of information for a clever person.

The real thing we need to think about here though is what’s actually happening. There is a bigger story for us to think about around all these named issues.
If you name an issue, you are making a claim that it’s very serious. There are literally thousands of security issues per year, and maybe ten gets fancy names. A name suggests this is something we should care about. That this issue is special. Except that’s not really the case all the time. There have been a lot of named issues that weren’t very impressive.
What happens in situations like this, when there is a near constant flow of information that’s not really important? People stop listening. The human brain is really good at filtering out noise. Named security issues are going to become noise at the current rate things are going. I’m not opposed to this, I think you should name your pets not your security issues.

Send your comments to Twitter: @joshbressers

Everything is fine, nothing to see here!

As anyone who reads this blog knows, I’ve been talking about soft skills in security for quite some time now. I’m willing to say it’s one of our biggest issues at the moment, a point which I get a lot of disagreement on. I have sympathy for anyone who thinks this stuff doesn’t matter, I used to be there. Until I had to start talking to people. As soon as you talk to most anyone outside the security echo chamber, you see what’s actually going on, and it’s not great.

I won’t say the security industry is one fire, but nobody is going to disagree many of the things we’re looking after aren’t in great shape.Outside of a few very large successful companies, most organizations have serious and significant security problems that could result in a massive breach, it’s just that nobody has tried, yet. I see a few reasons for many of our trouble, I always seem to come back to soft skills.

There is a skills shortage
But there’s training, look at all the training, there’s so much training everything is fine!

There is training. Some is good, some is bad (like anything). It’s not that training in itself is bad, I would encourage anyone to go get training. It’s not great though either. Most training today focuses on the symptoms of our problems. Things like pen testing, secure coding (which doesn’t exist), network defense. Things that while important, aren’t the real problems. I’ll talk more about this in a future post, but chew on this. There are about 96,000 CISSP holders, and about 5 million security jobs. That’s messed up.

Today everyone who is REALLY, I mean REALLY REALLY good at security got there through blood sweat and tears. Nobody taught them what they know, they learned it on their own. Many of us didn’t have training when we were learning these things. Regardless of this though, if training is fantastic, why does it seem there is a constant march toward things getting worse instead of better? That tells me we’re not teaching the right skills to the right people. The skills of yesterday don’t help you today, and especially don’t help tomorrow. By its very definition, training can only cover the topics of yesterday.

How do we skill up for the needs of today and tomorrow? The first thing we have to do is listen to the people running, building, and using the technology of today. They know things we don’t just as we know things they don’t. Security is still almost always an afterthought, even with everyone claiming it’s the most important thing ever. This is our failing, not theirs.

We build our skills by being an industry that doesn’t complain and belittle everyone who tries anything. We are notorious for being brutal to the new guys. Everyone starts somewhere, don’t be a jerk. I know a lot of people who are afraid to do almost anything in the security space because they know if they’re not 100% correct, they will have to deal with a torrent of negative comments. It’s not worth talking to us in many instances.

As an industry we are failing our customers
Things aren’t that bad, sure there are some breaches but in general everything is going pretty good!

If you read any news stories, you know things aren’t OK. There are loads of breaches and high profile security issues. Totally broken devices, phones that can’t be updated, light bulbs that can join a botnet. As an industry we like to stick to our echo chamber circles where we spin news and events into something that isn’t our fault. We laugh at the stupid people doing stupid things. We find a person or event that can explain away the incident as a singular event, not a systematic problem. The problems are growing exponentially, our resources are growing linearly, this means that our resources are actually decreasing every year.

Most organizations don’t have proper security and won’t even have a proper conversation until they end up on the wrong side of a major compromise. It’s our fault nobody is talking about this stuff, even if the breach isn’t technically our fault.

What advice are we giving people they can actually use? In almost every organization the security group is feared and hated. We’re not peers, we’re enemies, and they are ours. This isn’t helpful to anyone. How many of you actually sit down and have honest real discussions with those you are supposed to help. Do you actually understand their problems (not our problems with them, their actual problems, the ones they have to route around security to solve). Security shouldn’t be something bolted on later, we’re lucky if it’s even that in most cases.

Security is seen as a business prohibitor, not a business enabler
I know what needs to be done, nobody wants to listen!

We’ve all been here before. We suggest something to the group, they ignore us. We are the problem here, not the people we are supposed to help. We blame them for not listening when the real issue is we’re not talking to them properly. We throw information at people, complex hard to understand information, then rather than hold their hand when they don’t understand, we declare them stupid and go find someone who agrees with us, then we complain about how dumb everyone else is and how smart we are.

They aren’t stupid.

Neither are we.

The disconnect is one of talking. We have to talk to people, we have to engage with them. We have to build a relationship. You can’t expect to show up and be listened to if you’re not respected. People trust those they respect. If you’re in that circle of respect, you won’t be taken seriously. On a regular basis I hear security tell me “they’ll know I was right when we get hacked!” That doesn’t even make sense. It’s your failure for not creating a level of understanding for the issue, it’s not their fault for ignoring you.

Soft skills are hard
You don’t even know what you’re talking about, my skills are fine!

Maybe. I won’t say I’m an expert. I am constantly thinking about the state of things and how interactions go. What I do know though is the things I discuss here are based on my real world lessons. Every day is a new journey into being a new and better security person. I know how the technology works, what I don’t know is how people work. It’s a journey to figure this out. I’m pretty sure I’m on to something because people I respect are encouraging, yet there are some who are trying very hard to discourage this conversation. As the old saying goes, if nobody is complaining about what you’re doing, you’re not doing anything interesting.

Here’s what I do honestly believe. You can disagree with me or anyone you want. The industry isn’t solving the problems it needs to solve. Those problems will be solved eventually, there are many industry groups forming to start talking about some of these problems, the groups mostly talk though, that’s not a skill we’re good at. Even then I see a lot of criticism toward those groups. Problems won’t be solved quickly by doing the same thing we do today. I’m confident a big part of our future is humanizing security. Security today isn’t for humans, security tomorrow needs to be. We get there by cooperating, not by arguing and insulting.

Think I’m an idiot, let me know: @joshbressers

Containers are like sandwiches

During the RSA conference, I was talking about containers and it occurred to me we can think about them like a sandwich. Not so much that they’re tasty, but rather where does your container come from. I was pleased that almost all of the security people I spoke with understand the current security nightmare containers are. The challenge of course is how do we explain what’s going on to everyone else. Securtiy is hard and we’re bad at talking about it. They also didn’t know what Red Hat was doing, which is totally our own fault, but we’ll talk about that somewhere else.

But containers are sandwiches. What does that mean? Let’s think about them in this context. You can pick up a sandwich. You can look at it, you can tell basically what’s going on inside. Are there tomatoes? Lettuce? Ham? Turkey? It’s not that hard. There can be things hiding, but for the most part you can get the big details. This is just like a container. Fedora? Red Hat? Ubuntu? It has httpd, great. What about a shell? systemd? Cool. There can be scary bits hidden in there too. Someone decided to replace /bin/sh with a python script? That’s just like hiding the olives under the lettuce. What sort of monster would do such a thing!

So now that we have the image of a sandwich in our minds, let’s think about a few scenarios.

Find it on a bench
If you’re walking through the park and you see a sandwich just laying on a bench what would you do? You might look around, wondering who left this tasty delight, but you’re not going to eat it. Most people wouldn’t even touch it, who put it there, where did it come from, how old is it, does it have onions? So many questions and you honestly can’t get a decent answer. Even if someone could answer the questions, would you eat that sandwich? I certainly wouldn’t.

Finding a sandwich on a bench is the public container registry. If this is all you know, you wouldn’t think there’s anything wrong with doing this, but like the public registry, you don’t always know what you’re getting. I wonder how many of those containers saw update for the glibc flaw from a few weeks ago? It’s probably easier not knowing.

Get it from a scary shop with questionable ingredients
A long time ago I was walking around in New York and decided to hop into a sandwich shop for a quick bite. As I reached for the door, there was a notice from the health department. I decided to keep walking. Even if you can get your sandwich from a shop, if the shop is scary, you could find yourself in trouble.

There are loads of containers available out there you can download that aren’t trusted sources. Don’t download random containers from random places. It’s no different than trying to buy a sandwich from a filthy shop that has to shoo the rats out of the kitchen with a broom.

Get it from a nice shop that uses old ingredients
We’ve all seen those places selling sandwiches that look nice. The sign is painted, the windows are clean. When you walk in the tables are clean enough to eat off of! But then you order and it’s pretty clear everything is old and dried out. You might be able to sneak out the back door before the old man putting it together notices you’re not there anymore.

This is currently a huge danger in the container space. Containers are super hip right now so there are plenty of people doing work in this space. Many of these groups don’t even know they have a problem. The software in your containers is a lot like sandwich meat. After a few weeks it probably will start to smell, and after a month it’s going to do some serious damage to anyone who consumes it.

Be sure to ask your container supplier what they’re shipping, where it came from and how fresh it is. It would not be reasonable to ask “If this container was a sandwich would you eat it?”

Get it from a nice shop that uses nice ingredients
This is the dream. You walk into a nice shop. The nice person behind the counter takes your order and using the freshest ingredients possible constructs a sandwich shaped work of art. You take pictures and post them to all your friends explaining this sandwich is what your life was always missing and you didn’t know it before now.

This is why you need a partner you can trust when it comes to container content. The closer to the source you can get the better. Ask questions abut the content. Where did it come from? Who is taking care of it? How can I prove any of this? Who is updating it? Containers are a big deal, they’re new and exciting. They’re also very misunderstood. Only use fresh containers. If the content it more than a few months old, you’re eating a sandwich off a park bench. Don’t each sandwiches off park benches. Ask hard questions. If your vendor can’t answer them, you need to try the shop across the street. Part of the magic of containers is they are the result of truly commoditizing the operating system, you can get container content from a lot of sources, find a good one.

If we think about our infrastructure like we think about public health, you don’t want to be responsible for making everyone sick. You need to know what you’re using, where it came from, how fresh it is, who put it together, and what’s in it. It’s not enough to pretend everything is fine. Everything is not fine.

The interesting things from RSA are what didn’t happen, and containers are sandwiches

The RSA conference is done. It was a very long and busy show, there were plenty of interesting people there and lots of clever ideas and things to do.

I think the best part is what didn’t happen though. We love talking about the exciting things from the show, I’m going to talk about the unexciting non events I was waiting to happen (but thankfully they did not).

The DROWN issue came and went. It wasn’t very exciting, it got the appropriate amount of attention. Basically SSLv2 is still broken, don’t use it for any reasons. If you use SSLv2, it’s like licking the handrail at the airport. Nobody is going to feel bad for you.

There were keynotes by actors. The world continues to turn (pun intended). But really, these keynotes are about being entertaining, I didn’t go, because well, they’re actors 🙂 But I suspect they were entertaining. No doubt this will happen more and more as there are more and more security conferences, finding good keynotes will only get harder. They should hire that guy from the Hackers movie next.

There weren’t any exciting hacking events. Not that stunt hacking is a thing for RSA, I’m glad nobody tried anything new. I’m sure Blackhat will be a very different story. We shall wait and see.

And most importantly, I wasn’t booed off the stage 😛
I was pleased with how my talk went. Attendance was light but that’s expected on a Friday morning. The thing that made the happiest is that they had to kick our group out of the room for the next talk, not because I rambled on but because I got everyone in the room talking to each other. It was fantastic.

On to the interesting bit of the trip though. I found the most interest when I was talking about Red Hat’s concept of a trusted container registry. Today if you’re using the public registry it’s comparable to finding a sandwich on a bench at the park. You can look at it, you can tell it has ham and lettuce, but I mean, it’s a sandwich you found on a bench. Are you going to eat that?

If you want a nice sandwich you’re going to go to a sandwich shop, order a sandwich, and watch someone make it for you. You can then go and sit on the bench if you want.

The idea behind Red Hat’s trusted registry is we have a container registry for Red Hat customers. We control all the content in the registry, we know exactly what it is. We know where it came from. We control the sandwich supply chain from start to finish. No mystery meats here!

All the security people I talked to know that containers are currently a bit of a security circus. None of them knew what Red Hat was doing. This is of course a great opportunity for Red Hat to spread the word. Stay tuned for more clever sandwich jokes.

Let’s talk about soft skills at RSA, plus some other things

It’s been no secret that I think the lack of soft skills in the security space is one of our biggest problems. While usually I usually only write all about the world’s problems and how to fix them here, during RSA I’m going to take a somewhat different approach.

I’m giving a talk on Friday titled Why Won’t Anyone Listen to Us?

I’m going to talk about how a security person can talk to a normal person without turning them against us. We’re a group that doesn’t like talking to anyone, even each other. We need to start talking to people. I’m not saying we should stand around and accept abuse, I am saying the world wants help with security. We’re not really in a place to give it because we don’t like people. But they need our help, most of them know it even!

We’ve all had countless interactions where we give someone good, real advice, and they just completely ignore us. It’s infuriating sometimes. Part of the problem is we’re not talking to people, we’re throwing verbal information at them, and they ignore it. They listen to Oprah, if she told them about two factor auth everyone would be using it by the end of the week!

That’s just it, they listen to Oprah. They’re going to listen to anyone who talks to them in a way they understand. If it’s not us, it will be someone else, probably someone we don’t want talking about security.

I can’t teach you to like people (there are limits to my abilities), but I can help teach you how to talk to them. And of course a talk like this will need to have plenty of fun sprinkled in. How to talk to someone while being very important, can also be an extremely boring topic.

I touched on some of this during my Devconf.cz talk.

Red Hat is also putting on a Breakfast on Wednesday morning. I’m going to keynote it (I’ll keep it short and sweet for those of you attending, there’s nothing worse than a speaker at 8am who talks too much).

A coworker, Richard Morrell, is running a podcast from RSA called Locked Down. Be sure to give it a listen. I may even be able to convince him to let me on his show.

I have no doubt the RSA conference will be a great time. If you’re there come find me, Red Hat has a booth, North Expo #N3038. Come say hi, or not if you don’t like talking to people 😉

There or not, feel free to start a conversation on Twitter. I’m @joshbressers

Thinking about glibc and Heartbleed, how do fix things

After my last blog post Change direction, increase speed! (or why glibc changes nothing) it really got me thinking about how can we start to fix some of this. The sad conclusion is that nothing can be fixed in the short term. Rather than trying to make up some nonsense about how to fix this, I want to explain what’s happening and why this can’t be fixed anytime soon.

Let’s look at Heartbleed first.

There was a rather foul flaw found in OpenSSL, after Heartbleed the Linux Foundation collected a lot of money to help work on core infrastructure projects. If we look at the state of things it basically hasn’t changed outside of money moving around. OpenSSL cannot be fixed for a number of reason.

  1. Old codebase
  2. Backward compatibility
  3. Difficult API
  4. It is “general purpose”
The reality is that the only way to get what could be considered a safe library, you would have to throw everything out and start over with some very specific ideas in mind. Things like sun-setting algorithms didn’t exist when OpenSSL was created. There is no way you’re going to get even a small number of projects to move from using OpenSSL to some new “better” library. It would have to be so much better they couldn’t ignore it. As anyone who has ever written software knows, you don’t build a library like that over night. I think 5 years would be a conservative estimate for double digit adoption rates.
While I’m picking on OpenSSL here, the story is the same in virtually every library and application that exists. OpenSSL isn’t special, it just gets a lot of attention.
Let’s think about glibc.
Glibc is the C library used by most Linux distributions. If the Kernel is the heart, glibc is the soul. Nothing can even exist without this library. Glibc even strives to be POSIX compliant, for good reason. POSIX has given us years of compatibility and cooperation.
Glibc probably has plenty more horrible bugs hiding in the code. It’s wicked complex and really large. If you ever need some nightmare fuel, look at the glibc source code. Everything we do in C relies on a libc being around, glibc doesn’t have that luxury.
Replacing libc is probably out of the question, it’s just not something practical. So let’s think about something like golang. What if everything was written using golang? It’s not totally insane, there are substantial benefits. It’s not as fast as C though, that will be the argument most use. Golang will probably never beat C, the things that makes it safer also makes it slower. But now if we think about replacing UNIX utilities with golang, why would we want to do that? Why not throw out all the mistakes UNIX made and do something else?
Now we’re back to the legacy and compatibility arguments. Linux has more than twenty years of effort put into it. You can’t just replace that. Even if you had the best team in the world I bet 10 years would be wishful thinking for having half the existing features.
So what does this mean? It means we don’t know where to start yet. We are trying to solve our problems using what we know and the tools that exist. We have to solve this using new tools and new thinking. The way we fix security is by doing something nobody has ever thought of before. In 1920 the concept of the Internet didn’t exist, people couldn’t imagine how to even solve some of the problems we can easily solve using it. Don’t try to solve our problems with what you know. Solve the problems using new ideas, find out what you don’t know, that’s where the solution lives.

Change direction, increase speed! (or why glibc changes nothing)

The glibc issue has had me thinking. What will we learn from this?

I’m pretty sure the answer is “nothing”, which then made me wonder why this is.

The conclusion I came up with is we are basically the aliens from space invaders. Change direction, increase speed! While this can give the appearance of doing something, we are all very busy all the time. It’s not super useful when you really think about it. Look at Shellshock, Heartbleed, GHOST, LOGJAM, Venom, pick an issue with a fancy name. After the flurry of news stories and interviews, did anything change, or did everyone just go back to business as usual? Business as usual pretty much.

Dan Kaminski explains glibc nicely and has some advice. But let’s look at this honestly. Is anything going to change? No. Dan found a serious DNS issue back in the day. Did we fix DNS or did we bandage it up as best as we could? We bandaged it. What Dan found was without a doubt as bad or worse than this glibc issue, nothing changed.

I’ve said this before, I’ll say it again. I’m going to say it at RSA next week. We don’t know how to fix this. We think we do, you’re thinking about it right now, about how we can fix everything! We just have to do that one … Except you can’t. We don’t really know what’s wrong. Security bugs aren’t the problem, they are the result of the problem. We can’t fix all the security bugs. I’d be surprised if we’ve even fixed 10% of security bugs that exist. Even mitigation technologies aren’t going to get us there (they are better than constantly fixing bugs, but that’s a story for another day).

It’s like being obsessed about your tire pressure when there is a hole in the tire. If you only worry about one detail, the tunnel vision makes you miss what’s actually going on. Our tires are losing air faster than we can fix them, so we’re looking for a bigger pump instead of new tires.

We say things all the time about not using C anymore, or training developers, or writing better documentation. There’s nothing wrong with these ideas exactly, but the fact is they’ve all been tried more than once and none of them work. If we started the largest developer education program ever, if we made every developer sit through a week of training. I bet it would be optimistic if our bug rate would decrease by 5%. Think about that for a while.

We first have to understand our problem. We have lots of solutions, solutions to problems that don’t really exist. Solutions without problems tend to turn into new problems. We need to understand our security problem. It’s probably more like hundreds or thousands of problems. Every group, every company, every person has different problems. We understand none of them.

We start by listening. We’re not going to fix any of this with code. We need to see what’s happening, some big picture, some in the weeds. Today we show up, yell at people (if they’re lucky), then we leave. We don’t know what’s really happening. We don’t tell anyone what they need to know. We don’t even know what they need to know. The people we’re not talking to know what the problems are though. They don’t know they know, we just have to give them time to explain it to us.

If you’re at RSA next week, come talk to me. If not, hit me up on twitter @joshbressers

glibc for humans

Unless you’ve been living under a rock, you’ve heard about the latest glibc issue.
CVE-2015-7547 – glibc stack-based buffer overflow in getaddrinfo()

It’s always hard to understand some of these issues, so I’m going to do my best to explain it using simple language. Making security easy to understand is something I’ve been talking about for a long time now, it’s time to do something about it.

What is it?
The fundamental problem here is that glibc has a bug that could allow a DNS response from an attacker to run the command of that attacker’s choosing on your system. The final goal of course would be to become the root user.

The problem is that this glibc function is used by almost everything that talks to the network. In today’s hyperconnected world, this means basically everything is vulnerable to this bug because almost everything can connect to the network. As of this writing we have not seen this attack being used on the Internet. Just because there are no known attacks is no reason to relax though, constant vigilance is key for issues like this.

Am I vulnerable?
If you run Linux (most distributions use glibc), and you haven’t installed an update from your vendor, yes, you are vulnerable.

Are there workarounds?
No, there is no way to stop this issue. You have to install an update to glibc. Even the stack protector technology that is built into gcc and glibc will not stop this bug. While it is a stack overflow bug, the stack protector checks do not run before the exploit would gain control.

What about containers, VMs, or other confinement technology?
It is possible that a container, VM, or other technology such as SELinux could limit the possible damage from this bug. However it affects so many binaries on the system it should be expected that an attacker able to gain access to one applications could continue to exploit this bug to eventually become root and take over the entire machine.

Do I only need to be worried if I run a webserver or mailserver?
As stated previously, this bug affects virtually everything that talks to the network. Even if you think your webserver or mailserver are safe, everything from bash to your ssh client will use this library. Updating glibc is the only way to ensure you’ll be OK.

What if I run my own DNS server?
This point is currently under investigation. It is thought that it may be possible for a bad DNS request to be able to make it through a DNS server to a vulnerable host. Rather than find out, you should update your glibc.

What about …
No, just update your glibc 🙂

Do you have other questions? Ask me on twitter and I’ll be sure to update this article if I know the answer.
@joshbressers