Let’s talk about soft skills at RSA, plus some other things

It’s been no secret that I think the lack of soft skills in the security space is one of our biggest problems. While usually I usually only write all about the world’s problems and how to fix them here, during RSA I’m going to take a somewhat different approach.

I’m giving a talk on Friday titled Why Won’t Anyone Listen to Us?

I’m going to talk about how a security person can talk to a normal person without turning them against us. We’re a group that doesn’t like talking to anyone, even each other. We need to start talking to people. I’m not saying we should stand around and accept abuse, I am saying the world wants help with security. We’re not really in a place to give it because we don’t like people. But they need our help, most of them know it even!

We’ve all had countless interactions where we give someone good, real advice, and they just completely ignore us. It’s infuriating sometimes. Part of the problem is we’re not talking to people, we’re throwing verbal information at them, and they ignore it. They listen to Oprah, if she told them about two factor auth everyone would be using it by the end of the week!

That’s just it, they listen to Oprah. They’re going to listen to anyone who talks to them in a way they understand. If it’s not us, it will be someone else, probably someone we don’t want talking about security.

I can’t teach you to like people (there are limits to my abilities), but I can help teach you how to talk to them. And of course a talk like this will need to have plenty of fun sprinkled in. How to talk to someone while being very important, can also be an extremely boring topic.

I touched on some of this during my Devconf.cz talk.

Red Hat is also putting on a Breakfast on Wednesday morning. I’m going to keynote it (I’ll keep it short and sweet for those of you attending, there’s nothing worse than a speaker at 8am who talks too much).

A coworker, Richard Morrell, is running a podcast from RSA called Locked Down. Be sure to give it a listen. I may even be able to convince him to let me on his show.

I have no doubt the RSA conference will be a great time. If you’re there come find me, Red Hat has a booth, North Expo #N3038. Come say hi, or not if you don’t like talking to people 😉

There or not, feel free to start a conversation on Twitter. I’m @joshbressers

Thinking about glibc and Heartbleed, how do fix things

After my last blog post Change direction, increase speed! (or why glibc changes nothing) it really got me thinking about how can we start to fix some of this. The sad conclusion is that nothing can be fixed in the short term. Rather than trying to make up some nonsense about how to fix this, I want to explain what’s happening and why this can’t be fixed anytime soon.

Let’s look at Heartbleed first.

There was a rather foul flaw found in OpenSSL, after Heartbleed the Linux Foundation collected a lot of money to help work on core infrastructure projects. If we look at the state of things it basically hasn’t changed outside of money moving around. OpenSSL cannot be fixed for a number of reason.

  1. Old codebase
  2. Backward compatibility
  3. Difficult API
  4. It is “general purpose”
The reality is that the only way to get what could be considered a safe library, you would have to throw everything out and start over with some very specific ideas in mind. Things like sun-setting algorithms didn’t exist when OpenSSL was created. There is no way you’re going to get even a small number of projects to move from using OpenSSL to some new “better” library. It would have to be so much better they couldn’t ignore it. As anyone who has ever written software knows, you don’t build a library like that over night. I think 5 years would be a conservative estimate for double digit adoption rates.
While I’m picking on OpenSSL here, the story is the same in virtually every library and application that exists. OpenSSL isn’t special, it just gets a lot of attention.
Let’s think about glibc.
Glibc is the C library used by most Linux distributions. If the Kernel is the heart, glibc is the soul. Nothing can even exist without this library. Glibc even strives to be POSIX compliant, for good reason. POSIX has given us years of compatibility and cooperation.
Glibc probably has plenty more horrible bugs hiding in the code. It’s wicked complex and really large. If you ever need some nightmare fuel, look at the glibc source code. Everything we do in C relies on a libc being around, glibc doesn’t have that luxury.
Replacing libc is probably out of the question, it’s just not something practical. So let’s think about something like golang. What if everything was written using golang? It’s not totally insane, there are substantial benefits. It’s not as fast as C though, that will be the argument most use. Golang will probably never beat C, the things that makes it safer also makes it slower. But now if we think about replacing UNIX utilities with golang, why would we want to do that? Why not throw out all the mistakes UNIX made and do something else?
Now we’re back to the legacy and compatibility arguments. Linux has more than twenty years of effort put into it. You can’t just replace that. Even if you had the best team in the world I bet 10 years would be wishful thinking for having half the existing features.
So what does this mean? It means we don’t know where to start yet. We are trying to solve our problems using what we know and the tools that exist. We have to solve this using new tools and new thinking. The way we fix security is by doing something nobody has ever thought of before. In 1920 the concept of the Internet didn’t exist, people couldn’t imagine how to even solve some of the problems we can easily solve using it. Don’t try to solve our problems with what you know. Solve the problems using new ideas, find out what you don’t know, that’s where the solution lives.

Change direction, increase speed! (or why glibc changes nothing)

The glibc issue has had me thinking. What will we learn from this?

I’m pretty sure the answer is “nothing”, which then made me wonder why this is.

The conclusion I came up with is we are basically the aliens from space invaders. Change direction, increase speed! While this can give the appearance of doing something, we are all very busy all the time. It’s not super useful when you really think about it. Look at Shellshock, Heartbleed, GHOST, LOGJAM, Venom, pick an issue with a fancy name. After the flurry of news stories and interviews, did anything change, or did everyone just go back to business as usual? Business as usual pretty much.

Dan Kaminski explains glibc nicely and has some advice. But let’s look at this honestly. Is anything going to change? No. Dan found a serious DNS issue back in the day. Did we fix DNS or did we bandage it up as best as we could? We bandaged it. What Dan found was without a doubt as bad or worse than this glibc issue, nothing changed.

I’ve said this before, I’ll say it again. I’m going to say it at RSA next week. We don’t know how to fix this. We think we do, you’re thinking about it right now, about how we can fix everything! We just have to do that one … Except you can’t. We don’t really know what’s wrong. Security bugs aren’t the problem, they are the result of the problem. We can’t fix all the security bugs. I’d be surprised if we’ve even fixed 10% of security bugs that exist. Even mitigation technologies aren’t going to get us there (they are better than constantly fixing bugs, but that’s a story for another day).

It’s like being obsessed about your tire pressure when there is a hole in the tire. If you only worry about one detail, the tunnel vision makes you miss what’s actually going on. Our tires are losing air faster than we can fix them, so we’re looking for a bigger pump instead of new tires.

We say things all the time about not using C anymore, or training developers, or writing better documentation. There’s nothing wrong with these ideas exactly, but the fact is they’ve all been tried more than once and none of them work. If we started the largest developer education program ever, if we made every developer sit through a week of training. I bet it would be optimistic if our bug rate would decrease by 5%. Think about that for a while.

We first have to understand our problem. We have lots of solutions, solutions to problems that don’t really exist. Solutions without problems tend to turn into new problems. We need to understand our security problem. It’s probably more like hundreds or thousands of problems. Every group, every company, every person has different problems. We understand none of them.

We start by listening. We’re not going to fix any of this with code. We need to see what’s happening, some big picture, some in the weeds. Today we show up, yell at people (if they’re lucky), then we leave. We don’t know what’s really happening. We don’t tell anyone what they need to know. We don’t even know what they need to know. The people we’re not talking to know what the problems are though. They don’t know they know, we just have to give them time to explain it to us.

If you’re at RSA next week, come talk to me. If not, hit me up on twitter @joshbressers

glibc for humans

Unless you’ve been living under a rock, you’ve heard about the latest glibc issue.
CVE-2015-7547 – glibc stack-based buffer overflow in getaddrinfo()

It’s always hard to understand some of these issues, so I’m going to do my best to explain it using simple language. Making security easy to understand is something I’ve been talking about for a long time now, it’s time to do something about it.

What is it?
The fundamental problem here is that glibc has a bug that could allow a DNS response from an attacker to run the command of that attacker’s choosing on your system. The final goal of course would be to become the root user.

The problem is that this glibc function is used by almost everything that talks to the network. In today’s hyperconnected world, this means basically everything is vulnerable to this bug because almost everything can connect to the network. As of this writing we have not seen this attack being used on the Internet. Just because there are no known attacks is no reason to relax though, constant vigilance is key for issues like this.

Am I vulnerable?
If you run Linux (most distributions use glibc), and you haven’t installed an update from your vendor, yes, you are vulnerable.

Are there workarounds?
No, there is no way to stop this issue. You have to install an update to glibc. Even the stack protector technology that is built into gcc and glibc will not stop this bug. While it is a stack overflow bug, the stack protector checks do not run before the exploit would gain control.

What about containers, VMs, or other confinement technology?
It is possible that a container, VM, or other technology such as SELinux could limit the possible damage from this bug. However it affects so many binaries on the system it should be expected that an attacker able to gain access to one applications could continue to exploit this bug to eventually become root and take over the entire machine.

Do I only need to be worried if I run a webserver or mailserver?
As stated previously, this bug affects virtually everything that talks to the network. Even if you think your webserver or mailserver are safe, everything from bash to your ssh client will use this library. Updating glibc is the only way to ensure you’ll be OK.

What if I run my own DNS server?
This point is currently under investigation. It is thought that it may be possible for a bad DNS request to be able to make it through a DNS server to a vulnerable host. Rather than find out, you should update your glibc.

What about …
No, just update your glibc 🙂

Do you have other questions? Ask me on twitter and I’ll be sure to update this article if I know the answer.

Does the market care about security?

I had some discussions this week about security and the market. When I say the market I speak of what sort of products will people or won’t people buy based on some requirements centered around security. This usually ends up at a discussion about regulation. That got me wondering if there are any industries that are unregulated, have high safety requirements, and aren’t completely unsafe?

After a little research, it seems SCUBA is the industry I was looking for. If you read the linked article (which you should, it’s great) the SCUBA story is an important lesson for the security industry. Our industry moves fast, too fast to regulate. Regulation would either hurt innovation or be useless due to too much change. Either way it would be very expensive. SCUBA is a place where the lack of regulation has allowed for dramatic innovation over the past 50 years. The article compares the personal aircraft industry which has substantial regulation and very little innovation (but the experimental aircraft industry is innovating due to lax regulation).

I don’t think all regulation is bad, it certainly has its place, but in a fast moving industry it can bring innovation to a halt. And in the context of security, what could you even regulate that would actually matter? Given the knowledge gaps we have today any regulation would just end up being a box ticking exercise.

Market forces are what have kept SCUBA safe, divers and dive shops won’t use or stock bad gear. Security today has no such bar, there are lots of products that would fall under the “unsafe” category that are stocked and sold by many. Can this market driven approach work for our security industry?

It’s of course not that simple for security. Security isn’t exactly an industry in itself. There are security products, then there are other products. If you’re writing a web app security probably takes a back seat to features. Buyers don’t usually ask about security, they ask about features. People buying SCUBA gear don’t ask about safety, they just assume it’s OK. When you run computer software today you either know it’s insecure, or you’re oblivious to what’s going on. There’s not really a happy middle.

Even if we had an industry body everyone joined, it wouldn’t make a huge difference today. There is no software that exists without security problems. It’s a wide spectrum of course, there are examples that are terrible and examples that do everything right. Today both groups are rewarded equally because security isn’t taken into account in many instances. Even if you do everything right, you will still have security flaws in your software.

Getting the market to drive security is going to be tricky, security isn’t a product, it’s part of everything. I don’t think it’s impossible, just really hard. SCUBA has the advantage of a known and expected use case. Imagine if that gear was expected to work underwater, in space, in a fire, in the arctic, and you have to be able to eat pizza while wearing it? Nobody would even try to build something like that. The flexibility of software is also its curse.

In the early days of SCUBA there were a lot of accidents, by moving faster than the regulators could, they not only made the sport extremely safe, but probably saved what we know as SCUBA today. If it was heavily regulated I suspect much of the technology wouldn’t look all that different from what was used 30+ years ago. Software regulation would probably keep things looking a like they do today, just with a lot of voodoo to tick boxes.

Our great challenge is how do we apply this lesson from SCUBA to security? Is there a way we can start creating real positive change that can be market driven innovation and avoid the regulation quagmire?

Join the conversation, hit me up on twitter, I’m @joshbressers

Security and Tribal Knowledge

I’ve noted a few times in the past the whole security industry is run by magicians. I don’t mean this in a bad way, it’s just how things work. Long term will will have to change, but it’s not going to be an easy path.

When I say everything is run by magicians I speak of extremely smart people who are so smart they don’t need or have process (they probably don’t want it either so there’s no incentive). They can do whatever needs to be done whenever it needs doing. The folks in the center are incredibly smart but they learned their skills on their own and don’t know how to pass on knowledge. We have no way to pass knowledge on to others, many don’t even know this is a problem. Magicians can be awesome if you have one, until they quit. New industries are created by magicians but no industry succeeds with magicians. There are a finite number of these people and an infinite number of problems.

This got me thinking a bit, and it reminded me of the Internet back in the early 90’s.

If you were involved in the Internet back in the 90’s, it was all magic back then. The number of people who knew how things worked was incredibly small. There were RFCs and books and product documents, but at the end of the day, it was all magic. If your magician quit, you were screwed until you could find and hire a new magician. The bar was incredibly high.

Sounds a lot like security today.

Back then if you had a web page, it was a huge deal. If you could write CGI scripts, you were amazing, and if you had root on a server you were probably a magician. A lot of sysadmins knew C (you had to), a lot of software was built from source. Keeping anything running was a lot of work, infrastructure barely held together and you had to be an expert at literally everything.

Today getting a web site, running server side scripts, or root aren’t impressive. You can get much of those things for free. How did we get here? The bar used to be really high. The bar is pretty low now but also a lot more people understand how much of this works. They’re not experts but they know enough to get things done.

How does this apply to security?

Firstly we need to lower the bar. It’s not that anyone really plans to do this, it just sort of happens via better tooling. I think the Linux distribution communities helped a lot making this happen back in the day. The tools got a lot better. If you configured a server in 1995 it was horrible, everything was done by hand. Now 80% of the work just sort of happens, you don’t need super deep knowledge. Almost all security work done these days is manual. I see things like AFL and LLVM as the start but we have a long way to go. As of right now we don’t know which tools are actually useful. There are literally thousands of security products on the market. Only the useful ones will really make a difference in the long term.

The second thing we need to do is transfer relevant knowledge. What that knowledge is will take time to figure out. Does everyone need to know how a buffer overflow exploit works? Probably not, but the tools will really determine who needs to know what. Today you need to know everything. In the future you’ll need to know how to use the tools, interpret the output, and fill in some of the gaps. Think of it as the tools having 80% of the knowledge, you just need to bring the missing 20%. Only the tool writers need to know that missing knowledge. Today people have 100% or 0% of knowledge, this is a rough learning curve.

If you look at the Internet today, there is a combination of tons of howtos and much better software to setup and run your infrastructure. There are plenty of companies that can help you build the solution you need. It’s not nearly as important to know now to configure your router anymore, there are better tools that do a lot of this for you. This is where security needs to go. We need tools and documents that are useful and helpful. Unfortunately we don’t yet really know how to make useful tools, or how to transfer knowledge. We have a long way to go before we can even start that conversation.

The next step security needs to make is to create and pass on tribal knowledge. It’s still a bad place to be in, but it’s better than magicians. We’ll talk about tribal knowledge in the future.

Join the conversation, hit me up on twitter, I’m @joshbressers

OpenSSH, security, and everyone else

If you pay attention at all, this week you heard about a security flaw in OpenSSH.

Of course nothing is going to change because of this. We didn’t make any real changes after Heartbleed or Shellshock, this isn’t nearly as bad, it’s business as usual.
Trying to force change isn’t the important part though. The important thing to think about is the context this bug exists in. The folks who work on OpenSSH are some of the brightest security minds in the world. We’re talking well above average here, not just bright. If they can’t avoid security mistakes, is there any hope for the normal people?
The answer no.
What do we do now?

For the moment we will continue to operate just like we have been. Things aren’t great, but they’re not terrible. Part of our problem is things aren’t broken enough yet, we’re managing to squeak by in most situations.

The next step will be developing some sort of tribal knowledge model. It will develop in a mostly organic way. Long term security will be a teachable and repeatable thing, but we can’t just jump to that point, we have to grow into it.

If you look at most of the security conference content today it sort of falls into two camps.

  1. Look at my awesome research
  2. Everything is broken and we can’t fix it
Both of these content sets are taught by magicians. They’re not really teaching knowledge, they’re showing off. How do we teach? Teaching is really hard to do, it’s not easy to figure out.
Many people believe security can’t be learned, it’s just sort of something you have. This is nonsense. There are many possible levels of skill, there is a point where you have to be especially gifted to move on, but there is also a useful place a large number of people can reach.
Perhaps the best place to start is to think about the question “I want to learn security, where do I start?”
I’ve been asked that many times. I’ve never had a good answer.
If we want to move our industry forward that’s what we have to figure out. If someone came to you asking how to learn security, we have to have an answer. Remember no idea is too crazy, if you have thoughts, let’s start talking about it.
Join the conversation, hit me up on twitter, I’m @joshbressers

What the lottery and security have in common

If you live in the US you can’t escape the news about the Powerball lottery. The jackpot has grown to $1.3 Billion (with a capital B). Everyone is buying tickets and talking about what they’ll do when they win enough money to ruin their life.

This made me realize the unfortunate truth about security we like to ignore. Humans are bad at reality. Here is how most of my conversations go.

“You won’t win. The odds are zero percent”
“I might! You don’t know!”

I’m of course labeled as being some sort of party pooper because I’m not creating stories about how I will burn through hundreds of millions of dollars in a few short weeks.

What does this have to do with security? It’s because people are bad at reality. Let’s find out why.

Firstly, remember that as a species evolution has built us to survive on the African Savannah. We are good at looking for horrible beasts in the grass, and begin able to quickly notice other humans (even if they appear in toast). We are bad at things like math and science because math rarely hides in the grass and eats people. The vast majority of people live their lives unaware of this as a problem. What we call “intuition” is simply “don’t get eaten by things with big teeth”.

Keeping this in mind, let’s use the context of the lottery. The odds are basically zero percent once you take the margin of error into account. We don’t care though, we want to believe that there’s a chance to win. Our brain says “anything is possible” then marketing helps back that up. Almost nobody knows how bad their odds really are and since you see a winner on TV every now and then, you know it’s possible, you could be next! The lottery ticket is our magic gateway to fixing all our problems.

Now switch to security. People are bad at understanding the problems. They don’t grasp any of the math involved with risk, they want to do something or buy something that is the equivalent of a lottery ticket. They want a magic ticket that will solve all their problems. There are people selling these tickets. The tickets of course don’t work.

How we fix this if the question. Modern medicine is a nice example. Long ago it was all magic (literally). Then by creating the scientific method and properly training doctors things got better. People stopped listening to the magicians (well, most people) and now they listen to doctors who use science to make things better. There is still plenty of quack medicine though, we want to believe in the magic cures. In general most of humanity goes to doctors when they’re sick though.

Today all security is magic. We need to find a way to create security science so methods and ideas can be taught.

Between thinking about how to best blow my lottery winnings, I’ll probably find some time to think about what security science looks like. Once I win though you’ll all be on your own. You’ve been warned!

Join the conversation, hit me up on twitter, I’m @joshbressers

A security analogy that works

Over the holiday break I spent a lot of time reading and thinking about what the security problem really is. It’s really hard to describe, no analogies work, and things just seem to keep getting worse.

Until now!


Well, things will probably keep getting worse, but I think I’ve found a way to describe this almost anyone can understand. We can’t really talk about our problems today, which makes it impossible to fix anything.

Security is the same problem as World Hunger. Unfortunately we can’t solve either, but in theory we can make things better. Let’s look at the comparisons.

First, the problem we talk about isn’t just one thing. It’s really hundreds or thousands of other problems we lump together into one group and give it a simple yet mostly meaningless name. The real purpose of the name is to give humans a single idea they can relate to. It’s not meant to make the problem more fixable, it just makes it so we can talk about it.

Security includes things like application security, operational security, secure development, secure documentation, pen testing, hacking, DDoS, and hundreds of other things.

World hunger includes homelessness, hunger, malnutrition, lack of education, clean water, and hundreds of other things.

Lots of little things.

Second, the name isn’t really the problem. It’s what we can see. It’s a symptom of other problems. The other problems are what you have to fix, you can’t fix the name.

What we call “security” is really other things, and the real problem is rarely security, it’s something else, security is the symptom we can see, the real problem is less obvious and hard to see.

In the context of world hunger the real problems are things like clean water, education, equality, corruption, crime, and the list goes on. Hunger is what we see, but to fix hunger, we have to fix those other problems.

We can give people food, but that doesn’t fix the real problem, it makes things better for a day or a week. This is exactly how security works today. We run from fire to fire, fixing a single easy to see problem, then run off to the next thing. We never solve any problems, we put out fires.

So assuming this analogy holds, the sort of good news is that world hunger is slowly getting better. The bad news is progress is measured in decades. This is where my thinking starts to falter. Trade can help bring more progress to a given area. What is the equivalent in security? Are there things that can help make the situation better for a localized area? Will progress take decades?

If I had to guess, which I will, I suspect we’re in the dark ages of security. We don’t approach problems with a scientific mind, we try random things until something works, and then decide that spinning around while holding a chicken is what fixed that buffer overflow.

What we need is “security science”. This means we need ways to approach security in a formal reproducible manner. A practice that can be taught and learned. Today it’s all magic, some people have magic, most don’t. Remember when the world had magicians instead of doctors? Things weren’t better back then no matter what those forwards from your uncle claims.

This all leaves a lot of unanswered questions, but I think it’s a starting point. Today we have no starting point, we have people complaining everything is broken, people selling magic, some have given up and assume this is how everything will just always be.

What will our Security Renaissance be? What will security science look like?

Join the conversation, hit me up on twitter, I’m @joshbressers

Security reminds me of the gym on January 2

If you have any sort of gym membership you dread the month of January. Every year, there are countless people who make a resolution to get in shape, so the gym is flooded with people for much of January. I’m in favor of everyone staying in shape and having a gym membership, my point isn’t to claim how annoying the n00bs are. The point of this story is how few people stick around, and most give up because doing nothing is often easier than doing something.

What does this have to do with security?

The parallel here worries me. Let’s use Heartbleed for our context.

After Heartbleed (January 1), everyone was talking about security, it was super important and everyone wanted more security (flooding the gym). After a while (February) most people stopped obsessing over security, a few stick around, most don’t. As a species we’re not really doing any better now than we were before Heartbleed. You could make some arguments, but it’s a rounding error at best.

The real issue here is this is how humans work. We love running to whatever is popular, pretending we always knew it was cool, and watching for whatever next hip thing will pop up for us to latch on to.

Our current security problems aren’t technology problems, they are human problems. We have to assume we can’t change human nature. The vast majority of people will never take security seriously. They know it’s important, they might even want to do it right, but at the end of the day they’re not going to do anything about it.

The only solution is to make secure the default option.

This is probably harder than changing human nature.

Can this problem actually be fixed? I’m not sure. I need to think about it. I don’t want to say no, but my crystal ball is pretty fuzzy here. There are a lot of weird problems all tied together in bizarre ways. I’m always happy to listen to new ideas, let me know if you have any. The more I learn the less I know seems to be the only constant.

Join the conversation, hit me up on twitter, I’m @joshbressers