Everything is fine, nothing to see here!

As anyone who reads this blog knows, I’ve been talking about soft skills in security for quite some time now. I’m willing to say it’s one of our biggest issues at the moment, a point which I get a lot of disagreement on. I have sympathy for anyone who thinks this stuff doesn’t matter, I used to be there. Until I had to start talking to people. As soon as you talk to most anyone outside the security echo chamber, you see what’s actually going on, and it’s not great.

I won’t say the security industry is one fire, but nobody is going to disagree many of the things we’re looking after aren’t in great shape.Outside of a few very large successful companies, most organizations have serious and significant security problems that could result in a massive breach, it’s just that nobody has tried, yet. I see a few reasons for many of our trouble, I always seem to come back to soft skills.

There is a skills shortage
But there’s training, look at all the training, there’s so much training everything is fine!

There is training. Some is good, some is bad (like anything). It’s not that training in itself is bad, I would encourage anyone to go get training. It’s not great though either. Most training today focuses on the symptoms of our problems. Things like pen testing, secure coding (which doesn’t exist), network defense. Things that while important, aren’t the real problems. I’ll talk more about this in a future post, but chew on this. There are about 96,000 CISSP holders, and about 5 million security jobs. That’s messed up.

Today everyone who is REALLY, I mean REALLY REALLY good at security got there through blood sweat and tears. Nobody taught them what they know, they learned it on their own. Many of us didn’t have training when we were learning these things. Regardless of this though, if training is fantastic, why does it seem there is a constant march toward things getting worse instead of better? That tells me we’re not teaching the right skills to the right people. The skills of yesterday don’t help you today, and especially don’t help tomorrow. By its very definition, training can only cover the topics of yesterday.

How do we skill up for the needs of today and tomorrow? The first thing we have to do is listen to the people running, building, and using the technology of today. They know things we don’t just as we know things they don’t. Security is still almost always an afterthought, even with everyone claiming it’s the most important thing ever. This is our failing, not theirs.

We build our skills by being an industry that doesn’t complain and belittle everyone who tries anything. We are notorious for being brutal to the new guys. Everyone starts somewhere, don’t be a jerk. I know a lot of people who are afraid to do almost anything in the security space because they know if they’re not 100% correct, they will have to deal with a torrent of negative comments. It’s not worth talking to us in many instances.

As an industry we are failing our customers
Things aren’t that bad, sure there are some breaches but in general everything is going pretty good!

If you read any news stories, you know things aren’t OK. There are loads of breaches and high profile security issues. Totally broken devices, phones that can’t be updated, light bulbs that can join a botnet. As an industry we like to stick to our echo chamber circles where we spin news and events into something that isn’t our fault. We laugh at the stupid people doing stupid things. We find a person or event that can explain away the incident as a singular event, not a systematic problem. The problems are growing exponentially, our resources are growing linearly, this means that our resources are actually decreasing every year.

Most organizations don’t have proper security and won’t even have a proper conversation until they end up on the wrong side of a major compromise. It’s our fault nobody is talking about this stuff, even if the breach isn’t technically our fault.

What advice are we giving people they can actually use? In almost every organization the security group is feared and hated. We’re not peers, we’re enemies, and they are ours. This isn’t helpful to anyone. How many of you actually sit down and have honest real discussions with those you are supposed to help. Do you actually understand their problems (not our problems with them, their actual problems, the ones they have to route around security to solve). Security shouldn’t be something bolted on later, we’re lucky if it’s even that in most cases.

Security is seen as a business prohibitor, not a business enabler
I know what needs to be done, nobody wants to listen!

We’ve all been here before. We suggest something to the group, they ignore us. We are the problem here, not the people we are supposed to help. We blame them for not listening when the real issue is we’re not talking to them properly. We throw information at people, complex hard to understand information, then rather than hold their hand when they don’t understand, we declare them stupid and go find someone who agrees with us, then we complain about how dumb everyone else is and how smart we are.

They aren’t stupid.

Neither are we.

The disconnect is one of talking. We have to talk to people, we have to engage with them. We have to build a relationship. You can’t expect to show up and be listened to if you’re not respected. People trust those they respect. If you’re in that circle of respect, you won’t be taken seriously. On a regular basis I hear security tell me “they’ll know I was right when we get hacked!” That doesn’t even make sense. It’s your failure for not creating a level of understanding for the issue, it’s not their fault for ignoring you.

Soft skills are hard
You don’t even know what you’re talking about, my skills are fine!

Maybe. I won’t say I’m an expert. I am constantly thinking about the state of things and how interactions go. What I do know though is the things I discuss here are based on my real world lessons. Every day is a new journey into being a new and better security person. I know how the technology works, what I don’t know is how people work. It’s a journey to figure this out. I’m pretty sure I’m on to something because people I respect are encouraging, yet there are some who are trying very hard to discourage this conversation. As the old saying goes, if nobody is complaining about what you’re doing, you’re not doing anything interesting.

Here’s what I do honestly believe. You can disagree with me or anyone you want. The industry isn’t solving the problems it needs to solve. Those problems will be solved eventually, there are many industry groups forming to start talking about some of these problems, the groups mostly talk though, that’s not a skill we’re good at. Even then I see a lot of criticism toward those groups. Problems won’t be solved quickly by doing the same thing we do today. I’m confident a big part of our future is humanizing security. Security today isn’t for humans, security tomorrow needs to be. We get there by cooperating, not by arguing and insulting.

Think I’m an idiot, let me know: @joshbressers

Containers are like sandwiches

During the RSA conference, I was talking about containers and it occurred to me we can think about them like a sandwich. Not so much that they’re tasty, but rather where does your container come from. I was pleased that almost all of the security people I spoke with understand the current security nightmare containers are. The challenge of course is how do we explain what’s going on to everyone else. Securtiy is hard and we’re bad at talking about it. They also didn’t know what Red Hat was doing, which is totally our own fault, but we’ll talk about that somewhere else.

But containers are sandwiches. What does that mean? Let’s think about them in this context. You can pick up a sandwich. You can look at it, you can tell basically what’s going on inside. Are there tomatoes? Lettuce? Ham? Turkey? It’s not that hard. There can be things hiding, but for the most part you can get the big details. This is just like a container. Fedora? Red Hat? Ubuntu? It has httpd, great. What about a shell? systemd? Cool. There can be scary bits hidden in there too. Someone decided to replace /bin/sh with a python script? That’s just like hiding the olives under the lettuce. What sort of monster would do such a thing!

So now that we have the image of a sandwich in our minds, let’s think about a few scenarios.

Find it on a bench
If you’re walking through the park and you see a sandwich just laying on a bench what would you do? You might look around, wondering who left this tasty delight, but you’re not going to eat it. Most people wouldn’t even touch it, who put it there, where did it come from, how old is it, does it have onions? So many questions and you honestly can’t get a decent answer. Even if someone could answer the questions, would you eat that sandwich? I certainly wouldn’t.

Finding a sandwich on a bench is the public container registry. If this is all you know, you wouldn’t think there’s anything wrong with doing this, but like the public registry, you don’t always know what you’re getting. I wonder how many of those containers saw update for the glibc flaw from a few weeks ago? It’s probably easier not knowing.

Get it from a scary shop with questionable ingredients
A long time ago I was walking around in New York and decided to hop into a sandwich shop for a quick bite. As I reached for the door, there was a notice from the health department. I decided to keep walking. Even if you can get your sandwich from a shop, if the shop is scary, you could find yourself in trouble.

There are loads of containers available out there you can download that aren’t trusted sources. Don’t download random containers from random places. It’s no different than trying to buy a sandwich from a filthy shop that has to shoo the rats out of the kitchen with a broom.

Get it from a nice shop that uses old ingredients
We’ve all seen those places selling sandwiches that look nice. The sign is painted, the windows are clean. When you walk in the tables are clean enough to eat off of! But then you order and it’s pretty clear everything is old and dried out. You might be able to sneak out the back door before the old man putting it together notices you’re not there anymore.

This is currently a huge danger in the container space. Containers are super hip right now so there are plenty of people doing work in this space. Many of these groups don’t even know they have a problem. The software in your containers is a lot like sandwich meat. After a few weeks it probably will start to smell, and after a month it’s going to do some serious damage to anyone who consumes it.

Be sure to ask your container supplier what they’re shipping, where it came from and how fresh it is. It would not be reasonable to ask “If this container was a sandwich would you eat it?”

Get it from a nice shop that uses nice ingredients
This is the dream. You walk into a nice shop. The nice person behind the counter takes your order and using the freshest ingredients possible constructs a sandwich shaped work of art. You take pictures and post them to all your friends explaining this sandwich is what your life was always missing and you didn’t know it before now.

This is why you need a partner you can trust when it comes to container content. The closer to the source you can get the better. Ask questions abut the content. Where did it come from? Who is taking care of it? How can I prove any of this? Who is updating it? Containers are a big deal, they’re new and exciting. They’re also very misunderstood. Only use fresh containers. If the content it more than a few months old, you’re eating a sandwich off a park bench. Don’t each sandwiches off park benches. Ask hard questions. If your vendor can’t answer them, you need to try the shop across the street. Part of the magic of containers is they are the result of truly commoditizing the operating system, you can get container content from a lot of sources, find a good one.

If we think about our infrastructure like we think about public health, you don’t want to be responsible for making everyone sick. You need to know what you’re using, where it came from, how fresh it is, who put it together, and what’s in it. It’s not enough to pretend everything is fine. Everything is not fine.

The interesting things from RSA are what didn’t happen, and containers are sandwiches

The RSA conference is done. It was a very long and busy show, there were plenty of interesting people there and lots of clever ideas and things to do.

I think the best part is what didn’t happen though. We love talking about the exciting things from the show, I’m going to talk about the unexciting non events I was waiting to happen (but thankfully they did not).

The DROWN issue came and went. It wasn’t very exciting, it got the appropriate amount of attention. Basically SSLv2 is still broken, don’t use it for any reasons. If you use SSLv2, it’s like licking the handrail at the airport. Nobody is going to feel bad for you.

There were keynotes by actors. The world continues to turn (pun intended). But really, these keynotes are about being entertaining, I didn’t go, because well, they’re actors 🙂 But I suspect they were entertaining. No doubt this will happen more and more as there are more and more security conferences, finding good keynotes will only get harder. They should hire that guy from the Hackers movie next.

There weren’t any exciting hacking events. Not that stunt hacking is a thing for RSA, I’m glad nobody tried anything new. I’m sure Blackhat will be a very different story. We shall wait and see.

And most importantly, I wasn’t booed off the stage 😛
I was pleased with how my talk went. Attendance was light but that’s expected on a Friday morning. The thing that made the happiest is that they had to kick our group out of the room for the next talk, not because I rambled on but because I got everyone in the room talking to each other. It was fantastic.

On to the interesting bit of the trip though. I found the most interest when I was talking about Red Hat’s concept of a trusted container registry. Today if you’re using the public registry it’s comparable to finding a sandwich on a bench at the park. You can look at it, you can tell it has ham and lettuce, but I mean, it’s a sandwich you found on a bench. Are you going to eat that?

If you want a nice sandwich you’re going to go to a sandwich shop, order a sandwich, and watch someone make it for you. You can then go and sit on the bench if you want.

The idea behind Red Hat’s trusted registry is we have a container registry for Red Hat customers. We control all the content in the registry, we know exactly what it is. We know where it came from. We control the sandwich supply chain from start to finish. No mystery meats here!

All the security people I talked to know that containers are currently a bit of a security circus. None of them knew what Red Hat was doing. This is of course a great opportunity for Red Hat to spread the word. Stay tuned for more clever sandwich jokes.

Let’s talk about soft skills at RSA, plus some other things

It’s been no secret that I think the lack of soft skills in the security space is one of our biggest problems. While usually I usually only write all about the world’s problems and how to fix them here, during RSA I’m going to take a somewhat different approach.

I’m giving a talk on Friday titled Why Won’t Anyone Listen to Us?

I’m going to talk about how a security person can talk to a normal person without turning them against us. We’re a group that doesn’t like talking to anyone, even each other. We need to start talking to people. I’m not saying we should stand around and accept abuse, I am saying the world wants help with security. We’re not really in a place to give it because we don’t like people. But they need our help, most of them know it even!

We’ve all had countless interactions where we give someone good, real advice, and they just completely ignore us. It’s infuriating sometimes. Part of the problem is we’re not talking to people, we’re throwing verbal information at them, and they ignore it. They listen to Oprah, if she told them about two factor auth everyone would be using it by the end of the week!

That’s just it, they listen to Oprah. They’re going to listen to anyone who talks to them in a way they understand. If it’s not us, it will be someone else, probably someone we don’t want talking about security.

I can’t teach you to like people (there are limits to my abilities), but I can help teach you how to talk to them. And of course a talk like this will need to have plenty of fun sprinkled in. How to talk to someone while being very important, can also be an extremely boring topic.

I touched on some of this during my Devconf.cz talk.

Red Hat is also putting on a Breakfast on Wednesday morning. I’m going to keynote it (I’ll keep it short and sweet for those of you attending, there’s nothing worse than a speaker at 8am who talks too much).

A coworker, Richard Morrell, is running a podcast from RSA called Locked Down. Be sure to give it a listen. I may even be able to convince him to let me on his show.

I have no doubt the RSA conference will be a great time. If you’re there come find me, Red Hat has a booth, North Expo #N3038. Come say hi, or not if you don’t like talking to people 😉

There or not, feel free to start a conversation on Twitter. I’m @joshbressers

Thinking about glibc and Heartbleed, how do fix things

After my last blog post Change direction, increase speed! (or why glibc changes nothing) it really got me thinking about how can we start to fix some of this. The sad conclusion is that nothing can be fixed in the short term. Rather than trying to make up some nonsense about how to fix this, I want to explain what’s happening and why this can’t be fixed anytime soon.

Let’s look at Heartbleed first.

There was a rather foul flaw found in OpenSSL, after Heartbleed the Linux Foundation collected a lot of money to help work on core infrastructure projects. If we look at the state of things it basically hasn’t changed outside of money moving around. OpenSSL cannot be fixed for a number of reason.

  1. Old codebase
  2. Backward compatibility
  3. Difficult API
  4. It is “general purpose”
The reality is that the only way to get what could be considered a safe library, you would have to throw everything out and start over with some very specific ideas in mind. Things like sun-setting algorithms didn’t exist when OpenSSL was created. There is no way you’re going to get even a small number of projects to move from using OpenSSL to some new “better” library. It would have to be so much better they couldn’t ignore it. As anyone who has ever written software knows, you don’t build a library like that over night. I think 5 years would be a conservative estimate for double digit adoption rates.
While I’m picking on OpenSSL here, the story is the same in virtually every library and application that exists. OpenSSL isn’t special, it just gets a lot of attention.
Let’s think about glibc.
Glibc is the C library used by most Linux distributions. If the Kernel is the heart, glibc is the soul. Nothing can even exist without this library. Glibc even strives to be POSIX compliant, for good reason. POSIX has given us years of compatibility and cooperation.
Glibc probably has plenty more horrible bugs hiding in the code. It’s wicked complex and really large. If you ever need some nightmare fuel, look at the glibc source code. Everything we do in C relies on a libc being around, glibc doesn’t have that luxury.
Replacing libc is probably out of the question, it’s just not something practical. So let’s think about something like golang. What if everything was written using golang? It’s not totally insane, there are substantial benefits. It’s not as fast as C though, that will be the argument most use. Golang will probably never beat C, the things that makes it safer also makes it slower. But now if we think about replacing UNIX utilities with golang, why would we want to do that? Why not throw out all the mistakes UNIX made and do something else?
Now we’re back to the legacy and compatibility arguments. Linux has more than twenty years of effort put into it. You can’t just replace that. Even if you had the best team in the world I bet 10 years would be wishful thinking for having half the existing features.
So what does this mean? It means we don’t know where to start yet. We are trying to solve our problems using what we know and the tools that exist. We have to solve this using new tools and new thinking. The way we fix security is by doing something nobody has ever thought of before. In 1920 the concept of the Internet didn’t exist, people couldn’t imagine how to even solve some of the problems we can easily solve using it. Don’t try to solve our problems with what you know. Solve the problems using new ideas, find out what you don’t know, that’s where the solution lives.

Change direction, increase speed! (or why glibc changes nothing)

The glibc issue has had me thinking. What will we learn from this?

I’m pretty sure the answer is “nothing”, which then made me wonder why this is.

The conclusion I came up with is we are basically the aliens from space invaders. Change direction, increase speed! While this can give the appearance of doing something, we are all very busy all the time. It’s not super useful when you really think about it. Look at Shellshock, Heartbleed, GHOST, LOGJAM, Venom, pick an issue with a fancy name. After the flurry of news stories and interviews, did anything change, or did everyone just go back to business as usual? Business as usual pretty much.

Dan Kaminski explains glibc nicely and has some advice. But let’s look at this honestly. Is anything going to change? No. Dan found a serious DNS issue back in the day. Did we fix DNS or did we bandage it up as best as we could? We bandaged it. What Dan found was without a doubt as bad or worse than this glibc issue, nothing changed.

I’ve said this before, I’ll say it again. I’m going to say it at RSA next week. We don’t know how to fix this. We think we do, you’re thinking about it right now, about how we can fix everything! We just have to do that one … Except you can’t. We don’t really know what’s wrong. Security bugs aren’t the problem, they are the result of the problem. We can’t fix all the security bugs. I’d be surprised if we’ve even fixed 10% of security bugs that exist. Even mitigation technologies aren’t going to get us there (they are better than constantly fixing bugs, but that’s a story for another day).

It’s like being obsessed about your tire pressure when there is a hole in the tire. If you only worry about one detail, the tunnel vision makes you miss what’s actually going on. Our tires are losing air faster than we can fix them, so we’re looking for a bigger pump instead of new tires.

We say things all the time about not using C anymore, or training developers, or writing better documentation. There’s nothing wrong with these ideas exactly, but the fact is they’ve all been tried more than once and none of them work. If we started the largest developer education program ever, if we made every developer sit through a week of training. I bet it would be optimistic if our bug rate would decrease by 5%. Think about that for a while.

We first have to understand our problem. We have lots of solutions, solutions to problems that don’t really exist. Solutions without problems tend to turn into new problems. We need to understand our security problem. It’s probably more like hundreds or thousands of problems. Every group, every company, every person has different problems. We understand none of them.

We start by listening. We’re not going to fix any of this with code. We need to see what’s happening, some big picture, some in the weeds. Today we show up, yell at people (if they’re lucky), then we leave. We don’t know what’s really happening. We don’t tell anyone what they need to know. We don’t even know what they need to know. The people we’re not talking to know what the problems are though. They don’t know they know, we just have to give them time to explain it to us.

If you’re at RSA next week, come talk to me. If not, hit me up on twitter @joshbressers

glibc for humans

Unless you’ve been living under a rock, you’ve heard about the latest glibc issue.
CVE-2015-7547 – glibc stack-based buffer overflow in getaddrinfo()

It’s always hard to understand some of these issues, so I’m going to do my best to explain it using simple language. Making security easy to understand is something I’ve been talking about for a long time now, it’s time to do something about it.

What is it?
The fundamental problem here is that glibc has a bug that could allow a DNS response from an attacker to run the command of that attacker’s choosing on your system. The final goal of course would be to become the root user.

The problem is that this glibc function is used by almost everything that talks to the network. In today’s hyperconnected world, this means basically everything is vulnerable to this bug because almost everything can connect to the network. As of this writing we have not seen this attack being used on the Internet. Just because there are no known attacks is no reason to relax though, constant vigilance is key for issues like this.

Am I vulnerable?
If you run Linux (most distributions use glibc), and you haven’t installed an update from your vendor, yes, you are vulnerable.

Are there workarounds?
No, there is no way to stop this issue. You have to install an update to glibc. Even the stack protector technology that is built into gcc and glibc will not stop this bug. While it is a stack overflow bug, the stack protector checks do not run before the exploit would gain control.

What about containers, VMs, or other confinement technology?
It is possible that a container, VM, or other technology such as SELinux could limit the possible damage from this bug. However it affects so many binaries on the system it should be expected that an attacker able to gain access to one applications could continue to exploit this bug to eventually become root and take over the entire machine.

Do I only need to be worried if I run a webserver or mailserver?
As stated previously, this bug affects virtually everything that talks to the network. Even if you think your webserver or mailserver are safe, everything from bash to your ssh client will use this library. Updating glibc is the only way to ensure you’ll be OK.

What if I run my own DNS server?
This point is currently under investigation. It is thought that it may be possible for a bad DNS request to be able to make it through a DNS server to a vulnerable host. Rather than find out, you should update your glibc.

What about …
No, just update your glibc 🙂

Do you have other questions? Ask me on twitter and I’ll be sure to update this article if I know the answer.
@joshbressers

Does the market care about security?

I had some discussions this week about security and the market. When I say the market I speak of what sort of products will people or won’t people buy based on some requirements centered around security. This usually ends up at a discussion about regulation. That got me wondering if there are any industries that are unregulated, have high safety requirements, and aren’t completely unsafe?

After a little research, it seems SCUBA is the industry I was looking for. If you read the linked article (which you should, it’s great) the SCUBA story is an important lesson for the security industry. Our industry moves fast, too fast to regulate. Regulation would either hurt innovation or be useless due to too much change. Either way it would be very expensive. SCUBA is a place where the lack of regulation has allowed for dramatic innovation over the past 50 years. The article compares the personal aircraft industry which has substantial regulation and very little innovation (but the experimental aircraft industry is innovating due to lax regulation).

I don’t think all regulation is bad, it certainly has its place, but in a fast moving industry it can bring innovation to a halt. And in the context of security, what could you even regulate that would actually matter? Given the knowledge gaps we have today any regulation would just end up being a box ticking exercise.

Market forces are what have kept SCUBA safe, divers and dive shops won’t use or stock bad gear. Security today has no such bar, there are lots of products that would fall under the “unsafe” category that are stocked and sold by many. Can this market driven approach work for our security industry?

It’s of course not that simple for security. Security isn’t exactly an industry in itself. There are security products, then there are other products. If you’re writing a web app security probably takes a back seat to features. Buyers don’t usually ask about security, they ask about features. People buying SCUBA gear don’t ask about safety, they just assume it’s OK. When you run computer software today you either know it’s insecure, or you’re oblivious to what’s going on. There’s not really a happy middle.

Even if we had an industry body everyone joined, it wouldn’t make a huge difference today. There is no software that exists without security problems. It’s a wide spectrum of course, there are examples that are terrible and examples that do everything right. Today both groups are rewarded equally because security isn’t taken into account in many instances. Even if you do everything right, you will still have security flaws in your software.

Getting the market to drive security is going to be tricky, security isn’t a product, it’s part of everything. I don’t think it’s impossible, just really hard. SCUBA has the advantage of a known and expected use case. Imagine if that gear was expected to work underwater, in space, in a fire, in the arctic, and you have to be able to eat pizza while wearing it? Nobody would even try to build something like that. The flexibility of software is also its curse.

In the early days of SCUBA there were a lot of accidents, by moving faster than the regulators could, they not only made the sport extremely safe, but probably saved what we know as SCUBA today. If it was heavily regulated I suspect much of the technology wouldn’t look all that different from what was used 30+ years ago. Software regulation would probably keep things looking a like they do today, just with a lot of voodoo to tick boxes.

Our great challenge is how do we apply this lesson from SCUBA to security? Is there a way we can start creating real positive change that can be market driven innovation and avoid the regulation quagmire?

Join the conversation, hit me up on twitter, I’m @joshbressers

Security and Tribal Knowledge

I’ve noted a few times in the past the whole security industry is run by magicians. I don’t mean this in a bad way, it’s just how things work. Long term will will have to change, but it’s not going to be an easy path.

When I say everything is run by magicians I speak of extremely smart people who are so smart they don’t need or have process (they probably don’t want it either so there’s no incentive). They can do whatever needs to be done whenever it needs doing. The folks in the center are incredibly smart but they learned their skills on their own and don’t know how to pass on knowledge. We have no way to pass knowledge on to others, many don’t even know this is a problem. Magicians can be awesome if you have one, until they quit. New industries are created by magicians but no industry succeeds with magicians. There are a finite number of these people and an infinite number of problems.

This got me thinking a bit, and it reminded me of the Internet back in the early 90’s.

If you were involved in the Internet back in the 90’s, it was all magic back then. The number of people who knew how things worked was incredibly small. There were RFCs and books and product documents, but at the end of the day, it was all magic. If your magician quit, you were screwed until you could find and hire a new magician. The bar was incredibly high.

Sounds a lot like security today.

Back then if you had a web page, it was a huge deal. If you could write CGI scripts, you were amazing, and if you had root on a server you were probably a magician. A lot of sysadmins knew C (you had to), a lot of software was built from source. Keeping anything running was a lot of work, infrastructure barely held together and you had to be an expert at literally everything.

Today getting a web site, running server side scripts, or root aren’t impressive. You can get much of those things for free. How did we get here? The bar used to be really high. The bar is pretty low now but also a lot more people understand how much of this works. They’re not experts but they know enough to get things done.

How does this apply to security?

Firstly we need to lower the bar. It’s not that anyone really plans to do this, it just sort of happens via better tooling. I think the Linux distribution communities helped a lot making this happen back in the day. The tools got a lot better. If you configured a server in 1995 it was horrible, everything was done by hand. Now 80% of the work just sort of happens, you don’t need super deep knowledge. Almost all security work done these days is manual. I see things like AFL and LLVM as the start but we have a long way to go. As of right now we don’t know which tools are actually useful. There are literally thousands of security products on the market. Only the useful ones will really make a difference in the long term.

The second thing we need to do is transfer relevant knowledge. What that knowledge is will take time to figure out. Does everyone need to know how a buffer overflow exploit works? Probably not, but the tools will really determine who needs to know what. Today you need to know everything. In the future you’ll need to know how to use the tools, interpret the output, and fill in some of the gaps. Think of it as the tools having 80% of the knowledge, you just need to bring the missing 20%. Only the tool writers need to know that missing knowledge. Today people have 100% or 0% of knowledge, this is a rough learning curve.

If you look at the Internet today, there is a combination of tons of howtos and much better software to setup and run your infrastructure. There are plenty of companies that can help you build the solution you need. It’s not nearly as important to know now to configure your router anymore, there are better tools that do a lot of this for you. This is where security needs to go. We need tools and documents that are useful and helpful. Unfortunately we don’t yet really know how to make useful tools, or how to transfer knowledge. We have a long way to go before we can even start that conversation.

The next step security needs to make is to create and pass on tribal knowledge. It’s still a bad place to be in, but it’s better than magicians. We’ll talk about tribal knowledge in the future.

Join the conversation, hit me up on twitter, I’m @joshbressers

OpenSSH, security, and everyone else

If you pay attention at all, this week you heard about a security flaw in OpenSSH.

Of course nothing is going to change because of this. We didn’t make any real changes after Heartbleed or Shellshock, this isn’t nearly as bad, it’s business as usual.
Trying to force change isn’t the important part though. The important thing to think about is the context this bug exists in. The folks who work on OpenSSH are some of the brightest security minds in the world. We’re talking well above average here, not just bright. If they can’t avoid security mistakes, is there any hope for the normal people?
The answer no.
What do we do now?

For the moment we will continue to operate just like we have been. Things aren’t great, but they’re not terrible. Part of our problem is things aren’t broken enough yet, we’re managing to squeak by in most situations.

The next step will be developing some sort of tribal knowledge model. It will develop in a mostly organic way. Long term security will be a teachable and repeatable thing, but we can’t just jump to that point, we have to grow into it.

If you look at most of the security conference content today it sort of falls into two camps.

  1. Look at my awesome research
  2. Everything is broken and we can’t fix it
Both of these content sets are taught by magicians. They’re not really teaching knowledge, they’re showing off. How do we teach? Teaching is really hard to do, it’s not easy to figure out.
Many people believe security can’t be learned, it’s just sort of something you have. This is nonsense. There are many possible levels of skill, there is a point where you have to be especially gifted to move on, but there is also a useful place a large number of people can reach.
Perhaps the best place to start is to think about the question “I want to learn security, where do I start?”
I’ve been asked that many times. I’ve never had a good answer.
If we want to move our industry forward that’s what we have to figure out. If someone came to you asking how to learn security, we have to have an answer. Remember no idea is too crazy, if you have thoughts, let’s start talking about it.
Join the conversation, hit me up on twitter, I’m @joshbressers