Are you running a security scanner? It seems like everyone is doing it, maybe it’s time to get with it. It’s looking like automated security scanning is the next stage in the long winding history of the security industry. If you’ve never run one of these scanners that’s OK. I’m going to explain what they are, how they work, how we’re not using them correctly, and most importantly, what you can do about it. If you are running a scanner I’m either going to tell you why you’re doing it wrong, or why you’re doing it REALLY wrong. If you’re a vendor who builds a security scanner I assure you I understand there is a high probability I am indeed an idiot and don’t know what I’m talking about. I’m sure everything will be fine.

Automated scanning IS changing the world, but right now it’s not changing it for the better, it’s currently the security industry version of lead paint. The technology is still REALLY new, so it’s important we have proper expectations and work together to make things better. One of the challenges with new technology is understanding what you have now, and more importantly understanding what you need next. Like any tool, if you use it wrong it can make things worse than doing nothing at all. Let’s talk about how to make things better.

If you’ve never seen the sort of report an automated scanner generates you should probably consider yourself lucky. The best way to describe these reports is if you had a 10 page report that wasn’t very good, then you made 100 copies of every page, shuffled them around a bit and stapled it all together. There are some useful findings in the report, but they’re really hard to find. Expecting anyone to parse a 1000 page report for one or two findings has a terrible return on investment. It’s even less helpful if you send the report to someone else with unrealistic demands, such as requesting they fix all of the findings. By Friday. If you didn’t read the report, why should they?

There’s also the problem of incentives. Today a lot of report vendors talk about all the findings their scanner will … well, find. They fail to mention how many false positives are in those findings. In the case of security reports more findings is like bragging your house has the most lead paint. It’s not really a contest you want to win, and if you are winning you probably have a lot of work to do. That’s OK though, this is new technology trying to solve a REALLY hard and mostly unsolvable problem up to this point. Someday we’re going to look back on all this the same way we look back at food safety in the 1900’s. Asbestos was an ice cream flavor, motor oil counted as a vegetable, and nobody worried about where the meat came from.

There are a lot of moving parts in the scanner story, so I’m going to write a bunch of blogs posts to help explain the problem, explain what these scanners are doing, and finally what we can do about it.

The rough outline is going to look something like this

  1. Is your scanner running? You better go catch it!
    1. The quick win: I did something, and something is better than nothing!
    2. False positives
    3. Maybe positives
    4. Low quality positives
    5. False negatives
  2. Source code scanners
    1. I have software and nobody knows how it works
    2. The only thing harder than writing secure software is writing a code scanner
  3. Composition scanners
    1. Who let all this open source in?
    2. Updating dependencies is harder than not updating them
    3. Which of these security problems do I need to care about?
  4. Application scanning
    1. Scanning a running application is hard
    2. Scanning user interfaces
    3. Scanning APIs
  5. Which of these security problems do I need to care about?
    1. I ran the scan
    2. I was given a scan
    3. Falsifying false positives
    4. Parsing the positive positives
  6. What do we do now?
    1. Understand the problem we want to solve
    2. Push back on scanner vendors
    3. Work with your vendors
    4. Get involved in open source

Ending a blog post with an outline is pretty lame. Since writing good conclusions is hard work, I’m going to just link you to the first post of actual content in the series. I have a number of posts on this blog that talk about open source dependencies and the supply chain, I’m not going to torture you with links to all of them, I’m going to explain the problem with a slightly different angle in this series, if your time has very little value you can dig through the archives and see if there’s anything there worth reading.

Part 1: Is your security scanner running? You better go catch it!