This post is the first part in a series on automated security scanners. I explain some of the ideas and goals in the intro post, rather than rehashing that post as filler, just go read it, rehashing content isn’t exciting.

There are different kinds of security scanners, but the problem with all of them is basically the same. The results returned by the scanners are not good in the same way catching poison ivy is not good. The more you have, the worse it is. The most important thing to understand, and the whole reason I’m writing this series, is that scanners will get better in the future. How they get better will be driven by all of us. If we do nothing, they will get better in a way that might not make our lives easier. If we can understand the current shortcomings of these systems, we can better work with the vendors to improve them in ways that will benefit everyone.

The quick win: I did something, and something is better than nothing!

One of the easiest problems we can see when running a security scanner is the idea that doing anything is better than doing nothing. Sometimes this is true, sometimes it’s not. You have to decide for yourself if running a scanner is better than not running a scanner, every organization is different. I see a common theme in the security industry where we take actions but then never follow through. Just running a scanner isn’t the goal, the goal is making our products and services more secure.

If you’re running scans and not reviewing the output, doing something is not better than doing nothing. What I see on a regular basis is a security team handing a phone book sized scan report to the development team, demanding they fix everything clearly not having even looked at it. If you can’t be bothered to review the massive report, why should the dev team? A good dev team will push back with a firm “no”. Even if they wanted to review all the findings, practically speaking it can’t be done in a reasonable amount of time.

So what do we expect to see in the report we didn’t read?

False positives

False positive findings are probably the single biggest problem with security scanners today. A false positive is when the scanner flags a particular line of source code, or dependency, or application behavior, as being a security problem. A very high percentage of the time whatever findings the report spits out will be a false positive. This is problematic as dealing with false positive results have a cost.

No scanner will ever have zero false positives, but a scanner than has magnitudes more false positives than actual positives is a broken scanner. Anything it tells you should be treated with skepticism. I did some searching to find what other industries consider an acceptable false rate to be. I couldn’t find anything I want to link to, it’s all quite dull, but the vast majority of industries seem to find somewhere around 1%-10% in the acceptable category. If I had to guess, most scanners today have a false positive rate very close to 100%. I’ve seen many scan reports where all of the findings were false positives. This is not acceptable.

Our job as users of security scanners is to report false positives to whoever creates the scanner. They won’t get better if we don’t tell them what we’re seeing. When you’re paying a company for their product it’s quite appropriate to make reasonable requests. One challenge any product team has is receiving a large number of unrelated requests. When you have 100 customers with 100 different feature requests it’s really hard to prioritize. If you have 100 customers with the same request, the product team can have a razor sharp focus. We should all be asking our scanner vendors for fewer false positives. Every false positive is a bug. Bugs should be reported.

Maybe positives

There’s another call of issues I’ve seen from security scanners that I’m going to call “maybe” positives. The idea here is the scanner flags something and when a human reviews the results they can’t tell if it is or isn’t a problem.

These are tough to deal with, and they can be dangerous. I bring it up because this problem created some trouble for Debian some time ago. DSA-1571 was a security flaw that resulted in OpenSSL generating predictable secret keys. The patch that caused the bug was the result of fixing a warning from the compiler. Sometimes warnings are OK, it can be very difficult to know when.

The reason I include this is to warn that fixing errors just to make the scanner be quiet can be dangerous. You’re better off fixing the scanners than trying to fix code you don’t really understand just to make the scanner results go away.

Low quality positives

One of the other big problems you will see from security scanners is findings that are positives, but they’re not really security problems. I call these “low quality positives.” They are technically positives, but they shouldn’t be.

An easy example I saw recently was a scanner claiming a container had a vulnerable package in it. The vulnerability didn’t have a CVE ID. After some digging I managed to find a link to the upstream bug. The bug was closed by the upstream project as “not a bug”. That means the findings from the scanner wasn’t actually a vulnerability (or a bug). One could argue the scanner may have added the bug before it was marked “not a bug.” I would argue back they should be double checking these findings to know when the bug gets fixed (or closed in this instance).

These scanners should be getting CVE IDs for their findings. If a scanner isn’t a CVE naming authority (CNA), you should ask them why not. Findings that don’t have CVE IDs are probably low quality findings. Actual security issues will get CVE IDs. If the security issue can’t get a CVE ID, it’s not a security issue.

False negatives

The last point I want to cover is false negatives. A false negative is when there is a vulnerability present in a project but the scanner doesn’t report it. It is inevitable that there will always be false negatives with every scanner, and it’s very difficult to figure out false negatives.

Something we can do here is to try out different scanners every now and then. If you only ever run one scanner you will have blind spots. Running a different scanner can give you some insight into what your primary scanner may be missing.

Wrapping up

If you take away anything from this post I hope it’s that the most important thing we can do about the currently quality problems with security scanners is to report the low quality results as bugs. I think we are currently treating these unacceptable reports with more seriousness than they deserve. Remember this industry is in its infancy, it will grow up, but without guidance it could grow into a horrible monster. We need to help nurture it into something beautiful. Or at least not completely ugly.

Part 2 is when we start to talk about specific types of scanners. Source code scanners are up next.