These are certainly strange times we are living in. None of us will ever forget what’s happening and we will all retell stories for the rest of our days. Many of us asked “tell me about the depression grandma”, similar questions will be asked of us someday.
The whirlwind of confusion and chaos got me thinking about advice and who we listen to. Most of us know a staggering number of people who are apparently experts in immunology. I have no intention of talking about the politics of the current times, goodness knows nobody in their right mind should care what I think. What all this does have me pondering is what are experts and how can we decide who we should listen to?
So I’ve been thinking a lot about “experts” lately. Especially in the context of security. There have been a ton of expert opinions on how to work from home, and how to avoid getting scammed, which video conferencing software is the best (or worst). There are experts everywhere, but which ones should we listen to? I’m not an expert in anything, but there are some topics I know enough about to question some of these “experts”.
It seems like everyone has something to say about almost everything these days. It feels a bit like the market outside the train station. Whatever you need, someone is selling it, but you better buy it fast because everyone else also wants one!
I have a tweet from a few weeks ago when I really started to think about all this, I called it “distance to the work”
The basic idea is if someone is trying to post themselves as an expert on a topic, how close are actually to the topic. One of my favorite examples is when I see talks about DevSecOps. I’ve known people who have given DevSecOps talks that have never been developers or system administrators or worked in the field of security. In my mind you aren’t qualified to impart knowledge you don’t have. There are certain ideas they can grasp and understand, but part of being an expert at something is having done it, often for a long time. Would you let someone operate on you because they thought about the problem really hard and decided they are now a surgeon? Of course not!
So this brings to a place where we have to start deciding who we should be listening to. I like to break people up into a few groups in my mind when deciding if they should be listened to.
- Have they ever done actual work in this space?
- Do they have a history of doing work in this space, but aren’t currently?
- Are they doing work in this space now?
It’s not hard to see where I’m going with this. I think we all know people who fall into every group. It’s very related to my distance to the work idea. If someone has never done the work, I’m not going to consider them an expert. One the poster children for this is whenever someone titles themselves a “thought leader”. That’s usually double speak for “I have no idea what I’m doing but I have very nice clothes and speak very well”. For a number of these people, their primary skill is speaking well, so they can sound smart, but they can’t fool the real experts.
There are also groups of people who did a lot of work in a space long ago, but aren’t very active now. An easy example here would be the Apollo astronauts. Are these people experts on going to the moon? Yes. Are they experts on space? Yes. Would I trust them to help build a modern day rocket? Probably not.
There are plenty of parallels here in any industry. There are plenty of people who did amazing things a decade ago, but if you look at what they’ve done recently, a resume of “talking about the awesome thing I did a decade ago” doesn’t make them an expert on modern day problems. Look at what people are doing now, not what they did.
And lastly we have our group of people who are actual doing the work. These are the people who are making a real difference every day. Many of these people rarely talk about what they do, many don’t have time because they’re busy working. I find there are two challenges when trying to listen to the people doing the real work.
Firstly, they’re usually drown out by other making more noise. If your job is getting attention, your incentive is, well, getting attention. When your job is doing technical tasks, you’re not going to fight for attention. This means it’s up to us as the listener to decide who is full of gas and who can teach us new things. It’s a really hard problem.
The second problem is finding the people doing the work. They aren’t going to a lot of conferences. They’re usually not publishing blog articles 😎. You won’t find them on social media with millions of followers. A lot actively avoid attention for a variety of reasons. Some don’t have time, some got burnt and don’t want to stick their neck out, some just don’t want to talk to anyone else. The reason is unimportant, it is what it is.
I could end this one with some nonsense about getting outside your comfort zone and making more effort to encourage other to talk about what they’re doing, but I don’t want to. If people don’t want to give talks and write blogs, great, I’m tired of seeing an industry that bases success on how many conferences you attend each year. My suggestion this time is to just look around. You are working with people who are making a real difference. Find them. Talk to them (don’t be a pest). Go learn something new.
Well, we’ve made it to the end. What started out as a short blog post ended up being 7 posts long. If you made it this far I commend you for your mental fortitude.
I’m going to sum everything up with these 4 takeaways.
- Understand the problem we want to solve
- Push back on scanner vendors
- Work with your vendors
- Get involved in open source
Understand the problem we want to solve
In security it’s sometimes easy to lose sight of what we’re really trying to do. Running a scanner isn’t a goal in itself, the goal is to improve security, or it should be if it isn’t. Make sure you never forget what’s really happening. Sometimes in the excitement of security, the real reason we’re doing what we do can be lost.
I always hate digging out the old trope “what’s the problem we’re trying to solve” but in this instance I think it’s a good question to ask yourself. Defining problems is really hard. Staying on goal is even harder.
If we think our purpose is to run the scanners, what becomes our goal? The goal will be to have a clean scan. We know a clean scan is impossible, so what really happens is our purpose starts to twist itself around a disfigured version of reality. I’ve said many times the problem is really insecure applications, or at least that’s the problem I tend to think about. You have to figure this out for yourself. If you have a scanner running make sure you know why.
Push back on scanner vendors
When a scan has 80% false positives, that’s not because your project isn’t built well, it’s because the scanner has a lot of serious bugs. High false positive rates mean the product is broken, it doesn’t mean your project is broken. Well, it might be, but probably not. The security industry has come to accept incredibly high false positive rates as normal. We have to break this cycle. It holds the industry back and it makes our jobs horrible. If you are a scanner vendor go make a sign that says “ZERO FALSE POSITIVES” and hang it on the wall. That’s your new purpose.
Setup a weekly or monthly call with your vendor. Make sure they understand your purpose and goals (remember your purpose isn’t just to run a scanner). Make them help you through rough patches. If you feel pain because of their product, they should feel it with you. A vendor who won’t work with you is a vendor who needs to be replaced. Good vendors are your partner, your success is their success. Your pain is their pain.
Work with your vendors
Now, when you find a scanner that has a lot of bugs, you basically have two choices. You can give up (and sometimes this is an acceptable decision). Or you can work with the vendor. At this stage in the technology, it’s very important we start working with these vendors. Report every false positive as a bug. Make them answer hard questions about the results you see. If nobody pushes back we’re going to see worse results in the future, not better. Products improve because of feedback. They don’t improve if we all just pretend everything is fine. I do think part of the reason code and application scanning seems to have plateaued is because we accepted poor results as normal.
If you are a vendor, remember that reports about false positives are gifts. Make sure you treat false positives like they are important bugs. Like all new markets, there will be winners and there will be losers. If your scanner reports the most results, but most of those are false positives, that’s not progress. The scanners with the most false positives will be on the losing side of history.
Get involved in open source
And lastly, help. This sounds like the old “patches welcome” we all love to throw around, but in this case I’m quite serious. Your product is basically open source. Some of the projects you are working with could use a hand to fix some of these findings. As annoying as you think a huge scan report is, imagine getting one when you’re working for free. It’s insulting and degrading. If you have actual real scan results that need fixing in an open source project, don’t dump it over the fence, get in there and help fix problems. If you include a dependency in your project, sending a patch upstream is the same as patching your own application.
If you’ve never contributed to open source it can be a terrifying. I just spent longer than I want to admit trying to find a nice “getting started in open source” guide. I wasn’t terribly impressed with any of what came up (if you know of one let me know, I’ll link it at the bottom). I’m going to start writing a post titled “How to get involved in open source for security people”, but until then, my advice is just go help. You know how github works, get in there and help. Be patient and kind, apologize when you mess up and don’t be afraid to ask stupid questions. Some say there are no stupid questions. There totally are, but that’s not a reason to be a jerk when asking or answering.
The one ask I have of everyone reading this is to help educate other on this extremely complicated an important topic of security scanners. It’s important we approach others with empathy and understanding. Security has a long history of being ill tempered and hard to work with. If someone is misunderstanding how a security scanner works or what it does, it’s our opportunity to help them understand. These scanners are too important to ignore, and they need a lot of work. We don’t change an industry by being annoying idiots. We change it by being respected partners.
I think we have an opportunity to see composition scanners make things better in the future. Composition security has been a taboo topic for a very long time. Many of us knew what was happening but we didn’t do anything because we didn’t have a solution. Security through obscurity works, until it doesn’t. There’s a lot of work to do, and we have to do it together.
Now go and teach someone something new.
If you just showed up here, go back and start at the intro post, you’ll want the missing context before reading this article. Or not, I mean, whatever.
I’ve spent the last few posts going over the challenges of security scanners. I think the most important takeaway is we need to temper our expectations. Even a broken clock is right twice a day. So assuming some of the security flaws reported are real, how can we figure out what we should be paying attention to?
I ran the scan
If you ran a security scanner, running it is the easy part. What you do with the results of your scan is a challenge. I’ve seen teams just send the scan along to the developers without even looking at it. Never do this. This tells your developers two very important thing. 1) You think your time is worth more than theirs. 2) You aren’t smart enough to parse the scan. Even if one or both of these are true, don’t just dump these scans on someone else. If you ran it, you own it. Suddenly that phone book of a scan is more serious.
When you have the result of any security report, automated or human created, how you deal with the results depends on a lot of factors. Every organization has different process, different resources, and different goals. It’s super important to keep in mind the purpose of your organization, resolving security scan reports probably isn’t one of them. Why did you run this scan in the first place? If you did it because everyone else is doing it, reading this blog series isn’t going to help you. Fundamentally we want to run these scanners to make our products and services more secure. That’s the context we should read these reports. Which of these findings make my product or service less secure? And which findings should I fix to make it more secure?
I was given a scan
If you were given a scan, good luck. As I mention in the previous section. If you were given one of these scans and it’s pretty clear the person giving it to you didn’t read it, there’s nothing wrong with pushing back by asking for some clarification. There’s nothing more frustrating than someone handing you a huge scan with the only comment being “please fix”. As we’ve covered at length, a lot (almost all) of these results are going to be false positives. Now you have to weed through someone else’s problem and try to explain what’s happening.
I’ve seen cases where a groups claim they can’t run an application unless the scan comes back clean. That’s not a realistic goal. I would compare it to only buying computers that don’t crash. You can have it as a requirement, but you aren’t going to find one no matter how hard you try. Silly requirements lead to silly results.
Falsifying false positives
If you ran the scan or you were handed a scan, one of the biggest jobs will be figuring out which results are false positives. I don’t know of a way to do this that isn’t backbreaking manual labor. Every finding has a number of questions that you have to answer “yes” to in order for the finding to matter.
- Do you actually include the vulnerable dependency?
- Is version you’re using is affected by the issue?
- Are you use the feature in your application?
- Can attackers exploit the vulnerability?
- Can attackers use the vulnerability to cause actual harm?
As humans it’s hard work to do these steps, it’s likely you can’t do them by yourself. Find some help, don’t try to do everything yourself.
One really important thing to do as you are answering these questions is to document your work. Write down as much detail as you can because in three months you’re not going to remember any of this. Also, don’t use whatever scanner ID you get from the vendor, use the CVE ID. Every scanner should be reporting CVE IDs (if they don’t, that’s a bug you should report). Then if you run a second scanner you can know right away if something has already been investigated since you’ve already documented the CVE ID. Using only scanner IDs isn’t useful across vendors.
Parsing the positive positives
Let’s make the rather large leap from running a scan to having some positive positives to deal with. The false positives have been understood, or maybe the scanners have all been fixed so there aren’t any false positives! (har har har) Now it’s time to deal with the actual findings.
The first and most important thing to understand is all of the findings aren’t critical. There is going to be a cornucopia of results. Some will be critical, some will be low. Part of our job is to rank everything in an order that makes sense.
Don’t trust the severity the scanner gives you. A lot of scanners will assign a severity rating to the findings. They have no idea how you’re using a particular piece of code or dependency. Their severity ratings should be treated with extreme suspicion. They could be an easy way for a first pass ranking, but those rating shouldn’t be used for anything after the first pass. I’ll write a bit more on where these severities come from in a future post, the short version is the sausage is made with questionable ingredients.
It makes a lot of sense to fix the critical findings first, nobody will argue this point. A point that is a bit more contentious is not fixing low and moderate findings, at least not at first. You have finite resources. If fixing the critical issues consume all of your resources, that’s OK. You can mark low findings in a way that says you’re not fixing them now, but might fix them later. If your security team comes back claiming that’s not acceptable and you have to fix everything, I suggest a very hearty “patches welcome” be sent to them. In typical software development minor bugs don’t always gets fixed. Security bugs are just bugs, fix the important stuff first, don’t be afraid to WONTFIX silly things.
It’s also really important to avoid trying to “fix” everything just to make the scanner be quiet. If your goal is a clean report, you will suffer other consequences due to this. Beware the cobra effect.
Can’t we all just get along
The biggest takeaway from all of this is to understand intent and purpose. If you are running a scanner, understand why. If you’re receiving a report, make sure you ask why it was run and the expectations of whoever gave it to you. It’s generally a good idea not to assume malice, these scanners are very new and there is a huge knowledge gap, even with the people who historically would consider themselves security experts. It can get even more complicated because there’s a lot of open source thrown into the mix. The amount of knowledge needed for this problem is enormous, don’t be afraid to ask lots of questions and seek out help.
If you are doing the scanning, be patient.
If you are receiving a scan, be patient.
Remember, it’s nice to be nice.