Josh and David finish up the disaster recovery and emergency planning trilogy. In this one David tells us how to test the plan he told us how to build in the last episode. There are some great ideas in this one about how to test the process not the people. How to construct the plan, and even some tips to go from a plan to some actual real world testing. It’s another episode filled with great and practical advice.

This episode is also available as a podcast, search for “Open Source Security” on your favorite podcast player.

Episode Transcript

Josh Bressers (00:00) Today, open source security. Welcome back David Bernstein. David is a certified emergency manager and certified business continuity professional. This is a. ⁓

three-parter with Dave. is basically part three. Part one was kind of an explanation of like disaster recovery and emergency handling management. I’m not sure what the right term to use there is, but then the last episode, which was a couple of weeks ago, Dave explained to us kind of how to create ⁓ like a disaster recovery emergency plan. And today he’s going to kind of talk about once we have our plan, what do we do next? Cause making the plan is really just the beginning. And

I’m gonna try to give the TLDR for this one, Dave. So I would suggest, if you’re listening, you haven’t heard the other two episodes, go listen to them because they’re exceptionally good. I love them. And kind of in the last episode, I would say the quick summary of building a plan is like, don’t overcomplicate it, right? You wanna like, you list the people you want involved, list responsibilities, and then kind of just get everyone together and then start working on a plan versus trying to put a ton of effort in first. What do you think?

David Bernstein (01:02) Yep,

absolutely. You can go down a lot of roads for developing plans, but I’m a very strong advocate for the KISS ⁓ methodology. Keep it simple. Start simple, then you can start making it more complicated based on what you’re looking to do. yeah, emergency plan is just list of expectations at its core.

Josh Bressers (01:12) Yeah. Yeah.

Yeah, yeah, which funny enough, I had been over complicating this for literally decades at this point. But okay, this is good. This is why.

David Bernstein (01:32) Everyone does. Everyone thinks these

plans have to be some huge thing. No. What do you expect from me? What do I expect from you? At its crux, that’s all these emergency plans are.

Josh Bressers (01:35) Yeah.

Which is awesome. And this is also like, cannot stress enough how just timely this is because of all the like, if you are in the like open source universe, everything has been on fire for like months now.

David Bernstein (01:56) I thought you had a great LinkedIn post, Josh, about risks and hazards and how these things are on fire. I would add there’s a literal component to that as well, not only figurative.

Josh Bressers (02:03) It has been.

Indeed,

indeed. It’s been man, it’s been something. So anyway, okay. So I want to talk about what happens next because this is the part I’m kind of most excited for because like I’ve built a plan and I’ve literally done this right where I put a plan together and then we’ll do what we I call it table topping. I don’t know if you have a fancier term for it, but basically kind of like this is where I literally go back to my Dungeons and Dragons days of role playing and things and and we role play an incident.

And this is where I’m curious kind of what your suggestions are, or even if that’s the right thing to be doing, because it’s like, it’s all I know. So it’s kind of what I do. So like, I’ve got my plan. It’s not, I guess. Okay. So here’s part of my question, I guess, Dave is like, when I had overly complicated plans, I feel like the, the, like the, the, like the table topping was easier because I had a ton of complex detail in there. So I wasn’t, it wasn’t very open-ended, right?

So when we were trying to stress them and say like, how does this work? like, there were a lot of details in that plan, but I guess on the other side, we also weren’t preparing for something else to happen necessarily. So, all right, I’m going to let you take it from here. I think I’ve talked way too much already. like fill us in.

David Bernstein (03:16) Mm-hmm.

No, you’re great. And the more technical term I have for tabletopping is tabletop exercise. That’s just what it’s called. You’re right on the money. And it’s a little bit of a conundrum because there’s a full diatribe I’m going to go on, rest assured. But the benefit of having really complex exercises is when people are done exercising some component of the…

Josh Bressers (03:33) Okay, okay.

David Bernstein (03:53) of the program you’ve put together, you’re like, okay, great, we’ll just go on to the next one. There’s always something to continue progressing through with the exercise. The downfall is you start evaluating ⁓ such narrow components of the plan because you’re so specific on what you’re looking for on each of these outcomes. That’s not necessarily a bad thing. It just means that you’re only going to get the things that you test on.

There is, I’m going to use another federal program as an example. This one is more applicable than the other ones we’ve talked about. But again, I’m saying it not because it’s the best thing out there, not because it’s the only thing out there, but I’m saying it because it’s free, because it gives you all the tools you need, and it’ll talk through a methodology for pulling it through. And I think, ⁓ Josh, I included the link to this in all the other links of the business continuity stuff that I sent. So if I didn’t send that, I apologize, but…

Josh Bressers (04:38) I like that price.

David Bernstein (04:52) Josh is going to post it at some point whenever I get that to him.

Josh Bressers (04:54) What’s the

name of it? I can tell you if I have it.

David Bernstein (04:57) The name of the program is the Homeland Security Exercise and Evaluation Program, or HSEEP for short. ⁓ While Josh is looking for it, I’m just going to talk about the program and talk about the methodology for it. So basically, there’s this continuous improvement cycle that we’ve been talking about for developing your plan. The next piece after you do your planning and your training is the exercising. And the exercising is this validation of like, did the…

Are the people going to do what we expect them to do in the plan the way we’ve trained them? ⁓ If they do, great. If they don’t, glad we found out now. And then we can figure out why it is that that’s happening, what we’re going to move forward on. ⁓ In this program, the Homeland Security Exercise and Evaluation Program, I’m only going to refer to it as HSEEP moving forward because, that’s a mouthful. ⁓ Like any good…

Josh Bressers (05:47) So HC

like the letter C.

David Bernstein (05:52) H-S-E-E-P, as in it’s seeping through all of your programs.

Josh Bressers (05:55) H-seep. H-seep. Okay, I understand now. Okay, okay. It’s, I got you. It’s amusing to me that

now that you’ve told me that, I’ll hear it correctly, but because I had no idea what you trying to say, like my brain was not putting the letters together. It’s awesome.

David Bernstein (06:09) of course. Yeah,

well, can’t tell you how many times I’ve incorrectly pronounced CISA I’m just like…

Josh Bressers (06:16) Is there a right way?

Wait, you say CISA CISA is what most… CISA? Okay. So the people I know who used to work there called it CISA when I would talk to them. I don’t know. Whatever. Who cares? Who cares? Doesn’t matter.

David Bernstein (06:19) I thought it was always CISA. C-I-S-A?

Yeah, CISA, whatever,

it doesn’t matter. As long as we know we’re talking about the same thing. like whatever, maybe I’m still pretty proud of seeing it, who cares? Anyway, ⁓ the point is CISA. What was I saying before? All right, now I’m gonna have to play back the recording and like, know, wrap myself on the knuckles if I’m getting it wrong. ⁓ So the Homeland Security Exercise and Evaluation Program or HSEEP for short, HSEEP, ⁓ there’s a link that Josh is going to share with this. ⁓

talks about this ⁓ cycle of exercises specifically geared towards starting simple and getting more complex every time you do the exercise. And it’s not that you have to exercise the same thing every time, you’re actually not supposed to do that, but as you’re working with the same people, as you’re working on similar initiatives, ⁓ you can make the exercise more complex by getting more realistic with it, expanding the scope of the exercise, starting to

like, you ⁓ know, in your case, doing things in a sandbox and actually going through and clicking the keys you need to click. Or, you know, out in the physical realm, if it says you’re going to move people and equipment, start moving that people and equipment, which is different than you would do in a tabletop. And it basically starts from discussion-based exercises, games, workshops, seminars, up through a tabletop exercise where you’re actually working through a scenario as a group.

and then it would move into something called functional exercises where you’re doing those ⁓ operations-based exercises. So you take things out of the room and do it out in the world.

Josh Bressers (08:05) Okay, that makes sense.

You know what that reminds me of David actually is I remember back in my younger days when I worked in a data center when we had data centers, we would test the generator every year by turning off power to the data center. And one year the generator didn’t work. So would you like to guess what happened to the data center? Like, boo. We’re like, uh-oh, that’s bad. Yeah. It’s like that isn’t quite what we’d hoped was going to happen.

David Bernstein (08:30) that’s, that’s bad.

That’s not…

Yeah, that’s a great example of an exercise turning into a real incident, which is always not great.

Josh Bressers (08:40) That’s true. Yes, that’s exactly what it was. Yes, our exercise turned

into one of the biggest incidents we may have ever had in our lives up to that point.

David Bernstein (08:48) Yep.

Well, and we talk about this like if you are for talking business continuity and disaster recovery, and I’m going to date myself because this isn’t necessarily how it works anymore. But it’s the clearest example I can provide is when you’re in when you’re doing these discussion based exercises, it’s it’s literally just like the listener and me and Josh here on this call, like what is happening? What do we think is going to happen here? The plans and actions we’re planning to take. Does that sound good to everyone in the room? Great. You know, we’re all aligned. Everything kind of.

everything’s hunky-dory. What you might do for something that’s more operations-based, and this is where I’m going to date myself, is you’re at, from a continuity perspective, you’re at a hot site or a live site, and then you are going to fail over ⁓ to another hot site or a warm site or a cold site. For anyone who doesn’t understand the references, and I’m sure all your listeners do, but for the sake of it, hot sites are the ones that are totally live, everything’s double-backed.

I leave the computer in this room and I go to the computer in the other location and everything is totally up to date and up to speed. I just walk over and I continue doing what I’m doing. A warm site is that you have the computer set up, but you have to wake everything up and you have to go from the most recent backup, which might have been 12 hours ago, and you have to restart a little bit. A cold site is it’s just a room and you have to fill it with computers and do whatever else you need to do.

An operations-based exercise may be testing the physical element of that failover process or restart process. It does not make sense after we’ve first written a plan to do something that extreme because there’s so much that can fail in the process that we don’t really understand if it’s working or not. All we know is that things have either gone okay or they’ve gone terribly wrong. ⁓

Josh Bressers (10:20) Gotcha.

Yes.

David Bernstein (10:42) First, we want to start off with these workshops or these games or a tabletop exercise because it’s a great opportunity to say, all right, my responsibility is X, Josh’s responsibility is X. Josh, how are you going to do that thing? Do you have all the resources you need? you have the people you need? Yeah, you think you do? Great. All right, well, the next exercise, we’ll test that for real. Like when you say you have all the resources you need, let’s see if that’s actually adequate or more than you need to do the thing that you said you’re going to do.

When we start planning these exercises, it’s not always just go, because there has to be something for us to react to. I think you were alluding to this before, also, Josh. The benefit to having a really complex plan is you can keep pushing on specific things, but you don’t need a complex plan to do that. Having something that is at a high level, a little bit more general that allows you to respond to…

a hazard or a process without getting to every nitty-gritty detail in the plan. Because the plan is supposed to be like, how does the organization operate? It’s not like every single policy and procedure to support what you’re actually doing. I know how to turn on a computer. I don’t need the emergency plan to tell me I have to turn on a computer. That’s the protocol for me starting to respond is turn on the computer. It’s kind of like a…

Josh Bressers (11:53) Right. Right.

David Bernstein (12:07) Every time someone call, have you ever seen the IT crowd, the VIA crowd? They answer the phone like, hi IT, you if you should try turning it on and off again. Like, if that’s just part of your process, we don’t need to make that part of the emergency plan. That’s just a process, right? ⁓ But one of the things that HSEEP calls out and that is just a good practice for planning and implementing these exercises is what’s called ⁓

Josh Bressers (12:10) Yes.

I know.

Right. OK, that makes sense.

David Bernstein (12:36) exercising to the objective. That’s not the actual term for it. Basically, what it means is come up with what you want to do. It’s objective-driven scenarios. Here’s the things we want to test from the plan. What is a scenario that allows us to do these tests without a bunch of extraneous detail that we need to pull in? If you have a plan that’s a little bit more general because you want it to allow you to respond to a number of other things, the emergency plan is

Josh Bressers (12:54) ⁓ I understand.

David Bernstein (13:05) here’s who’s allowed to make decisions, here’s what they’re allowed to do, like here’s kind of the broad expectations we have of them to be able to do it. And then policies and procedures support that. like the objective might be, what do we do when this person won the lottery and they’re not here anymore? Like what do we do with this? You know, we do this a lot in like pandemic planning is the people that you have in charge of the operations just aren’t available anymore. You know, they’re sick, they’re whatever, like they can’t respond.

Josh Bressers (13:30) ⁓

David Bernstein (13:34) But you start with what was the purpose of the plan? What are we trying to achieve with the objective of the plan? That might be, you might remember from the last call, like, why are we doing this? Right? Same question for an exercise. What are we trying to accomplish with the plan? Like, what are some of the things we really want to test? So we want to test that people are starting to respond within 15 minutes. That’s one of our key objectives because in a downtime, you can’t let it go on for a day. let’s, so, you know, if there’s a,

a policy or procedure that says they have to respond within 15 minutes in support of the plan? Great, let’s test it. Can they respond within 15 minutes? Is all the equipment charged and ready to go? Do people know what to do? And then one more objective that meets the core element of the exercise will say, they’re able to start responding to customer demands within 24 hours of this downtime. 24 hours, I know, can be…

forever, but we’ll say, put something arbitrary in there. Yeah. And then we build the scenario around that. So what is this scenario? What’s the evolution that’s taking place that will allow us to test these things? Because if we just say arbitrarily, all right, the network is down. We didn’t want to see what you respond to in 15 minutes. People are going to turn around and they’re going to say, well, why is it down? Because the reason for it being down is going to

Josh Bressers (14:35) Yeah. Yeah. No, that, I mean, that makes sense.

David Bernstein (15:02) I’m just going to prompt what I do to address this issue. And that’s your opportunity to say, as part of the scenario that you’re going to create, what do we want them to do? Do we want it to be down because we don’t know why and we’re trying to assess their ability to decode the issue? Or are we saying like,

We can give them a scenario, we’ll tell them why it is, and that gives them an opportunity to figure out how they’re actually going to set up and respond. There’s a lot of different directions you can go just based on what is the objective you’re trying to address with the exercise. ⁓

Josh Bressers (15:41) So I have a question for you actually, and this

is something I’ve had in the past, is where, like, why is the network down? It’s a great example, right? Where I would give people an answer to something like that, and they would end up trying to go down a rat hole, and they’re like, no, that is not why we’re here. And like, I’m curious, is there a trick to this, or is it just tell them to stop it and get back on the road?

David Bernstein (15:58) Mm-hmm.

Yeah, so there’s a couple reasons the exercise can fail. The first reason is because people just do not know what to do and training was totally ineffective and you’re like, all well, like, there’s no point in continuing this. Like, there’s no, we’re not going to accomplish anything. The second reason is exactly what you mentioned, like, ⁓ you know, I understand where they’re trying to go, but that’s not what we’re trying to accomplish. And they’re getting actually way off track. And yeah, in that case, well, and the third reason is someone’s doing unsafe and you have to like,

Josh Bressers (16:17) That’s fair. Yeah, and that’s fair.

Yeah, yeah.

David Bernstein (16:35) just time out, like, you you’re going to hurt yourself. We can’t allow that to happen. ⁓ But exactly as you said, ⁓ we can and we should put, ⁓ my gosh, I really wanted to use a specific term and I can’t forget what it is, the things that go on either side of the bowling lane.

Josh Bressers (16:36) Right.

⁓ gutters?

David Bernstein (16:56) The thing that protects it from the gutters. Boundaries. Yeah. This is what I get for trying to talk about bowling. I was into it in high school and I can’t remember what it is now. ⁓ Anyway. You know what? That works. Yeah. So, yeah. Nice little tangent there. We talked about sports that I haven’t even thought about in a long time.

Josh Bressers (16:58) What protects it from the gutters?

I don’t know.

We’ll just call it guardrails like everyone else would do. It works, but it’s boring.

David Bernstein (17:25) Bowling is super fun. should get back into that. Anyway, ⁓ so yeah, you can and should put guardrails around the exercise. ⁓ the reason is because you can totally derail the exercise ⁓ and lose the reason that you’re trying to do it. Now, just because you try and reorient someone doesn’t mean that ⁓ it’s a missed opportunity. You should note that they were going to go down a rabbit hole.

Josh Bressers (17:27) Yes.

⁓ right.

David Bernstein (17:55) Because that might be something that you want to tighten up in the plan. And you say, all right, the priority when this thing happens isn’t whatever you were trying to do. It isn’t trying to resolve the initial issue. You have to be in response, not fix ⁓ in whatever, in the scenario that you’re trying to address. And ⁓ that might expose a new gap for you. It’s that people are misaligned on the priorities of what they should be doing.

which is actually a really great finding for an exercise. That is a really good finding. And I should note, ⁓ a huge component of the exercise planning that I haven’t gotten to yet is ⁓ what we would call a no-fault exercise. I’m not testing your ability to do the work. I’m testing if this thing is going to happen. And so no one should feel any fear or concern that they are being evaluated personally.

Josh Bressers (18:26) That’s true. That’s a good point. Yeah.

David Bernstein (18:53) on their job performance, the reason we’re doing the exercise is to see what happens. It’s not a test for you as an individual, it’s a test for the process itself. And so if someone doesn’t do what we’re expecting them to do, they shouldn’t get a finger wagged at them. We should figure out, should we instead be, should we change our training? Should we change the staffing that we’re committing to this thing? What do we wanna fix? Because I guarantee you,

Josh Bressers (18:53) Yeah, yeah.

Yeah.

David Bernstein (19:22) someone feeling like they’re under the gun to perform is not going to perform as well as they usually would anyway.

Josh Bressers (19:28) So I have a question for you about this actually. So in the world of InfoSec, of security especially, there’s kind of this attitude of the first time you do a tabletop, you make sure like everything goes well, right? It’s like you’re playing Dungeons and Dragons, everyone wins. But then after that, you kind of make, you fail them on purpose, right? Or like your.

Goal is to not let them win. You keep putting up roadblocks to make everything go wrong no matter what they’re doing. And I’m curious if that is a good idea or a bad idea from your perspective.

David Bernstein (20:01) Well, there’s a big difference between a demonstration and an exercise. A demonstration is meant to just show people that something will work. An exercise is meant to show where things will fail. yeah, so well, we don’t necessarily want to fail. We want to follow the process to the point, we want to follow the process and see where gaps exist along the way. we want to, this is we talk about building complexity. The first thing, the first exercise should not

Josh Bressers (20:12) Okay, so you want to fail.

David Bernstein (20:30) be pushing everyone to fail. But by the time you’re doing bigger and bigger exercises, by the time you’re at a tabletop exercise, you want people to start really thinking about what they’re doing, right? And start to push and start to push on the expectations a little bit, make the exercise more complex, like start to really push on the resources that they’re using so that you’re not just failing for the sake of failing, you’re pushing because you think this might be a critical part of the of your response and you want to understand where it’s going to fail and like what the limitations are.

Josh Bressers (20:46) Okay.

David Bernstein (21:00) so that when a real incident happens, you can say, all right, this is kind of the boundary that we have based on the exercises that we’ve done. And anything beyond this, we’re really out in open water. ⁓ And that being said, if you set up a Kobayashi Maru of an exercise where you’re just going to get people to fail for the sake of failing because they need to understand that they’re going to fail, ⁓ that isn’t necessarily helpful. The point is, what are you learning from this?

Josh Bressers (21:10) Okay.

Yes!

Yeah, yeah.

David Bernstein (21:30) pushing them like you want to understand where it breaks not break it for the sake of breaking it.

Josh Bressers (21:34) Okay, that makes sense.

David Bernstein (21:35) Yeah. I mean, there’s also, if I can pull out another Star Trek quote here, there’s also, yeah. So there’s also a line from Jean-Luc Picard where he says, he’s talking to Q at the moment, and he says, you can do everything right and still fail. That’s just a part of being human. We said it’s a data?

Josh Bressers (21:39) Please, I love Star Trek. Star Trek’s the best

He says that to Data in the episode where

he loses the game of… I can’t remember the name of the game. It’s the one where they put the weird cow milkers on their fingers and they play… I can’t remember the name of it. No, that was Data. Because Data lost to the guy. ⁓ It’s the episode where they have a mock battle in space where Riker takes a bunch of people onto the other ship.

David Bernstein (22:05) Yeah. I thought he was talking to Q when he was doing that. Thank you for, thank you for correcting me.

Yes!

Josh Bressers (22:22) Yes. And then Data loses and Data

David Bernstein (22:23) That’s right.

Josh Bressers (22:24) thinks he’s broken because he lost this game. And then that’s when Picari tells him that. I, I, we, we totally could. Like I love, so actually, actually, David, I have a talk I give called next generation enterprise security, where I watch all of Star Trek, the next generation multiple times. Mind you, I took incredibly detailed notes about all of the threats and risks in every episode. And then I go through them and I map them to the MITRE ATT &CK framework.

David Bernstein (22:28) Yeah. So are we going to have a separate podcast just about Star Trek? Is that?

Mm-hmm.

Josh Bressers (22:53) And it is like, it’s the deck is like 90 slides long. And when I give it, I’m like, I’m just going to talk until someone takes the hook out and kicks me off because like, it’s just, it’s so much fun to talk about. It’s so good. And I gave it actually at B-Side Las Vegas one year. And they have this like, this like request box when you send in the talk and they were like, what is your ridiculous request? And I asked for someone dressed in a Starfleet uniform to heckle me. And Daemon Hunter showed up with a trombone.

David Bernstein (23:02) I love it.

Josh Bressers (23:21) in a red uniform. was amazing. It was so much fun. And he like, and he knows a lot about Star Trek, so it was really good. It was so good. Anyway, okay, back to our actual topic.

David Bernstein (23:23) ⁓ my gosh.

Oh my gosh. Yeah, go back to exercise. But

this is something you and I have to talk about offline for the podcast But yeah, mean, these exercises, they’re not meant to make a person fail. They’re meant to stress the process. At what point do we think this is going to break so that we know where the boundaries are for our exercise? And also where we need to reinvest. So at the end of the exercise,

Josh Bressers (23:37) Yeah, yeah, yeah.

David Bernstein (23:58) The typical staffing for an exercise, this all scales with the complexity of the exercise, ⁓ is you’ll have people that are just kind of like watching because they’re learning, those are called observers, and you’ll have people that are evaluating. And so they’ll have something called an Exercise and Evaluation Guide or EEG. I’m not going to go through all of the acronyms or naming conventions of all of this because it’s a federal program, so holy cow, I’m never going to be able to go through all of it.

Josh Bressers (24:24) yeah, they love it.

David Bernstein (24:26) Yeah, they do. But in this exercise evaluation guide, you’ll have written, what’s the objective, what’s the expected outcome, and how do they perform to that? And so as they’re going through, you have someone that is intentionally watching to try and assist, where did this thing happen, and why did it happen? So that the people participating in the exercise are playing in the exercise, just get to play in the exercise, not trying to accomplish everything. There’s a couple other layers, like, you

controllers and exercise directors and everything else, or an exercise director, usually just one. But we’re not going to go into that too much. But ⁓ the point is there are people that are there also to assist in understanding what happens, what happened in the exercise. That cycles through at the end of the exercise ⁓ towards ⁓ the evaluation program. And this is different than evaluation of the overall program that can be used in improvement cycle we talked about before. This is specifically to the exercise.

You’ve done the exercise. Now it’s time to develop your after-action report and improvement plan using all the notes. at the end of the exercise, you’ll do a hot wash. You’ll take feedback from everyone that was playing, like what worked, what didn’t, where did you feel pressure that we might not have noticed, and then all the notes from these evaluators and observers to come up with, all right.

The after action report is here’s all the objectives we’re trying to accomplish, here’s why we were trying to do those objectives, and here’s where we performed well, and here’s where we struggled. ⁓ And then depending if you want to continue using the HSEEP modeling, there’s like a whole evaluation criteria you can use. Again, it’s free and it makes sense, and it’s all kind of out there, but also feel free to use your own. There’s no downside if you think you have something that works better for you. ⁓ And then you’ll write your improvement plan, which is

Basically, everyone getting back in a room and saying, all right, here’s where we failed, or here’s where we felt we needed to improve. How are we going to do that and who’s responsible for it? Because it’s important to have accountability in this as well. And that is your opportunity to say, all right, we’re going to change the plan in this way, we’re going to change the procedures in this way, we’re going to reinvest because we found out we needed equipment that we didn’t have. ⁓

⁓ That’s where it will cover all of that. And you don’t want too many objectives because if you’re trying to measure too much, that improvement plan is going to be like 10 pages long, which can be really hard to manage as well. And so it’s important to test things in a way that’s going to be meaningful, but also the same way you don’t want a plan that is overcomplicated, you don’t want an exercise that is overcomplicated and trying to accomplish too much. Like usually, you know…

five, seven objectives is like a good place to be.

Josh Bressers (27:16) Yeah, that seems fair.

David Bernstein (27:18) Yeah. And again, that changes with the scale. Now, the one soapbox moment I’ll have on this is that there is as casually as we use some of this language, there is a difference between a drill and an exercise. And so I’ll keep this at the end. A drill is only testing one component of a thing.

It’s super small, super direct. So like an emergency notification drill doesn’t have a big scenario behind it. It’s literally what happens when you press the button? Like that’s it, right? And so anything that you would list as a drill should be really, really small in scope. You’re only testing one thing. Like it does not require a lot of stakeholders. It’s just, you’re just doing one thing. And exercise, the way we’ve been talking about exercises for the last 20 minutes or whatever it is, is

There’s a group of stakeholders, it’s multifaceted, it’s going to incorporate ⁓ a lot of different components of a response that you’re going to pull off, whether you’re talking about it or actually actioning it.

Josh Bressers (28:26) Okay, okay. I mean, that makes sense. I guess I’ve never, I didn’t know that, but it makes sense.

David Bernstein (28:32) You know what? And no

one does and no one cares, but like, since I have an audience, I’m gonna say it.

Josh Bressers (28:36) Well, no, no, I’m look one of,

mean, we complain about this all the time, especially in the security industry is where people don’t necessarily understand the terms we’re using. like vulnerability and exploit are two of my favorites where to most normal people, vulnerability and exploit, they think mean the same thing, but in the security world, they mean extremely different things. And that language matters a lot because if I’m talking about a vulnerability or I’m talking about an exploit, those have extremely different.

I guess process and outcomes versus if I’m a normal person, I’m like, what’s the difference? They’re the same. So like the language matters a lot in this stuff, especially when like health and human safety is involved because it can literally be a life or death decision that happens because of the wrong term.

David Bernstein (29:07) Mm-hmm.

Yeah. Yeah, I mean, like, and you’re not going to hit that in the exercises itself, but the things you’re testing, might that might be a great thing to exercise is like, people using the right language as you’re talking about how you’re going to respond, right? Like if that’s something that’s called out in a policy or a procedure or an emergency plan, like that is actually a really good thing to test.

Josh Bressers (29:30) Yeah. Yeah.

David Bernstein (29:46) So, you know, stay on that horse.

Josh Bressers (29:46) Yeah, yeah, for sure. No,

it’s, security people love nothing more than being overly pedantic. Like, it’s fantastic. All right, now.

David Bernstein (29:56) There’s something

else, there’s another Z-Version where they’re like, I’m the best kind of write. Technically write. That’s right. ⁓

Josh Bressers (30:01) Correct. That’s from Futurama. That’s when Hermes

goes to the central bureaucracy and he says, it’s technically correct. And then the bureaucrat number one says the best kind of correct as he flies down on his desk. I watch too much science fiction is the lesson here.

David Bernstein (30:08) that’s right, yeah.

Yeah, his fly is through the room. I have to say, I mean,

my, I shouldn’t say this to a whole bunch of tech people, but like, ⁓ a good amount of my profile is dedicated to Dr. Zoidberg Even though he’s not the one I identify with, like, I can’t, I can’t get away from him. He’s just, he’s so great. Anyway.

Josh Bressers (30:30) Fantastic.

Zoidberg is like the best character. I love Zoidberg so much. He’s so good.

Okay, so I’ve got kind of one

last question for you then on all of this. Is the cool thing people are starting to do now is to really gamify a lot of kind of the response process. And what I mean by that is so like literally creating a game with like dice and cards or like a certain amount of kind of randomness built in, right? Where people are reacting and they have to roll dice to see did something work? Did something not work? Or you’re flipping cards over to be like, what happens next?

I’m curious, you as an actual professional, like me as a just like kind of armchair security person who does this stuff, cause he has to, this, I love this stuff cause I think it’s awesome. Cause I’m also not real smart, so it’s hard to do, but I’m curious what you think of all that.

David Bernstein (31:24) I love the idea of a choose-your-own-adventure kind of exercise. It’s like, well, one of three things is going to happen. None of us know what it is. Let’s roll the dice and see what happens. I love the concept of that. The only hesitation I would give to anyone going down that path is that you still have to align with your exercise objectives. So you can’t totally deviate the scenario to the point where you’re not actually accomplishing the objectives or testing the objectives that you’re

Josh Bressers (31:27) Yeah, right.

Yeah. Yeah.

Okay, yeah.

David Bernstein (31:53) or set out the test. And that because that can be so much fun, don’t let creating the gamification overshadow the actual scenario that you’re building. Don’t spend so much time building the game that you don’t actually build the plan and the training and the other stuff that you actually need to do. And to be fair, we’re talking about exercise validation, but if you gamify it or if you…

Josh Bressers (31:54) That’s fair.

Okay, that’s fair.

David Bernstein (32:21) or if you don’t have a plan and you just need to see where people are at, you can do an exercise just to gather that information. We don’t actually know what’s going to happen. We haven’t built this plan before. We’re getting ready for some event that’s coming up. Let’s just do an exercise and figure out what we’re going to do. In that case, I would hesitate to call it an exercise. I would probably call it a planning seminar or something like that or a planning workshop.

Josh Bressers (32:28) true.

Sure. Yeah.

David Bernstein (32:50) Also, not the correct terminology to use for any of those things, but be honest with people and be like, we don’t have a plan, we can go through a full planning process or we can just sit in a room and throw stuff against the wall and see what happens. ⁓ The point is, again, if you’re going to gamify it, that gamification will lead you to specific outcomes. That’s never going to be as good as a more formal planning process, but you’ll get a lot of information really quickly. It’s just going to be based on…

Josh Bressers (33:03) Okay, that makes sense.

David Bernstein (33:18) the specific inputs you’re giving to people instead of saying, all right, let’s talk about what happens for each of these three kinds of situations. That’s going to, gamification will push you down one of them.

Josh Bressers (33:28) Sure. Okay, that makes sense. Okay, yeah, I dig it.

David Bernstein (33:32) Yeah, but

other than that, I love like, exercises are not meant to be like, you know, push you into the mud and like, you know, really make sure everything goes terrible. So you can react to it like, yeah, have fun with it. Like do the game. That sounds awesome. That sounds great. Just like, yeah, it sounds super cool. Just don’t let it distract from the real reason you’re trying to do this.

Josh Bressers (33:44) Yeah, yeah, they’re pretty cool. I’m not gonna lie.

That’s a good point. Okay, okay. Man, I have a lot in my head right now, David. What do you want us to know? How should we end this one?

David Bernstein (34:01) Yeah, I think I’ll leave people with a couple of quick thoughts. The first is, ⁓

do objective driven exercises and then build the scenario, right? Like figure out what you’re trying to test and how you’re trying to test it. And then you can build out the scenario that drives you to that to give it some realism ⁓ and no fault exercises. If people feel like they are being tested, then like you’re not going to get good performance. People are going to be like really sweating bullets on this. Like it’s not meant to be sweating bullets. It’s not always fun, but it’s also you’re not meant to be sweating bullets. Like really test.

Test the process, not a person.

Josh Bressers (34:42) Yes, yes, ⁓ yes, 100%. That’s always, I mean, that’s always tough. And obviously someone feels like they’re being attacked. That exercise is not worthless because they aren’t going to do any of the things they would actually do. only gonna react in a way they think you want them to react to make you go away.

David Bernstein (34:59) Yeah, and this is easier said than done. We didn’t talk about underperformers and the personalities that you would be dealing with as part of this anyway. So those are all things you also have to manage and mitigate to some extent. But the worst thing you can do, even with your top performers, is say, Josh, you’re really good. And that’s why we’re focusing the entire exercise on you. And the CEO is watching. That’d be brutal. That’d be the worst. Don’t do that.

Josh Bressers (35:01) Yes.

man, that would be brutal. Right? Yes.

Yes. That. ⁓ anyway. All right. All right, David, this has been awesome. I want to thank you so much. I have learned an amazing amount from you over the last couple of shows. So this is like this is what my hope was. Like secretly your audience is me. Like I wanted to learn, but I know there’s there’s a lot of value in this for everyone. So I hope everyone enjoys it.

David Bernstein (35:47) I mean, this is

just us nerding out. Hopefully people also get the Star Trek and Futurama references we’re making through this. So.

Josh Bressers (35:54) I guarantee it,

yes, guaranteed. Who doesn’t love Star Trek and Futurama, right? I can’t wait for all the messages now that’s like, I don’t like Star Trek. That’s all right, that’s all right. All right, awesome. David, thank you. Yeah, this has been amazing. Thank you so much. I truly appreciate the time.

David Bernstein (35:59) Seriously.

Thanks, this was super fun.

Yeah, I appreciate the opportunity. And thanks for having me on these sessions. This was great.

Josh Bressers (36:18) Awesome.