A Guide to Cybersecurity Incident Simulation Options
What is the best way to prepare your team to respond to cybersecurity incidents?
While the first step is having the right plans, processes and tools without testing and practice it is impossible to know if they will work as expected. Even if everything is in place, if the team is unfamiliar with the tools and plan, they will not operate at peak performance.
Asking your responders to read the plan and manuals is a start, but you would not prepare for a marathon by reading a book on running. As with training for a marathon, the best way to prepare for incident response is to exercise! There are a number of exercise options available to you, including table top, capture-the-flag, cyber ranges, purple team and red team exercises.
In this blog post, I will share some lived experience working with each option and consider the costs and benefits of each.
About my experience
Over the years, I have been involved in running variations of these exercises for a range of clients, from relatively small organizations with 10 or so people on the response team to large enterprises with a dedicated security team of 80 or more.
I have used a range of different types of exercise over the years, including:
- Table Top
- Live Fire
- Training platform & test data
- Cyber Range
- Purple Team
- Red Team
To better understand the different exercise types, see the section at the end of this post.
When it comes to return on investment, the humble table top is ahead of the rest by several orders of magnitude. While many organizations run these exercises, there are many variations on how they can be implemented and, to a certain extent, the outcomes.
Comparing the costs
There are two easily measured costs associated with IR exercises: the direct financial cost of hiring consultants and facilitators and the time cost staff have to spend on the exercise.
The table below gives some idea of the costs of each exercise for a medium-sized organization. With the caveat that direct costs can vary depending on the complexity of the exercise and network.
|Direct Cost 1
|Hours per participant 2
|$5,000 to $20,000
|2 to 4
|$12,000 to $50,000
|8 to 16
|$500 to $8,000
|$50,000 to $220,000
|16 to 32
|$50,000 to $100,000+
|8 to 32
|$50,000 to $100,000+
|16 to 40
1 This is assuming that an external third party is developing and facilitating the exercise. The prices are indicative of the average range, there are certainly outliers beyond these ranges, with associated varying levels of quality and complexity.
2 This time may not be contiguous, for example in a Red Team exercise participants are likely to be performing regular work functions in parallel with the exercise.
Benefits of IR exercises
There are two primary benefits of conducting IR exercises: making the response team familiar with the response process and confirming that the process actually works! Which of these receives the higher focus will be determined by the maturity of your IR capabilities. For organizations in the early stages of IR capability development, Table Top Exercises provide an extremely high rate of return.
As with any form of exercise, the more frequently you practice, the better you will perform. If IR exercises are run as an annual event, while the benefits may be significant immediately following the exercise, the familiarity will decline.
It is better to run a two-hour table top four times a year rather than an eight-hour exercise once a year. Even better is to use a mix of exercise types to ensure that all the required skills are practiced.
I recommend that most teams start with one or more table top exercises and then progress to more advanced simulations as they are ready. TTX often identifies several areas that need work, and once changes have been made, it’s good to run another TTX to ensure the change worked as intended.
Once you are confident that the theory side of things is working, then it is time to test your systems using test data and live exercises. This may be a cyber range or using your production systems. This can then be followed by a purple team exercise, culminating in a red team exercise. This progression allows for maximum value from each exercise, as many issues will be identified by the simpler and less expensive exercises while at the same time familiarizing participants with the IR process.
Keys for a Successful IR Exercise
- Proper preparation
You are taking the right first step by reading this! Make sure you allow enough time to prepare the exercise, the standard rule of IT projects applies here, figure out how long you think it will take and triple it!
- Clearly defined scope and objectives
A critical part of planning a successful IR exercise is first identifying what you want to achieve from it. This will heavily influence the exercise design. Questions to ask include:
- Have we made any changes to our IR process or systems?
- Do we have new people on the team that need training?
- What issues were raised in recent post incident reviews?
- Realistic scenario
Designing a realistic scenario that addresses your objectives can be challenging. It is important to remember that you do not have to run through the entire IR lifecycle for the exercise but can focus on specific stages if that will better address your needs. For example, if one of your objectives is to test a new SIEM then a scenario built around the detection stage may be enough.When designing a scenario, I draw from my experience and my team’s experience in recent investigations and will generally build the scenario using elements from three or four cases. If you are not lucky (or unlucky enough) to be conducting regular investigations, case studies from sites like the DFIR Report, Recorded Future, and ReversingLabs are invaluable. If you use an external consultant to build and facilitate the exercise, ensure they have current IR experience.
- Engagement of all participants (this also means limiting the number of participants)
A table top exercise can be rapidly derailed if even one participant is not engaged in the spirit of the exercise, and there are a number of things that can disengage participants. One time we had been engaged by the PCI team to run a table top. Since it was PCI the decision was made to build a scenario around a compromise of the website, resulting in compromise of credit card data. This exercise was run with around 20 participants, including a number of the website developers. As soon as the scenario was presented the developers became defensive about the security of the website, resulting in fifteen minutes or more spend discussing the scenario and why it would not work.This response was perfectly predictable and not unreasonable, as the developers felt their expertise was being criticized. This was a valuable learning experience for me, and following it and other more positive exercises, I now use the following to help ensure that the participants are positively engaged in the exercise:
- Limit participant numbers — Group size will vary depending on the focus on identifying and resolving issues vs training. If the focus is on identifying issues and resolving them a smaller group is better, as the discussion will be more focused on training the number of participants can be larger, but the exercise will be more prescriptive.
- Communicate from the outset that this is a theoretical exercise — Focus on how the incident would be handled in a real situation. Emphasize that the scenario is not real nor a reflection on the teams involved.
- Seek input from all — Ensure that all teams to be involved in the exercise are consulted when developing the scenario (without giving the scenario away in advance).
- Manage discussion during the exercise — Ensure that everyone has a chance to be heard, but don’t be afraid to shut down discussions that are going way off-topic.
- Facilitator with real Incident Response experience
Using a facilitator with Incident Response experience results in a scenario that is realistic and more importantly can provide real-life context to the responses from participants. They will know what containment actions are most likely to be effective and have a realistic understanding of analysis outcomes and timeframes.
There are a number of different ways of exercising your Incident Response capabilities. Like any form of exercise the more often it is repeated the better the performance. For a great example of a well-planned Incident Response exercise schedule, have a look at this post from Uber. Different types of exercise require different levels of effort, and the humble table top can be an excellent low cost approach to regular exercising, backup up by training, test data sets and less frequent major exercises such as Cyber Range, Purple and Red teaming.