How to Design an Effective Cyber Exercise

In today’s blog, we’ll discuss how to design an effective cyber exercise to follow up on our post on the benefits of cyber exercises and how to get approvals.

A good starting point is to consider why you want to have an exercise. A lot of times, cyber security teams and even upper management consider cyber exercises as a “nice to have,” not a “must-have.” However, a simulated cyber security simulation, which is often structured as a half-day or full-day tabletop or roundtable event, can be a very useful way to see how prepared your organization is – or not. Consider what your organization’s biggest threats or concerns are: is it a ransomware attack? Insider threats? Nation-state actors?

Do you need help learning how to design an effective cyber exercise? Get expert assistance from Delta Risk. Contact us now.

Three Phases of Cyber Exercise Development

One way to look at this from a different perspective is to consider professional sports teams. Why do they practice when they’re already pros? Could you imagine how badly they would perform if they never practiced? They’d never make it to the playoffs, much less to superstar status.

Developing a simulated cyber exercise gives you the opportunity to figure out strengths, weaknesses, and any differences of opinion before an actual security incident happens. The more specific you can be with your goals and objectives, the more useful the exercise will be. Let’s take an in-depth look at three phases you can use to design a successful cyber exercise.

We recommend a three-phase approach to cyber exercise development: Pre-Planning, Event Planning, and Post Planning. Each phase serves an important role in creating an effective exercise.

1. Pre-Planning consists of items that need to be decided and coordinated prior to starting the exercise. This is the most critical step, since it will determine the exercise’s value to the participants and your organization.
2. Event Planning consists of items that are needed during the event to make it flow as envisioned. This phase defines how well the pre-planning goal or goals can be achieved versus how much time is spent overcoming any limitations of an exercise.
3. Post Planning consists of items that will close out the exercise while reinforcing its value to the organization. This is time to gather all the results, the feedback, and give the teams time to discuss their roles with each other.

Pre-Planning

Consider the following during the pre-planning phase:

• Scope of Participants: Who is the exercise supposed to exercise? This list should include active participants, support participants, and exercise managers. Determine which support participants will participate and which will be simulated.
  • Example: If the exercise will include an Incident Response (IR) activity, the cyber personnel that would find the incident, do the preliminary investigation, and/or mitigation are active participants. The legal team, upper management, a representative from your insurance company, and law enforcement personnel would support participants and/or simulated.
  • Who will manage the exercise? In most instances, there are people who know what is supposed to happen in the exercise to monitor what’s actually happening. They should be ready to coordinate with people outside the exercise in case of technical issues or if something gets missed by the participants.
  • Another thing to consider is how many personnel will participate. In most companies, there are teams involved in a single role, and they each have various skills. Will the cyber exercise actively utilize a member from each team, an entire team, multiple teams, or some blend? This discussion often includes cost arguments and manning constraints. Regulatory requirements can also play a role in this topic.
• What type of actors are needed and how will they be included in the exercise?
  • Define an actor as a person who’s needed to support the realism of a task within the exercise without being part of the metrics. As this article recommends, ”ask yourself what they would know in a given situation, what’s important to them and what motivates them.”
    • For example, will you simulate users or have a group of actual users to role-play daily tasks? If you use a group of actual people, then they are actors.
  • If remote sites, partners, and/or external customers exist, how will they be included/simulated?
  • Note: Adding different “political” actors and different levels of “demanding” actors adds to the realism and the fun of the exercise.
    • For example, you could simulate a vice president on a business trip who wants to get something done as quickly as possible.
  • Scope of Facilities: If the exercise occurs in the players’ normal offices, there’s usually an established communication pattern. However, daily work and interruptions are hard to control. Flow is hard to observe. Is there a classroom or training space available? If so, what needs to be considered to make it usable for the exercise?
    • Tip: Consider power, HVAC, communication methods, evacuation, restrooms, and access needs.
  • Scale: The better an exercise can duplicate reality, the more value it will have for the participants. Scale encompasses all the details required for the exercise: access to policies, procedures, checklists, “normal traffic”, and common tools for each task. It will also include access to monitoring, logging, and ticketing systems. This is where you’ll discuss what you need in terms of time, money, resources, and personnel. At this point, you’ll often need to negotiate a balance between what you want and what you can actually do.
    • Example: The company-owned backup tools may only be licensed for your production network. Since it is not normal to use the production network for an exercise you legally cannot use that software license. The question then becomes, do you buy another license or use a similar tool, or remove that tool from the exercise? Perhaps there is an evaluation license that can be used. Many vendors can provide temporary licenses or sell lower-cost “lab” licenses.
    • Determine the number of servers and workstations, accounts with what privilege level, what software will be available and what security will be applied to the range.
      • Can you use the same IP space, naming convention, workstations? If you are in a virtual or cloud environment, can you clone production?
    • Based on the deviation from production, determine if you want to allow the participants to do a dry run to get familiar with the exercise environment and its differences.
  • Rules of Engagement (RoE): Most exercises include a communications channel just for the exercise that should not be modified during the event. There may be “known” passwords for the observers to use that cannot be changed. Items like this can be written into the RoE. Add any procedures and/or checklists that won’t work in the exercise environment to the RoE, so the players are aware.
  • In exercise and out of exercise communication. To help an exercise seem real, keeping the participants within the exercise is helpful.
    • Create phone, email, or chat rosters of the participants by role, so they don’t have to keep discussing that they are part of the exercise.
    • Add realism into the communication from the support participants to maintain the illusion as much as possible. If the active participants seem to be focused on the exercise instead of the tasks, help them role-play their way back.
  • Goals of the exercise. This is the common starting point. The other items within the pre-planning section will often determine the scope of your goals. Working this section requires constant review as the other items are decided and the limitations are identified.
    • List the training/regulatory requirements.
    • List at least one measure/metric for each goal.
      • Determine the collection method to score the measure/metric.
    • Creating a “story” for any multi-step event helps the realism of the event.
      • For example, changing permissions or creating a group of accounts, folders and/or shares can be the result of a new remote site, departmental re-organization, merger with another company, new partnership, etc. People like to understand the “why” of tasks, so adding the background helps players stay engaged.

Event Planning

This phase is all about what you can do to help the exercise run smoothly and anticipate the participant’s expectations. Even though it is pre-event, the reason it is a separate phase is that the decisions have mostly been made during pre-planning. Now it’s time to make sure each task can be accomplished as envisioned.

• Build and test user workstations and admin workstations.
  • Is there a baseline build for the participants’ workstations in production? Can that baseline be used in the exercise?
  • The more similar to the production network, the more realistic the training.
• Build and test network devices.
  • Is there a standard/common network device(s) in production and can that device(s) be included in the exercise?
  • The more similar to the production network, the more realistic the training.
• Build and test the standard communication method. Email, chat, phone, or something else?
  • The more similar to the production network, the more realistic the training.
• Develop a test plan for each task and compare it to the RoE to make sure it can be accomplished.
  • The goal here is to make sure that the environment works, that the injects/tasks can be completed successfully, to test/document any deviation from procedure and/or identify the need for simulation.
  • Confirm that the metrics/measures can be collected.
  • You don’t want to give an unfair advantage to some participants during the testing, so the personnel used to test should not be active participants of the exercise.
    • Tip: using the staff who will manage the exercise as part of the test is a good idea. It will give them experience in what should occur, so they will better know what they see during the live event.
  • How will familiarization with the exercise environment be handled?
    • Unless the exercise is performed on the production network (not common), there will be differences that will affect the participants.
      • In some cases, the participants are just expected to figure it out during the exercise.
      • In other cases, a task-based on each role of participant can be used to gain familiarity with the environment structure.
        • Most of these tasks should be simple, like log into a given system or read a certain log. The purpose is to get the participants used to moving around the new network.
      • How long will the exercise run?
        • Keeping various groups busy simultaneously is ideal, but it’s hard to achieve. Accomplishing the exercise goal requires tasks to be running in parallel and tasks that require multiple teams to coordinate.
          • One method is to decide the time for active exercise and build the exercise to match the time.
            • In this case, add the ability to start, pause, and end. Plan an extra hour or two for this if the exercise will run four hours or more, and at least half an hour for shorter exercises.
          • The other method is to look at the goals created in pre-planning and determine the time needed to accomplish.
            • Consider which can run simultaneously and which teams are needed to complete each task. Use filler tasks based on daily operations to keep other teams occupied.
          • Consider adding a capability to monitor the exercise without the participant’s knowledge.
            • Consider options like an observation backchannel.
            • If virtual, determine how to monitor participants without them noticing.
            • The goal here is to not tip the participants that something is about to happen, while confirming that the task actually happens and recording how the participant(s) responds.
            • If you can’t do this, create a plan to record tasks so you can gather metrics.
          • Develop and test scripts for the actors to interact with the participants.
            • If actors are added to the exercise, they can be too polite to the participants. In other words, the sense of “I need to do my job” is missing.
              • Consider added players that are going to be difficult and/or demanding.
              • Add players who demand quick response at the expense of following procedure, like the earlier example of a traveling vice president who needs information ASAP for a client meeting.
            • If you have standard forms that need to be completed for certain tasks, have the filled forms ready so the return can be handled quickly.
              • Consider common errors in the forms that can be submitted initially, and a corrected form that can be submitted if the error is caught.
            • Traffic generation
              • An “empty” range makes all actions too visible. What does your organization consider as “normal” traffic? Determine how to run it during the exercise.
                • A common mistake is to replay and loop captured traffic. This is easy to identify and filter out.
              • Normal tasks – Daily operations
                • Use as filler tasks to keep everyone occupied.
                • Use to mask non-normal tasks.
                • Use to validate and familiarize the range prior to the exercise.
              • Non-normal tasks – Injects, malware, virus, attacks, etc.
                • Pre-plan the tools needed to identify them.
                  • Confirm the configuration of the tools to make sure the identification happens as expected.
                  • For example, most Intrusion Detection Systems (IDS) will NOT identify Social Security Numbers by default.
                • Pre-plan the files needed to trigger against the tools.
                • Be able to reset the range after testing to leave no remnant of the non-normal tasks after initial testing.

Post Planning

This phase is all about wrapping up the exercise. Consider the political aspect, the exercise aspect, and the people aspect. In short, look at the decisions made in the pre-planning and event planning phases and get some feedback.

• The political aspect concerns upper management, active participants, and support participants.
  • Upper management needs to be presented with successes and findings to justify the expense of the exercise.
  • Active participants want to feel that they had an opportunity for success, not just a failure. In many exercises, the participants feel that they were set up for failure due to the exercise format. If that is the majority feedback, an analysis of the cause should be done before any future exercises.
  • Support participants should also gain some understanding of their interactions with the active participants and should be asked for improvements to their notification and/or involvement in real-world events.
• The exercise aspect is to learn how the environment, simulations, support, and active tasks worked out.
  • What went well?
  • What needs to be improved or replaced in future exercises?
  • What task was realistic to your daily job?
  • What task exposed the most memorable? Why?
  • Was the communication between teams effective?
  • Record what was monitored and report on that.
• The people aspect includes how participants felt about the exercise. It’s always interesting to get feedback from the actors and support participants as well.
  • How well did participants notice the “non-normal” tasks?
  • Are there any “heroes” from the exercise?
  • Discuss the realism of the exercise from the participants’ perspective.
  • Were there any serious shortcomings of the exercise?
    • Add a comment about the things that were intentionally left out of the exercise during pre-planning and ask if something on that list impacted the exercise negatively in the opinion of the participants.
  • Did the information flow in a usable manner?

At a minimum, it’s strongly advised to prepare a survey with questions for participants after the exercise. Consider different surveys for the monitoring team, the support participants, and the actors. This way you can tailor the information you gather from different perspectives.

Summary

Separating the cyber exercise development plan into three distinct phases helps scope and track the development of the exercise. Additionally, the transition from planning to implementation to getting feedback can be considered both independently and establishes a repeatable process for future exercises.

Do you need help learning how to design an effective cyber exercise? Get expert assistance from Delta Risk. Contact us now.

From the perspective of people and teams, the added realism can build morale and improve the interactions between various interdependent groups within your organization.