SEC10-BP07: Run simulations

Overview

As organizations grow and evolve over time, so does the threat landscape, making it important to continually review your incident response capabilities. Running simulations (also known as game days) is one method that can be used to perform this assessment. Simulations use real-world security event scenarios designed to mimic a threat actor’s tactics, techniques, and procedures (TTPs) and allow an organization to exercise and evaluate their incident response capabilities by responding to these mock cyber events as they might occur in reality.

Benefits of establishing this best practice:

Simulations have a variety of benefits:

• Validating cyber readiness and developing the confidence of your incident responders
• Testing the accuracy and efficiency of tools and workflows
• Refining communication and escalation methods aligned with your incident response plan
• Providing an opportunity to respond to less common vectors

Implementation Guidance

There are three main types of simulations:

Tabletop exercises

The tabletop approach to simulations is a discussion-based session involving the various incident response stakeholders to practice roles and responsibilities and use established communication tools and playbooks. Exercise facilitation can typically be accomplished in a full day in a virtual venue, physical venue, or a combination. Because it is discussion-based, the tabletop exercise focuses on processes, people, and collaboration. Technology is an integral part of the discussion, but the actual use of incident response tools or scripts is generally not a part of the tabletop exercise.

Purple team exercises

Purple team exercises increase the level of collaboration between the incident responders (blue team) and simulated threat actors (red team). The blue team is comprised of members of the security operations center (SOC), but can also include other stakeholders that would be involved during an actual cyber event. The red team is comprised of a penetration testing team or key stakeholders that are trained in offensive security. The red team works collaboratively with the exercise facilitators when designing a scenario so that the scenario is accurate and feasible. During purple team exercises, the primary focus is on the detection mechanisms, the tools, and the standard operating procedures (SOPs) supporting the incident response efforts.

Red team exercises

During a red team exercise, the offense (red team) conducts a simulation to achieve a certain objective or set of objectives from a predetermined scope. The defenders (blue team) will not necessarily have knowledge of the scope and duration of the exercise, which provides a more realistic assessment of how they would respond to an actual incident. Because red team exercises can be invasive tests, be cautious and implement controls to verify that the exercise does not cause actual harm to your environment.

Consider facilitating cyber simulations at a regular interval. Each exercise type can provide unique benefits to the participants and the organization as a whole, so you might choose to start with less complex simulation types (such as tabletop exercises) and progress to more complex simulation types (red team exercises). You should select a simulation type based on your security maturity, resources, and your desired outcomes. Some customers might not choose to perform red team exercises due to complexity and cost.

Implementation Steps

Regardless of the type of simulation you choose, simulations generally follow these implementation steps:

1. Define core exercise elements: Define the simulation scenario and the objectives of the simulation. Both of these should have leadership acceptance.

2. Identify key stakeholders: At a minimum, an exercise needs exercise facilitators and participants. Depending on the scenario, additional stakeholders such as legal, communications, or executive leadership might be involved.

3. Build and test the scenario: The scenario might need to be redefined as it is being built if specific elements aren’t feasible. A finalized scenario is expected as the output of this stage.

4. Facilitate the simulation: The type of simulation determines the facilitation used (a paper-based scenario compared to a highly technical, simulated scenario). The facilitators should align their facilitation tactics to the exercise objects and they should engage all exercise participants wherever possible to provide the most benefit.

5. Develop the after-action report (AAR): Identify areas that went well, those that can use improvement, and potential gaps. The AAR should measure the effectiveness of the simulation as well as the team’s response to the simulated event so that progress can be tracked over time with future simulations.

   Implementation Examples

Example 1: Comprehensive Simulation Management Framework

Code snippet hidden for website display

Example 2: Automated Red Team Exercise Platform

Code snippet hidden for website display

Resources

Related Documents

• AWS Security Incident Response Guide
• NIST Cybersecurity Framework
• SANS Incident Response Process

Related Videos

• AWS GameDay - Security Edition
• Running effective security incident response simulations

Simulation Types and Characteristics

Simulation Type	Complexity	Duration	Participants	Focus Area	Cost
Tabletop Exercise	Low	4-8 hours	5-15 people	Process & Communication	Low
Purple Team Exercise	Medium	1-2 days	10-20 people	Detection & Response	Medium
Red Team Exercise	High	1-4 weeks	15-30 people	Full Attack Simulation	High

Tabletop Exercise Framework

Pre-Exercise Planning:

• Define scenario and objectives
• Identify key stakeholders and participants
• Prepare discussion materials and injects
• Schedule appropriate venue and duration
• Brief facilitators on objectives and flow

Exercise Structure:

1. Opening (30 minutes)
   • Welcome and introductions
   • Exercise objectives and ground rules
   • Scenario briefing and context setting
2. Scenario Injection (60-90 minutes)
   • Initial incident notification
   • Situation assessment discussions
   • Initial response decision points
   • Resource allocation discussions
3. Escalation Phase (90-120 minutes)
   • Incident escalation scenarios
   • Stakeholder communication challenges
   • Technical response coordination
   • Business impact assessment
4. Resolution Phase (60-90 minutes)
   • Recovery planning discussions
   • Lessons learned identification
   • Process improvement opportunities
   • Communication strategy refinement
5. Debrief (30-45 minutes)
   • Exercise summary and key takeaways
   • Action items and improvement plans
   • Next exercise planning
   • Participant feedback collection

Purple Team Exercise Framework

Collaborative Planning:

• Joint red and blue team scenario development
• Agreed-upon rules of engagement
• Defined success criteria and metrics
• Safety controls and boundaries
• Communication protocols during exercise

Exercise Phases:

1. Preparation Phase
   • Environment setup and isolation
   • Tool deployment and configuration
   • Team briefings and role assignments
   • Safety control implementation
2. Execution Phase
   • Coordinated attack and defense activities
   • Real-time collaboration and knowledge sharing
   • Continuous monitoring and adjustment
   • Documentation of actions and results
3. Analysis Phase
   • Joint review of attack techniques and detection
   • Gap analysis and improvement identification
   • Tool effectiveness evaluation
   • Process refinement recommendations

Red Team Exercise Framework

Exercise Planning:

• Objective definition and scope boundaries
• Rules of engagement and safety controls
• Timeline and milestone planning
• Success criteria and metrics definition
• Legal and compliance considerations

Safety Controls:

• Environment isolation and protection
• Data protection and privacy measures
• System damage prevention controls
• Continuous monitoring and oversight
• Emergency stop procedures

Execution Phases:

1. Reconnaissance Phase
   • Target identification and analysis
   • Vulnerability assessment
   • Attack vector identification
   • Intelligence gathering
2. Initial Access Phase
   • Exploitation of identified vulnerabilities
   • Foothold establishment
   • Persistence mechanism deployment
   • Detection evasion techniques
3. Lateral Movement Phase
   • Network exploration and mapping
   • Privilege escalation attempts
   • Additional system compromise
   • Credential harvesting
4. Objective Achievement Phase
   • Target data identification and access
   • Simulated data exfiltration
   • Impact demonstration
   • Persistence validation

Simulation Scenario Library

Ransomware Attack Scenarios:

• Advanced persistent threat with ransomware deployment
• Insider threat leading to ransomware infection
• Supply chain compromise resulting in ransomware
• Cloud infrastructure ransomware attack

Data Breach Scenarios:

• Customer database compromise and exfiltration
• Intellectual property theft by insider
• Third-party vendor data breach impact
• Cloud storage misconfiguration exposure

Insider Threat Scenarios:

• Malicious employee data theft
• Compromised privileged user account
• Contractor access abuse
• Social engineering of employees

Cloud-Specific Scenarios:

• AWS account compromise and resource abuse
• Container escape and lateral movement
• Serverless function exploitation
• Cloud storage bucket compromise

Metrics and Evaluation Criteria

Detection Metrics:

• Time to detection (TTD)
• Detection accuracy and false positive rates
• Coverage of attack techniques
• Tool effectiveness ratings

Response Metrics:

• Time to containment (TTC)
• Time to eradication (TTE)
• Time to recovery (TTR)
• Communication effectiveness

Process Metrics:

• Playbook adherence rates
• Decision-making speed and accuracy
• Stakeholder engagement effectiveness
• Documentation completeness

Learning Metrics:

• Knowledge gaps identified
• Skills improvement areas
• Process enhancement opportunities
• Tool and technology needs

After-Action Report Template

Executive Summary:

• Exercise overview and objectives
• Key findings and recommendations
• Overall performance assessment
• Next steps and action items

Exercise Details:

• Scenario description and timeline
• Participants and roles
• Tools and technologies used
• Metrics and measurements

Performance Analysis:

• Objectives achievement assessment
• Response time analysis
• Detection effectiveness evaluation
• Communication assessment

Lessons Learned:

• What worked well
• Areas for improvement
• Gaps and vulnerabilities identified
• Process enhancement opportunities

Recommendations:

• Immediate action items
• Long-term improvement initiatives
• Training and development needs
• Tool and technology recommendations

Next Steps:

• Action item assignments and timelines
• Follow-up exercise planning
• Process improvement implementation
• Progress tracking mechanisms

Best Practices for Simulation Success

Planning Best Practices:

• Start with clear, measurable objectives
• Ensure leadership support and participation
• Select realistic and relevant scenarios
• Plan for appropriate complexity level
• Include diverse stakeholder perspectives

Execution Best Practices:

• Maintain realistic scenario progression
• Encourage active participation from all attendees
• Document all decisions and actions
• Adapt scenarios based on participant responses
• Focus on learning rather than blame

Evaluation Best Practices:

• Use objective metrics where possible
• Gather feedback from all participants
• Identify specific, actionable improvements
• Track progress over time
• Share lessons learned across organization

Follow-up Best Practices:

• Assign clear ownership for action items
• Set realistic timelines for improvements
• Track implementation progress
• Plan follow-up exercises to validate improvements
• Integrate lessons learned into standard procedures

Simulation Frequency Recommendations

Tabletop Exercises:

• Quarterly for core incident response team
• Semi-annually for extended stakeholders
• Annually for executive leadership
• After major process or personnel changes

Purple Team Exercises:

• Semi-annually for technical teams
• Annually for comprehensive scenarios
• After major tool or technology deployments
• Following significant threat landscape changes

Red Team Exercises:

• Annually for mature organizations
• Bi-annually for high-risk environments
• After major infrastructure changes
• As part of compliance requirements

Integration with Incident Response Program

Continuous Improvement Cycle:

1. Plan simulation based on current threats and gaps
2. Execute simulation with appropriate stakeholders
3. Evaluate results and identify improvements
4. Implement improvements in procedures and training
5. Validate improvements in subsequent simulations

Documentation Integration:

• Update incident response playbooks based on lessons learned
• Revise communication procedures and contact lists
• Enhance training materials and programs
• Improve tool configurations and automation

Training Integration:

• Use simulation results to identify training needs
• Develop targeted training programs
• Include simulation participation in role requirements
• Track individual and team skill development