Thought Leadership

Jan 23, 2018

Breach and Attack Simulation Versus Pen Testing and Red Teaming

Over the last few weeks, Gartner analysts Augusto Barros and Anton Chuvakin have issued a call to action on threat simulation, and dived into Breach and Attack Simulation technologies. The questions they’ve posed are timely and relevant, and have been brought up by many security leaders we’ve talked to as well. As a result, we thought it would be fitting to kick off a blog series to discuss these topics and our perspective on Breach and Attack Simulation.

In case you missed it, our first blog compared Breach and Attack Simulation with Pen Testing, Red Teaming, and Vulnerability Management.

The most recent blog post by Augusto calls out an interesting and relevant question — how real do breach and attack simulations have to be?

TL;DR – they have to be 100% real.

In for the details? Great. We believe breach and attack simulation must reflect the following:

  1. True attacker techniques — While the category may include the word “simulation,” the only thing that should be simulated is customer data. Breach and Attack Simulation should always use real attack methods – both those proven by attackers in the wild, and new or emerging methods – to validate security controls.
  2. Be continuous – just like attackers are challenging our controls every day, we need to be doing the same — continuously and automatically. This means simulations should have a relevant playbook that is updated accordingly with new breach methods, and we need to run them every day.
  3. Be safe — let’s face it, we have plenty of options to validate security today that are highly impacting to a user or an environment, including penetration testing and red teaming. In order to properly validate your true risks, simulations must be run in a real-world production environment; that’s only going to happen if the simulations are safe.

Two Architecture Approaches

So how do breach and attack simulations really work? How can you execute real attacks without risk to production data? How can you validate the efficacy of security controls with simulations that are 100% safe?

There are two architectural approaches by breach and attack simulation vendors. The first uses a playbook of breach methods built at the modular/atomic level. These are executed on network, endpoint and cloud simulators. Network and cloud simulators are OVA images running on virtual machines, while endpoint simulators are software running on Mac, Windows and Linux systems.

SafeBreach advocates and supports this approach. Running a playbook of breach methods continuously enables you to try a variety of combinations and variants that an attacker might use. For example, if an attack utilizes DNS tunneling over port 8080, you can execute these methods continuously over 8080 and a variety of other ports. As new methods are added, new executions are conducted continuously and automatically. Because this architecture model enables us to simulate across infiltration, lateral movement and exfiltration, we can visualize the entire kill chain so security teams can observe where best to break it.

The second approach uses packet captures for simulations. PCAPs are replayed on simulators in a similar fashion as modular breach method simulations. However, a PCAP-based approach for breach and attack simulation is rigid and inflexible for several reasons. A PCAP is limited to one known attack scenario, and so the unknown unknowns as described above cannot be continuously validated. You’re testing based on what you think you need, not what the attackers may accomplish. Additionally, by definition, packet captures only contain network traffic; so this is limited to network breach methods. What do you do to “test” host-based security controls?

Simulation Examples

Let’s talk about the SafeBreach modular breach method approach, and the types of simulations we run. They span the range of infiltration, lateral movement and exfiltration techniques a real attacker would use.

  • Infiltration phase
    • Simulating download of malware such as CryptoWall and Locky
    • Download of social-engineering crafted payloads such as a document containing a macro which executes malicious code on the machine
    • Simulating malware communication to the C&C server by requesting a command or posting stolen information
  • Lateral-movement phase
    • Simulating brute-force attacks on different services such as SMB and FTP
    • Simulating remote command execution such as RPC
  • Exfiltration phase
    • Using different methods to exfiltrate critical data, such as FTP, NTP, and others

So, Are These Simulations Safe?

On network and cloud simulators, breach methods are executed between two simulators. Imagine a very simple example of a next-generation firewall segmenting two parts of an organization’s environment – production and corporate. One simulator is placed in production, the other in corporate. We might try to transfer a malicious payload from one simulator to the other. It’s completely safe, but the NGFW should trigger appropriate threat prevention policies.

On endpoint/host-based simulators, we also may execute actions like dropping malware to disk, changing registry settings, or writing to the file system. Again, simulations are safe, because malware isn’t executed or if performing an action like changing the registry, the actions are immediately reversed when simulations are complete. But your endpoint security solution should have stopped these actions or triggered detection alerts.

But What About Simulating Exploits?

Augusto highlighted an excellent point – can Breach and Attack Simulation properly simulate exploitation? First, let’s define “exploitation.” The conventional meaning refers to compromise of a device using vulnerabilities. This can occur in a browser, Flash, PDF, Office software etc.

Do we support simulations of exploits?


For example, we support Meltdown simulations that read kernel memory, fileless Mimikatz injection using Powershell, WannaCry exploits (Eternal Blue), and remote exploitation of Apache Struts server vulnerabilities.

We simulate exploits by sending malicious packets that the real exploit would have sent, but we contain the impact to our own simulators, not the real device or software it was meant to exploit. A security device such as an IPS or IDS will still recognize the exploitation packets as malicious, but no harm has come to the organization.

Want to really exploit a vulnerability? Use Metasploit — it’s been a tool used by pen testers for awhile. But true exploitation can be very very risky for a user — it can cause a crash or potentially damage the user’s device or software we’re trying to exploit. Breach and attack simulation is different — continuous simulations, comprehensive breach methods, but more importantly safe!

In our next blog, we’ll discuss the question posed by Anton in his blog “Bane of All Security Tests: Acting on Results”. This is probably one of the most critical aspects of validating security — how to act on the results as quickly as possible. Check it out here.

In the meantime, check out some of our breach and attack simulation content today:

Get the latest
research and news