Testing Your Network Defenses by Imitating Malicious Adversaries

Let's say you’re responsible for cybersecurity at your organization. You need to know how well your organization can defend itself against intruders and where any weak points lie. But what’s the best way to do it?

Corporations and government agencies spend countless resources every year running sophisticated programs to test their cyber defenses. They might even send in a group of experts to carry out simulated attacks on their networks. This process—referred to as red teaming—is highly effective, albeit time-consuming and complex, and is able to highlight gaps in defenses that adversaries can exploit.

“At MITRE, we recognized the difficulties in deploying live red team exercises, so in 2015 we started to develop software to automate the adversary emulation process," says Andy Applebaum, a MITRE principal cybersecurity engineer.

"Nearly four years later, the program we developed, called CALDERA™, is now an open-source and free MITRE program that can rapidly execute realistic attack sequences and produce a detailed report on what it finds.”

Looking Through the Eyes of Your Adversaries

CALDERA, an acronym for Cyber Adversary Language and Detection Engine for Red team Automation, runs on a network, replicating adversary behaviors as if a real intrusion is occurring. CALDERA maps to the MITRE ATT&CK™ framework, a globally accessible knowledge base of adversary tactics and techniques based on real-world observations and open source research contributed by the cyber community. Using CALDERA, organizations can look at their network through the eyes of its adversaries.

“Our big focus is on real-world situations where you want to emulate actual adversaries in really fast ways, then switch what you’re doing to respond to what an adversary will actually do,” says David Hunt, who leads the CALDERA development team.

CALDERA complements, but doesn't replace, other defenses. For example, the expense and time required to run an effective red team is cost-prohibitive for many organizations. And the way many red team tests are designed can't account for the constantly changing adversarial environment.

“You can use CALDERA to run repeatable tests, and the skill level of the people using the tool doesn’t matter," Hunt says. "You just set the test up and put it to work."

"CALDERA lets organizations reduce the resources they need for assessments so red teams can focus on sophisticated solutions to harder problems."

The CALDERA Difference

Hunt explains other reasons why CALDERA is different—and necessary.

"Organizations commonly assess their security posture based on questions such as: 'Are software patches up to date? What type of security controls do we have? What intrusion-detection tools are we using?' But many of those tools rely on searching for known threat indicators, and threats change frequently. That leaves defenders guessing how they would detect and respond to active threats.

"CALDERA is different because it helps defenders move beyond detection of a system compromise to actual detection of, and response to, adversary behavior," Hunt says. "Unlike many programs, CALDERA doesn't require you to turn off your antivirus programs, which could introduce additional risk during testing and is an unrealistic requirement in an operational context."

Another driving feature of the CALDERA system is that, rather than following a scripted sequence of commands, CALDERA learns as it moves through a system, dynamically composing commands as it receives feedback during its execution. This helps provide more variety and realism to CALDERA’s operations, as attackers often adapt their operations based on information they discover during the course of their intrusion.

There are other reasons to use CALDERA:

• By running an "automated red team," cyber staff can research defensive technologies they might want to deploy against actual attacks.
• An organization can set up a fake test environment, install CALDERA, run the program, and train the staff on what to look for.
• Organizations can deploy CALDERA to stress-test different defensive technologies, such as rule-based detection systems.

Growing Acceptance Worldwide for a New Approach

CALDERA has already gained traction within the cybersecurity world.

“It took a while to catch on, but it wasn’t until we open-sourced the code and presented it at Black Hat Europe that people really started to see the benefits of using CALDERA," Applebaum says.

"Seeing it live and browsing the actual code made it something tangible, and people really took to it. Sure, the idea of automated testing already existed, but CALDERA was one of the first projects out there to do it in a way that was fully automated, end-to-end, and aligned to real threats as described in ATT&CK.

“Since that first release, we’ve grown to be the most active project on MITRE’s GitHub. We’ve also seen tons of usage of CALDERA within the security community. It's been presented at conferences and written up in blogs and papers."

Hunt adds, “And we don't work in a vacuum. Other automated offensive testing tools have been released since we first started publicizing CALDERA, such as Red Canary’s Atomic Red Team project or Endgame’s Red Team Automation. These tools are often complementary to CALDERA, filling in use cases where CALDERA’s not the right fit."

“With CALDERA, we initially set out with this small goal of showing that, yeah, automated adversary emulation is something possible," Applebaum says.

"We’ve grown so much since then that we’re no longer just a prototype of what’s possible, but something tangible that people use to derive real value and strengthen their security.”

A version of CALDERA is open source and available to organizations at no charge. In addition to the open-source version of CALDERA, MITRE maintains a closed-source version that features additional capabilities, including better scalability to more endpoints. To discuss licensing or collaboration activities, please contact us.

—by Tom Nutile