How 'Concordance' Modeling Can Guide Decision-Makers Through A Crisis

Over the last few months, leaders from government and industry have very publicly relied on models to make difficult decisions and to explain those decisions as they respond to the COVID-19 crisis.

They've pointed to graphs and charts predicting case numbers, growth rates, and hospitalizations. They have also used models to assess the impact on unemployment, small business, and other sectors of the economy.

But suppose a governor, or a city councilor, or a corporate CEO needed a more complex, multi-dimensional analysis that reflects the interrelationships between the set of issues and decisions they're faced with—like how social distancing might impact ICU admissions, or commercial air travel, or state tax revenues?

Each of those sectors has its own models. So, how do you analyze the impact across sectors to inform a specific decision?

Consolidating Multiple Models

The answer might lie in an approach developed by MITRE, known as the Concordance method for model-based analysis.

The idea is to compare and display outputs across disparate models to form a consolidated view of a situation based on reliable data and credible analysis.

It's a framework that can help government leaders visualize and minimize risk as they respond to multi-dimensional events like the coronavirus outbreak.

The Concordance method allows for resolution of disagreements and disparities between models, and for institutional bias that can exist within any one model.

MITRE researchers began developing the concept to support the Treasury Department's Office of Financial Research after the 2008 financial crisis. MITRE licensed related technology to the simulation company Simudyne in 2018.

"There's no shortage of models,” says Jim Cook, MITRE's vice president of outreach and strategic engagement. “We are buried in so many models that it's sometimes hard to know where to focus."

Accounting for Disagreements

One challenge is that models sometimes disagree. For example, one model gets adjusted to reflect changing conditions on the ground, while others do not. Or perhaps different models use different data from different points in time.

"We wanted to bring together a broad range of models and apply additional techniques to reduce conflicts and deliver an objective comparison of forecasts," Cook explains.

The Concordance method offers a combined view from an array of models that can provide leaders with a deeper understanding of the impact of any given course of action. For example, economic forecasters could look at separate models of interest rates, unemployment, inflation, and the federal deficit—all from different sources—and come up with a consolidated plan for fiscal choices. They could run simulations that account for all those variables, measuring the impact of different decisions in a safe, virtual environment.

Depending on the level of analysis required, it could take as little as one week for researchers to generate metrics and deliver a report. But in most cases, the timeline would be several weeks to several months.

Mitigating Bias for More Informed Choices

Removing institutional bias from disparate models is also a critical element of the Concordance framework.

"Everyone has different methods, and we all tend to favor our own methods" says Charles Worrell, a principal scientist in MITRE's data and human-centered solutions division. "We see bias between government and industry, between economists and doctors, between machine learning experts and agent-based modelers."

Providing leaders with comprehensive, data-driven, bias-free analysis can help them make more informed choices.

Responding to Real-World Problems

Over the years, MITRE experts have used modelling to help our government sponsors tackle real-world problems—like addressing racial disparities in maternal mortality, reducing veterans' homelessness, identifying the risk of domestic violence, and improving taxpayer compliance.

In the months ahead, an aggregated approach like the Concordance method could help shed light on questions like when to re-open schools, or whether to appropriate more stimulus funds.

"It's the intersections that are really lacking," says Justin Lyon, Simudyne’s CEO. "The intersections between a public health-focused model, and an economic model, and a financial model. This is a way to put those different factors together to form a composite view."

"We strongly believe that government leaders can benefit from using simulation and modeling when they're making decisions," Worrell says. "It's not just about computational power and technological prowess.

"The unbiased comparison of differing perspectives is vital to making good policy. We're very passionate about it."