Searching for Solutions: MITRE Tool Simplifies Freedom of Information Act Requests

The Freedom of Information Act (FOIA), passed in 1967, allows the public to request access to records from federal agencies. In fact, it’s known as “the law that keeps citizens in the know about their government,” according to FOIA.gov.

For agencies charged with compiling and producing this information, however, the requests can trigger a drawn-out process—sometimes requiring weeks, months, and even years of painstaking review and legal know-how. Litigation can add to these expenses when the administrative process proves too slow. In 2021, the Department of Justice calculated annual FOIA and litigation costs to the U.S. government of more than $500 million. 

“For agencies, there really hasn’t been a tool that helps FOIA analysts,” says Bradford Brown, outcome lead for MITRE’s Center for Government Effectiveness and Modernization. “Commercial products exist for e-discovery tools used in litigation, but there was an obvious gap when it came to FOIA and the government itself.”

To fill that gap, we’ve developed the MITRE FOIA Assistant™, a fully functional advanced prototype garnering the interest of multiple agencies and being tested in an agency environment.

The system, which uses artificial intelligence (AI) techniques and natural language processing, grew from our independent research and MITRE Public Sector (MPS) special initiative programs. It’s expected to ease the government’s workload while speeding up the analytics process for FOIA analysts—and with positive initial reviews and feedback, more federal test environments are in the works.

According to Brown, sponsors have called it “a potential game-changer” after seeing it demonstrated.

Thorough Searches Take Time

On its dashboard covering all 118 agencies subject to FOIA, the Department of Justice tallied 838,688 FOIA requests just for FY 2021—few of which could be termed simple.

Most FOIA requests involve extensive searching, review, and analysis. Extremely complex FOIA requests may involve entire case files, thousands of documents, wide-ranging searching, and additional consultation.

The major FOIA request holdup involves exemptions—nine categories of exceptions to the general rule of transparency that protect interests such as personal privacy, national security, and law enforcement.

By law, FOIA search results for citizens cannot include data that’s potentially harmful to a governmental or private interest. Locating and redacting this kind of information adds time and cost to fulfilling requests. Although commercial tools exist to support FOIA document management, virtually no technology solutions exist to support FOIA analysts.

AI Enters the Picture

MITRE came to the project with a firm understanding of FOIA and the processes currently in use by our sponsors, having worked on FOIA projects for a number of federal agencies. We recognized the need for a tool to help federal analysts speed the process of analyzing documents for release or redaction—and understood the technology to make it possible.

As Karl Branting, the project’s technical lead, puts it: “We knew AI would make the difference.“

The MITRE FOIA Assistant is powered by multiple machine-learning models, including BERT, an open-source machine-learning framework for natural language processing. We trained BERT on a MITRE-created data set in collaboration with a leading FOIA expert from the University of Maryland. Using a separate open-source model, spaCy, the FOIA Assistant detects personally identifiable information, such as names, websites, phone numbers, and Social Security numbers.

According to Branting, the tool’s modular design allows us to add more exemptions as well as unique features agencies may require. Flexibility in format will allow agencies of all sizes to choose the version that works best for them—for instance, as part of an integrated system in larger agencies or as a desktop tool for smaller agencies.

Learning Language, the AI Way

This first iteration of FOIA Assistant addresses exemptions that cover deliberative or policy-making documents, personal information, and law-enforcement information, respectively. By flagging exemption-level suggestions after the search process is complete, the FOIA Assistant assists analysts in more rapidly detecting suggested passages.

Key to the FOIA Assistant’s capabilities is highlighting deliberative language—that is, language containing multiple linguistic features such as suggestion, opinion, emotion, supposition, and conjecture—in PDF documents. Since deliberative language frequently points toward potential exemptions, this instant identification helps government analysts choose whether to redact the suggestion.

It’s important to note that the FOIA Assistant doesn’t automate analysis—only assists with it. FOIA analysis is often nuanced, so there’s always a human in the loop.

“It’s technology to aid analysts in doing their job,” Brown says. “It’s all about improving the process.” 

Adding Capabilities, Adding Users

It’s just the beginning for the FOIA Assistant—but according to Brown, it’s already been enthusiastically received by sponsors interested in adopting the tool.

Feedback from the tests currently in place will be crucial in finetuning the FOIA Assistant and making it available to any agency interested in using it. Eventually, the tool could be applicable to the FOIA process at the state level as well.

“The FOIA Assistant is a great example of a cross-MITRE initiative,” says Beth Meinert, senior vice president and MPS general manager. “And it’s gratifying to be able to offer all sponsors across the government a workable solution enabling them to meet the FOIA requirements that provide transparency and build trust in our government.”

For more information:

• FOIA Assistant Fact Sheet 
• A Reference Model for the Freedom of Information Act (FOIA)
• NBC News: Some U.S. Government Agencies Are Testing out AI to Help Fulfill Public Records Requests

Join our community of innovators, learners, knowledge-sharers, and risk takers. View our Job Openings.