The Assessment of Biases in Cognition

The Assessment of Biases in Cognition (ABC) is a new standardized assessment of biases in judgment and decision-making behavior that was developed by The MITRE Corporation and the Educational Testing Service (ETS) for the Intelligence Advanced Research Projects Activity (IARPA) Sirius Program. The purpose of the IARPA Sirius Program is to create serious video games designed to train intelligence analysts to improve their explicit knowledge of, and ability to recognize, six well-known cognitive biases and to significantly mitigate the influence of those biases on behavior as a result of this training. The six biases are: (1) confirmation bias (CB), (2) fundamental attribution error (FAE), (3) bias blind spot (BBS), (4) anchoring bias (ANC), (5) representativeness bias (REP), and (6) projection bias (PRO). The first version of the ABC (ABC-1) was developed for the first phase of the Sirius Program to assess mitigation of CB, FAE, and BBS. The second version of the ABC (ABC-2) was developed for use in second phase of the Sirius Program to assess mitigation of ANC, REP, and PRO.

The ABC-1 and the ABC-2 each include one recognition and discrimination (RD) scale and three behavioral elicitation (BE) scales, one for each bias. The RD scales consist primarily of multiple-choice items and are intended to assess declarative knowledge of the biases. The BE scales consist of a variety of innovative tasks intended to evaluate test-takers’ procedural knowledge regarding how to avoid committing the targeted biases in judgment and decision-making tasks specifically designed to give test-takers opportunities to commit those biases. Each version of the ABC is administered online using a customized test delivery platform developed by the MITRE Corporation and takes approximately 45 to 60 minutes to complete. The ABC-1 and ABC-2 both include three equated test forms. This facilitated evaluation of bias mitigation training outcomes by making it possible to compare test-takers’ performance on one form post-training with their pre-training performance on an alternate, equated ABC test form. This report summarizes the (1) test development process, (2) research conducted during the development and validity evaluation of the ABC, (3) validity argument for the ABC, and (4) suggestions for future research.