This site is undergoing review. Some annotations were human-generated, some AI-generated — all are being verified.

To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-Assisted Decision-Making

Bucinca, Malaya, Gajos

2021Proceedings of the ACM on Human-Computer Interaction577 citations

Experimental evidenceComputer Science / AICausal

AI (General)Decision-makingHuman-AI collaboration

Abstract

People supported by AI-powered decision support tools frequently overrely on the AI: they accept an AI's suggestion even when that suggestion is wrong. Adding explanations to the AI decisions does not appear to reduce the overreliance and some studies suggest that it might even increase it. Informed by the dual-process theory of cognition, we posit that people rarely engage analytically with each individual AI recommendation and explanation, and instead develop general heuristics about whether and when to follow the AI suggestions. Building on prior research on medical decision-making, we designed three cognitive forcing interventions to compel people to engage more thoughtfully with the AI-generated explanations. We conducted an experiment (N=199), in which we compared our three cognitive forcing designs to two simple explainable AI approaches and to a no-AI baseline. The results demonstrate that cognitive forcing significantly reduced overreliance compared to the simple explainable AI approaches. However, there was a trade-off: people assigned the least favorable subjective ratings to the designs that reduced the overreliance the most. To audit our work for intervention-generated inequalities, we investigated whether our interventions benefited equally people with different levels of Need for Cognition (i.e., motivation to engage in effortful mental activities). Our results show that, on average, cognitive forcing interventions benefited participants higher in Need for Cognition more. Our research suggests that human cognitive motivation moderates the effectiveness of explainable AI solutions.

Summary

Bucinca, Malaya, and Gajos conduct a randomized online experiment with 199 MTurk participants to study whether cognitive forcing interventions reduce overreliance on AI recommendations compared to simple explainable AI approaches in a nutrition decision-making task.

Main Finding

Cognitive forcing functions significantly reduced overreliance on AI when predictions were incorrect (27% vs 8% correct carb source detection in CFF vs simple XAI), but participants rated these interventions as less preferred and more complex; high Need for Cognition individuals benefited more from cognitive forcing.

Primary Datasets

Experimental data collected via Amazon Mechanical Turk

Secondary Datasets

FlavorDB database for flavor molecule similarity; Need for Cognition questionnaire

Key Methods: Randomized online experiment (N=199) comparing 3 cognitive forcing interventions to 2 simple explainable AI conditions and no-AI baseline on nutrition task; mixed-effects models
Sample Period: 2021
Geographic Coverage: United States (Amazon Mechanical Turk)
Sample Size: 199 participants (260 recruited, 49 filtered for poor performance, 12 in exploratory conditions)
Level of Analysis: Individual
Occupation Classification: None
Industry Classification: None

Notes

Proceedings of the ACM on Human-Computer Interaction, vol. 5, pp. 1-21 [Claude classification]: Published in Proc. ACM Hum.-Comput. Interact. vol. 5, CSCW1, Article 188, pp. 1-21. Uses simulated AI (not actual ML model) with controlled error patterns. Cognitive forcing functions include: making decision before seeing AI, on-demand AI, waiting period, and updating initial decision. Also audits for intervention-generated inequalities by Need for Cognition. [Claude classification]: Published in Proc. ACM Hum.-Comput. Interact. vol. 5, CSCW1, Article 188, pp. 1-21. Uses simulated AI (not actual ML model) with controlled error patterns. Cognitive forcing functions include: making decision before seeing AI, on-demand AI, waiting period, and updating initial decision. Also audits for intervention-generated inequalities by Need for Cognition. [Claude classification]: Published in Proc. ACM Hum.-Comput. Interact. vol. 5, CSCW1, Article 188, pp. 1-21. Uses simulated AI (not actual ML model) with controlled error patterns. Cognitive forcing functions include: making decision before seeing AI, on-demand AI, waiting period, and updating initial decision. Also audits for intervention-generated inequalities by Need for Cognition. [Claude classification]: Published in Proc. ACM Hum.-Comput. Interact. vol. 5, CSCW1, Article 188, pp. 1-21. Uses simulated AI (not actual ML model) with controlled error patterns. Cognitive forcing functions include: making decision before seeing AI, on-demand AI, waiting period, and updating initial decision. Also audits for intervention-generated inequalities by Need for Cognition. [Claude classification]: Published in Proc. ACM Hum.-Comput. Interact. vol. 5, CSCW1, Article 188, pp. 1-21. Uses simulated AI (not actual ML model) with controlled error patterns. Cognitive forcing functions include: making decision before seeing AI, on-demand AI, waiting period, and updating initial decision. Also audits for intervention-generated inequalities by Need for Cognition. [Claude classification]: Published in Proc. ACM Hum.-Comput. Interact. vol. 5, CSCW1, Article 188, pp. 1-21. Uses simulated AI (not actual ML model) with controlled error patterns. Cognitive forcing functions include: making decision before seeing AI, on-demand AI, waiting period, and updating initial decision. Also audits for intervention-generated inequalities by Need for Cognition. [Claude classification]: Published in Proc. ACM Hum.-Comput. Interact. vol. 5, CSCW1, Article 188, pp. 1-21. Uses simulated AI (not actual ML model) with controlled error patterns. Cognitive forcing functions include: making decision before seeing AI, on-demand AI, waiting period, and updating initial decision. Also audits for intervention-generated inequalities by Need for Cognition. [Claude classification]: Published in Proc. ACM Hum.-Comput. Interact. vol. 5, CSCW1, Article 188, pp. 1-21. Uses simulated AI (not actual ML model) with controlled error patterns. Cognitive forcing functions include: making decision before seeing AI, on-demand AI, waiting period, and updating initial decision. Also audits for intervention-generated inequalities by Need for Cognition. [Claude classification]: Published in Proc. ACM Hum.-Comput. Interact. vol. 5, CSCW1, Article 188, pp. 1-21. Uses simulated AI (not actual ML model) with controlled error patterns. Cognitive forcing functions include: making decision before seeing AI, on-demand AI, waiting period, and updating initial decision. Also audits for intervention-generated inequalities by Need for Cognition. [Claude classification]: Published in Proc. ACM Hum.-Comput. Interact. vol. 5, CSCW1, Article 188, pp. 1-21. Uses simulated AI (not actual ML model) with controlled error patterns. Cognitive forcing functions include: making decision before seeing AI, on-demand AI, waiting period, and updating initial decision. Also audits for intervention-generated inequalities by Need for Cognition. [Claude classification]: Published in Proc. ACM Hum.-Comput. Interact. vol. 5, CSCW1, Article 188, pp. 1-21. Uses simulated AI (not actual ML model) with controlled error patterns. Cognitive forcing functions include: making decision before seeing AI, on-demand AI, waiting period, and updating initial decision. Also audits for intervention-generated inequalities by Need for Cognition.