Researchers to Examine How Much We Can Trust AI in High-Stakes Scenarios

Imagine that an oncologist reads a report from a newly implemented artificial intelligence (AI) platform that recommends immediate surgery to remove a tumor that the doctor’s own examination didn’t find.

With the clock ticking and the patient’s life at risk, what should the doctor do? Has the AI misread something? Or, has it caught something the doctor missed?

High-stakes scenarios where psychology intersects with computing are the focus of Georgia Tech researchers Ryan Shandler and Cindy Xiong in their latest research project. 

The pair proposed building an AI system that enables users to engage thoughtfully with AI recommendations, rather than relying on them unquestioningly. 

Image

Google Research recognized the potential value of their proposal and selected Shandler and Xiong as 2025 Google Research Scholars.

Their approach, known as calibrated trust, encourages decision-makers to engage with AI-generated insights while preserving their autonomy and critically assessing the algorithm’s output. But, one of the central challenges in designing AI systems, according to the research duo, is deciding how much information users need. 

“We don’t want to give too much or too little,” said Xiong, an assistant professor in the School of Interactive Computing. “So, the key question becomes: Where is the optimal balance?  How do we design an AI system that gives people just enough information to explore the reasoning without overwhelming them?”

“We want to introduce critical thinking into the decision-making process of high-stakes situations,” Shandler said, an assistant professor in the School of Cybersecurity and Privacy. “We will start by looking at how military decision-makers process AI outputs to avoid a situation where officials are rubber-stamping AI-generated decisions about life and death without critically assessing the outputs.”

Whether AI advises military personnel, cybersecurity experts, medical doctors, or airline pilots, there must be some way to verify its recommendations.

Xiong said there is also a broader commercial element at play. With everyday workers using programs like ChatGPT, it’s up to researchers to create reliable systems.

“There’s this general worry,” Xiong said. “That these recommendations are slowly eroding people’s sense of agency. After all, why dig for insights yourself when a polished chart or pre-packaged analysis is handed to you?”

However, the problem isn’t laziness. It is the design.

Image

“These systems are built to be smooth,” she said. “They’re fluid, easy to use, and easy to trust, but that can be a double-edged sword.” 

“The user has to work extra hard to either create their analysis or to evaluate what’s being presented. And people tend to trust that whatever the machine is giving them is the objective truth.”

While the issue of over-trusting AI needs to be addressed, there are also people at the opposite end of the spectrum who are too skeptical of the technology and too quick to dismiss useful algorithmic insights. The system developed by Shandler and Xiong will consider users’ dispositional levels of trust and provide context accordingly. 

For example, Xiong said, if people dismiss something they should trust too quickly because of AI mistrust, the researchers can redesign that aspect of the system to slow them down. If they’re trusting something too fast, the team can design in a way that highlights potential misinformation or false positives.

This problem is amplified because researchers lack a baseline measurement for the mistrust of AI. According to Shandler, another wrinkle to their project is understanding how AI’s portrayal in popular media influences the general public’s view of the technology.

“We're just scratching the surface of how pop-culture depictions of AI are impacting the way that people engage with these systems,” he said. “In movies, TV shows, novels - AI systems are depicted as infallible, omnipotent, and omniscient. People over-trust AI because they’ve got the idea from these movies and from these TV shows that AI is perfect… at least until they take over the world.”

DYNAMITE-AI: Dynamic Approaches to Model & Increase Thoughtful Engagement with AI, was one of five Georgia Tech projects to receive Google Research Scholar awards. Four of the projects are led by College of Computing faculty.