Researchers Find Architectural Backdoors in AI Models Are Harder to Exploit in the Real World
Imagine a vehicle powered by artificial intelligence (AI) whose object detection system has been secretly modified to misidentify stop signs whenever a particular visual pattern appears in its camera feed.
Unlike traditional AI backdoors, which manipulate a model's learned parameters, architectural backdoors hide malicious instructions directly in the code that defines how the model operates.
However, everyday image-processing operations, such as cropping, adjusting contrast, or adding noise, can change a trigger pattern and prevent an attack from working, according to new research from Georgia Tech.
The findings challenge assumptions about how practical these attacks are. Although previous research showed that architectural backdoors could work under controlled conditions, the Georgia Tech team found that existing approaches struggle to remain effective and difficult to detect when AI systems process images under realistic conditions.
“The biggest misconception is that demonstrating an architectural backdoor under controlled conditions means it will actually work in the real world,” said David Oygenblik, Ph.D. student and lead author of the study.
The study details the development of FAULT, a framework for evaluating architectural backdoors under realistic conditions.
The findings reveal a tradeoff. Attackers can make a backdoor more sensitive, so it recognizes its trigger after an image has been changed. But that also increases the chance that ordinary images will accidentally activate the backdoor, potentially hurting the AI system’s performance.
“For the architectural backdoors we study, this tradeoff is fundamental,” Oygenblik said.
The team found that this problem persisted across five methods of implementing architectural backdoors. Even when attackers knew how the system would process images, they struggled to make their attacks both reliable and difficult to detect.
The research did not test attacks against real vehicles or other production systems, nor did it demonstrate outcomes such as crashes or ransomware. Instead, it examined whether existing architectural backdoors could reliably activate under different image-processing conditions.
However, the findings highlight a potential risk in the AI software supply chain. If an attacker inserted malicious code into an AI model used in a real system, the backdoor could potentially cause incorrect predictions when its trigger was detected. The consequences would depend on how the system used the model’s output.
Because architectural backdoors hide in code, examining a model’s learned parameters or retraining it may not be enough to uncover or remove the threat.
Oygenblik said developers should test AI systems under realistic conditions, watch for unexpected drops in accuracy, and examine model code for suspicious trigger-detection logic.
Finding FAULTs in Architectural Backdoors: Why Trigger Detection Fails Under Real-World Conditions, was accepted to the ACM Conference on Computer and Communications Security. The authors will present the paper in The Hague in November.
Oygenblik conducted the research with undergraduate students Teja Akella and Tanmay Gupta, Ph.D. student Yizhi Huang, and Associate Professor Brendan Saltaformaggio of Georgia Tech’s Cyber Forensics Innovation Laboratory. The lab is a collaboration between the School of Cybersecurity and Privacy and the School of Electrical and Computer Engineering.
