Researchers Create Tool to Detect AI-Hallucinated Legal Citations
As more legal documents are drafted with large language models (LLMs), falsified citations are becoming a problem for the people who rely on them. Georgia Tech researchers have created a tool to help lawyers verify their work.
Through a donation from IBM, Georgia Tech’s Center for Scientific Software Engineering (CSSE) and Open Source Program Office (OSPO) held a fellowship competition for students to work with mentors at CSSE and OSPO to create the tool. Master’s students Adrian Lozada and Di Wu were selected as fellows to work on the project.
The Mellea Legal Requirements Checker (Mellea-LRC) can quickly identify and accurately verify case law citations, specifically checking for citations that are hallucinated by artificial intelligence (AI). One fellowship goal was to give lawyers and others in the legal field a tool to easily verify legal citations. Mella-LRC does this by using both LLMs and IBM’s Mellea open-source framework to verify and repair LLM prompts.
The team tested the tool on 26 public court-filing documents containing falsified citations. Mellea-LRC can process a document in about eight minutes. The tool first extracts the case law citations from the legal document. It then verifies that each citation is authentic and supports the document's content.
“IBM’s Mellea framework was the foundation we built it on. Mellea’s job is to make model calls predictable: you write the rules the output must satisfy as ordinary code, and the framework retries until LLM output meets them. Our job was to leverage the framework to build the legal application from end to end,” Wu said.
An important aspect of the project was ensuring the tool’s code was open source. The team supported this goal by also creating and releasing a new test and validation dataset on Hugging Face–a popular open-source AI development community–for future benchmarking studies.
“Agentic AI is growing quickly, and much of the growth happens in open-source projects, where anyone can read, use, and improve the work,” Lozada said.
OSPO Director and Principal Research Scientist Jeff Young noted that IBM’s Mellea framework was critical to this work in that its “instruct, verify, repair” paradigm allows for the creation of reliable LLM loops as well as the use of small language models (SLMs), such as IBM’s Granite, which can be run on a laptop.
“This project demonstrates the power of open-source models like IBM Granite as well as the importance of using novel software engineering and algorithmic techniques from the Mellea project to improve the reliability of our existing LLMs and SLMs.
“Mellea-LRC provides a compelling example of AI’s potential to evolve from a source of error in the legal profession into a trusted ally by helping lawyers work faster and cite precedents more accurately,” Young said.
