New Benchmark Measures Memory Capabilities of VLM-Powered Robots

In-home assistive robots might be able to clean and organize your home, but if they can’t tell you where they put your keys or wallet, they can become a major inconvenience.

To improve robot performance and advance memory capabilities needed for reliable home use, Georgia Tech researchers have developed a new benchmark to evaluate the memory capabilities of vision-language models (VLMs).

That includes Gemini 2.0-flash and GPT-4o.

In-home assistive robots could clean and organize a home while the owner is away, but without enhanced memory capabilities, they could cause more problems than they solve.

They could, for example, cause a major inconvenience if they move important objects like wallets and keys but are unable to tell the owner where they put them.

A new benchmark created by Georgia Tech researchers tests the memory capability of vision-language models (VLMs), including Gemini 2.0-flash and GPT-4o, to determine whether they could improve robot performance.

Zsolt Kira, an associate professor in Tech’s School of Interactive Computing, said that most active VLMs can process only a few hundred images at a time before forgetting them.

“There wasn’t a benchmark that could test these specialized capabilities that we might want from a robot,” Kira said. “Our purpose in creating one is to spur research in this area so that others can develop methods to solve this problem.”

The team’s benchmark consists of memory evaluations based on 60 tasks that require continuous engagement and contextual and environmental awareness. Three task categories — spatial, temporal, and multi-goal — are highly difficult.

“You can chat with a language model like ChatGPT, and it remembers all the text from your previous conversations with it because the text data is small in terms of storage,” Kira said. 

“Embodied agents and robots observe a lot of videos, and those tend to be a lot harder to recall. If you ask it where your diary is, it must efficiently store images of your diary and where it is.

“That’s just a few frames out of a million that it recorded since yesterday, and it must know where that is relative to the spatial layout of the house.”

Karmesh Yadav, a Ph.D. student in Kira’s lab, said users don’t want to repeat information about their preferences after deployment. They also expect the robot to maintain long-term memory.

“In reality, you would want it to have a memory of the entire time it has been in your house, whether that’s days or years,” Yadav said.

The team’s benchmark evaluates memory performance over an average one-day period. It challenges the models with commands like “navigate to an object you did not interact with yesterday” or “navigate to a room you did not visit yesterday.”

The results show that current state-of-the-art models have a long way to go before they can be used in embodied agents. The best model achieved roughly a 50% success rate on high-level difficulty tasks.

Surprisingly, this model wasn’t Gemini or GPT VLMs. It was an open-source reasoning model from Qwen. 

“If you use an open-source reasoning model, even one that’s small in size, it can match the performance of closed-source non-reasoning models,” Yadav said. “Reasoning models can look at the video and translate it into text. It describes what’s going on in the video in words and determines the right frame to choose in its final output.”

Kira said that converting images into descriptive text that language models easily understand could be one way to circumvent the memory storage dilemma. Another way could be to assign a hierarchy that teaches the model to identify irrelevant or redundant images and erase them from storage.

Whatever future progress researchers make in those areas, Kira and his students are confident that their benchmark will remain relevant. 

“This benchmark is scalable,” said Ph.D. student Yusuf Ali. “If two years from now, videos are easy for a model to understand and store, you can use the same infrastructure we propose and scale up the problem’s difficulty.”

Yadav and Ali are co-first authors of a paper on the benchmark, presented in early September at the 2026 European Conference on Computer Vision in Sweden.

For more information about the project, click here.