Research often begins as mere curiosity, but sometimes there’s something more personal at stake.
Professor Gregory Abowd was inspired to conduct research related to autism after one of his sons was diagnosed with the condition. His work helped earn Abowd the 2009 ACM Eugene L. Lawler Award for Humanitarian Contributions in Computer Science.In 2002, Professor Gregory Abowd was working on research at the AWARE Home, which he had founded two years prior, designed to help people age in place. The project involved annotating family movies. As he began annotating his own family movies, he noticed something peculiar, something that would drive his research for years to come.
His oldest son was 5 years old at the time and had been diagnosed with autism. As Abowd watched the movies of his son through the years, he noticed that when the boy was 18 months old he behaved just like any other toddler, but in a video shot just eight months later, he seemed like a completely different child.
“He was exhibiting classic signs of autism,” said Abowd, Regents' Professor and J. Z. Liang Chair in the School of Interactive Computing at Georgia Tech. “He wasn't speaking. Instead he was doing these verbal whooping sounds and was shaking his hand, and he was generally not that happy to be around other people and engaging with them.
“Right in front of my eyes, my child was changing from a child that was behaving like most other children to one who was clearly not like them,” he said.
That observation launched a research focus that spread to a host of other professors, scientists, and students in the College of Computing. Autism work at Tech now runs the gamut from helping clinicians diagnose and manage the disorder to informing research in artificial intelligence.
“I didn’t really have a clear goal of what I was going to accomplish,” Abowd said, “but I knew I was going to connect to people who presented challenges in the autism space and figure out how technology might play a role.”
Building Tools for Now
Autism and autism spectrum disorder are names for a complex group of disorders of brain development characterized by repetitive behaviors and difficulties in social interaction and communication. A prevalent disorder in the United States, the Centers for Disease Control and Prevention released data in 2014 that estimate that one in 68 children is affected.
Senior research scientists Rosa Arriaga (left) and Agata Rozga are both involved in multiple autism-related research projects, often involving Georgia Tech’s Child Study Lab. The lab, which Rozga founded and directs, helps researchers examine social, communication, and play behavior in children, including those who are experiencing problems in their development.“So many times, people think of research as just the future of technology, but if we can build something that works now, that's even better,” said Rosa Arriaga, senior research scientist in the School of Interactive Computing. “I think that's one of our strengths, that we can make things better now.”
One of the challenges in treating autism is understanding whether or not a particular treatment approach is working for a particular child. Operating out of Georgia Tech’s Child Study Lab, Professor Jim Rehg and Senior Research Scientist Agata Rozga are developing a novel technology for measuring eye contact during therapy sessions, using a pair of glasses containing a camera that is worn by the therapist. By automatically measuring the attention the child pays to their partner and how that attention shifts from object to object, the team can enable clinicians to more effectively monitor the progress of their clients and improve their treatment. The research paper describing this work was the winner of a Best Student Paper Award at the International Conference on Face and Gesture Recognition last May. With funding from a grant from the Simons Foundation Autism Research Initiative, Rozga and Rehg are further developing their method. They are also beginning a clinical evaluation of the technology at the Marcus Autism Center.
Professor Jim Rehg’s $10 NSF Expeditions grant supported a range of autism-related research that led to advancements in understanding autism and screening techniques to identify children displaying behavioral signs of autism. Rehg is shown with Child Study Lab manager Audrey Southerland.
Another project involves helping clinicians at the Center for Discovery in upstate New York manage the deluge of data they have to gather and interpret each month as they oversee hundreds of adults and children. Professor John Stasko’s Information Interfaces Group is helping the center collect and display the data using visualization tools so the staff can be more efficient and do more with the information.
Still another project aims to speed the diagnosis of autism. Across the country, autism centers are so overwhelmed with requests that it can take six months from the time a parents seeks a diagnostic evaluation to the time the child is actually seen. In research led by Rozga and conducted in collaboration with Behavior Imaging Solutions and the Southwest Autism Research and Resource Center in Phoenix, scientists developed an app to guide parents to record video examples of their child at home and share it with clinicians for remote diagnoses of autism. In a soon-to-be-published paper, they report an 85 percent agreement between the outcome of the remote diagnosis and the traditional in-person assessment.
Georgia Tech has also led a national effort to engage the computer science and engineering communities in developing technology for autism. With a $10 million, five-year award from the National Science Foundation’s Expeditions in Computing Program, Rehg led a team of 13 faculty members across seven universities, including Georgia Tech, MIT, CMU, and UIUC, from 2010-2016. The Expedition team made contributions in a wide range of areas, from using machine learning for autism diagnosis to methods for analyzing home audio recordings to understand the influence of the home language environment on the development of autism.
Connecting People through Technology
Using technology to connect people is another way scientists can help those with autism and their caregivers. Connecting people with high-functioning autism to a social support network is one way Arriaga envisions to help them live more independently.
“One of the questions we ask is, how can we have other humans provide the support? How can we start thinking about the kind of socially mediated support that we can give?” said Arriaga.
“You can imagine, if I have one person and they're taking care of a person with autism 24/7, that's a big job. But if that person has a trusted support network, that will actually serve to provide this individual with opportunities for independence.”
One idea Arriaga, Abowd, and their graduate student conceptualized is a system known as Social Mirror. The system might work by allowing an adult with high-functioning autism to be connected to a support network of family and friends using a video system. That person could check in with their support system at important times of the day to get care-giving instructions or advice—for example, just before a job interview, or prior to a social engagement.
“A lot of these individuals have a very hard time with common knowledge,” said Arriaga. “But common knowledge is something the crowd provides readily, and so a couple of our studies have shown that we can use a crowd to give people support, which might give them independence.”
Still another area of inquiry involves looking at the way people with autism might think and using that knowledge to inform other computing research, such as artificial intelligence. Professor Ashok Goel in the School of Interactive Computing said his autism interest began when a student, Maithilee Kunda, came to him after reading the book, Thinking in Pictures, by Temple Grandin.
Grandin, a professor at Colorado State University, is a high-functioning person with autism, and she writes that she thinks not in words but in images. Using this as inspiration, Kunda began working with Goel to understand how visual thinkers process information, and then she built a computational model of that thinking for use in artificial intelligence.
“The surprising thing in artificial intelligence is that almost all theories have been purely word-based, not image-based,” said Goel. “She built one of the first AI models that was purely picture-based. There were no words in it, which was a very interesting finding, and we're getting a lot of pushback from AI researchers and theorists who don't like the fact that we are challenging their purely word-based models.”
Kunda isn’t the only doctoral student that has had a profound effect on autism research at Tech. The college has graduated 10 Ph.D. students in this area over the past decade, eight of whom are women, and six of those women now faculty positions at universities. In addition to Kunda, they are Gillian Hayes, Julie Kienz, Ping Wang, Tracy Westeyn, Fatima Boujarwah, Nazneen, Hwajung Hong, Keith McGreggor, and Arri Ciptadi. Another, Yi Han, is set to graduate this year. In addition, Hayes and Nazneen’s work led to commercial products provided by the company Behavioral Imaging Solutions.
While autism research at Georgia Tech began with just one researcher for the most personal of reasons, nearly 14 years later it involves a network of scientists, labs, and students. Whether it’s using computing technology to do the rote work of making measurements and processing data, using it to connect people so they can do the sensing and social work they do best, or even utilizing autism research to inform investigations back into computing, the explorations surrounding this disorder are vast and promise only to expand.
“The problems of the real world can drive a lot of interesting and varied computing research,” said Abowd. “I'm convinced that if you pick the right kinds of real-world problems, there are lots of ways computing can provide great research opportunities.”