Future Computing Environments

PUI Workshop Submission

Computational Perception in Future Computing Environments

Gregory D. Abowd, Christopher G. Atkeson and Irfan Essa
Future Computing Environments Group
Graphics, Visualization & Usability Center
Georgia Institute of Technology
Atlanta, GA 30332-0280 USA
{abowd, cga, irfan}@cc.gatech.edu

Automation of our daily environment requires going beyond installing ubiquitous computing and networking capabilities. In the Future Computing Environments Group (FCE) at Georgia Tech, we are making the environment more aware of the user. This awareness will play a major role in the computational augmentation of our everyday activities.

To provide the environment with an awareness of the user, we are developing ways for the environment to perceive and (re)act. In daily interactions, we are capable of perceiving the activity and the events around us, answering questions to do with who, what, when, where, why and how. We want to provide the environment a similar understanding.[5]

Our interest is in developing systems that can understand, interpret and recognize human actions to perceive what the user's intentions are and how these intentions can be supported. [2, 3] We will use audio processing technologies to aid in understanding on a verbal level and video and other technologies to aid in understanding on a non-verbal level what the users needs and interests are.

In the initial stages of our work on perceptual interfaces, we are working towards providing our environments with an ability to be context aware so that they can aid in the capture, integration and access of sensory data that we acquire from the surroundings. Perception systems can help with interpreting the identity and the location of the user within the context of the environments. Some simple assumptions can also be made about what the user is doing from the data captured from audio and video sensors. This form of context awareness can aid in annotation of the data that is captured.

Over the past 2 years, the FCE group has developed a number of applications that can benefit from intelligent environment activity. These applications have involved three different environments:

We describe these different environments below in the context of ongoing FCE projects.

The Classroom

The Classroom 2000 project [1] is investigating the impact of ubiquitous computing technology in education. We have built a special classroom that makes it easy to capture the activities of a lecture so that students and teachers can more easily access the record of a class. A typical classroom generates a number of different streams of information: people talking and demonstrating, presentations on a whiteboard, software simulations and the like. Students can spend a lot of time frantically taking notes to capture their understanding of all of the information. Classroom 2000 is trying to enable the environment to augment the student record of activities.

As a simple example, Classroom 2000 provides the ability to integrate different streams of activities together. For example, words that are written on an electronic whiteboard are automatically linked to a digital video recording of the audio in the class. Take a look at a a sample class that integrates the output from an electronic whiteboard with the audio in a lecture. This simple integration can be useful, but it is clear from our experience with Classroom 2000 that there are limitations to a simple timestamp integration technique. What we are lacking in our captured classes is some notion of the focus of the class at a particular time. To enable us to preserve more semantic information from the class, we are analyzing the audio and video signals recorded to provide higher level information.

The Home

The Domisilica project is aimed at producing a virtual community that mirrors and supports some real physical community. Our initial efforts are targeted toward the home and the extended family. We have built a prototype virtual home environment that is tied to home settings of a number of researchers in FCE. We are making the two worlds, physical and virtual, work in concert with each other. So, for example, when some produce is placed inside the physical refrigerator in the kitchen, the contents of a virtual refrigerator, CyberFridge, is automatically updated as well. We are also experimenting with how activity in the virtual world can affect the physical world. For example, when multiple people virtually visit a room in Domisilica that is associated to a physical room, say a living room, the physical environment is enabled to produce more ambient noise to inform the physical occupants of the room of the presence of the virtual visitors.

Personal space

The previous two examples dealt with fairly well-defined physical spaces, the classroom and the home. In this last environment, the physical space is defined as the unfamiliar territory that surrounds a mobile user. In the Cyberguide project [4], we are interested in developing mobile tour guides that are aware of the location and orientation of their user and provide information about the surrounding space.


  1. Gregory D. Abowd, Chris Atkeson, Ami Feinstein, Cindy Hmelo, Rob Kooper, Sue Long, Nitin "Nick" Sawhney and Mikiya Tani. Teaching and Learning as Multimedia Authoring: The Classroom 2000 Project. In the Proceedings of the ACM Multimedia'96 Conference, November 1996. HTML version
  2. A. Bobick, S. Intille, J. Davis, F. Baird, C. Pinhanez, L. Campbell, Y. Ivanov, A. Schutte and A. Wilson. The KidsRoom: A Perceptually-based Interactive and Immersive Story Environment. MIT Media Laboratory, Perceptual Computing Section Technical Report 398, 1997. (Available online).
  3. Irfan Essa and Alex Pentland. Coding, Analysis, Interpretation, and Recognition of Facial Expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, Volume 19, Number 7, 1997.
  4. Sue Long, Rob Kooper, Gregory D. Abowd, and Christopher G. Atkeson. Rapid Prototyping of Mobile Context-Aware Applications: The Cyberguide Case Study. In the Proceedings of the 2nd ACM International Conference on Mobile Computing and Networking (MobiCom'96), November 1996. PostScript version
  5. Alex Pentland. Smart Rooms. Scientific American, Volume 274, Number 4, pages 68-76, 1996.

Future Computing Environments Georgia Institute of Technology