PUI Workshop Submission
Computational Perception in Future Computing Environments
Gregory D. Abowd, Christopher G. Atkeson and Irfan Essa
Future Computing Environments Group
Graphics, Visualization & Usability Center
Georgia Institute of Technology
Atlanta, GA 30332-0280 USA
+1-404-894-7512
{abowd, cga, irfan}@cc.gatech.edu
Automation of our daily environment requires going beyond installing
ubiquitous computing and networking capabilities.
In the
Future Computing Environments Group (FCE) at Georgia Tech, we are
making the environment more aware of the user. This
awareness will play a major role in the computational augmentation of
our everyday activities.
To provide the environment with an awareness of the user, we are developing
ways for the environment
to perceive and (re)act. In daily interactions,
we are capable of perceiving the activity and the events around us,
answering questions to do with who, what, when, where, why and how.
We want to provide the environment a similar understanding.[5]
Our interest is in developing systems that can understand,
interpret and recognize human actions to perceive what the user's
intentions are and how these intentions can be supported. [2, 3] We will use audio
processing technologies to aid in understanding on a verbal level and
video and other technologies to aid in understanding on a
non-verbal level what the users needs and interests are.
In the initial stages of our work on perceptual interfaces, we are
working towards providing our environments with an ability to be context
aware so that they can aid in the capture, integration and access of
sensory data that we acquire from the surroundings. Perception
systems can help with interpreting the identity and the location of
the user within the context of the environments. Some simple
assumptions can also be made about what the user is doing from the
data captured from audio and video sensors. This form
of context awareness can aid in annotation of the data that is
captured.
Over the past 2
years, the FCE group has developed a number of
applications that can benefit from intelligent environment activity.
These applications have involved three different environments:
- the classroom
- the home
- the space immediately surrounding a traveler
We describe these different environments below in the context of
ongoing FCE projects.
The Classroom
The Classroom 2000 project
[1] is
investigating the impact of ubiquitous computing technology in
education. We have built a special classroom that makes it easy to
capture the activities of a lecture so that students and teachers can
more easily access the record of a class. A typical classroom
generates a number of different streams of information: people talking
and demonstrating, presentations on a whiteboard, software
simulations and the like. Students can spend a lot of time frantically
taking notes to capture their understanding of all of the
information. Classroom 2000 is trying to enable the environment to
augment the student record of activities.
As a simple example, Classroom 2000 provides the ability to integrate
different streams of activities together. For example, words that are
written on an electronic whiteboard are automatically linked to a
digital video recording of the audio in the class. Take a look at a
a
sample class that integrates the output from an electronic
whiteboard with the audio in a lecture.
This simple integration can be useful, but it is clear from our
experience with Classroom 2000 that there are limitations to a simple
timestamp integration technique. What we are lacking in our captured
classes is some notion of the focus of the class at a particular
time. To enable us to preserve more semantic information from the
class, we are analyzing the audio and video signals recorded to
provide higher level information.
The Home
The Domisilica project is aimed at
producing a virtual community that mirrors and supports some real
physical community. Our initial efforts are targeted toward the home
and the extended family. We have built a prototype virtual home
environment that is tied to home settings of a number of researchers
in FCE. We are making the two worlds, physical and virtual, work in
concert with each other. So, for example, when some produce is placed
inside the physical refrigerator in the kitchen, the contents of a
virtual refrigerator, CyberFridge, is automatically updated as well.
We are also experimenting with how activity in the virtual world can
affect the physical world. For example, when multiple people
virtually visit a room in Domisilica that is associated to a physical
room, say a living room, the physical environment is enabled to
produce more ambient noise to inform the physical occupants of the
room of the presence of the virtual visitors.
Personal space
The previous two examples dealt with fairly well-defined physical
spaces, the classroom and the home. In this last environment, the
physical space is defined as the unfamiliar territory that surrounds a
mobile user. In the Cyberguide project [4],
we are interested in developing mobile tour guides that are aware of
the location and orientation of their user and provide information
about the surrounding space.
-
Gregory D. Abowd, Chris Atkeson, Ami Feinstein, Cindy Hmelo, Rob Kooper, Sue
Long, Nitin "Nick" Sawhney and Mikiya Tani. Teaching
and Learning as Multimedia Authoring: The Classroom 2000 Project. In
the Proceedings of the ACM Multimedia'96 Conference,
November 1996.
HTML version
- A. Bobick, S. Intille, J. Davis, F. Baird,
C. Pinhanez, L. Campbell, Y. Ivanov, A. Schutte and A. Wilson. The
KidsRoom: A Perceptually-based Interactive and Immersive Story
Environment. MIT Media Laboratory, Perceptual Computing Section
Technical Report 398, 1997. (Available online).
- Irfan Essa and Alex Pentland. Coding, Analysis,
Interpretation, and Recognition of Facial Expressions. IEEE
Transactions on Pattern Analysis and Machine Intelligence,
Volume 19, Number 7, 1997.
-
Sue Long, Rob Kooper, Gregory D. Abowd, and Christopher
G. Atkeson. Rapid Prototyping of Mobile Context-Aware Applications:
The Cyberguide Case Study. In the Proceedings of the 2nd ACM
International Conference on Mobile Computing and Networking
(MobiCom'96), November 1996.
PostScript version
- Alex Pentland. Smart Rooms. Scientific
American, Volume 274, Number 4, pages 68-76, 1996.