Future Computing Environments

PUI Workshop Submission

CyberDesk: The Use of Perception in Context-Aware Computing

Anind K. Dey, Gregory D. Abowd
Graphics, Visualization & Usability Center
Georgia Institute of Technology
Atlanta, GA 30332-0280 USA
{anind, abowd}@cc.gatech.edu

The CyberDesk project [1] is concerned with providing relevant context-aware services to users. Context-aware services are computational processes whose behavior is adapted based on changes in the perceptions of a user. These perceptions include a user's physical and emotional state as well as his actions. CyberDesk provides a framework for gathering perceptual (or contextual) information and processing this information. When the user's state changes, CyberDesk uses the updated information to find services that the user may find useful. More information on CyberDesk can be found in [2]. Currently in CyberDesk, we integrate information about the data and applications a user is working with, the user's location and time of day to find appropriate services. We are working to incorporate speech, gestures, and models of the user to make the framework more powerful.

Types of Context

A user's context or perceptions can be divided into three categories: physical, action-based, and emotional.

Physical context consists of objective information about the user. It includes:

Action-based context consists of actions the user takes that leads to determining a focus of attention. It includes:

Emotional context includes:

These three categories apply not only to the user (first person perspective) but also apply to other people in the environment (third person perspective), as well as objects in the environment. Context also involves a time perspective, since context can be viewed at a point in time or over a period of time. All these perspectives are important for providing useful services to the user.

Use of Context

The general mechanism for context-aware computing is summarized in the following steps:
  1. collect information on the user's state
  2. analyze the information, either treating each piece individually or combining it with others, collected in the past or present
  3. perform some action based on the analysis
  4. repeat from step 1, with some adaptation based on previous iterations.

We have found that the greater the knowledge of the user's context, the better CyberDesk is able to support the user in his task. Step 1 is important for gathering as much information as possible about the user and his environment. Step 2 appears to be the most difficult. This step involves a type of sensor fusion: the integration of multiple sources (and types) of information into a useful entity that CyberDesk can use to locate and present relevant actions to the user (Step 3). CyberDesk currently supports Steps 2 and 3. We are currently upgrading CyberDesk to support Step 4. This will help CyberDesk be more effective in finding relevant actions.


Examples of CyberDesk's functionality are shown below, using both action-based and physical perceptual information.

Figure 1: Screenshot: knowledge of the user's applications and data beting used with CyberDesk. The user selects the string "Andy Wood" in the e-mail tool (a). CyberDesk offers some suggestions (b): search using Altavista, look up a phone number using Switchboard (c), and look up the name in the contact manager (d).

Click on the screenshot to get an image with better resolution.

Figure 2: Screenshot of position service. (a) is where changing GPS coordinates are displayed , causing changes in the ActOn Button Bar (b) when the coordinates correspond to a different Georgia Tech building. The user is keeping track of his trip in the scratchpad (c), and is able to view the building URLs in the web browser (d).
Click on the screenshot to get an image with better resolution.

Future Work

Currently, CyberDesk uses knowledge of limited action-based and physical context. It will support the use of social and emotional context, once we can find reliable mechanisms for capturing this context. We are currently working on integrating the use of speech recognition, computer vision, and user behavior.

Speech recognition would allow the user's speech to be used as input to CyberDesk, in much the same way as user-selected information was used in Example 1 above. Large vocabulary, continuous speech speech recognizers with high recognition rates (95% and greater) are now commercially available. The textual output from one of these recognizers could be input into CyberDesk, which would search for data that it can use: names, dates, phone numbers, URLs, e-mail addresses, etc. The difficulty is in determining what parts of the speech information is important and should be acted upon by CyberDesk. Two potential solutions are the use of keyword spotting and word frequencies. Keywords could be taken from the user's traditional personal information sources (e-mail, contact manager, scheduler) along with any personal history or documents. An even better, although more complex, solution is to use statistical information retrieval algorithms (much like the Remembrance Agent [3]) to determine what data is relevant.

We are also experimenting with computer vision for use with CyberDesk. We are interested in using facial recognition (along with speaker identification) to determine the identity of people in the user's environment, and using vision for tracking these people and their actions. Currently, an infra-red based positioning system is being used to track people in the environment, requiring them to wear infra-red emitters. We have also begun work on a virtual billboard. The billboard consists of a large, networked, interactive surface and cameras. As users walk up to the billboard, the cameras capture their image. The billboard will recognize them and provide information specific to them on the portion of the billboard that they're standing in front of. The CyberDesk infrastructure will be responsible for taking the identity as input and determining what information is relevant to the user.

Finally, we want to dynamically generate a model of the user as they use the CyberDesk system. By keeping track of a user's actions, the system can take advantage of patterns in behavior. The patterns will be useful in determining what a user is likely to do in a familiar situation she's been in before, as well as predicting what a user is likely to do in a new situation.


  1. CyberDesk Home Page. Available at http://www.cc.gatech.edu/fce/cyberdesk.
  2. Dey, A., et al. CyberDesk: A Framework for Providing Self-Integrating Ubiquitous Software Services. Technical Report, GVU Center, Georgia Institute of Technology. GIT-GVU-97-10, May, 1997.
  3. Rhodes, B. J. and Starner, T. The Remembrance Agent: A continuously running automated information retrieval system. Proceedings of The First International Conference on The Practical Application of Intelligent Agents and Multi Agent Technology (PAAM '96), London, UK, April 1996, pp. 487-495.

Future Computing Environments Georgia Institute of Technology