The CyberDesk project [1] is concerned with providing relevant context-aware services to users. Context-aware services are computational processes whose behavior is adapted based on changes in the perceptions of a user. These perceptions include a user's physical and emotional state as well as his actions. CyberDesk provides a framework for gathering perceptual (or contextual) information and processing this information. When the user's state changes, CyberDesk uses the updated information to find services that the user may find useful. More information on CyberDesk can be found in [2]. Currently in CyberDesk, we integrate information about the data and applications a user is working with, the user's location and time of day to find appropriate services. We are working to incorporate speech, gestures, and models of the user to make the framework more powerful.
Physical context consists of objective information about the user. It includes:
Action-based context consists of actions the user takes that leads to determining a focus of attention. It includes:
Emotional context includes:
These three categories apply not only to the user (first person perspective) but also apply to other people in the environment (third person perspective), as well as objects in the environment. Context also involves a time perspective, since context can be viewed at a point in time or over a period of time. All these perspectives are important for providing useful services to the user.
We have found that the greater the knowledge of the user's context, the better CyberDesk is able to support the user in his task. Step 1 is important for gathering as much information as possible about the user and his environment. Step 2 appears to be the most difficult. This step involves a type of sensor fusion: the integration of multiple sources (and types) of information into a useful entity that CyberDesk can use to locate and present relevant actions to the user (Step 3). CyberDesk currently supports Steps 2 and 3. We are currently upgrading CyberDesk to support Step 4. This will help CyberDesk be more effective in finding relevant actions.
Figure 1: Screenshot: knowledge of the user's applications and data beting used with CyberDesk. The user selects the string "Andy Wood" in the e-mail tool (a). CyberDesk offers some suggestions (b): search using Altavista, look up a phone number using Switchboard (c), and look up the name in the contact manager (d).
Click on the screenshot to get an image with better resolution.
Figure 2: Screenshot of position service. (a) is where changing GPS coordinates are displayed
, causing changes in the ActOn Button Bar (b) when the coordinates
correspond to a different Georgia Tech building. The user is keeping track of
his trip in the scratchpad (c), and is able to view the building URLs in the
web browser (d).
Click on the screenshot to get an image with better resolution.
Speech recognition would allow the user's speech to be used as input to CyberDesk, in much the same way as user-selected information was used in Example 1 above. Large vocabulary, continuous speech speech recognizers with high recognition rates (95% and greater) are now commercially available. The textual output from one of these recognizers could be input into CyberDesk, which would search for data that it can use: names, dates, phone numbers, URLs, e-mail addresses, etc. The difficulty is in determining what parts of the speech information is important and should be acted upon by CyberDesk. Two potential solutions are the use of keyword spotting and word frequencies. Keywords could be taken from the user's traditional personal information sources (e-mail, contact manager, scheduler) along with any personal history or documents. An even better, although more complex, solution is to use statistical information retrieval algorithms (much like the Remembrance Agent [3]) to determine what data is relevant.
We are also experimenting with computer vision for use with CyberDesk. We are interested in using facial recognition (along with speaker identification) to determine the identity of people in the user's environment, and using vision for tracking these people and their actions. Currently, an infra-red based positioning system is being used to track people in the environment, requiring them to wear infra-red emitters. We have also begun work on a virtual billboard. The billboard consists of a large, networked, interactive surface and cameras. As users walk up to the billboard, the cameras capture their image. The billboard will recognize them and provide information specific to them on the portion of the billboard that they're standing in front of. The CyberDesk infrastructure will be responsible for taking the identity as input and determining what information is relevant to the user.
Finally, we want to dynamically generate a model of the user as they use the CyberDesk system. By keeping track of a user's actions, the system can take advantage of patterns in behavior. The patterns will be useful in determining what a user is likely to do in a familiar situation she's been in before, as well as predicting what a user is likely to do in a new situation.
