Bringing People and Places Together

This document has been copyrighted by the American Association for Artificial Intelligence, and is being made available solely for the use and benefit of authors preparing papers for AAAI Press publications. No other use of this template or document may be made without permission in writing from AAAI.

ABSTRACT

Keywords

INTRODUCTION

For many of us, a home is not simply a single physical location where we relax, eat and generally spend time with family and other loved ones. There are a large number of activities that define life outside of the work environment and, although those activities are part of our home life, they often occur in places outside the actual home location. When we shop, we are purchasing goods to replenish the inventory at home. Even while at work, we often do things that involve taking care of matters on the home front. Some of us are fortunate enough to have the freedom to work at home, in which case the boundary between home and office is yet more blurred.

With the increasing affordability of high bandwidth network connectivity and the ability to use computational devices to sense and affect our physical environment, we no longer need to distinguish the different environments in which we live simply because of physical separation. We can use ubiquitous computing technology to break down the physical barriers in our lives and bring people and places together. In this paper, we outline work done in the Future Computing Environments Group and Broadband Telecommunications Group at Georgia Tech that focuses on how technology can be used to augment our daily lives outside of the workplace. This project is entitled Domisilica (Mankoff & Abowd 1997) and is a demonstration of what we call dual augmentation, in which a physical environment is mirrored by a virtual reconstruction. The physical world of the home and its virtual counterpart serve to augment the capabilities of each other.

After providing an overview of augmented reality, augmented virtuality and ubiquitous computing, we will describe two specific topics in this paper. First, we will outline the software architectural approach to dual augmentation through an extension to MUD technology. Second, we will describe our initial efforts on dual augmentation in the home that revolved around a refrigerator, CyberFridge, which provides inventory management and supports communication activities within a household. We will show how a physical/virtual cooperation with CyberFridge provides the potential for a more useful and entertaining appliance. We conclude this paper by discussing our ongoing work to expand dual augmentation throughout the home and between homes in a virtual community.

Background and Related Work

Research into augmented reality (Feiner, MacIntyre, & Sellgmann 1993; Fitzmaurice 1993; Wellner, Mackay, & Gold 1993; Virtual LA) has a broad history. Most approaches involve complementing the user's perception of and interaction with the physical world in some way, either through visual means (Feiner et al. 1995), sound (Mynatt et al. 1997), or touch (Weiser 1991). Ubiquitous Computing (Want et al. 1995) takes a slightly different tack on the matter:

1. In some cases, the environment is augmented simply by changing the behavior of devices in it (such as when a light is turned on). The Neural Network House (Mozer et al. 1996) is an example of this approach. Physical spaces are monitored so that they can be controlled for optimal behavior (such as minimize fuel or electricity consumption). There are a variety of mechanisms available for controlling the environment. A simple and cheap approach is to use X10 (Also see ftp://ftp.seruz.net/users/cichlid/public/x10faq) to control the power supply to devices combined with repeating IR for more programmable features. When mixed with some direct serial-port sensors, this allows rapid prototyping of a variety of behaviors. A more sophisticated control mechanism is CEBUS, which allows actual programs to sit on physical devices. This is an industry standard that we are working toward as our long-term solution.
2. In other cases, computers and displays are added throughout the real world. The ParcTab/Pad ubiquitous computing project (Want et al. 1995) and the PALplates project (Mankoff & Schilit 1997) both demonstrate adding new computing objects to the environment in this way. The idea is to provide access to new services and information ``on location''. Another approach to this is to bring the interaction into the hands of the user. Two projects which demonstrate physical interfaces are the Tangible Bits work (Ishii & Ullmer 1997) and the Digital Desk (Wellner 1993). These take the idea of adding displays to the real world to an even more integrated level.

Domisilica complements the notion of augmented reality with a corresponding augmented virtuality, a mirror world (Gelernter 1991) whose computational state depends in part on the physical world. In the past, virtual spaces have been augmented by other virtual spaces (Guzdial 1997) and by physical spaces (Virtual LA). Combining augmented reality and augmented virtuality yields what we call dual augmentation. The Jupiter project (Nichols et al. 1995) demonstrates this by tracking users and some devices (augmented virtuality) and playing sounds in real spaces (augmented reality).Our research with Domisilica involves exploring the benefits of dual augmentation for the support of communication, and its advantages in creating a rich, user-friendly environment.

Architecture of Domisilica

Figure 1

A physical object (P) and its virtual counterpart (V). The virtual object is ``augmented'' both by Internet information (eg a URL) and by update information about its physical counterpart. The physical object is ``augmented'' if the virtual object can affect its state.

An understanding of dual augmentation benefits from a conceptual model of the connections between the real space and its virtual model. We have chosen to store a model of the physical world in a MUD (Multi User Dungeon) (Curtis & Nichols 1994) . The canonical MUD is an object-oriented database which contains objects representing things, people, and spaces and uses a coherent metaphor which maps nicely onto a physical space. Objects in our database correspond to physical things and spaces as well as to virtual information (such as URLs or notes), and services with no physical counterpart (such as a recipe finder). Figure 1 demonstrates the relationship between the physical (real) world and virtual model. Information sensed in the physical world is used to update information in the virtual world. Information from the virtual world is used to produce some affect in the physical world.

In the real world

Domisilica's augmentation takes two noticeable forms in physical spaces: one is a change in the behavior of devices (when they turn on and off, for example); the other is displays, or ``windows'' into the virtual world. These can be used to access parts of the virtual world which have no real-world counterpart. For example, our extended refrigerator, CyberFridge, has a GUI display attached to its freezer door. This display lets the user see virtual information such as posted notes that have no physical instantiation. It also gives them access to a recipe program and other virtual services. Displays could potentially be more lightweight, in the style of Audio Aura (Mynatt et al. 1997).

In the virtual world

Users can open windows into the virtual world from any networked computer. Since the virtual world is augmented by (models) a real space, this allows users to monitor the state of that space. They can also change the space by changing virtual information associated with it, or interact with other users (both real and virtual) in the same area. The CyberFridge project is an in-depth example of some of the uses of this kind of remote access for a real-world device.

A Digital Watering Hole

CyberFridge is a project which demonstrates the extent to which one device (a refrigerator) can be enriched by dual augmentation. We began by observing the diverse uses to which a refrigerator is put in most households. Like a watering hole, it becomes a central place for communication and information. People cover the door of their refrigerator with photographs, ads, notes, bills, shopping list, and recipes. They use magnets to write poems, as decorations, and to hang things. And they think of new uses for their refrigerators all the time.

All of the uses described above take place in the absence of any augmentation. Many of them, though, have obvious extensions given a networked computer: Notes could be posted by users from any (authenticated) computer account. Pictures could be loaded from URLs anywhere on the Web. Given a sensor which can keep track of the contents of the refrigerator (we use a bar code scanner for packaged goods combined with a vision system for fruit), the shopping list could be generated dynamically once a recipe is chosen. Recipes could be downloaded from web pages and compared to the known refrigerator contents.

Figure 2

(left) The top section has a Notes, Find Recipes, Suggestions, and Navigation button. The bottom has a refrigerator and pantry. In the refrigerator a variety of food items are displayed. (right) A display attached to a refrigerator door.

Each of the extensions in the previous paragraph has been implemented. We have created a digital watering hole. Remote users can view the state of the refrigerator and add new digital information such as notes and pictures using the interface in the right side of Figure 2. Local users currently use the same interface to view virtual information. The interface is loaded in the display on the refrigerator door shown on the left. This is an example of a ``window'' into a virtual space (the model of the kitchen).

Conclusion

In summary, we have discussed how dual augmentation can support a broader range of interactions across an augmented physical and virtual space. On-location users reap the standard benefits of augmented reality, and remote users can access real-world information not normally available. We have used a ubiquitous computing approach within the home combined with a virtual model for remote users. Two projects demonstrate our approach: Domisilica focuses specifically on an augmented home, and CyberFridge is an example of one appliance which supports dual augmentation.

Future Work

We are currently extending the techniques demonstrated in CyberFridge to augment an entire home, the Regency. Currently, we have control of most home appliances through X10. We also use repeating IR where applicable. Users can control parts of the system through the home phone system, and some automatic behaviors are also present. The major space-wide output we use is audio although lighting and temperature can also be controlled. We are working on using space-wide output to supply peripheral information. This is especially important for support of awareness between local and remote people.

Motion sensors are the main input available in the Regency to date, although we are exploring uses for other sensors. Eventually, we hope to use video and audio for input as well. We also plan to extend the amount to which we have augmented the virtual model of the Regency.

Although augmenting just one home is a tough problem on it's own, we feel that the most interesting applications arise in a situation when many homes are joined in a virtual village. How does a home support virtual visitors? Our augmented spaces could potentially allow visitors to interact with individual devices in a room such as a game board or the lights. We'd like to experiment with the effectiveness of such subtle interactions in conveying a sense of presence.

Acknowledgments

The authors would like to thank colleagues in the GVU Center, particularly those involved in the Future Computing Environments group, and the Broadband Telecommunications Center, for their support and brainstorming that lead to the Domisilica project. Special thanks to Ken Calvert, Chris Atkeson and the many undergraduate students who have contributed to various stages of development of Domisilica and its precursor, CyberFridge. Many thanks also to Joe Bayes, and the many other people who have helped to augment Jen's hands. Thanks also to Ben for help with this submission. This work has been sponsored in part by a grant from Intel Corporation. Jennifer Mankoff is supported by a National Science Foundation HCI Traineeship Fellowship Grant \# GER-9454185.

REFERENCES

Feiner, S.; Webster, A.; Krueger, T.; and MacIntyre, B. 1995.
Architectural anatomy.
Presence 4(3):318- 325.

Feiner, S.; MacIntyre, B.; and Sellgmann, D. 1993.
Knowledge-based augmented reality.
Communications of the ACM, 36(7):53--61.

Fitzmaurice, G. W. 1993.
Situated information spaces and spatially aware palmtop computers.
Communications of the ACM, 36(7):39--49.

Gelernter, D. 1991.
Mirror worlds, or, the day software puts the universe in a shoebox - how it will happen and what it will mean.
Oxford University Press.

Guzdial, M. 1997.
A shared command line in a virtual space: The working man's MOO.
In Proceedings of UIST 97, 73-74. ACM.

Ishii, H., and Ullmer, B. 1997.
Tangible bits: Towards seamless interfaces between people, bits and atoms.
In Proceedings of CHI '97, pages 234--241. ACM.

Mankoff, J., and Abowd, G. D. 1997.
Domisilica: Providing ubiquitous access to the home.
Technical Report GIT-GVU-97-17, Georgia Institute of Technology: Graphics, Visualization and Usability Center.

Mankoff, J., and Schilit, B. 1997.
Supporting knowledge workers beyond the desktop with PALPlates.
In Proceedings of CHI '97, pages 550--551. ACM.

Mozer, M. C.; Dodier, R. H.; Anderson, M.; Vidmar, L.; III, R. F. C.; and Miller, D. 1996.
The Neural Network House: An overview.
Technical report, University of Colorado.

Mynatt, B.; Back, M.; Want, R.; and Frederick, R. 1997.
Audio aura: Light-weight audio augmented reality.
In Proceedings of UIST 97, 211-212. ACM. Nichols, D. A.; Curtis, P.; Dixon, M.; and Lamping, J. 1995.
High-latency, low-bandwidth windowing in the Jupiter collaboration system.
In Proceedings of UIST '95, pages 111--120. ACM

Want, R.; Schilit, B.; Adams, N.; Gold, R.; Petersen, K.; Goldberk, D.; Ellis, J.; and Weiser, M. 1995.
The ParcTab ubiquitous computing experiment.
Technical Report CSL-95-1, Xerox PARC.

Weiser, M. 1991.
The computer for the 21st century.
Scientific American, 265(3):94-104.

Wellner, P.; 1993. Mackay, W. and Gold, R.
Computer-augmented environments: Back to the real world.
Communications of the ACM, 36(7):24-26.

Wellner, P. 1993.
Interacting with paper on the digitaldesk.
Communications of the ACM, 36(7):87-96.