User Interfaces for Augmented Environments

Introduction

In recent years, there has been a growing interest in developing computer systems that move ?off the desktop? to more naturally support people?s activities as they navigate through their work and personal lives. Many research efforts are underway to explore and understand different approaches to computing that move beyond the 2D windowing metaphors that dominate current computer interfaces. Such efforts cover a wide range of designs and philosophical views of the future, and go by many names: Ubiquitous Computing [5], Information Appliances [3], Tangible Media [2], Wearable Computing [4], and Augmented Reality [1], to name a few.

While all of these visions are compelling in their own way, it is unlikely that any single one of them will completely define the future of human-computer interaction. Rather, a rich hybrid interactive environment that combines elements of all of these approaches will probably emerge. Such an environment will contain a multitude of computing devices, both general purpose [5, 4, 1] and special purpose [2, 3]. Some of them will be embedded in the environment [5, 3, 2] and others carried by their users [4, 1]. Regardless of the balance between general and special purpose devices, or between stationary and mobile ones, the key to the success of such a hybrid interactive environment will be the existence of a common infrastructure that allows people to use this assortment of devices as their working style, environment, and aesthetic preferences dictate.

What does this mean, to allow people to ?use them as their needs dictate?? At a basic level, users must be able to easily add and remove devices to and from the environment, and the devices that are there must work together and allow themselves to be used by people as they see fit. Consider a simple example, that combines elements of Tangible User Interfaces and Ubiquitous Computing. A user obtains two ambient displays [6], a water lamp [7] for their office, that casts shadows on the ceiling based on a function of some value, and a vibrating pager-like device that they will carry in their pocket. This user has an application that can tell them the activity of a close friend or relative, both on a continual basis and when there are sudden changes in their friend?s activity level. The user wants to tie the output of the application to these devices: the continuous state should be displayed with the water lamp, and any significant changes should be displayed using the vibrating pager. The infrastructure should make this easy, even if the application designer knew nothing about these devices, and the device designer knew nothing of this application.

This simple example becomes more interesting when elements of other new interaction paradigms are added. For example, consider the addition of spatially located information (from Augmented Reality) and contextual information (from Ubiquitous Computing). Assume the application can provide additional detailed information about the friend, and wants this information to appear ?near? the other displays. If the infrastructure has access to the user?s context (including their position and direction of gaze) and the user is wearing a see-through Augmented Reality display, the infrastructure should be able to have the detailed information appear near the water lamp when the user looks at it, without the application knowing the physical details of any of the display devices (e.g., it does not need to know were the lamp is physically located).

Contextual information could be used in other ways. For example, it should be possible for the user to specify that the water lamp should be turned off if others are in the room, or that if they are not carrying the vibrating pager, any device in their vicinity could be used as a display for the application to notify them of significant changes in the activity of their friend.

While similar examples of the usefulness of these interaction paradigms have been presented by other researchers, this example serves to highlight the central motivation for this research: to develop infrastructure that allows these exciting, new interaction paradigms to be used together in a straightforward way.

The main questions we are pursuing in this research project are:

• What are the principals underlying a general purpose user interface infrastructure that will enable practical hybrid interactive environments?
• What are the appropriate interaction techniques and interface metaphors for each of these devices, and for environments that contain groups of them?
• How will such an infrastructure change how we think about these user interaction paradigms and how they can be used together?

Projects

There are a number of proposed projects in this area. Dr. MacIntyre is open to other project ideas, so if you have one, please contact him.

Background

• Augmented Environments Lab web page
• References listed below
• (Optional) Some knowledge of, or practical experience with, one or more of the following:
  • GUI software
  • Software engineering
  • Distributed systems

REFERENCES

1. Feiner, S., MacIntyre, B., and Seligmann, D. (1993) "Knowledge-based augmented reality," Communications of the ACM, 36(7):52-63.
2. Ishii, H. and B. Ullmer (1997) "Tangible Bits: Towards Seamless Interfaces Between People, Bits and Atoms," In Proceedings of ACM CHI?97 Conference on Human Factors in Computing Systems, pp. 234-241, 1997.
3. Norman, D. (1998) The Invisible Computer: Why Good Products Can Fail, the Personal Computer Is So Complex, and Information Appliances Are the Solution. MIT Press, 1998.
4. Starner, T. S. Mann, B. Rhodes, J. Levine, J. Healey, D. Kirsch, R. Picard, and A. Pentland (1997), "Augmented Reality Through Wearable Computing," In Presence 6(4), 1997.
5. Weiser, M. (1991) The computer of the 21st century. Scientific American, 265(3):66-75, September.
6. Wisneski, C., Ishii, H., Dahley, A., Gorbet, M., Brave, S., Ullmer, B. and Yarin, P., "Ambient Displays: Turning Architectual Space into an Interface between People and Digital Information," in Proceedings of International Workshop on Cooperative Buildings (CoBuild '98), (Darmstadt, Germany, February 1998), Springer Press, pp. 22-32.
7. Dahley, A., Wisneski, C. and Ishii, H., Water Lamp and Pinwheels: Ambient Projection of Digital Information into Architectural Space (short paper), in Summary of Conference on Human Factors in Computing Systems (CHI '98), (Los Angeles, April 1998), ACM Press, pp. 269-270.