Goals

The ubiquitous video/audio project explores potential implementations of video and audio technology that seeks to become transparent to everyday activities. As a part of this project, we are developing video and audio processing techniques and exploring their applications within the context of an “Aware Home.” During the first year of the project, we developed prototypical systems for high and low bandwidth video and audio processing.  In the second year we propose to explore and prototype a spectrum of applications in the home that uses these systems.

Impact on Technologies and Applications:

We expect the use of video in the home to grow from isolated applications such as videophones, smart front doors, and surveillance devices to become pervasive. Such pervasive use of video and audio technology will make the entire volume of the house viewable, and permit synthesis of desired views from existing views.  Once we have established an infrastructure to support such a pervasive use of video and audio technology, a homeowner will be able to install video hardware for one purpose and use it for many others. For this reason, we expect an explosion of demand for video applications. This proposal attempts to anticipate that demand and explore the technologies to provide ubiquitous sensing of video and audio and explore candidate video applications now.  Making video and audio sensing transparent to everyday activities and distributing it everywhere in an environment is essential in the development of an Aware Home.

Research Foci

There are three different classes of video and audio processing technologies that we are exploring:

1. Low cost, low bandwidth technologies that emphasize the transmission of single images or low frame rates. These applications could use power lines or wireless links to transmit images. Camera does little processing besides reformatting data for transmission.

2. High frame rate applications that require the processing and transmission of images at 30 Hz or better. This is specifically important in applications where high fidelity information is required to track activities.

3. Camera and sensors with local onboard processing.  These applications use task specific image processing right at the camera. This increases the cost of each installed camera, but decreases the requirement for high bandwidth information transmission. 
   

In addition to the above enabling technologies, we are studying following methods to develop aware space (ie. Aware Home).

• methods for video and audio-based tracking of people
• multiple person, multiple sensor tracking
• representations for recognizing human activity
• techniques for audio-based interactions
• multi-modal interactions
• health-care and personal monitoring applications
• security applications

 Related Publications (In reverse chronological order):

• A. Gardner and I. Essa, "Prosody Analysis for Speaker Affect Determination", In Proceedings of PUI 1997 Workshop, in conjunction with UIST 1997, Banff, Canada, October 1997.
• G. Abowd, C. Atkeson, I. Essa, "Computational Perception in Future Computing Environments", October, 1997. Perceptual User Interfaces Workshop in Conjunction with UIST 1997.
• G. Abowd and I. Essa, Ubiquitous and Aware Computing, In GVU Newsletter, November 1997.
• I. Essa, G. Abowd,  C. Atkeson, Ubiquitous Smart Spaces, A white paper submitted to DARPA, February 1998.
• S. Stillman, R. Tanawongsuwan, and I. Essa, "A System for Tracking and Recognizing Multiple People with Multiple Cameras", Submitted (August 1998) to AVBPA Conference, Washington DC. March 1999. [Available as GVU Tech Report # 98-25 (Abstract | PDF | PS)].
• D. Moore, I. Essa, and M. Hayes, "Object Spaces: Context Management for Human Activity Recognition", Submitted (August 1998) to AVBPA Conference, Washington DC. March 1999.[Available as GVU Tech Report # 98-26 (Abstract | PDF | PS)].
• Essa, I. "Computers Seeing People", in AI Magazine, Volume 20 (1), 69--82, Summer 1999. (Abstract | PS | PDF (You need to be an AAAI member))
• I. Essa, "Ubiquitous Sensing for Smart and Aware Environments", A Position paper at DARPA/NIST/NSF Workshop on Smart Environments, Atlanta, GA., July 1999. (PDF)
• D. Moore, I. Essa, and M. Hayes, "Exploiting Human Actions and Object Context for Recognition Tasks", In Proceedings of IEEE International Conference on Computer Vision 1999 (ICCV’99), Corfu, Greece, March 1999.[Also available as Georgia Tech, GVU Center Tech Report No. GIT-GVU-TR-99-11 (Abstract | PS.Z | PDF)].
• Kidd. C., G. Abowd, C. Atkeson, I. Essa, B. MacIntyre, E. Mynatt, T. Starner, “The Aware Home: A Living Laboratory for Ubiquitous Computing Research”, In Proceedings of Second International Workshop on Cooperative Buildings 1999, Editors, Streitz, J. Siegel, V. Hartkopf, S. Konomi, Pittsburgh. LNCS 1670. Springer: Heidelberg, Pittsburgh, PA, October 1999.(HTML)

• Haro, A., I. Essa, and M. Flickner, “A Non-invasive Computer Vision System For Reliable Eye Tracking”, In Proceedings of ACM CHI 2000, (Late Breaking Short Paper). The Hague, Netherlands, April 2000 (To Appear).

• Haro, A., I. Essa, and M. Flickner, “Detecting and Tracking Eyes by Using their Physiological Properties, Dynamics and Appearance”, In Proceedings of IEEE CVPR 2000, December 2000 (To Appear).

• Essa, I. “Ubiquitous Sensing for Smart and Aware Environments”, Submitted to IEEE Personal Communications, Special Issue on Networking the Physical World, January 2000.

Presentations

• I. Essa, "Intelligent and Aware Environments", at CHI 1997, Ubiquitous Computing Workshop, Atlanta, GA., March 1997.  (Position Paper)
• I. Essa, "Ubiquitous Video and Audio: Steps towards an Aware Home", BTC Brown Bag, May 1999. (HTML)
• I. Essa, "Building an Aware Home", at Mitsubishi Electric Research Laboratory, Cambridge, MA, August 1999. (HTML)

Proposals

• G. Abowd, C. Atkeson, I. Essa and K. Ramachandran, "Building a Large-Scale Ubiquitous Computing Interface",  A proposal submitted to the National Science Foundation Major Research Instrumentation Program, January 1998.
• I. Essa, "Ubiquitous Video and Audio in the Home", Submitted to Broadband Telecommunications Center, Georgia Tech. April 1999.

People

• Irfan Essa (PI),
• Gregory Abowd,
• Chris Atkeson,
• Darnell Moore,
• Scott Stillman

Funding

• Broadband Institute, Georgia Institute of Technology, 1998-present
• National Science Foundation, ESS Program, 1998-present.

  Home Projects Publications People Courses Sponsors Co-Web CPL Swiki GVU Copyright © 1997-2000

Last Updated April 6, 2000.