Automated Understanding of Captured Experiences

The Automated Understanding of Captured Experiences project is aimed at reducing substantially the human input necessary for creating and accessing large collections of multimedia, particularly multimedia created by capturing what is happening in an environment.

This project is Funded by the National Science Foundation (Grant # 9806822) [Sept 1, 1998 - Aug 31, 2001]
(The material presented on this WWW page  is based upon work supported by the National Science Foundation under Grant No. 9806822)

Description:

The objective of this research is to reduce substantially the human input necessary for creating and accessing large collections of multimedia, particularly multimedia created by capturing what is happening in an environment. The existing software system which is being used as the starting point for this investigation is Classroom 2000, a system designed to capture what happens in classrooms, meetings, and offices. Classroom 2000 integrates and synchronizes multiple streams of captured text, images, handwritten annotations, audio, and video. In a sense, it automates note-taking for a lecture or meeting. The research challenge is to make sense of this flood of captured data. The project explores how the output of Classroom 2000 can be automatically structured, segmented, indexed, and linked. Machine learning and statistical approaches to language are used to attempt to understand the captured data. Techniques from computational perception are used to try to find structure in the captured data. An important component of this research is the experimental analysis of the software system being built. The expectation is that this research will have a dramatic impact on how humans work and learn, as technology aids humans by capturing and making accessible what happens in an environment.

Publications:

• Gregory D. Abowd, "Classroom 2000: An Experiment with the Instrumentation of a Living Educational Environment", IBM Systems Journal
• Gregory D. Abowd and Elizabeth D. Mynatt, "Charting past, present and future research in ubiqutous computing", ACM Transactions on Computer-Human Interaction (special issue on HCI research in the new millenium), p. 29, vol. 7/1, (2000).
• Gregory D. Abowd and J. Sterbenz, "Report on the Inter-Agency Workshop on Research Issues for Smart Environments", IEEE Personal Communications, p. 36, vol. 7, (2000). ) Published
• Jennifer Mankoff, Gregory D. Abowd, Scott E. Hudson, "OOPS: A toolkit supporting mediation techniques for resolving ambiguity in recognition-based interfaces", Computers and Graphics (special issue on Calligraphic Interfaces: towards a new generation of interactive systems), p. 819, vol. 24/6, (2000).
• Maria da Graca Pimentel, Yoshihide Ishiguro, Bolot Kerimbaev, Gregory D. Abowd, Mark Guzdial, "Supporting long-term educational activities through dymanic web interfaces", Interacting with Computers (special issue on interfaces for the active Web), p. 353, vol. 13/3, (2001).
• Anind K. Dey, Daniel Salber, Gregory D. Abowd, "A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications", Human-Computer Interaction (HCI) Journal (anchor article of a special issue on Context-Aware Computing), p. , vol. 16, (2001).
• James M. Rehg, Kathleen Knobe, Umakishore Ramachandran, Richiyur S. Nikhil, "Integrated Task and Data Parallel Support for Dynamic Applications", Journal of Scientific Programming, p. 1, vol. 1, (2000).
• K. Knobe, J. M. Rehg, A. Chauhan, R.S. Nikhil, U. Ramachandran, "Scheduling Constrained Dynamic Applications on Clusters", Supercomputing 99, p. 1, vol. 12, (1999).
• S. Nikhil, U. Ramachandran, "Garbage Collection of Timestamped Data in Stampede", Principles of Distributed Computing 2000, p. 1, vol. 7, (2000).
• I. Essa, "Computers Seeing People", AI Magazine, p. 69, vol. 20/1, (1999).
• Irfan Essa, "Ubiquitous Sensing for Smart and Aware Environments", IEEE Personal Communications (special issue on Networking the Physical World, p. 47, vol. 7/5, (2000).
• Jennifer Mankoff, Scott Hudson, Gregory D. Abowd, "Providing integrated toolkit-level support for ambiguity in recognition-based interfaces", Proceedings of CHI 2000, p. 368, vol. 1, (2000).
• Robert Orr, Gregory D. Abowd, "The smart floor: A mechanism for natural user identification and tracking", Companion Proceedings of CHI 2000 (short paper), p. 1, vol. 2, (2000).
• Gregory D. Abowd, Christopher G. Atekson, Aaron Bobick, Irfan A. Essa, Blair MacIntyre, Elizabeth D. Mynatt, Thad Starner, "The Future Computing Environments Group at the Georgia Institute of Technology", Proceedings of CHI 2000 (companion proceedings) / Organizational overview, p. 1, vol. 2, (2000).
• Maria Pimentel, Gregory D. Abowd, Yoshihide Ishiguro, "Linking by Interacting: A paradigm for authoring hypertext", Proceedings of the 11th ACM Conference on Hypertext and Hypermedia 2000, San Antonio, Texas, p. 1, vol. 1, (2000). 
• A.K. Dey, G.D. Abowd (principal research by Dey, shared authorship), "CyberMinder: A Context-Aware System for Supporting Reminders", Proceedings of the 2nd International Symposium on Handheld and Ubiquitous Computing (HUC2K), p. 172, vol. 1, (2000).
• Jennifer Mankoff, Scott E Hudson, Gregory D Abowd, "Interaction techniques for ambiguity resolution in recognition-based interfaces", Proceedings of the Symposium on User Interface Software Technology - UIST 2000, p. 11, vol. 1, (2000).
• D Moore, I Essa, M Hayes, "Exploiting Human Actions and Object Context for Recognition Tasks", Proceedings of IEEE International Conference on Computer Vision 1999, p. 1, vol. 1, (1999).
• A. Haro, I. Essa, M. Flickner, "A Non-invasive Computer Vision System for Reliable Eye Tracking", Proceedings of ACM CHI 2000 Conference, late breaking short paper, p. 1, vol. 2, (2000).
• A. Haro, I Essa, M. Flickner, "Detecting and Tracking Eyes by Using their Physiological Properties, Dynamics and Appearance", Proceedings of IEEE Computer Vision and Pattern Recognition 2000 Conference, p. 1, vol. 1, (2000).
• S. Stillman, R.Tanawongsuwan, and I. Essa, "A System for Tracking and Recognizing Multiple People with Multiple Cameras", in Proceedings of AVBPA Conference, Washington DC. March 1999. [Available as GVU Tech Report # 98-25 (Abstract | PDF | PS)].
• D. Moore, I. Essa, and M. Hayes, "Object Spaces: Context Management for Human Activity Recognition", in Proceedings of AVBPA Conference, Washington DC. March 1999.[Available as GVU Tech Report # 98-26 (Abstract | PDF | PS)].
• Gregory D. Abowd, “Classroom 2000: An Experiment with the Instrumentation of a Living Educational Environment”, IBM Systems Journal, Accepted
• J. Brotherton, K. Truong, and G. Abowd. “Supporting capture and access interfaces for informal and opportunistic meetings.” Technical Report GIT-GVU-99-06, GVU Center,  Georgia Institute of Technology, January 1999.
• A. Chervenak, V. Vellanki, I. Yanasak, N. Harel, R. Rodenstein, G. Abowd, J. Brotherton, and K. Ramachandran. “A scalable workload model of media-enhanced classrooms.” Technical Report GIT-CC-99-06, College of Computing, Georgia Institute of Technology, February 1999.
• Jason Brotherton, Janak R. Bhalodia, and Gregory D. Abowd, “Automated Capture, Integration, and Visualization of Multiple Media Streams”, (1998). conference, Published Collection: Proceedings of the IEEE Multimedia and Computing Systems '98 Conference Bibliography: pages 54-63
• Daniel Salber, Anind Dey and Gregory Abowd, "The Context Toolkit: Aiding the Development of Context-Enabled Applications", (1999).conference, Accepted Collection: Proceedings of CHI'99 Bibliography: To appear
• Gregory D. Abowd, “Software engineering issues for ubiquitous computing", (1999). conference, Accepted Collection: Proceedings of the International Conference on Software Engineering - ICSE'99 Bibliography: To appear
• Heather Richter, Pascal Schuchhard and Gregory A, "Automated capture and retrieval of architectural rationale", (1999). position paper at conference, Published, Collection: First Working IFIP Conference on Software Architecture
• R. S. Nikhil, K. Ramachandran, et al., “Space-Time Memory: A parallel programming abstraction for interactive multimedia applications,” In Proc. ACM Principles and Practices of Parallel Programming, May 1999, Atlanta, GA.
• R. S. Nikhil, K. Ramachandran, et al., “Space-Time Memory,” In New Ideas Sesson held at the Eighth International Conference on Architectural Support for Programming Languages and Operating Systems}, October 1998, San Jose, CA.
• R. S. Nikhil, K. Ramachandran, et al., “Stampede: A programming system for emerging scalable interactive multimedia applications,” In The 11th International Workshop on Languages and Compilers for Parallel Computing, August 1998, Chapel Hill, NC.