Aware Home: Recognizing what people are doing?

Problem
As we move towards computers that are embedded in the environment, one main question arises, "What are these computers going to do for us?"  In order for them to "DO" anything for us, they need to be Aware of what we are doing and how our activity can be supported.  Towards this end, We are interested in building Aware Environments (e.g. the Aware Home) that can recognize the activities of the people in a space and yield some sort of an understanding of what is going on and how it can be supported.

Along this direction, we have developed vision-based activity recognition system that can track a person in a know setting and build detailed models of what a person is doing and how the person is interacting with various objects in the environments.  The basic insight in this work is that we can use the relationships between visual features of objects in the scenes and the visual measurement of an activity to build a link between the action and the object. This link is a first step at learning the context in a space and can result in a very detailed interpretation of what is happening.

The bottom-line
The specific interest with this 7001 project is to study the applicability of the above technique to a real situation like the Aware Home. There are very interesting and challenging research questions associated with this and it is not just a simple "apply someone else's work to a new domain."  Darnell Moore, who did most of this work has finished his PhD and making good money at TI.  I am now looking for someone to work on this area and develop detailed representations of the sensory data in various long-term scenarios and develop theories to support the specialized domains.

Here is what you will need to do.

• Read the papers by Darnell on Action Recognition.
  • D. Moore, I. Essa, M. Hayes, "ObjectSpaces: Context Management for Human Activity Recognition," Proceedings of the 2nd Annual Conference on Audio-Visual Biometric Person Authentification, Washington, D.C., March 1999. Compressed Poscript | PDF
  • D. Moore, I. Essa, M. Hayes, "Exploiting Human Actions and Object Context for Recognition Tasks," Proceedings of ICCV 1999, Corfu, Greece. Compressed Poscript | PDF
  • See the project WWW page ObjectSpaces
• Read the following paper, which gives an excellent review of methods used in speech recognition and how they are applied to activity recognition.
• Rabiner, L., "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition", In Proceedings of the IEEE, 77(2), 257--286, 1989.
• Evaluate the results and discuss the differences, strengths, weaknesses of the approach.
• Suggest how this method could be applied to recognition of activities on the home.
  • In the kitchen, while cooking
  • Over a day (not hours).
  • [Both these are very good PhD Topics, I am just looking for your insights as new person in this area]

• Write a 4-5 page Report addressing the following issues:
  • Evaluate critique the methods developed by Darnell, what are the limitations, what is wrong (I know some of what is wrong).
  • Suggest how to extend this to work with cooking
  • Suggest how to make it over extended time (days).