Propagation Networks for Recognizing Partially Ordered Sequential ActivityYifan Shi, Yan Huang, David Minnen, Aaron Bobick, Irfan Essa


This work is presented in CVPR'2004, (PDF, BIB)
Represent and fuse human knowledge of daily activities with noisy perceptual features  
Detect and recognize an activity  
Pinpoint components of the activity and detect missing or improperly performed steps 
Difficulty in modeling parallel streams
 
Difficulty in handling noisy sensor data

Handle multiple, parallel streams  
Naturally represent partially ordered activity components  
Directly model duration of components  
Deal with noisy underlying vision sensor data  
Classify whole sequence as well as identify individual components  
Applied to recognition of actions in the medical domain 
Use links to represent temporal partial ordering;  
Use nodes with duration model to represent temporal interval;  
Represent parallel activity stream with multiple, simultaneously active nodes;  
Use conditional probability function to represent temporal/logical relationship between nodes;  
Embed duration model in activation triggering function;  
Use hidden observation model for underlying vision uncertainty 
Local Maximal Search Algorithm
 
DCondensation

Model the complex LifeScan glucose calibration process with a PNet using simple vision detectors 3 subjects with 41 videos 3 task categories:
Correct: all nodes present. PNet identifies them all Almost right: one node missing. PNet incorrectly accepts 20% and correctly reports missing steps in the other 80%. Negative: missing many steps. PNet correctly rejects half of the test sequences, others are accepted, but PNet correctly identifies the missing steps.
PNet Definition for Glucose Meter Calibration Task 
System Architecture 
Detection rate for individual nodes over all samples 
Ability to locate critical moment 
TurnOn [44] 
Unscrew Cap [229]  Drop Solution [256]  Screw Cap [288]  Insert Strip [329]  Mark Ctrl [655] 
Snapshots from example sequence and corresponding detection results 
Subject1 correct sequence 1 subject1_right1.avi  
Subject1 correct sequence 2 subject1_right2.avi  
Subject1 correct sequence 3 subject1_right3.avi  
Subject1 correct sequence 4 subject1_right4.avi  
Subject1 correct sequence 5 subject1_right5.avi  
Subject1 correct sequence 6 subject1_right6.avi  
Subject1 correct sequence 7 subject1_right7.avi  
Subject1 correct sequence 8 subject1_right8.avi  
Subject1 correct sequence 9 subject1_right9.avi  
Subject1 correct sequence 10 subject1_right10.avi 
Subject2 correct sequence 1 subject2_right1.avi  
Subject2 correct sequence 2 subject2_right2.avi  
Subject2 correct sequence 3 subject2_right3.avi  
Subject2 correct sequence 4 subject2_right4.avi  
Subject2 correct sequence 5 subject2_right5.avi  
Subject2 correct sequence 6 subject2_right6.avi  
Subject2 correct sequence 7 subject2_right7.avi  
Subject2 correct sequence 8 subject2_right8.avi  
Subject2 correct sequence 9 subject2_right9.avi 
Subject2 missingonestep sequence 1 subject2_noshake1.avi  
Subject2 missingonestep sequence 2 subject2_noshake2.avi  
Subject2 missingonestep sequence 3 subject2_noshake3.avi  
Subject2 missingonestep sequence 4 subject2_noshake4.avi  
Subject2 missingonestep sequence 5 subject2_noshake5.avi  
Subject2 missingonestep sequence 6 subject2_noshake6.avi  
Subject2 missingonestep sequence 7 subject2_noshake7.avi 
Subject2 missingsixstep sequence 1 subject2_noapply1.avi  
Subject2 missingsixstep sequence 2 subject2_noapply2.avi  
Subject2 missingsixstep sequence 3 subject2_noapply3.avi  
Subject2 missingsixstep sequence 4 subject2_noapply4.avi  
Subject2 missingsixstep sequence 5 subject2_noapply5.avi 
Subject3 correct sequence 1 subject3_right1.avi  
Subject3 correct sequence 2 subject3_right2.avi 
Subject3 missingonestep sequence 1 subject3_noshake1.avi  
Subject3 missingonestep sequence 2 subject3_noshake2.avi  
Subject3 missingonestep sequence 3 subject3_noshake3.avi 
Subject3 missingsixstep sequence 1 subject3_noapply1.avi  
Subject3 missingsixstep sequence 2 subject3_noapply2.avi  
Subject3 missingsixstep sequence 3 subject3_noapply3.avi  
Subject3 missingsixstep sequence 4 subject3_noapply4.avi  
Subject3 missingsixstep sequence 5 subject3_noapply5.avi 
Copyright © 19972005 
Last Updated Dec 31, 2004.