Social Game Retrieval from Unstructured Videos

Ping Wang     Gregory D. Abowd    James M. Rehg

            Project description       Publications     Dataset   Acknowledgements    Contact

Examples of the mined patterns from social games



Project description 

Video-based behaviors, especially social interactions, are widely analyzed in screenings and diagnosis of developmental disorders, as well as studies on child development. Existing methods rely on human's frame-by-frame tagging and editing of video content, which is time-consuming and labor-intensive. Automatic retrieval of social games is a logical first step towards automatic analysis of social interactions for three reasons. First, these games typically arise between 8 and 12 months of age, when infants have not yet achieved full mobility, making it easier to monitor the game with a single relatively-stationary camera. Second, many infant social games such as peek-a-boo and so-big are characterized principally by gross motor movements. Third, these games are fairly structured interactions and they typically follow a regular turn-taking pattern. It suggests that analysis of repetitive patterns of gross motion in video could be sufficient to identify the instances of social games in a video collection, and distinguish social games from other video content.

Selected results

1.      Parsing semantic stages of a game

2.      Retrieval of social games from laboratory videos

3.      Retrieval of social interactions from home movies


1.      Quasi-Periodic Event Analysis for Social Game Retrieval, Ping Wang, Gregory D. Abowd and James M. Rehg, in Proceedings of IEEE International Conference on Computer Vision, Kyoto, Japan, 2009. [Oral presentation, acceptance rate 3.6%]

2.      Automatic Retrieval of Mother-Infant Social Games from Unstructured Videos, Ping Wang, James M. Rehg, Gregory D. Abowd and Rosa I. Arriaga, International Meeting for Autism Research (IMFAR), 2009. [Poster presentation]


Please email me your contact information (email address, name and research institute) if you need data 2 and 3. Thanks!

1.      Social games from YouTube (1 pattycake, 3 peek-a-boo). Download

2.      Children’s play in a laboratory setting.

a.      HD format. Video 1 (3.54 GB), Video 2 (2.56 GB), Video 3 (5.47 GB)

b.      Resolution of 160-by-90. This is the resolution we used in our experiments for ICCV ‘09. Download

3.      Home movies. The videos are of resolution 720 by 480. In our experiments, we used a reduced resolution of 180 by 120. Download (2.56 GB)


We thank our collaborators from the autism research community. This work was supported in part by Children’s Healthcare of Atlanta, an award from the Microsoft Research program in Intelligent Systems for Assisted Cognition, and Google Research.

Dr. Opal Ousley, Emory Autism Center

Dr. Agata Rozga, Georgia State University

Marcus Autism Center


pingwang at

Last updated on 12/11/2009