Facial Modeling/Animation: Predicting visual information from acoustic for Lip Synch

Problem
In facial animation, Lip Synch is known as the process of synchronizing a pre-recorded sound track with the lip and facial movements of a synthetic character. To realize a successful animation, it is required to perform an accurate timing alignment between the acoustic signal and the visual rendering. The animation film "Alien Song" by Victor Navone shows a good example of lip synch. The traditional approach in computer graphics consists in labeling the sound track with visual targets, these targets corresponding to the pronounced phonemes (sound units in speech). This time-consuming step can be helped with speech recognition  but the available systems for lip synch making use of speech recognition still require some "art work" to correct the automatic labeling. An alternative approach consists in trying to estimate the facial geometry directly from the acoustic signal. The goal of this project is to investigate an example of this approach.

For this project, the visual rendering will be supported by the facial animation system available at the CPL (ask Lionel Reveret). This system uses a characterization of facial movements in speech production, based on a set of visual speech parameters (see below article of Reveret et al.). The work of this project will be to implement a prediction of these visual speech parameters from an acoustic signal (multilinear regression, neural networks, HMM,...)
 

Here is what you will need to do.

• Read the following paper for a description of the facial animation system available at the CPL and the visual speech parameters :
• Reveret, L., Bailly G., Badin, P., "MOTHER: A new generation of talking heads providing a flexible articulatory control for video-realistic speech animation", Proc. of the 6th Int. Conference of Spoken Language Processing, ICSLP'2000, Beijing, China, Oct. 16-20, 2000.
• Read the following papers for examples of prediction of facial geometry from acoustic signal :
• Brand, M., "Voice Puppetry", SIGGRAPH'99.
• Bregler, C., Covell, M., Slaney, M., "Video Rewrite: Driving Visual Speech with Audio", SIGGRAPH'97.
• Study the HTK, Waves and Transcriber system by Entropics Inc. which are toolkits for HMM and speech recognition available in the Computational Perception Lab. (One of grad students will show you how).
• Develop a demo of prediction of visual speech parameters. This demo will be connected with the facial animation system available at the CPL.
• Evaluate the results and discuss the strengths, weaknesses of the approach.

• Talking heads web site
• Above papers will be provided

• Write a 4-5 page Report addressing the following issues:
• Give a brief overview of what you understand is the Lip Synch problem. Is it tractable ?
• Report your progress with the recognition demo for lip synch and the choice of the prediction method.
• Compare all the systems you tried out, discuss the differences, strengths, weaknesses.