Robust Speech Recognition using Audio and Visual Information

Problem
One of the most promising innovation to increase the robustness of Automatic Speech Recognition systems (ASR) consists in making an efficient use of the visual information that can be perceived from the speaker's face, in addition to the standard acoustical information. In human communication, it is well known that subjects with hearing impaired are able to retrieve some linguistic information from a speaker's face (lip-reading). In the past decades, a lot of perceptual experiments have proved that even listeners with no hearing loss can also take benefit of visual information to correctly perceive speech, especially when the acoustical environment is degraded (background noise, other speakers). It turned out from these experiments that the perception of speech by humans is always higher when both modalities, audio and video, are used. This property of speech for being intrinsically bimodal has been employed to implement automatic Audio-Visual Speech Recognition (AVSR) system. The choice of what are the best visual features to represent speech and how to combine them with acoustical data are still open questions.

In a recent article, Reveret et al. propose a representation of facial movements for speech production, based on a set visual speech parameters. Using visual tracking, these parameters can be extracted from a video sequence of a speaker. The goal of this project is to implement an Audio-Visual Speech Recognition system that combines acoustical information and these visual speech parameters as visual information.
 

Here is what you will need to do

• Read the followings papers, which gives an excellent review of methods used in speech recognition.
• Reddy, R., "Speech Recognition by Machine: A Review", In IEEE Proceedings 6(4), 502--531, 1976.
• Rabiner, L., "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition", In Proceedings of the IEEE, 77(2), 257--286, 1989.
• Read the following paper for a description of the facial speech parameters used as visual information :
• Reveret, L., Bailly G., Badin, P., "MOTHER: A new generation of talking heads providing a flexible articulatory control for video-realistic speech animation", Proc. of the 6th Int. Conference of Spoken Language Processing, ICSLP'2000, Beijing, China, Oct. 16-20, 2000.
• Read the following paper for a recent overview on audio and visual information integration for speech recognition
• "Continuous Audio-Visual Speech Recognition", J. Luettin and S. Dupont, in "Proc. 5th European Conference on Computer Vision",1998.
• Study the HTK, Waves and Transcriber system by Entropics Inc. which are toolkits for speech recognition available in the Computational Perception Lab.(One of grad students will show you how).
• Develop a demo of audio visual speech recognition.
• Evaluate the results and discuss the strengths, weaknesses of the approach.

• Commercial Speech Recognition Systems
• Above papers will be provided

• Write a 4-5 page Report addressing the following issues:
• Give a brief overview of what you understand is the speech recognition problem and what new solutions and problems arose using visual information. Is it tractable ?
• Explain the Hidden Markov Models (HMM) and the multistream approach.
• Report your progress with the HTK package and the audiovisual recognition demo.