Kim Xu


Telesign is a system designed for Deaf adults attempting to carry out service transactions with a hearing person such as visiting the veterinarian or getting their oil changed. American Sign Language (ASL) is the native language of the Deaf in the United States. ASL has a completely different grammar than English. This makes it difficult for a Deaf person to communicate in English with a hearing person.

Traditional methods of communication between the Deaf and hearing are writing on paper and passing it back and forth or typing onto a Sidekick-like device and showing that. Both of these methods require the Deaf individual to create grammatically correct or at least semantically understandable English phrases which may be difficult. Telesign is a one-way phrase book-based system into which the Deaf individual can sign a phrase and the system provides a number of phrases in English which can be selected.

The interface for Telesign is shown in the image below. First, the user ensures that his or sign space (the three-dimensional space that the person will use for signing) is within the view of the camera. The user then presses the "Start Signing" button. After the user has signed the desired phrase, "Stop Signing" is pressed. A screen with three phrase choices is shown in English. The user chooses the correct phrase and can then display that phrase to the hearing individual. Interface design for the Telesign device

I designed and carried out a study to compare Telesign with the more traditional communication methods of writing or typing. Although a functional ASL recognizer has been created for Telesign, I used a "Wizard of Oz" approach in the study. In Telesign, videos captured by the device (A) are transmitted to another computer where a human translator with knowledge of ASL types in the appropriate English translation (B). This translation is then sent to the mobile device along with other appropriate phrases (C).
An illustration of how Telesign's Wizard of Oz functions

A paper with the results from the Telesign study has been submitted as a long paper to CHI 2010. A poster explaining more about Telesign can be seen here.