Making the Benefits of Reading Aloud
Prof. Tony Scarlatos
Communication within the hearing-impaired and deaf community has historically been done using the American Sign Language (ASL). Most deaf children learn their first ASL words when they are just a few months old, therefore ASL becomes their first language and the process of learning to read English is, for them, completely equivalent to the process of becoming bilingual. Hearing-impaired children learn to read by recognizing written words as images associated with a meaning. They learn to form words by observing a speakerís mouth and lip movements (as well as facial expression). They have to simultaneously associate images of words with the ASL meaning of the word. Outside of the classroom, most parents and caregivers do not know ASL, and thus are limited in their ability to supplement instruction provided in school.
The software iSign has been designed to solve this problem by providing a tool that translates English spoken words into ASL video clips providing all the information necessary to the child in order to make the correct visual associations.
The prototype system responds to a limited set of semantically related words (semantic web), such as are found in board books for young children. These board books typically focus on some theme such as farm animals, foods, colors, shapes, or parts of the body. Each page contains an illustration and a single word.
The system presents a simple interface on the screen, since most of the interaction is done using vocal commands. An example of a complete interaction with the program is summarized here:
The application runs on a Macintosh computer running the operating system OS X. It uses the very advanced voice recognition capability built into the operating system to recognize the words in our vocabulary. The advantage of this approach is that the system does not need to be trained for a particular voice. Users do not need to install any software that does not come with the operating system, nor do they need any special hardware except for a microphone.
The application is going to be tested at the Cleary School for the Deaf in Nesconset, NY. It will be installed in the computer lab, where it will be used by teachers. We will also loan the application (on a laptop computer with a microphone) to parents of hearing-impaired children through the school, for further testing in the home environment. This work is being supported by the National Science Foundation under Grant No. 0203333.