The SYNFACE project

Synthesised talking face derived from speech for hearing disabled users of voice channels

Funded by CEC Framework V IST program (reference number IST-2001-33327)

October 2001 - December 2004.

SYNFACE project home page at KTH

This work is an EU-funded project in collaboration with:

• KTH Dept. Speech Music and Hearing, Stockholm, Sweden (project leader)
• Babel Infovox, a Swedish developer of speech technology for telecommunications
• Research and Development department of the Instituut voor Doven, Sint-Michielsgestel, Netherlands:
• Royal National Institute for Deaf People, UK.

 The project aimed to develop a real-time system that delivers phonetically informative lipreading cues derived from parameters extracted from the acoustic speech signal. The starting point was a non-real time prototype developed for the Swedish language. This comprises a computer-generated wire-frame head model whose movement is controlled by a look-up table linked to the output of a Hidden-Markov-Model phoneme recogniser.

During the course of the project the head model was been developed further to add movements for sounds that occur in English and Dutch but not in Swedish. A series of studies of lipreading of the face by hearing-impaired and normally hearing people guided this development and verified the usefulness of the head-model

A second major development has been to assemble a real-time speech recogniser that operates sufficiently accurately with telephone speech. Because the head has to keep up with running speech, the recogniser is unique in working not only fast but on very short time samples of speech.

The project ran trials of a real-time version of the complete system that ran on a PC connected to a normal telephone - trials with hearing-impaired users showed great interest, even though the speech recognition performance was shown to be in need of further improvement

UCL's contribution to the project was led by Dr Andrew Faulkner. Other UCL staff associated with the project include Professor Ruth Campbell, Dr Mark Huckvale, and Dr Valerie Hazan.