UK boffins work on lip-reading machine

Researchers are working on technology that can automatically convert video of lip-motions into text

Researchers at the University of East Anglia (UEA) are starting a new project to develop computerised lip-reading systems that could be used to tackle crime.

The three-year project will collect data for lip-reading and use it to create a system that can automatically convert video of lip-motions into text in a variety of languages.

Researchers are taking advantage of work already carried out at UEA to develop state-of-the-art speech recognition systems.

The project has received £400,000 from the Engineering and Physical Sciences Research Council, and is headed up Dr Richard Harvey, senior lecturer at UEA's School of Computing Sciences.

"To be effective the systems must accurately track the head over a variety of poses, extract numbers or features that describe the lips, and then learn what features correspond to what text," said Dr Harvey.

"To tackle the problem we will need to use information collected from audio speech, so this project will also investigate how to use the extensive information known about audio speech to recognise visual speech."

The university is teaming up with the Centre for Vision, Speech and Signal Processing at Surrey University, which has built accurate and reliable face and lip trackers, and the Home Office Scientific Development Branch, to investigate the feasibility of using the technology for crime fighting.

"The Home Office Scientific Development Branch is interested in anything that helps the police gather information about criminals," added Dr Harvey.

"It appears that the best lip-readers are the ones who learned to speak a language before they lost their hearing and who have been taught lip-reading intensively. It is a very desirable skill.

"We all lip read, for example in noisy situations like a bar or party, but even the performance of expert lip readers can be very poor."

Other potential uses for the technology include installing a camera in a mobile phone or on the dashboard for in-car speech recognition systems.