Home > News > Software

Disney software for automatic lip syncing in real time

Gianluigi Piludu 10/08/2017

Disney software for automatic real-time lip synchronization

Submitted to SIGGRAPH 2017 in Los Angeles, this software uses a very realistic and "believable" technique to generate animated speeches in real time.

Automatic lip syncing in a cartoon, animated film or video game (called Lip Sync in English) is not a new technique, but Disney, along with a group of researchers from the University of East Anglia (England), Caltech and Carnegie Mellon University have added a small and big innovation: deep learning. Researchers are using an even more refined and precise technique to generate animated speeches in real time. In addition to automatically generating lip sync for English-speaking actors, the new software can be applied to make the animated character sing or adapt him to speak foreign languages.

Realistic speech animation is key to effective character animation, "said researcher Dr. Sarah Taylor, of UEA School of Information Sciences." If this is done poorly, it can generate disorientation and lead to a flop. at the box office of the entire film. If done well, it requires excessive processing times and costs on the part of the animator. Our goal is to automatically generate a quality animated language for any character style, based on a speech given by an audio file. "

 

The team of researchers has designed software that can teach the computer how to interpret the words spoken by an actor's voice, shape the shape of the mouth and then animate it by synchronizing the character's lip with the actor's voice.

The process requires both the recorded audio file, and eight hours of video reference from a single speeker who recites a collection of over 2.500 sentences with different phonetic phrases.

Audio is transcribed into speech sounds (phonemes) using off-the-shelf speech recognition software. These phonemes are then applied to the reference face, and these results can be reported in any 3D character, who can thus speak in real time.

“Our automatic speech animation therefore works for any style of speech and can also work in other languages,” said Taylor. "Our results so far show that our approach achieves cutting-edge performance in visual speech animation. The real beauty is that it is very simple to use and easy to edit and stylize using standard editing software."

While it would be easy to upset the quality of the technology based on these early examples, it's not hard to imagine how in 10 or 20 years automatic lip syncing could be the standard for most computer-generated characters. In its current state, real-time lip syncing could play an invaluable role for video game applications, location-based animation projects, and television series that require large volumes of animation on modest budgets.

The Walt Disney Company isn't the only production company exploring automatic and facial animation techniques. An entire section was devoted to the topic at the most recent SIGGRAPH 2017 , with the most innovative techniques.

Source: http://www.cartoonbrew.com/tech/disney-researchers-developing-new-method-automated-real-time-lip-sync-152737.html

<< Previous