Matthias Niebner from Stanford University along with his fellow researchers has created a computer application as a part of their ongoing thesis that can do a real-time facial reenactment of a monocular target video sequence. In simple words, you can use Donald Trump’s face and mimic the facial expressions of another, in real-time video.
Technologies related to facial recognition applications like motion capture (mo-cap) is not new to the world especially in Hollywood movies like Gollum in “Lord of the Rings” trilogy, Apes in the “Rise of the planet of the apes”, or those tall blue Na’vi clan from “Avatar“. Check out the video linked below for more elucidation on this topic.
But, this technology is not cheap and requires extremely skilled engineers to operate. In contrast to this, Matthias Niebner’s team’s goal is to modify the target video in a photo-realistic fashion, such that it is virtually impossible to notice the manipulations.
Real-time Face Capture and Reenactment of RGB Videos techniques are bit different than the technology we referred above. If the product is out for commercial use, people and especially filmmakers from all over the world can utilize this to create mimicking videos in real time; and can be operated by anyone. According to the researchers, their goal is to do an online transfer for the facial expressions of a source actor captured by an RGB sensor to target the actor.
- According to their thesis, their key contributions are
- Dense, global non-rigid model-based bundling
- Accurate tracking, appearance, and lighting estimation in unconstrained live RGB video
- Person-dependent expression transfer using subspace deformations
- Novel mouth synthesis approach
By minimizing the distance to the target expression, above image clearly shows you how they used appearance graph to retrieve new mouth frames.
What’s the connection with YouTube?
Apart from YouTube being their only source of data feed they used in their videos, this technology when implemented can give rise different content creators. As this system works with any camera and any recorded video, YouTube channels like Bad Lip Reading (5 M subs), who put words into other people’s mouths can use this technology to enhance his videos.
To conclude, this Real-time Face Capture can pave the way for many new and exciting applications in the fields of VR/AR, teleconferencing, or on-the-fly dubbing of videos with translated audio. But, there is a chance that this application can go into wrong hands that can use this Face Capturing method to propagate wrong statements from famed politicians and celebrities.
Check out the video they uploaded on YouTube
Being an artist, movie buff and a media enthusiast, content writing is my career train. I am a proud alumni of Symbiosis Institute of Media Communication (Pune) and currently working for Vidooly.