In Depth: Brilliant (and Possibly Dangerous) Future Video and Audio Technologies

Audio Replacement

We’ll start with audio replacement. There have been several developed over the years and they’ve gotten to the point where these tools can create human voices from samples and it’s completely believable.

WaveNet by Google DeepMind

Sample someone’s voice and you can make someone say anything. Google’s speech synthesis is truly impressive technology. It generates wave forms too. The technology can make a voice speak another language that they are unable to speak, which will be great for dubbing films to foreign languages and having American actors pitch products in Japan without learning the language. If Morgan Freeman is unavailable for a voice over project, perhaps he could allow a company to use his voice from a generator and no one would be the wiser.

Here’s more about the WaveNet system.

Adobe VoCo

VoCo, which is explained as the “Photoshop for Voice”, was shown at Adobe MAX 2016 and is a technology that can replace the audio of someone’s speech by editing the transcripts. It sounds pretty darn flawless. The idea is that this technology would be used in replacing flubbed or changed words in film production without having to re-record the audio.

Adobe MAX 2016 (Sneak Peeks) | Adobe Creative Cloud

This demo is so convincing at Adobe Max 2016. The technology is really cool and will be so useful for sound replacement in films, however, the comments on this video show how skeptical the public is about this technology being used to make people say things they never said.

Synthesizing Obama: Learning Lip Sync from Audio

This video was for SIGGRAPH 2017 from the University of Washington, which happened just last week for a presentation by Supasorn Suwajanakorn, Steven M. Seitz, and Ira Kemelmacher-Shlizerman. They used a target video of Obama too so it’s a bit confusing at first but they’re adding lip sync from one Obama video to another video.  Read more about the mathematics and technology behind this: Synthesizing Obama: Learning Lip Sync from Audio.

Breaking News from Radiolab

Radiolab talks about VoCo in the beginning of the podcast on to explain some video technology and how they can track someone’s facial movements and use that data to create a video that didn’t happen. This could be used to bring celebrities back from the dead for commercials or for replacing something in a movie that isn’t quite right, without a full re-shoot. This could also be used to create fake news or to blackmail people. It’s all very interesting technology but it will come with a price.

De-Aging and Aging Actors

I remember watching A League of Their Own when I was a kid and that was really the first time I remember seeing movie stars aged. They used different actors for the older characters though, not Geena Davis and Lori Petty, but their voices are played by Geena and Lori. Totally believable.

I’ve seen so much de-aging of actors lately from Brad Pitt as Benjamin Button to Robert Downey, Jr. in Captain America: Civil War. I thought they did a fantastic job with both actors. They also de-aged Kurt Russell in Guardian’s of the Galaxy and most notably, Carrie Fisher, at the end of Star Wars: Rogue One. I will admit, it’s a bit jarring. Although it looks good, my brain knows how the actors really look. Many of these examples are cited in this article at The Ringer, The De-aging of Old Actors Needs to End and Film School Reject’s The Larger Implications of Hollywood’s De-Aging Fascination.

Personally, I have no issue at all with de-aging actors. It puts VFX artists to work and most of the time, it looks really good. The actors who are involved know full well what is happening.

Captain America Civil War Young Tony Stark

Robert Downey, Jr. as young Tony Stark may be one of the most impressive de-aging I’ve seen yet. He looks just like he did in Tuff Turf (a guilty pleasure, I know)!

How Benjamin Button got his face – Ed Ulbrich

Ed Ulbrich from Digital Domain talks about the amazing VFX in The Curious Case of Benjamin Button.

Bringing Actors Back from the Dead with CGI

Fred Astaire Dirt Devil Commercial

I remember the Fred Astaire Dirt Devil commercial with him dancing with a vacuum from about 10 years ago and there was a bit of controversy about the ethics of bringing him back to life and this was approved by his daughter.

Bruce Lee in Johnnie Walker commercial “Game Changer”

Another example is the legendary Bruce Lee in a Chinese ad for Johnny Walker whiskey. Would Bruce Lee be okay with pitching booze, considering he didn’t drink? Most of the comments on YouTube are complaining about him not speaking Cantonese in the ad. From the Atlantic, Johnnie Walker Offends by Using Bruce Lee in Chinese Ad.

Star Wars Rogue One – CGI Tarkin Explained

In Star Wars: Rogue One, they took it much further by resurrecting actor Peter Cushing back, who died in 1994, to reprise his role as Grand Moff Tarkin with a technology known as Light Stage. They scanned hours of footage of Cushing and then had a living actor play the character and used motion capture technology to record the movement. Cushing’s face was mapped on this actors face. CBS Sunday Morning did an article on this technology, Digital doubles: Bringing actors back to life.

Real-Time High Fidelity Facial Performance Capture

This is from Disney Research Hub to give you an idea of how facial expressions are captured.

Real-time Facial Reenactment

This could go further than just bringing dead actors back to life. This technology could be used to improve dubbing of foreign films, which is a plus, but it could be used to make living people, like world leaders, say things that they never said. Fake news is bad already, but imagine when there is a video to support it!

Face2Face: Real-time Face Capture and Reenactment of RGB Videos (CVPR 2016 Oral)

Face2Face uses live capture to generate a re-enactment. While this looks slightly off and gets into the uncanny valley, I’m looking for abnormalities. If I didn’t realize they were doing this, if I was passively watching the news, I may just think this was really George Bush or Vladimir Putin.

Creating Video of things that Never Happened

This is incredible technology and while I don’t think it’s 100% perfect, it’s pretty darn close, and it’s getting better. Right now, some of the software is proprietary and unavailable to anyone outside of a production. you can do face replacements using software like Mocha, but it takes skill, time, and money. What if this technology were cheap and easy to obtain and anyone could get their hands on it? You can’t trust the news as it is, but imagine if there’s video support? I can see a future of propaganda and misinformation campaigns from political parties. For entertainment purposes, like de-aging an actor or bringing someone back to life, it definitely has its purpose, but it could potentially be used to make politicians say things they didn’t say or used to get revenge on your ex.

AMAZING Facial Recognition Breakthrough Announced

Flash Forward: Unreel

Another podcast that I regularly listen to is Flash Forward, which is about future scenarios. They have experts weigh in on each topic and it’s always very interesting. This episode is about a future where anyone can create a video of you doing anything that they want with software that is cheap and easy to get. They talk about problems with things like revenge porn or creating false video confessions.

They reference a video by MIT in the program and it’s here: Creating Videos of the Future. 

The Future of Video?

The future is already here. The fantastic work by CG artists is only getting better. Facial tracking technology has gotten insanely good and is only improving. Used in the right way, it’s incredible technology. Used in the wrong way, it’s dangerous. Audio can be manipulated and be undetectable. I see this as a major issue that will be upon us in the next few years. How can we know if something is fake? I have a feeling that the amount of fake news and propaganda is going to be getting out of hand in the next few years. There needs to be some sort of watermark or some other method to show that a video has been manipulated. Imagine what top-secret technology is being developed that we know nothing about? Is this already being used in the media and we don’t know it? It’s both exciting and scary!