Intel’s deepfake detector tested on real and fake videos

Last March, a video was released showing President Volodymyr Zelensky calling on Ukrainians to lay down their weapons and surrender to Russia.

It was a very obvious deepfake. It’s a kind of fake video that uses artificial intelligence to swap faces or create a digital version of someone.

However, as AI advances make it easier to create deepfakes, it becomes increasingly important to detect deepfakes quickly.

Intel believes it has a solution, but it’s all blood in the face.

The company called the system “FakeCatcher”.

At Intel’s fancy, mostly empty offices in Silicon Valley, I met Ilke Demir, a research scientist at Intel Labs, to explain how it works.

“We ask ourselves what’s real about real video? What’s real about us? What’s the watermark of being human?” she says.

At the heart of this system is a technology called photoplethysmography (PPG) that detects changes in blood flow.

Faces created by deepfakes don’t emit these signals, she says.

The system also analyzes eye movements to check authenticity.

“Normally, when a human stares at a point, when I look at you, it feels like my eyes are shooting rays of light at you.

By examining both of these characteristics, Intel believes it can tell the difference between real and fake video within seconds.

The company claims FakeCatcher is 96% accurate. So I decided to give this system a try. Intel agreed.

I used dozens of clips of former President Donald Trump and President Joe Biden.

Some were real, while others were deepfakes created by the Massachusetts Institute of Technology (MIT).

In terms of finding deepfakes, the system seemed pretty good.

We mainly chose lip-syncing fakes, i.e. real videos with altered mouths and voices.

And all but one answer was correct.

But when I watched the actual real video, I started having problems.

Several times the system said the video was fake, even though it was actually real.

The more pixelated the video, the harder it is to capture blood flow.

The system does not analyze audio either. As such, some videos were assigned as fakes that clearly looked real when heard the audio.

What worries me is that if the video is real, but the program claims it’s fake, it can cause real problems.

When we bring this up to Demir, she says, “Verifying that something is fake carries a different weight than saying, ‘This might be fake, so be careful.'”

He claims the system is overly cautious. It’s better to catch all the fakes and some real videos than to miss the fakes.

Deepfakes can be incredibly sophisticated. For example, a two-second clip of him in a political campaign ad. It may be of poor quality. You can create a fake just by changing your voice.

In this regard, FaceCatcher’s ability to “actually”, i.e., work in real-world situations, has been questioned.

Matt Groh is an assistant professor at Northwestern University in Illinois and an expert in deepfakes.

“I doubt the stats they gave in their first evaluation,” he says. “But what I do wonder is whether that statistic is relevant to real-world situations.”

This is where FakeCatcher’s technology is difficult to assess.

Programs like facial recognition systems often offer very generous statistics in terms of accuracy.

However, actual testing in the real world may result in lower accuracy.

Essentially, accuracy depends entirely on test difficulty.

Intel claims that FakeCatcher has passed rigorous testing. This includes a “wild” test in which the company compiled 140 fake videos, as well as the real ones.

Intel said the system had a success rate of 91% in this test.

But Matt Groh and others would like to analyze the system independently. They don’t think it’s enough for Intel to challenge itself.

“We would love to evaluate these systems,” Groh says.

“I think this is very important when designing audits and trying to understand how accurate something is in real-world situations,” he says.

It’s amazing how hard it is to tell fake videos from real ones, but this technology certainly has potential.

However, our limited testing shows that we still have a long way to go.