Netflix’s AI-assisted green screen paints actors a stunning magenta hue

Image credit: Netflix

The process of compositing—the process of placing actors in front of a background that doesn’t really exist—is as old as filmmaking itself, and has always been painful. Netflix is ​​introducing a new technique that relies on machine learning to do some of the hard work, but it requires lighting the actors in a flashy magenta.

For decades, chroma keying has been the easiest method of compositing. In this method, the actors stand against a light-colored background (initially blue, later green) that can be easily identified and replaced, from weather maps to fighting Thanos. The foreground is said to be “matte” and the background is a transparent “alpha” channel operated along with the red, green and blue channels.

This is easy and cheap, but has some drawbacks. Among these are problems with transparent objects, fine details such as hair, and of course anything else similar in color to the background. However, this is usually sufficient, and attempts to replace it with more sophisticated and expensive methods (such as light field cameras) have stagnated.

But Netflix researchers challenge the possibility of combining old and new for a simple, perfect composition, but at the cost of hellish lighting setups on set. Become.

As described in a recently published paper, their “magenta green screen” produces impressive results by essentially placing the actor in a sandwich of lighting. Behind them, bright green (actively lit, not background). On the front, red and blue mix to create a dramatic contrast.

On-set results can daunt even the most seasoned post-production artist. Generally, you want a fairly natural light to brightly illuminate your actors. So it might need a little punch here and there, but the actor’s appearance on camera is relatively normal. But when illuminated with only red and blue light, its appearance is completely distorted. Because, of course, normal light doesn’t cut a large part of the spectrum.

However, this technique is also clever in that it simplifies the process of separating the two by having only red/blue foregrounds and only green backgrounds. A normal camera that normally captures these colors will instead capture red, blue and alpha. This made the resulting matte very accurate and eliminated artifacts from having to separate the full spectrum input from the spectrally limited key background.

Of course, it seems that they just replaced one difficulty with another. The process of compositing is now easier, but restoring the green channel to a magenta lit subject is difficult.

Different subjects and compositions have to be done systematically and adaptively, but the “simple” linear approach of injecting green gives a faded, yellowish look. How can we automate this? AI can help!

The team trained the machine learning model on its own training data, which was essentially a “rehearsal” take of a similar scene shot normally. A convolutional neural network is fed a patch of a full-spectrum image to compare with the magenta-lit image and develops a process that rapidly restores the missing green channel in a way that is more intelligent than a simple algorithm. To do.

As such, it can recover color surprisingly well in post (“virtually indistinguishable” from in-camera ground truth). But the problem still remains that the actors and set have to be lit in this horrible way. Many actors are already complaining about how unnatural it is to work in front of a green screen. Imagine doing it in a harsh, inhuman light.

However, in this paper this is addressed by the possibility of “time multiplexing” the illumination, essentially allowing the magenta/green illumination to be turned on and off multiple times per second. This can be distracting (and even dangerous) when you do it 24 times per second (that’s the frame rate most movies and TV are shot at), but it can be distracting (and even dangerous) when you switch lights faster (144 per second). times), it appears to be “nearly constant”.

However, this requires complex synchronization with the camera and only captures light during the moment when the scene is magenta. And you also have to account for missing frames of motion…

As you can see, this is still very experimental. But it’s also an interesting way to tackle a long-standing problem in media production with a fresh, high-tech approach. This he was not able to do five years ago. I don’t know if it will be adopted on set, but it’s definitely worth a try.