Beauty of screen composting in Nuke -- Part I

 Hey guys, 

I am here with another post regarding Screen compositing in a green or blue screen footages. 

        

As everyone knows blue screen/green screen is specifically and mostly used for keying or matte extracting or whatever you may call.

Offcourse we would go through keying in different post, but here we are just going to see my way of comping monitor or mobile screens when you are given with a green/blue screen plate. The catch here is we are going to do  this without keying.

Lets first analyse the nature of green and blue screen.

The main thing what green screen does is isolate an element of the input apart from other things, so that we can comp it with other BG elements or other stuff.

But we are going to use some other things from the green screen so that we maximize the laws of physics.

For now we are going to seperate the screen comping process into layers.

1) Compositing

2) Bringing back the reflections

3) Interacting with surrounding elements.

So These are the basic stuff we do in compositing when we comp a screen, But here we are going to combine the first two steps. 

We are going to use the green information in the footage to comp and bring back the reflections at the same time.

For example, we are going to a simple shot.

      

Normally what everyone normally does is, place a screen above the footage using merge over operation.

Then, they would try to key the reflections separately, so that we can add on the top layer.

Yeah this works, but I feel this method is too fakey and disobeying the laws of physics.

This is the process of the traditional methods of screen comps.

But to understand our method, we need to understand colors in composting.

In the above method, what happens basically is we try to key the reflections separately, most of the times we couldn't key it, but when we could, it there will be transparency issues, this is where physics of colors theory comes into play.

The above image is when we try to key the reflections.

When we see a reflection in a footage, what actually happens is each color reacts with the green differently and with different intensities. The black will be different from whites. 

So when we comp, we should do the same. we should add the layer above it based on the laws of physics and colors. whereas this cant be achieved by the traditional methods.

Firstly, We need the information of how the colors fell on the green screen. Basically, This information is provided by the green screen. Lets see how its done.

We need to get both types of information, -- intensity and color.

for intensity it can be achieved by despilling the green.

Basically, I prefer this to be done by a gizmo called despill madness. Although manual despilling using hue correct node have more control and better for comp despilling. but here despill madness is enough.

After we despill we should use merge minus operation to seperate the greens from the the footage.

Now if we view the greens in green channel we would get a data of the intensities of greens in black and white. We can feed this data into alpha channel using a shuffle node. As we got the alpha of the greens. 

    

    

Multiply the alpha with the screen element. This process brings and matches the color with intensity. So we got the exact intensity and color a screen will be placed. Note is we don't need to bring back reflections, as the intensity info we have will be less in the reflected part, So it would bring back the reflections in a more mathematical and realistic way. 

And as for this technique, I discovered this all by myself just while trying stuff. So do try new stuff. Thats where magic happens.

Now we have the intensity and the color information of the screen element. Now we just have to add this to the despilled plate using plus operation.

Basically I have used keylight for the mask to make this affect only the screen area.

So here, we have basically comped, Just used some advanced techniques to blend the edge enough. Yeah, we will see those in other video for sure. The edge can be improved even more, we will save it for another blog.

Yeah that's it. We don't even need to add grain. everything will be matched perfectly. Literally what happened is we comped the plate like as a real screen is placed in the plate. Now we can come to the next part. -- INTERACTION IN THE ENVIRONMENT

For the next part, the same green screen is coming into play. For the interaction, we need the information which area would have been affected if a screen was there in the plate. For that purpose, we are going to use the spill information. It would give a basic information on how the environment is affected if a screen of green wavelength is placed. So we can tweak it later.

First of all, what we should do is to use the same method, like separating the greens from the plate and feeding it into alpha using despill and minus operation, But here we don't need to affect the screen area, So What we are going to do is basically stencil the screen area using a hard key and some blur.

Now we got an alpha consisting information of the intensities of environmental interaction. To add the interaction just add another layer of the screen. By scaling it up, blurring, and masking with the spill info we have, the interaction would have been perfectly placed obeying the laws of physics.

Quality can be increased even higher through layer based interaction to be more precise. for example, first layer with a slight scale up and second with a higher scale factor. Finally touching with a overall blur gradually decreasing nearby the screen and higher as far away.\

For the final fact, even now we don't have an exact replication of the nature and laws due to the slight  difference in the intensity and wavelength of green and our screen element. We can tweak it manually.

For that plugin a grade node in alpha channel, and erode node if you need a higher rate of environmental interaction. Every one of these should be connected to the alpha pipe line. So that we can control the intensities in the favor we want.

Traditional method:

I named it PBK(Physics based keying):

(knowing someone would have found this already)

Note : I have used a hue correct before the plus operated merge, to reduce the despill extras. For example if we despill green, it would produce a blue tone, to remove that I used minor hue correct.

You know why that IBK is the best in keying. Thats because it's algorithm is more similar to the thing we used here.

IBK, Das Grain, PBK all of these.

Thanks,

So that's it for the blog!

See you at other post.