MART Term 2, Lecture 9
Render Layers and Passes

Render Layers

One way to reduce the time a scene takes to render is to split it up into parts that are easier for the computer to deal with. These are referred to as passes, and are usually put back together in a compositing package, such as Shake. The added benefit of splitting the scene up into passes is that much more control is available in post production, allowing last minute changes to the overall look of the scene.

Splitting Up The Scene

Imagine you have a scene containing a character in the foreground, a background, and two lights. How do you split it up? How many passes should you use? The answer to that depends on how much control you want to have at the end of the process:

This allows adjustment of the colour, brightness, etc in the character separately from the environment. However, if you have shadows in your scene, you'll also need a shadow pass.

This allows adjustment of the intensities of the two lights in post production.

This allows the brightness of the specular highlights to be adjusted independently from the colour. It does, however, mean that if you have any reflections or refractions these can (and probably should) also be rendered in separate passes.

Let's assume that our scene has shadows and reflection but no refractions. The following table indicates the number of passes required for each type of splitting:

Splitting Type

Passes required

FG/BG Layers

3: foreground, background, shadows


2: key light, fill light

Light components

3: diffuse, specular, reflections

The way a scene gets split up will depend greatly on the scene. Imagine the background is incredibly complicated but stationary: splitting by FG/BG will be very beneficial, as only one frame of background needs to be rendered. If the specular is too bright at one point, but just right in the rest of the shot, splitting the shot up into light components allows the brightness of the specular to be keyframed easily.

It is important to point out that these ways of splitting the scene are not mutually exclusive: we can use more than one if we want to. In fact, we can use all three, by splitting each pass up into its components:

Foreground, key light, diffuse

Foreground, key light, specular

Foreground, key light, reflections

Foreground, fill light, diffuse

Foreground, fill light, specular

Foreground, fill light, reflections

Background, key light, diffuse

Background, key light, specular

Background, key light, reflections

Background, fill light, diffuse

Background, fill light, specular

Background, fill light, reflections

Background, shadow pass

This gives a list of 13 passes, which is a little over the top for a simple scene like this, but it does give a tremendous amount of flexibility in post production. Don't forget, as well, that the passes required are not set in stone: for example, the fill light might not be required to give specular highlights, so "Foreground, fill light, specular" and "Background, fill light, specular" can be removed. Likewise, the effect of the fill light in reflections might be negligible, so "Foreground, fill light, reflections" and "Background, fill light, reflections" can be removed. Thus we are left with:

Foreground, key light, diffuse

Foreground, key light, specular

Foreground, key light, reflections

Foreground, fill light, diffuse

Background, key light, diffuse (single frame)

Background, key light, specular (single frame)

Background, key light, reflections (single frame)

Background, fill light, diffuse (single frame)

Background, shadow pass

This is a much more respectable number: 5 full passes, and 4 single frame passes. The organization of all these passes is likely to be quite tricky, but it will maximize the flexibility available to the compositor.

Splitting By Foreground & Background

Load up the following scene file:

For this we're going to be using the layer editor: the section below the channel box. You'll see at the top there are a pair of radio buttons: previously we have been making display layers, but now we're going to make render layers, so switch over to Render.

[screenshot: MART_T2L09_html_m10b66d2d]The easiest way to set up render layers is to click the Create new layer and assign selected objects button. From the outliner select the Tarquin locator (the parent of the Tarquin hierarchy), and click the button. Confusingly, this creates two layers: layer1 and masterLayer. The masterLayer is simply created so that there is an easy way to render all objects.

Now select layer1 and rename it to TarqLayer by double clicking on the name. You'll see that this only shows the Tarquin object.

Now we'll create the background layer, but this time we'll start off with an empty layer by clicking the other button. Rename the layer to BkgLayer. If you hold the RMB on the layer name, you'll notice that there is an Add Selected Objects item on the menu: we could use this, but we're not going to. Go to the Layers menu and click on Membership (or, if you prefer, go to Window -> Relationship Editors -> Render Layers...), and use the editor that pops up in the same way as any other relationship editor: select a layer on the left (BkgLayer in our case), and the objects in that layer will be selected on the right for you to modify at will. Notice that lights should also be added to every layer you want them to appear in.

So we have our two layers, and if we look at each in turn in the Render View, it looks correct. We now need to change the render settings for BkgLayer so that it only renders one frame. There are three buttons next to each layer: the right-hand one of these opens a render settings window that is specific to that layer. Do this for BkgLayer now.

[screenshot: MART_T2L09_html_4f2d1f4a]Change Frame/Animation Ext to name.#.ext, and close the window (we'll come back to it later). Now open the equivalent window for TarqLayer, and see what it says: the value in this window has changed too. In this case that's good, we want both to have this setting, but why is this? The reason is that all of the settings are shared between windows, unless we specify otherwise. Get the settings for BkgLayer up again, and hold the RMB on the words End Frame: one of the options that comes up is Create Layer Override. Select this, and you should see the words go orange. This means that the attribute has a different value in this layer to the rest of the render layers. Change the value to 1, and then go and check the render settings of TarqLayer: they haven't changed. Notice also that the picture on the Render Layers button for BkgLayer has changed, to indicate that the render settings are no longer the same as all others.

Now we need to be able to render everything out. The first thing to do is make sure that when we ask it to do a batch render, it renders all render layers instead of just the current one. This is done by going to the options menu on the layer editor, and selecting Render All Layers. Now go to Render -> Batch Render. When it has finished, you should find that you can take the images into Shake and composite the character onto the background.


The one thing that is missing from our composite, though, is shadows. We have shadows on the background, but Tarquin does not cast a shadow as he walks along. Let's correct this.

[screenshot: MART_T2L09_html_m4c39e5bd]Create a new Render Layer in Maya that contains all the objects and lights, and name it Shadows. Hold the RMB on it, and go to Presets -> Shadow. Now render it out, and look at it. You should get an image with an alpha channel (Maya always renders shadow passes to the alpha channel) that looks something like that shown. This will work, and if you use it in Shake1 it will create a shadow of Tarquin on the background.

[screenshot: MART_T2L09_html_m6b1f783e]However, the image that we have is not perfect. If we look closely at the composited image, we have an error known as double shadowing: the shadow of the window sill is darker than the shadow of Tarquin. This happens because the shadow under the window sill is present in both the background pass and the shadow pass. To get round this, we have to render our individual layers without shadows.

[screenshot: MART_T2L09_html_m30d84fc5]Select BkgLayer, and click the middle of the three buttons (the one that looks like a flag). This brings up the layer's attributes. We want to override the layer so it neither casts nor receives shadows: turn on both of the relevant Override buttons, then turn off the Casts Shadows and Receive Shadows buttons (you only need to do one of these, but we'll do both to be sure).

Before we re-render, let's have another look at our Shadows layer. When we rendered it before, we got an image that was dark grey in the alpha channel. The ideal shadow pass should have the maximum range of colours, i.e. going from black to white.

The reason that our shadow pass is grey is due to the number of lights in the Shadows layer: we put all three lights in the layer, so Maya would make the pass completely white only if an area was in shadow from all three lights. As only one of our lights even casts shadows, that's never going to happen. So get the membership of the Shadows layer up, and remove the other two lights from it.

If we now re-render and re-composite, you should find that the problem is no longer there.


I am going to assume that, using the methods that we just used, you can figure out how to get one render layer set up for each light. The one thing I haven't shown you is how to composite together the passes from two separate lights, but that is the same as in the next section. Thus I am going to move swiftly on to...

Light Components

Open the following file:

Do a quick render. You should see a very hastily created pebble with a large specular highlight on it. We are going to separate this scene up into light components: we will make diffuse and specular passes, and then recombine them in Shake.

Maya makes this as simple as possible for us. Go to the render layer control, and go to Edit -> Select All. Then click on Create New Layer and Assign Selected Object twice, and rename one to Diffuse and one to Specular. Hold the RMB on Diffuse, and go to Presets -> Diffuse, and do the equivalent for Specular.

[screenshot: MART_T2L09_html_m7e58b919]You should find, if you try rendering both of these out, that you get two images that look something those shown here. We're going to try putting them together in Shake, but before we do, we'll do something else.

Make another Specular layer (call it Specular2 or something more original), and click on the Render Settings button (the right-hand one) on it. When you choose one of the presets, it will automatically set the renderer to be Maya Software. We don't want this, we want to use mental ray, so set it to that. Now set the image format to OpenEXR (exr), and then swap to the mental ray tab. Scroll down to the Framebuffer section, open up Primary Framebuffer, and hold the RMB on the words Data Type and Create Layer Override. Now change the Data Type to RGBA (Float) 4x32 Bit. Now, instead of using a value from 0-255 for each colour, Maya will use a floating point number: any number from a 1.4×10-45 to 3.4×1038. This gives us much more flexibility in Shake.

Now render it by going to Render -> Batch Render.

[screenshot: MART_T2L09_html_2a990956]Open Shake now, and bring in the three images as FileIn nodes (you might want to check the render settings or the script editor to see where it saved the images). The way we combine the diffuse and specular components of a light is to add them together: go to the Layer tab and get an IAdd node, and plug the diffuse into one side and one of the speculars into the other. Now get another IAdd node, and plug the diffuse and the other specular into it.

If you compare the two IAdd nodes, you'll find them to be pretty much the same at this stage. But let's try something else: double-click on each specular pass picture in turn, and the get a Brightness node from the Color tab. Turn the value on each of them down to about 0.4, and compare the two IAdd nodes again.

[screenshot: MART_T2L09_html_m2dfc0a77]This difference is caused by the fact that the highlight was so bright to begin with. An 8-bit image cannot cope with anything brighter than white, so it just makes it white. Darken white down, and it becomes grey. A floating point image, however, has the ability to store any colour (e.g. ten times as bright as white) so when it gets displayed on screen unaltered the two look the same, but when you try and reduce the brightness of them, the floating point image keeps all of the detail.

1There are many ways to use a shadow pass in Shake: the easiest is to plug a brightness node onto the bottom of your network, turn the brightness down a bit, and plug your shadow pass into the side of it.