Vray compositing a render

[Seraph135]

My result looks the same as the "Real RGB" channel that is in the frame buffer, however this is not the result I am after.

So my question is how do I comp my renders to get similar results to the "Real Rgb" channel?

I'm a little confused by the comments above...But I'm assuming this is a typo and you want the results to look like the max/vray framebuffer result...Or the "RGB color" pass.


The Real RGB is just a non-clipped version of the RGB color. If you render your passes in floating point then those jagged edges around overbright objects will not go away, and your result will look like the Real RGB pass. You have to put a soft glow or something on it to blur it and soften the edges. The RGB color pass clips overbright pixels and anti-aliases the result (assuming you have clamp output checked).

Assuming you don't have any reflections/refractions/specular...The only passes you need are the GI and Lighting passes (Not the Raw versions of them) which you add together. You don't need a shadow pass or diffuse pass (necessarily).

Tim J

[dark_tranquiltiy]

Hey Tim,

Thanks alot for the reply, You are right about the rgb colour pass, (I do get the same results if clamp out put is checked or not). I dont understand why I am getting the overbright area in my render. Is there a way to get rid of these overbright areas without affecting the rest of the image? Like the Real RGB does?

I tried the soft glow which does blur the jagged edges but I still have overbright areas that need attension.

Also "The only passes you need are the GI and Lighting passes (Not the Raw versions of them)" what do you mean by "not the raw versions of them"?

Thanks alot Tim, loved your dvd's by the way!

Jim.

[dprgb]

There's a good tutorial on the vray forums by Chris Nichols about comping a vray render:

http://www.chaosgroup.com/forums/vbulletin/showthread.php?t=32485&highlight=cpnichols

...and you do want to use the raw passes as these don't carry any of the diffuse color information in them. That way, if you need to correct the diffuse color of an object you can just change it in the diffuse pass.

Take a look at that tutorial, the basic equation to combine the vray channels is:

[ (Diffuse * raw GI) + (Diffuse * raw Light) ] + Specular + Reflection

The (unfortunately) easiest way to get rid of aliasing on multipass renders is to render at a higher resolution then scale down to delivery size as the last step. It's just the function of the way antialiasing works.

The other thing you might be missing is tonemapping... if you're multipass compositing floating point images there's no tonemapping on the final image, so you end up with overbrights.

[Seraph135]

...and you do want to use the raw passes as these don't carry any of the diffuse color information in them. That way, if you need to correct the diffuse color of an object you can just change it in the diffuse pass.

I'm not sure I agree with that exactly. Raw passes are necessary ONLY if you need to have specific control over the diffuse pass. The major downside to using Raw passes is that if you actually do the math in that equation above (which is correct) you will get either problems with pre-multiplied alphas causing halos around your mattes and/or an even more difficult problem to fix around the edges of reflective/refractive objects. You will not be able to put those passes back together and get an "exact" match with the Max/Vray frame buffer (unless you render without AA, Render larger without AA and resize down, or do some interesting math to some of the passes which I'm currently creating a tutorial for). Vlado points out this problem as well at the end of that thread. It may be close enough for your needs though. Just don't expect an exact match to the max/vray frame buffer.

...(I do get the same results if clamp out put is checked or not). I dont understand why I am getting the overbright area in my render. Is there a way to get rid of these overbright areas without affecting the rest of the image? Like the Real RGB does?

There are a few too many unknowns still to solve your problem. Are you rendering to Vrimg and converting to EXR? Are your passes clipped or are they float? Tone mapping?.

Tim J

[dark_tranquiltiy]

Hey thanks alot for that link, it looks like it will be very useful for me, I will take a good look at it when I get a chance.

Tim, I am rendering a still image and saving out each channel as a exr format (so I presume I am not converting to a exr format). The exr options are Float bits 32 per channel, under the vray gbuffer/colour mapping tab colour mapping type is exponetial and clamp output is turned off.

Im sorry I am not sure what you mean by tone mapping, is that a format saved in the image itself?

Thanks alot for the replies guys!

Jim.

[Seraph135]

Tim, I am rendering a still image and saving out each channel as a exr format (so I presume I am not converting to a exr format). The exr options are Float bits 32 per channel, under the vray gbuffer/colour mapping tab colour mapping type is exponetial and clamp output is turned off.

Your workflow could be part of the problem. I haven't tried doing it that way, but I'm wondering if your passes are actually being clipped. As an experiment take a pass that you know has values that should be above 1.0 and see if they actually are. Even though your saving out 32 bit exr's they may not have any information above 1.0.

The color mapping is the same thing as tone mapping. Change it back to the defaults (linear) and make sure all the values for it are at their default. Tone mapping should be done after the composite is finished.

Tim J

[dark_tranquiltiy]

Ok I looked over Chris’s tutorial and am wondering how to comp this in Fusion (With the Raw passes).

http://www.chaosgroup.com/forums/vbulletin/showthread.php?t=32485&highlight=cpnichols&page=18


Here is a screen shot of my flow but I am not getting the right results. I must be doing something wrong

http://img81.imageshack.us/img81/9906/myfusionflowku4.jpg



It seams the over bright areas happen with linear multiply as opposed to exponential.



Your workflow could be part of the problem. I haven't tried doing it that way, but I'm wondering if your passes are actually being clipped. As an experiment take a pass that you know has values that should be above 1.0 and see if they actually are. Even though your saving out 32 bit exr's they may not have any information above 1.0.

Which pass would have values over 1.0 the raw lighting? How can I see if they actually are above 1.0?

Thanks Tim.

[Seraph135]

Take a look at the first page of the tutorial by Chris. It shows his node set up in Nuke, but that exact same setup can be followed in fusion.

Just about any pass "could" have values over 1.0. The Raw lighting pass would probably be a good one to check first. You could always turn the multipliers up on your lights and blow the scene out. Then open your passes in fusion and check them. That way you can be certain if your clipping or not.

The easiest way to check in fusion would be to create a background node and use the color picker to select really bright areas of your pass. If any of the RGB channel values go above 1.0 then your fine. If none go over 1.0 then you will likely have problems compositing the passes.

Tim J

[dark_tranquiltiy]

Ok, that was my problem I didn’t use linear colour space. I comped the render in Fusion using Chris work flow and got these results:

fusion comp

http://img215.imageshack.us/img215/7546/fusioncomposition0000ga3.jpg

max render

http://img89.imageshack.us/img89/3254/rgbcolourframebuffer000us9.jpg




As you can see the renders aren’t identical as you said “Just don't expect an exact match to the max/vray frame buffer.”



dapeter2 (http://forums.cgsociety.org/member.php?u=65115) suggested “The other thing you might be missing is tonemapping” could this be the reason my renders aren’t the same? If so how can I enable the tone mapping?



Thanks again Tim!



Jim.

[Seraph135]

There is still a problem in your workflow somewhere. The difference I was talking about is not as extreme as the difference between your comp and max render.

There are three possible problems here.

1. Your passes are not actually floating point
2. Your tone mapping your vray passes before compositing.
3. Your compositing math is incorrect.

You need to make sure all three are correct or you won't get a match. Based on what I see I'm guessing your problem is #2. But so far I don't think we can rule out 1 and 3 either. Make sure your actually using the default values for color mapping.

Tim J

[dark_tranquiltiy]

In fusion I am gettting values higher than 1.0 in the RGB channels in overbright areas. So they must be float images?

My tone mapping (Colour mapping) is set to linear multiply, dark multiplyer 1.0, bright multiplyer 1.0 (I assume are the default settings) like when you first switch to the vray render.

I am using the open exr plugin (ver 1.01) and am using the following settings.

compression none
Format: float 32 bit
RGBA ticked
Use real pixels ticked
Pre multipied alpha ticked.

My Fusion flow is as follows,

backGround-Mrg-RawGi-Mrg-diffuse(multiply)-Mrg-RawLighting(Gain 0)-Mrg-diffuse(multiply)

Does this flow look correct to you?

Thanks

Jim.

[Seraph135]

In fusion I am gettting values higher than 1.0 in the RGB channels in overbright areas. So they must be float images?

Cool...You can rule that out as a problem.

My tone mapping (Colour mapping) is set to linear multiply, dark multiplyer 1.0, bright multiplyer 1.0 (I assume are the default settings) like when you first switch to the vray render.

That also looks correct. Gamma is also 1.0?


My Fusion flow is as follows,

backGround-Mrg-RawGi-Mrg-diffuse(multiply)-Mrg-RawLighting(Gain 0)-Mrg-diffuse(multiply)

Does this flow look correct to you?

I'm confused by the way you've written out your flow description. It should be like this..

(Diffuse * RawGI) + (Diffuse * RawLighting) + Refraction + Reflection + Specular

Can you post a screen capture of your fusion flow and mark the merges with what blend mode your using on them?

Tim J

[dark_tranquiltiy]

Hi Tim,

I did have the comp in the wrong order. I changed it around and got these results.

http://img329.imageshack.us/img329/8679/fusionflowmh3.jpg

http://img444.imageshack.us/img444/5065/maxrendermq4.jpg

for some reason the raw lighting pass is more intense and not yellow and has lots of noise in it, do you know why this could be?

Also you said "Gamma is also 1.0?" I am not sure what you mean by this could you elabirate?

thanks again bud.

Jim

[Seraph135]

This looks very much like your comp is still not set up correctly.

Can you please post a screen capture of your fusion flow and mark the merges with what blend mode your using on them?

Tim J

[dark_tranquiltiy]

Hi Tim,

Sorry I didnt post the screenshot before, here is a screengrab of my flow:

http://img291.imageshack.us/img291/8500/fusionflowll5.jpg

I have also tried the diffuse passes before the lighting passes e.g diffuse*gi + diffuse*lighting.

Thanks

Jim.

[Seraph135]

Ok...Its your comp that is the problem. The math in your comp is not correct. I'm going to explain the issue using a normal math equation. Hopefully it will make more sense that way.

Each number will represet one of your passes. This first equation is how you have your passes set up.

2*5+8*5 = 90

The correct set up looks like this...

(2*5) + (8*5) =

or to simplify it.

10 + 40 = 50

All the math in parenthasis is done first.

Here is what your fusion flow should look like.

http://www.seraph3d.com/FlowExample.jpg

Tim J

[dark_tranquiltiy]

http://img390.imageshack.us/img390/5485/fusionflow0000ne2.jpg (http://img390.imageshack.us/img390/5485/fusionflow0000ne2.jpg)



http://img390.imageshack.us/img390/6633/maxrender0000ti2.jpg



Ok that looks better, now here are my results (without specular or reflection).



I notice the white halos you where talking about. My options to get rid of them are to:



Render a higher resolution and resize (what size would you recommend for a 720x576 render double res?)



Also you said in the vray forums



“Depending on the situation, you can also un-premultiply your diffuse and raw passes. Multiply there RGB channels together, then pre-multiply the result to get rid of the halos.”



Would this work in my situation I see that the halos only appear against the background. It sounds really complicated could you please explain or screengrab what you are saying?



Thanks once again Tim, You are my savior.



Jim

[Seraph135]

Would this work in my situation I see that the halos only appear against the background. It sounds really complicated could you please explain or screengrab what you are saying?

Almost there. :)

Yes this is a situation that can be fixed by un-premultiplying. Here is a screen grab that shows how I like to set this up. In this example I'm using channel booleans to multiply the passes together instead of merges. You can get the same results using a merge node instead, but you need to feed the merges solid alpha channels and uncheck affect alpha in the merge nodes or else you'll get incorrect result. With channel booleans you don't have to worry about that. You can just multiply the RGB channels and leave the alpha alone.

http://www.seraph3d.com/Un-Premult_example.jpg

Tim J

[dark_tranquiltiy]

Hey Tim,

Thanks alot for your effots! Really appreciate it, I will give that a go, one last question though, I have been playing around in fusion with my passes to try and get rid of the overbright areas in the render (using brightness contast, ccing the passes). How would you go about this without affecting the rest of the image (trying to get similar results to what the rgb colout pass in vray. Because some of the areas are really really bright (even without spec reflect, e.t.c.

[dark_tranquiltiy]

Your method got rid of the white halos, thanks alot. Still trying to find the best solution to get rid of the overbrights though.

[Seraph135]

I have been playing around in fusion with my passes to try and get rid of the overbright areas in the render (using brightness contast, ccing the passes). How would you go about this without affecting the rest of the image (trying to get similar results to what the rgb colout pass in vray. Because some of the areas are really really bright (even without spec reflect, e.t.c.

If you have your flow set up exactly like in my last post, then you should get identical results to the max frame buffer. If its not the same then somewhere along the line, one or more of your passes was not rendered correct. It looks to me like its your raw light pass, but I don't know that for sure.

However, if you have floating point passes and it appears that your raw light pass is just too bright, then you should be able to put any tool that has a "gain" slider after your channel boolean (the dividing one) and lower the gain. That should remove the overbright areas and get you closer to the output from max. Something is still wrong with your original set up, but this should get you closer.

Tim J

[dark_tranquiltiy]

but the overbright areas only appear when using linear colour space even the max render has overbright areas. This is something you explained in one of the first posts (when using exponential it clamps and anti ailising the overbright areas) I assumed this is what you meant.

Chris also has these overbright areas in comp.

[Seraph135]

but the overbright areas only appear when using linear colour space even the max render has overbright areas. This is something you explained in one of the first posts (when using exponential it clamps and anti ailising the overbright areas) I assumed this is what you meant.

Maybe I'm not understanding what your looking for exactly. Overbrights can appear in the max frame buffer or the fusion comp no matter what color space you use. Using a gamma color space is only affecting the visible range of colors between 0 and 1. Also, the values that are exactly 0 or 1 are not being affected, only the values inbetween. Your passes should always be in floating point if your going to composite your passes this way because some passes will very likely need values over 1 in order for the math to all work out to be the correct final image.

I read through all my previous posts and I never said that using exponential will clamp your image. Only the clamp output checkbox, or saving your file to a non-floating point format will clamp your image. I did see that dapeter2 said....

"if you're multipass compositing floating point images there's no tonemapping on the final image, so you end up with overbrights."

I think what he means is that if your multipass compositing there should be no tone mapping on your image, and its very likely you will end up with overbrights.

Tim J

[dark_tranquiltiy]

Ok, here are some results I have come up with:

http://img356.imageshack.us/img356/1827/fusioncompunpremultiplycx3.jpg

http://img230.imageshack.us/img230/2156/maxlineardw5.jpg

and the results with colour mapping set to exponential:

http://img230.imageshack.us/img230/5338/maxexponentialoy5.jpg


For the exponential render (straight out of max) I changed the colour mapping parameter from linear (which I use the float passes) to exponential (that is what I meant by the over bright areas going away in the previous post).



Anyway, so there is still a difference between the max render (linear) and the fusion comp (It does look like the raw lighting pass is brighter). I have double checked my flow with the example you gave (unpremultiply example) and it is the same.



When you say “add” in your example of merge nodes this means turn the alpha gain to zero?



Is there anything I can do to change the way vray renders the raw lighting pass?



I have played around with colour corrects and turned the gain down (after the channel booleans) and I can get the results very similar to the max renders if I try to much to get rid of the over brights like the worst in the image I start to affect the overall lighting to much.



I suppose I could always do a separate colour correct and mask the bad over brights.

Thanks Tim,

[Seraph135]

For the exponential render (straight out of max) I changed the colour mapping parameter from linear (which I use the float passes) to exponential (that is what I meant by the over bright areas going away in the previous post).

Yep. I see what your doing. This sort of thing is best handled in fusion if you plan on compositing your passes.

When you say “add” in your example of merge nodes this means turn the alpha gain to zero?

Yes...and make sure that the slider about the gain (Subtractive/Additive) is all the way to the additive side.

Is there anything I can do to change the way vray renders the raw lighting pass?

Yes, but you don't want to do that. The default settings for it are correct. If you wanted to change it you could select the element itself from your list of elements and a rollout should appear that gives you a few options.

I have played around with colour corrects and turned the gain down (after the channel booleans) and I can get the results very similar to the max renders if I try to much to get rid of the over brights like the worst in the image I start to affect the overall lighting to much. I suppose I could always do a separate colour correct and mask the bad over brights.

I'm pretty sure you can get the exact same tone mapping in fusion with a floating point comp that you could get directly rendering in max with tone mapping. I like to use curves to do that. Tone mapping should be done in the comp.

Tim J

[dark_tranquiltiy]

Thanks for you so much for your help! I am now getting the right results, one last qustion if you dont mind (I am soo close to finishing).

http://rapidshare.com/files/106620413/Fog_Issue.rar.html (http://rapidshare.com/files/106620413/Fog_Issue.rar.html)


I am having a problem when compositing my fog layer on top of my scene, basically I have followed the steps in your Environment Lighting For Production Dvd.



I have the same results as you have in terms of alpha information with the luma keyer. Yet my results are very different from yours. Shouldn’t the far crane on the left of the image be completely covered in fog like the max render for suggests?



Note: the alpha images have been screen grabbed and cropped from fusion that’s why they have a weird size



Do you have any ideas on what could be causing this to happen?



You’re the best Tim!



Jim.

[Seraph135]

Nope...what I see here looks exactly right.

Luma keying takes the RGB intensity, makes it greyscale and copies it to the alpha channel. The brightest value in your fog pass is about 50% grey (with a little extra in the blue channel). So when you key it your alpha channel is only 50% grey and so only half of your fog is going on top of the final comp.

I'd recommend you render a black and white fog pass (no color) and apply levels to it if needed so that the far away areas are white and the close areas are black. Luma key that. Then you can use a color gain node to tint the color of the fog before merging it over the scene. Adjust the gain of each channel with the color gain tool.

Tim J

[juand731]

can someone explain how to do the post production with photoshop, how do i set up my stack of different render elements?

[Aritz]

Almost there. :)

Yes this is a situation that can be fixed by un-premultiplying. Here is a screen grab that shows how I like to set this up. In this example I'm using channel booleans to multiply the passes together instead of merges. You can get the same results using a merge node instead, but you need to feed the merges solid alpha channels and uncheck affect alpha in the merge nodes or else you'll get incorrect result. With channel booleans you don't have to worry about that. You can just multiply the RGB channels and leave the alpha alone.

http://www.seraph3d.com/Un-Premult_example.jpg

Tim J

Any idea to how reproduce that in AE ? :rolleyes:

[hugoguerra]

Hi all, i have read this forum with great detail and many thanks for all this precious info.

I though I could share my VRay setup also.

For all of you wanting to comp VRay in Nuke here is some of my tips and gizmos to make it happen

(P.S: I am currently finishing a Raw version of my gizmo, so for now these gizmos only comp the normal passes from VRay)

The "3dsMax Tools" Gizmos:

http://www.hugo-guerra.com/hugo-guerra.com/Nuke_gizmos.html

The Test Image:

http://www.hugo-guerra.com/Gizmos/Test_Images.zip

How to use my Gizmo:

http://www.hugo-guerra.com/Tutorials/Tutorial_003_3dsMaxTools.mov

Hope it helps, any issues just send me a message or email.

Thanks

[AroundTheFur]

Hey guys,

I'm very happy I found this thread because I've been following Tim's DVD and I can't for the life of me get this working. Even using the sample Fusion file's flow, my results are not coming out correct at all. I assume this means I'm doing something wrong out of max.

Following your exact workflow, I am rendering a simple scene out with the V-ray frame buffer kicking our an EXR with all the passes I need. My color mapping is set to the default Linear Multiply with no clamping or sub-pixel mapping.

When I go to comp this in Fusion, the diffuseFilter and rawGI (or GI/diffuseFilter) calculation is giving me a much darker result than I'd expect, which is leading to a much much darker image in the areas without reflection. This makes sense to me because looking at the diffuseFilter, the values are much darker than the GI lighting values, so multiplying them together is going to give me values that are too dark.

Here is what I'm talking about (control on the left, flow on the right):

Fusion Screenshot (http://img128.imageshack.us/img128/6348/fusionre9.jpg)

and I've posted my max file (2009 vray 1.5 sp2) and the exr file with the render passes in it:

Max 2009 File (http://rapidshare.com/files/158483339/compTest.rar.html)

Any help would be GREATLY appreciated and great DVD Tim! Only have to figure out how to do this myself :)

-Steve

PS: You'll also notice the overbrights that were mentioned earlier. Is there no way to get rid of those using this workflow?

[hugoguerra]

Hi, try my Nuke comp below, first comp is a regular Raw composite, and it is not working because of some strange Raw GI pass, second comp works fine, and 3rd comp is basically using my gizmo but in my gizmo I only use the regular passes, no Raw ones (I am still finishing the update to have Raw also)

You know Vray could be trick, aldo Tim DVD is amazing, real life situations in Vray are far from perfect.

For example: regarding the over bright GI passes, I have made a gizmo to fix that, it basically minus all the passes from the RGB to get a proper GI, the reason for a over bright GI could be several, I found that if you use standard materials in VRay you get this problem, but other things may be causing it.

Regarding your EXR, the only thing i see that looks strange is it's alpha, since it is the wrong alpha, since it should be the windows and not a dark version of the RGB, also you need the VRayBackground pass to complete the comp.

I have found that problems in passes with VRay could come from using standard materials, regular lights, regular cams, sun systems, etc, and some times it could come without explanation, so I just try my best to adapt and move forward, since it will never be perfect. :cool:

[Seraph135]

Hi, try my Nuke comp below, first comp is a regular Raw composite, and it is not working because of some strange Raw GI pass, second comp works fine, and 3rd comp is basically using my gizmo but in my gizmo I only use the regular passes, no Raw ones (I am still finishing the update to have Raw also)

I'm thinking the problem is the diffuse pass and not the Raw GI pass. If the GI and Light pass add together properly then its the diffuse pass that is wrong. If you take the RawGI and the GI pass and divide them you should end up with the correct diffuse pass which looks nothing like the one saved in the exr.

Did you happen to change any settings in the render elements themselves before they got rendered out of max Steve? I'd look but I don't have access to Max2009. I would definitely check all the things Hugo is saying could be the problem.

Tim J

[AroundTheFur]

Hey guys,

First, thanks for the quick reply. I am gonna pull the demo of nuke to try and open your file to make some sense of what you are saying Hugo. Tim, the only thing I have checked is the color mapping parameter in the render elements option rollout for each render element. Even though my color mapping settings are set to default, could this be the problem?

I'll try to render it without that parameter checked and I'll post my results.

Thanks again guys,
-Steve

[AroundTheFur]

OK good news!

It seems that while i was messing with the color mapping settings, trying to get my passes to respect my color mapping, I checked some boxes that shouldn't have been checked. When I removed all the passes and added them again, and left them at defaults everything seemed to work perfectly. Hugo I tried to look at your file but it seemed as though some connections were broken, not really sure though.

If anyone has any idea how to get rid of those nasty overbrights and their dark halo wrappers I'm all ears. Seems like different anti-aliasing filters produce different results. The filters with edge sharpening exaggerate the dark halo.

Thanks guys, much appreciated.
-Steve

[hugoguerra]

Hi, apparently PLE version of Nuke dosent really read my file, I guess it could be a bug or a limitation because of the gizmos. I will call the foundry and ask them and get back to you.

Great that it work for you.

[hugoguerra]

So now it should work with PLE version, I had some stuff in the comp that was not possible to have in the PLE version, sorry about that, I never used it before.

[AroundTheFur]

Hey Hugo,

I'd say that's pretty impressive if you were able to match the control render with the borked color mapped passes. I will take a look today when I get into work.

-Steve

[AroundTheFur]

So I took a look at your file Hugo, and this appears to be the same flow I have in Fusion. The only thing is, My flow seems to be clamping all of the values over 1.0, whereas your Nuke file does not. Any idea why that would be happening?

Fusion file + updated EXR (http://rapidshare.com/files/158664787/compTest_02.rar.html)

-Steve

Edit: I found the problem. Thanks guys!

[CGTalk Moderation]

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.