Normal / AO Baking Way Too Fuzzy


It looks like The Max AO uses a much shorter tracing distance and maybe a narrower angle.
The cage/baking angles might play a role, too.


Hi @Noren,

thanks, well, the difference between the intensities comes from varying AO settings in 3DS and XNormal.

But the AO from 3DS isnt calcualated correctly at all for this particular object (LP_1 and HP_1), some small faces (at UV seams, if that corresponds to sth) are not calculated at all (the horizontal ones, as noted on the Unity screenshots).

The AO from XNormal ist calculated correctly.

Cage might play a role in this setting as I dont know wether XNormal makes use of the projection modifier from 3DS (exported with the FBX) or if it sets up its own cage or other projection /calculation method.

I tried many different things to narrow down the reason for this issue but I#m out of ideas now… :roll_eyes:

Maybe I can export a FBX without projection setups to XNormal to see if it makes a difference - I GUESS XNormal does its own calculation for that…


No idea what’s happening. If you upload a file, I or someone else can have a look over the weekend.


That would be great!

I did already attach the FBX exports but you’re right, I’ll attach a sample 3DS file as this is where the error occurs, not in XNormal.

Thanks a lot - and dont spend too much of your weekend time please :wink:


Sample File_Missing AO on some little horizontal faces of object 1_LP_001.max (828 KB)

EDIT Seems like the XNormal calculation makes no difference if the Projection modifier is included in the FBX export or not - and so doesnt the cage. There is no cage being exported, XNormal does its own calculation.

That would speak for a cage related issue in Max. Actually the cage is a bit problematic in the case of object LP_1_001 as the little faces (horizontal and vertical) get a tilted cage face angle and depending on the distance a weird (but understandable) overlap. That might be it.

Can you confirm that? Any idea how to avoid / solve that?


There’s no "AO settings in 3DS (3ds Max) "
Every render engine/plugin have it’s own settings (and naming) for that effect.

And, if If I understood you correct, you are baking AO with Vray engine and there is no Vray RTT element for that!?
And you’re testing/switching from Vray to Scanline and Scanl. can’t render Vray dirt map…

Yours proc. is ok, I’m running max with not much faster (and was much cheaper AMD proc.) just fine.

Some guidelines:
1) There are render presets, make some and use them

There are non - shading render elements/geometry rend. el. (normals, Z-depth…)
No need to “torture” renderer with ilum., lights, and/or other things where not needed -make 1) and use it

Use suitable renderer for different elements -make 1) and use it
You already got some answers what is suitable for what here - read your post again


Hi @domos,

thanks, maybe there was some misunderstanding or some of my information misleading… :wink:

Sure there are no 3DS AO settings, I use VRay and once the correct render settings are found Ill save it as a preset for RTT for sure. Right now the VRay settings are saved along with the 3DS file.

I did use the right RTT render elements for each render engine approach. To render AO with Scanline you’d have to set up the scene with some skylight / or LightTracer plugin, otherwise youd get no AO.

VRay wise its a little different, there is the Vray Normal element and AO is being rendered through a VrayDirt map inside a self illumination slot of a Vray global override material.

As far as I know Vray IS ONLY using the same Vray raytraycing method for Normals as well - which can cause more noise than in other calculation methods (like XNormal) and longer rendertimes.

Actually my problem with noisy Normal maps is solved, there was a render setting (sampling) I was missing in that case.

So besides rendertimes Im pretty much where I wanted to be in setting up the pipeline - but for some reason there are those AO errors (missing on faces) as described above.


The cage might play a small role (don’t use the initial cage automatically set up by 3ds max, that’s almost always a mess. Reset it and apply a small push. If you want to automate it, you need to build your LP accordingly, so a small push of the cage gets the desired result. Here e.g. the inner panel of the LP is way below the panel of the HP.

The bigger factor seems to be scale, though. You’ve set your scene up in meters, which means that small details like the ridges in question will get a lot of decimals. Centimeters would be more appropriate. In theory this shouldn’t affect AO at all (using appropriate AO distances), but maybe there are some optimizations in Vray under the hood. More likely it’s a bug of Vrays RTT integration, though, as there’s no problem when rendering normally.
When I scaled the meshes and AO distance up 100 times, the ridges came out fine.

Avoid using GI if you don’t need it, you can render directly to a Vray selfillumination RTT element.


Scanline can render AO just fine. Just use a skylight and render out a shadow map, you don’t need to enable light tracer. Make sure to disable cast shadows on the low poly source object. This method produces excellent results. The only drawbacks are that it’s slow-ish, and it doesn’t work for interior scenes.

Set shadow rays in the skylight to 3-5 for testing, and 20 for final quality results.


You’re right, Skylight does indeed work without the lighttracer. Wasn’t aware of that.


Can You explain that?

What if he pay someone to take all his RTT furniture outside and bring it back after rendering :smiley:



thanks to y’all for your quick replies!

@Noren: Yes, that will probably be the key! Stupid me I did not consider scale / units issues. ^^ Absolutely possible, the real world object’s size in this particular region is only about 3 mm. I’ll try setting up different scene units and work my way from there. Maybe I’ll have scale things up before render, the render results will be proportional anyways! :+1:

Yes, I did change the cage values to slightly cover the objects so they dont overlap too much. The basic autocage is not great but does work in many cases alright.

@Noren and @Senorpablo: I thought you could render Scanline AO with skylight OR the LightTracer (without skylight) plugin?! But as I’m not too deep in that renderer I might be mistaken. If that wouldnt work for interiors (as wall however would probably block calculation rays) wouldnt it be a problem if there are overlapping parts / overhangs too? Just for interest :wink:

@domos: Pffffff… :roll_eyes: :rofl:

Thanks guys!

Have a nice weekend!


Yes, you got that more or less right. I wrongly thought that Scanline doesn’t render skylights without the lighttracer and suggested to use a manual set up light dome instead (lights arranged in a dome shining down at the object that’s to be lit) but using a skylight is obviously easier (the lighttracer seems to be a bit faster with the same quality). And while I’m not sure what’s going on under the hood, at least in theory AO is calculated from the inside out and the tracing distance can be limited, while the skylight shines from the outside in, so overhangs/overlaps can indeed be a problem. An unlimited AO tracing distance should behave pretty much like a skylight.


I looked your file, there’s too many mistakes before projecting/ rendering process so you will not find problem nor solution by experimenting that way.

Here’s Occlusion map from Render Surface Map tools
Need unwrapped editable poly, 1 click on button and 1 sec. to render and with other maps from tools can get different results (composite mapping or whatever)

Lowpoly mesh - L to R:
Normal - Normal + Oc - Occlusion

Normal map from RTT Scanline without any supersampling.
Clean and smooth highres mesh projected on 1x1x1 box with “hand” packed UVs and (too big for this) 1024 map so no need for aditional antialiasing.


@Noren: Seems like the system / scene units settings were the culprit. AO is calculated correctly now, I did not even have to scale up the objects. Scene is set to mm / mm now, works! Thanks a lot! Should have thought of that myself - but thats how it is sometimes :slight_smile:

Actually the cage does work fine so far, we’ll check out some better values, eg. as you mentioned just a slight push. LP objects should be just the same size of the HP versions (dont know why or if the inner panel of the LP version is lower than the HP, shouldnt be like that, I’ll have to check that). In this case here the automatic cage did work fine, I just adjusted the flatten angle to 89° instead of 45°.

Cool, getting somewhere :wink:

@Domos: Thanks for your effort on the weekend as well! What do you mean by there are too many mistakes in that file? Objects are not modelled in 3DS for specific reasons but they should follow all neccessary conventions for a successful batch.

Your results look good for sure, but seems like I have to follow a different approch though. I have 3DS 2016 installed, render surface map tools seems to be a feature introduced in 2017 version. Also the proposed objects are just an example, I will have to process a whole bunch of objects later, so hand-unwrapping those objects just wouldnt work. The LP objects still need to remain as depending on the later platform these objects might even be used without Normal maps. Therefor a 1x1x1 box would not work as well. The texture size is set by the later batch process and intended use too, these texture will be downsized if neccessary :wink:


2017? So you have concluded that just by taking look at first result
Bet you didn’t even bother to open it

BTW ther’s universal shortcut F1 in Max, I’can’t force you to use it
So looks like no point to write you about something that you won’t read


Not the best bet then.

I did actually read about render surface map tools, did not come across using it before. Checked the version requirements AFTER reading it and maybe Max help is a bit misleading at this point. Seems like its still not the right tool for me in this case, I will keep it in mind for later though.

I still would be interested to know the many mistakes I have done one my models. Maybe Ive overlooked sth, maybe its a way to go.

Anyways, I appreciate your effort, but please dont get personal.