Normal / AO Baking Way Too Fuzzy

yellowpinkfox · 2019-01-24 16:25:19 UTC

Hi guys,

I seriously dont know why this is happening…

I’m baking a HP object onto a LP target object using the standard render to texture tool. All set up correctly I assume, I’ve set up several presets render and bake settings wise. Normally I use Vray and maybe would like to stick to it in this case too - and I’d expect Vray to do that job properly anyways…!

Basically I set up the render to texture tool for a LP object, use the HP object as projection object, the cage is set up correctly and automatic unwrap / flatten works fine too.

For some reason the render turns out totally fuzzy, all edges are super pixelated, texture is useluess like this.

I did some tests with Xnormal, render came out perfectly clean in no time! (Actually some strange things happening there now too, normal is just a plain color - where it comes at least properly layed out in VRay…)

Long story short: I would assume a normal map should be quickly rendered and perfect, as there is no bouncing and interpolation of rays at all anyways. Switching to Scanline did show the same strange flat color results as XNormal - but last time it did work properly, BUT as well having the fuzzy pixelated edge artifacts.

Please see attached render, HOPEFULLY some of you can shed some light on this.

THANKS in advance
Niko

darthviper107 · 2019-01-24 16:28:14 UTC

For baking normal maps I use the Default Scanline Renderer, it’s faster and you can get sharper results my setting the supersampling to Hammersley

yellowpinkfox · 2019-01-24 16:55:11 UTC

Hey, thanks for that super quick reply!

Tried setting up the whole scene again from scratch, used Scanline and did set the sampling values. Strange enough I get the same results as from XNormal - just a plain color output. Super weird…

Probably its me who made a mistake, just back to the topic after some weeks off. Still cant find anything wrong here. Maybe a guess about this too? There HAS to be something Im missing !

Normals%20problem%20flat%20color%20only|690x329

I’ll be gone now for today, getting back to the problem tomorrow, have a nice evening!

N

darthviper107 · 2019-01-24 17:19:41 UTC

If it’s the solid blue color (127,127,255) then you didn’t set up the projection mapping so the normals are coming out the exact same as the surface. The high poly and low poly objects need to be in the same location and you need to go into the Projection Mapping section of the low poly object and enable it and select the high poly object to project to. Also, I usually reset the cage and then push it out a little bit, the auto cage that it creates can be a mess.

Also, when you’re switching from Vray to scanline to render the maps you need to make sure your map type is changed from the Vray types to the default types.

Noren · 2019-01-25 11:36:40 UTC

Are your objects mostly identical with coplanar faces? If so, try to give one of them a tiny Push so that that’s no longer the case. (Can cause pixelated artefacts)
Did you deactivate GI for Vray manually and set appropriate sampling values/time?

yellowpinkfox · 2019-01-25 16:03:31 UTC

Hey guys,

Thanks for your effort!

Had a lot of office communcation to do today but focused on the problem now again. Please see attached screenshot - I assume everything should be set up properly for baking. And I do get proper normal results through Vray, but still in a bad quality. Which I seriously dont understand… In my eyes a normal map should be a data-only texture, there should be no raytracing involved. Which makes totally sense as GI is turned off globally and it still renders the (fuzzy) output.

The screenshot is set up for Vray this time, if I switch back to Scanline all object settings should remain the same for sure, should be nothing wrong with those. There are still remains of Vray elements I could choose from in the render elements drop down, I’m just ignoring those. If I choose the standard normal map element type it still renders this plain color. Which is even more strange as the same result comes from Xnormal.

I did some baking tests some weeks ago and Xnormal did deliver 100% proper results. Still it doesnt really fit in the actual pipeline as I need to bake a load of elements, which ideally will be supported by a custom script inside 3DSMax. There is a discontinued script to batch bake objects with 3DS and Xnormals, but this isnt properly working and not the preferred way to go.

Ideally I try to stay within 3DS only to fulfil that task…

The strange thing about Xnormals and Scanline deliviering the same wrong (plain color only) results is, that for Xnormals the object’s setups need to be wrong already, somehow.

But to be honest, there IS not that much that CAN be done wrong, right?? :)

I’m already guessing that there is an evil bug somewhere, something I just CANT find… ^^

To adress your answers directly:

@darthviper107: HP and LP objects share the same space, the LP object being a totally reduced version of the HP object, all proper quad face modeled. Projection and cage is being set up by render to texture tool, tried changing the cage size, didnt help. Should work though anyways as I’m bulding a pipeline which should work semi-automatic with presets. Still, right way to find the errors - cage doesnt seem to be an issue. I’m using automatic unwrap / flatten option as well, had some good test results with it before so far. (Actually the objects will be pretty simple, so no need for perfect unwrapping or ideal UV placement, does the job ) Sure when I switched back and forth between Scanline / Vray I did set up the correct elements for each type…

@Noren: Actually yes, at least in this case - but in many future cases too. Its going to be a higher detailed version of specific furniture elements being projected on simple boxes of the same size. We need to be sure that both versions are exactly the same size for the later use. Still interesting to know - I DID have some artifacts problems but kind of managed to solve them out. GI wasnt deactivated this time, but had it deactivated before while trying to solve things - didnt make any difference at all as well. Question sideways: why should it make a difference, except speed improvements? It might happen that you would want to bake several element types at once, some need raytracing, some are data textures and dont. Still should work together, right?

Well, overall, (sorry for that long text, proves my confusion ) its still SO strange that is does work with Vray and doesnt in other ways, and Vray giving this crappy results.

Any other thoughts?

I will get back to the drawing board and start again from scratch, still I dont know what I could have done wrong…

Best,
Niko

yellowpinkfox · 2019-01-25 16:31:17 UTC

Guys, something (for me) totally strange happend here.

Maybe there is a reason for this, probably, maybe you know:

I just played around with the VrayDirt options to generate a better AO setup for these flat objects (furniture doors).
Normally there is always a VrayDirt texture material plugged into the global material override slot (set to VrayComplete element type) to generate AO along Normals (ideally later). For sure it does slow down the render, still it will be needed later anyways.

What I did now, which apparently “solved” the fuzzy edges error: I just did checked the VrayDirt options “invert normal”, only as a test to improve the intensity of AO on the surfaces. Normally I wouldnt want that option to be on.

Strange enough - the normal map rendered out is perfect!

Not yet THAT ideal solution as the rendertime went way too high, but that something I could investigate further - sure a VRay AO render (with VrayDirt) is pretty time consuming. Maybe Scanline / Mental Ray AO would be quicker.

Anyways, could you imagine WHY this is happening??

N

Front_XY_LP_001_AO

Normal: OK
AO: Weird, something to be tweaked

Still strange.

Noren · 2019-01-25 17:02:39 UTC

Can you upload an example file? Which version of Vray are you using?

Have you tried doing a reset xform and checked if both or one or parts of your objects is/are inverted to begin with (e. g. if it was mirrored at one point)?

I don’t think the normals are rendered separately, I’d guess a complete render is done and the single elements extracted from that.

Vray uses sampling either way, GI or not, so if your settings are too low, it might lead to slightly noisy results. Maybe Dirt forces more samples by triggering the noise threshold.
Or it’s something about the material/map/blending of materials itself.

The Push modifier was meant only temporally, for the baking process. If there are coplanar faces, you’ll probably need it unless they fixed that bug.

darthviper107 · 2019-01-25 20:25:58 UTC

It might have to do with sampling for anti-aliasing. Vray isn’t good for rendering normal maps since it’s designed to render things with GI and reflections and stuff and the sampling isn’t optimized for when you want the normals really sharp, which is why I use Scanline for that.
For AO, you always have to turn up sampling to get it to look less noisy, I think Mental Ray does a bit better for AO passes, but recent versions of 3ds Max don’t have Mental Ray anymore.

ACiD80 · 2019-01-26 17:01:04 UTC

At this point a sample scene would be helpful

yellowpinkfox · 2019-01-28 10:02:20 UTC

Hey guys,

the scene was for testing only anyways, I attached it here as a sample scene, nothing changed or excluded.

I’ll try removing the VrayDirt AO material and start from scratch there now too, maybe I can narrow things down a bit.

The materials on the objects is a standard Arch&Design material which comes through the import from Sketchup. I’m applying some kind of placeholder materials with a 1x1 m mapping in Sketchup to address material channels in Unity later. This comes in 3DS as a custom UV map, tiles are mainly layed out aside UV tile 1,1, sometimes overlaying each other. Shouldnt affect the Normal / AO baking as the UVs get automatically flattened on another UV channel (2). (In the sample scene its on UV 1 as I dont need to keep the original mapping on UV 1 )

Just four you to know the background, why things are like that.

I need to stick to a automatic workflow, supported later by a script as I have to bake some hundreds of these objects like in the sample scene.

One of the strange things with that scene is, that if I switch from Vray to Scanline it SHOULD reset all renderer related things, eg. global override material. And as there is no Vray material applied to the objects there should be no interference.

Maybe you can find some in that scene I’m missing.

Vray version is 3.60.03, maybe updating to Next soon, but thats not planned yet.
Max version is 2016.

Sample_Scene_Front_XY.max (604 KB)

Best,
Niko

yellowpinkfox · 2019-01-28 10:26:16 UTC

Ah, you shouldnt be wondering about the dirt / AO radius, this number comes from other settings where AO was intended to match the scale of larger furniture cases. AO radius for those objects here need to be tweaked still.

Noren · 2019-01-28 14:01:03 UTC

Thanks for the file.

You need to increase your minimum subdivisions for the image sampler. Vray doesn’t take the normal map into consideration when looking for contrast changes. Rendering the normal map with fixed 4 subdivisions should work. (You could render the normal map separately if that’s an option).

And then Scanline can’t render the Arch&Design material, so that’s why it didn’t show any normals. Use a standard material instead (no idea what happened in Xnormal).

yellowpinkfox · 2019-01-28 16:37:50 UTC

Thanks!

Did not expect the object material itself would effect normal calculation…! I had some success, some with Vray, some Scanline, some XNormal. I’ll put those results together tomorrow.

Still not at 100% but getting there slowly.

Could not really find the right way to generate AO in Scanline along with Normals. Maybe I’m too blind to see now for today. Maybe you can point me somewhere

I’ll post an update tomorrow, thanks again!

Niko

Noren · 2019-01-28 18:51:59 UTC

Unfortunately Scanline can’t do AO. You can try a skylight with the lighttracer or a dome of many spotlights without, but you are probably far better off using Mental Ray for AO and Normals. (Or Vray, if you optimize the Dirt some more). Don’t know what’s quicker.

yellowpinkfox · 2019-01-29 17:43:43 UTC

@Noren : I was almost expecting that Scanline cant do it. I’m not that much familiar with it, not even Mental Ray.

Actually my main area of work was / is more in Archviz, modelling buildings and some assets if needed, compositing scenes with vegetation, rendering stills and sometimes animations. And to be honest, got lazy and used to Solid Rocks since a while which is just too easy to use So producing normal maps for best quality vs speed is a little new to me, though I came across it every now an then.

I did several tests in various directions as mentioned, Scanline Normals as well as the workflow with skylight / light tracer, did not get very far. As well as with XNormals - where I did yield some pretty good results some weeks ago, super clean Normal maps as well as good AOs.

But its a little hard to control the various settings properly when you’re in a new software, as well as the whole setup for batch exporting the proper file types, for XNormals in this case.

Thats why I’d prefer to stick to VRay, which I’m mainly able to handle, and the workflow within 3DS. So I guess the main thing was your hint with the subdivision value, normals turn out way better so far. Not yet super perfect but might get it. Rendertime is incomparably slow, but that should not be a big thing then.

Actually this time Solid Rocks was probably just not the right tool - I first tried to raise quality settings - which increased rendertimes but did not produce that much better results… thats why I did not focus on render settings much / at all.

I guess I have to investigate further into the relation between VrayDirt and Normal maps, or try see if I can seperate those and get proper results then.

Still, ideally one day there is a simple foolproof template / preset that batch renders out all maps at once in a good quality. Today some devs were around, we’ve got some other things to focus on too, but I’ll post a final update here if everything is solved!

Thanks for your support

Best,
Niko

yellowpinkfox · 2019-01-30 11:50:16 UTC

It might have something to do with the interpolation of pretty little normal information along the edges. This could create those mixed pixels as well…

I’m once again wondering why I can’t change the “Element Background” color in render to texture. Is this a limitation of using Vray? I tried using Max internal frame buffer, still no effect. As this might have certain impact on the use of the later normal map it would be better to have a RGB 127,127,255 color as an element background too.

Any idea?

Noren · 2019-01-30 14:36:12 UTC

Too lazy to check right now, but I think that should work. Did you look at the saved images or the frame buffer? Did you activate the crop alpha option? Either way adding some generous padding should make the background color irrelevant.

darthviper107 · 2019-01-30 22:28:27 UTC

Scanline should work properly, as long as you don’t have any Vray settings or materials still active when doing it. I usually render my normals with Scanline since it’s faster and gets sharp results. Sometimes I’ll do a pass with just the Skylight with shadows turned on. Vray is a bit slow for baking textures so I mostly don’t use it. MR can do AO passes a bit faster.

yellowpinkfox · 2019-02-07 13:44:26 UTC

Hi guys,

I solved most of the problems and the results are quite good so far.

I decided to stick to Vray and tweaked te settings so the quality is… ok.
The quality through the RTT tool is NOT as good as a standard viewport render - AO there is MUCH cleaner, in a quicker time.

Rendertime for a combined AO / Normal bake is now about 8-10 minutes on a i7 2600k (yep, not the newest )
Thats pretty slow, even more slow compared to Xnormal with some seconds rendertime for a super crisp result - but this is a totally different calculation method.
I could live with that slow rendertime, if all is set up properly and a machine would render some hours for a bunch of maps, that would be ok.

I’m still investigating both methods for production, 3DS+Vray VS Xnormal…

XNormal:

I’ve found this great XNormal Batcher Tool which creates a XML file and automates the baking process in XNormal (https://polycount.com/discussion/126610/xnormal-batcher-beta/p1). Had some trouble installing it on one machine due to some certificate / security issues but on another machine it works fine.

Advantages: XNormal is super fast and super clean.

Downside: all maps need to be reassigned to the materials in Unity by hand (maybe by some naming algorithm later to automate that step)

3DS+VRay:

Workflow remains inside 3DS, if obejcts are set up properly, bake results are acceptable. Problem right now is the automatic assignment of the HP projection objects. This MIGHT be done through a custom script which sets up the corresponding projection modifier.

Advantages: RTT tool can automatically create a bake material and plug the baked maps into chosen material slots - which can then be exported inside the FBX file to Unity.

Downsides: rendertime and quality.

Recent problem:

For some reason (after countless time of trying we already call it “the error which actually is not existing”) in certain parts of the model the AO is not calculated properly in VRay (and not in Scanline, too, same problem). The AO is calculated on all faces properly, only where a small vertical face meets a small horizontal face AO is only calculated on the vertical face, leaving the horizontal one untouched.

There is (might be) a relation between the UV seams and the output - but XNormal does calculate it all perfectly. So it somehow has to have sth to do with 3DS and / or VRay…
Please have a look a the attached images, maybe you have an idea??

There are 3 different kinds of objects, 2 of them render totally fine from 3DS and XNormal, 1 is being rendered finde from XNormal but has those strange AO errors at this small 90° corners / faces.

The screenshot from Unity shows the AO overlay, this white pixel line beween two correct AO faces is due to some XNormal settings, please ignore that.

UV Layout with Map underlay from the object