I solved most of the problems and the results are quite good so far.
I decided to stick to Vray and tweaked te settings so the quality is… ok.
The quality through the RTT tool is NOT as good as a standard viewport render - AO there is MUCH cleaner, in a quicker time.
Rendertime for a combined AO / Normal bake is now about 8-10 minutes on a i7 2600k (yep, not the newest )
Thats pretty slow, even more slow compared to Xnormal with some seconds rendertime for a super crisp result - but this is a totally different calculation method.
I could live with that slow rendertime, if all is set up properly and a machine would render some hours for a bunch of maps, that would be ok.
I’m still investigating both methods for production, 3DS+Vray VS Xnormal…
I’ve found this great XNormal Batcher Tool which creates a XML file and automates the baking process in XNormal (https://polycount.com/discussion/126610/xnormal-batcher-beta/p1). Had some trouble installing it on one machine due to some certificate / security issues but on another machine it works fine.
Advantages: XNormal is super fast and super clean.
Downside: all maps need to be reassigned to the materials in Unity by hand (maybe by some naming algorithm later to automate that step)
Workflow remains inside 3DS, if obejcts are set up properly, bake results are acceptable. Problem right now is the automatic assignment of the HP projection objects. This MIGHT be done through a custom script which sets up the corresponding projection modifier.
Advantages: RTT tool can automatically create a bake material and plug the baked maps into chosen material slots - which can then be exported inside the FBX file to Unity.
Downsides: rendertime and quality.
For some reason (after countless time of trying we already call it “the error which actually is not existing”) in certain parts of the model the AO is not calculated properly in VRay (and not in Scanline, too, same problem). The AO is calculated on all faces properly, only where a small vertical face meets a small horizontal face AO is only calculated on the vertical face, leaving the horizontal one untouched.
There is (might be) a relation between the UV seams and the output - but XNormal does calculate it all perfectly. So it somehow has to have sth to do with 3DS and / or VRay…
Please have a look a the attached images, maybe you have an idea??
There are 3 different kinds of objects, 2 of them render totally fine from 3DS and XNormal, 1 is being rendered finde from XNormal but has those strange AO errors at this small 90° corners / faces.
The screenshot from Unity shows the AO overlay, this white pixel line beween two correct AO faces is due to some XNormal settings, please ignore that.
UV Layout with Map underlay from the object
Markups from Unity
FBX files of all 3 objects
1_LP_001.fbx (25.6 KB)
3_LP_001.fbx (22.5 KB)
2_LP_001.fbx (26.0 KB)
Maps from 3DS / VRay with this error
Maps from XNormal without error
Sorry for file spamming, maybe it helpf to identify the problem
Thank you very much, best,