when I open my fbx unwrapped model in substance painter and apply a texture the result is really awful. I think this has something to do with texel density, How do I increase it? Do I just have to scale up the map in 3dsmax unwrap UVW?
Maximise Space in the UVUnwrap, so scale up parts you need the highest res and that part will receive more pixels for painting.
Output at a larger size 4000px x 4000px for example if you are going to include this all on 1 UV island.
Use UDIMS. This means you can put different parts on different UV islands so you have 4k res for the head, 4k for the hands/legs etc (if you really need that much).
How do I do that? I do not export the the UVs into a file like you do when you edit them in Photoshop. I just export the FBX
When you export from Substance Painter export as 4096px by 4096px.
If you want to increase the texture size within substance you should adjust the Texture Set Settings
If you want high resolution per area I suggest breaking off parts of the body and give them a different name in the material editor in 3dsmax, unwrap to the scale you need maximising the UV island. This way you can have a 4k for the head, 4k for the arms etc if its really neccessary.