JamesDeschenes
12-09-2002, 12:10 AM
Well this is the third time I had to rework this post… the first time it was a reply to a question about cel-shading and it got deleted because of the forum rollback... the second time it was going to be a new thread but it got mangled up because the server told me I had too many images and then deleted the message so I could not go back and just edit out some stuff… and now this will be my third attempt at making my time more then just wasted… hehe.. Well I hope it works out.
Well this may not be the fanciest solution but it works pretty damn well if you tweak it a bit.. and best of all you are doing all the work yourself out of the box with max :) (and er.. Photoshop :P )
Also I used 3dsmax4.2 so I don't have the fancy ink and paint tools.. so thats why I have spent time on this :) (also I have heard that ink and paint in max 5 is ass slow)
There are plenty of ways to get a nice flat shading effect in Max or other shading effects..
For just solid colours with borders.. I would apply just a material with a solid colour in the diffuse slot and set the self-illumination to 100% so you kill the shading. You set up all your materials with the correct colours... and you render out your sequence (most likely an image sequence if you want to batch process it in Photoshop)... this is set of files will be using for colour...
Now you need to go back to your scene and replace all your objects with a new material I will now describe.. basically apply a material with a falloff map in the diffuse slot.. set the falloff map to perpendicular parallel (bad spelling) and then go back to the material and set its self-illumination to 100% and render out another sequence of images. This set of images will be used to define the outlines.
Once that’s done you will need to take these into Photoshop (or any other paint package with this feature) for some manipulation... open up one of the images that you rendered (with the falloff material) and apply the high-ass filter to it (filters>other>high-pass)... I’m not sure of the specific settings.. but it will depend on what you want... set the value for high-pass somewhere in the low range.. basically this filter is getting rid of the shading in the image and only leaving the detail... (yeah it's a lot more technical then that but I’m not going in to that for now) after the high-pass filter you need to go into the image>adjustments>threshold and adjust the threshold of your image… you will need to play around with the threshold and high pass amount in order to get clean outlines. Now there are a few problems left do address… the main one being is that you have bad aliasing on the outlines.. you can fix this by doing some compound blurs and fading.. or even using the diffuse filter with the anisotropic setting (in Photoshop 7.0)... or even another solution could be to render your falloff pass at a higher resolution and then downsample it. If you are happy with the quality of the outlines.. then all you need to do is grab the outlines and composite them with the color render.. Basically just throw them into a layer on top of the color render and set the layer transfer mode to multiply. Walla.. you have a free solid colored.. outlined render… I will explain at the bottom how to do this for animation. Here is a diagram of all process.
http://www.accesscable.net/~jamie/npr/mix1.jpg
If you want to have distinct shading marks and tones in your color render.. then that’s easy to and also uses a fall off map. Just go back to your color materials and add a falloff map to the diffuse slot (you can make the material 100% self-illuminated too) and set the falloff map to shadow/light. This makes the falloff map shade areas the object with the ramp depending on the amount of light. We want to manipulate the mix curve (or ramp) so that it shows distinct shading marks when going from your darkest colour to your brightest colour on the object (this is also a good time to tell you to set the colours on the falloff map.. the top color will be your darkest colour.. and the bottom colour will be your brightest color). I have include an image below to show you graphically how the mix curve can be setup to show some distinct shades…
http://www.accesscable.net/~jamie/npr/mix.jpg
You can make the areas of shading even more distinct by killing off any blending between these ‘steps’ on the mix curve… you can add more.. or take away some.. with is method you are limited to blending between two colours based on the amount of light. If you want to get even more advanced I will show you how now.
With this method you can control the entire color shading range of an object.. which is much better for doing skin and stuff.. Plus it is a lot more visual too. What you need to do is create a material.. set it’s self-illumination to 100. Now put a color ramp in the diffuse slot and set the gradient type under the color ramp to ‘mapped’ now put a falloff map which is set to shadow/light in the source map slot. You now have the diffuse set up to apply the color ramp based on the amount of light on the surface. You can now throw all the information on how you want to shade the surface into the color ramp.. you can throw discreet shading steps it by turning off any interpolation between the flags.. or you can have a smooth range of colors.. Like a light blue for bright areas of skin to a deep red/brown for the dark areas... and whatever you want in-between. Here is an example.. this is just an example.. you can do whatever you want and get a lot of different effects.
http://www.accesscable.net/~jamie/npr/mix2.jpg
As I said earlier, you can combine this method of coloring with the outlining method.
Now there is even another method of out of the box NPR shading (not photorealistic-rendering) and it is damn easy too! This can be combined with the other methods too but it will require a lot of tweaking... Well all you basically need to do is make your scene.. apply some colored materials to everything.. But here the trick is when you are lighting the scene.. There is a contrast setting on all the max lights as shown below. If you set this anywhere between 90 and 100 you can get some cool effects.
http://www.accesscable.net/~jamie/npr/mix3.jpg
Messing with some bump maps and color maps can also add to the ‘snazyness’. Well that’s basically all I have to say right now… about non photorealistic rendering. There are tons of things you can do with max out of the box. The outlines may not be perfect but they are good for being free. Here are some shots and example stuff with these methods.
http://www.accesscable.net/~jamie/dac/celsample.jpg
If you are going to use this method for animation.. You will need to render out multiple passes and run a batch process in Photoshop using actions... I would say use AfterFX but for some damn reason they decided to not include the High Pass filter in it and the plug-in file can not be found in the Photoshop directory anywhere so there is not hope of taking it from Photoshop and putting it in After-FX... so the only way I can see it being used for animation is with a batch process in Photoshop.
Well that was a lot of writing… I’m sure there are tons of errors so just message me if you have any questions or you see something that is horribly wrong.
Well after reading all this again I have noticed that I have not talked about specular shading.... well for now you can use the same method as you did for coloring your objects (stepped shaders and stuff)... but once I get some time I will reply to this post with more detail on how to get a good specular effect (well 'good' all depends on what you think )
:beer:
Well this may not be the fanciest solution but it works pretty damn well if you tweak it a bit.. and best of all you are doing all the work yourself out of the box with max :) (and er.. Photoshop :P )
Also I used 3dsmax4.2 so I don't have the fancy ink and paint tools.. so thats why I have spent time on this :) (also I have heard that ink and paint in max 5 is ass slow)
There are plenty of ways to get a nice flat shading effect in Max or other shading effects..
For just solid colours with borders.. I would apply just a material with a solid colour in the diffuse slot and set the self-illumination to 100% so you kill the shading. You set up all your materials with the correct colours... and you render out your sequence (most likely an image sequence if you want to batch process it in Photoshop)... this is set of files will be using for colour...
Now you need to go back to your scene and replace all your objects with a new material I will now describe.. basically apply a material with a falloff map in the diffuse slot.. set the falloff map to perpendicular parallel (bad spelling) and then go back to the material and set its self-illumination to 100% and render out another sequence of images. This set of images will be used to define the outlines.
Once that’s done you will need to take these into Photoshop (or any other paint package with this feature) for some manipulation... open up one of the images that you rendered (with the falloff material) and apply the high-ass filter to it (filters>other>high-pass)... I’m not sure of the specific settings.. but it will depend on what you want... set the value for high-pass somewhere in the low range.. basically this filter is getting rid of the shading in the image and only leaving the detail... (yeah it's a lot more technical then that but I’m not going in to that for now) after the high-pass filter you need to go into the image>adjustments>threshold and adjust the threshold of your image… you will need to play around with the threshold and high pass amount in order to get clean outlines. Now there are a few problems left do address… the main one being is that you have bad aliasing on the outlines.. you can fix this by doing some compound blurs and fading.. or even using the diffuse filter with the anisotropic setting (in Photoshop 7.0)... or even another solution could be to render your falloff pass at a higher resolution and then downsample it. If you are happy with the quality of the outlines.. then all you need to do is grab the outlines and composite them with the color render.. Basically just throw them into a layer on top of the color render and set the layer transfer mode to multiply. Walla.. you have a free solid colored.. outlined render… I will explain at the bottom how to do this for animation. Here is a diagram of all process.
http://www.accesscable.net/~jamie/npr/mix1.jpg
If you want to have distinct shading marks and tones in your color render.. then that’s easy to and also uses a fall off map. Just go back to your color materials and add a falloff map to the diffuse slot (you can make the material 100% self-illuminated too) and set the falloff map to shadow/light. This makes the falloff map shade areas the object with the ramp depending on the amount of light. We want to manipulate the mix curve (or ramp) so that it shows distinct shading marks when going from your darkest colour to your brightest colour on the object (this is also a good time to tell you to set the colours on the falloff map.. the top color will be your darkest colour.. and the bottom colour will be your brightest color). I have include an image below to show you graphically how the mix curve can be setup to show some distinct shades…
http://www.accesscable.net/~jamie/npr/mix.jpg
You can make the areas of shading even more distinct by killing off any blending between these ‘steps’ on the mix curve… you can add more.. or take away some.. with is method you are limited to blending between two colours based on the amount of light. If you want to get even more advanced I will show you how now.
With this method you can control the entire color shading range of an object.. which is much better for doing skin and stuff.. Plus it is a lot more visual too. What you need to do is create a material.. set it’s self-illumination to 100. Now put a color ramp in the diffuse slot and set the gradient type under the color ramp to ‘mapped’ now put a falloff map which is set to shadow/light in the source map slot. You now have the diffuse set up to apply the color ramp based on the amount of light on the surface. You can now throw all the information on how you want to shade the surface into the color ramp.. you can throw discreet shading steps it by turning off any interpolation between the flags.. or you can have a smooth range of colors.. Like a light blue for bright areas of skin to a deep red/brown for the dark areas... and whatever you want in-between. Here is an example.. this is just an example.. you can do whatever you want and get a lot of different effects.
http://www.accesscable.net/~jamie/npr/mix2.jpg
As I said earlier, you can combine this method of coloring with the outlining method.
Now there is even another method of out of the box NPR shading (not photorealistic-rendering) and it is damn easy too! This can be combined with the other methods too but it will require a lot of tweaking... Well all you basically need to do is make your scene.. apply some colored materials to everything.. But here the trick is when you are lighting the scene.. There is a contrast setting on all the max lights as shown below. If you set this anywhere between 90 and 100 you can get some cool effects.
http://www.accesscable.net/~jamie/npr/mix3.jpg
Messing with some bump maps and color maps can also add to the ‘snazyness’. Well that’s basically all I have to say right now… about non photorealistic rendering. There are tons of things you can do with max out of the box. The outlines may not be perfect but they are good for being free. Here are some shots and example stuff with these methods.
http://www.accesscable.net/~jamie/dac/celsample.jpg
If you are going to use this method for animation.. You will need to render out multiple passes and run a batch process in Photoshop using actions... I would say use AfterFX but for some damn reason they decided to not include the High Pass filter in it and the plug-in file can not be found in the Photoshop directory anywhere so there is not hope of taking it from Photoshop and putting it in After-FX... so the only way I can see it being used for animation is with a batch process in Photoshop.
Well that was a lot of writing… I’m sure there are tons of errors so just message me if you have any questions or you see something that is horribly wrong.
Well after reading all this again I have noticed that I have not talked about specular shading.... well for now you can use the same method as you did for coloring your objects (stepped shaders and stuff)... but once I get some time I will reply to this post with more detail on how to get a good specular effect (well 'good' all depends on what you think )
:beer: