View Full Version : Questions about the Cel-Shader

04 April 2003, 01:41 PM
Hi everybody,
I'm new to this forum so if this topic has been discussed before please refer me to the appropriate links where I can read up on it :)

I had some questions about the cel shader in lightwave that is used to make cel-shaded style renders.

I read up some of the tutorials and help provided at on how to use the cel-shader in lightwave.

However, the cel-shade renders I'm getting dont have smooth shading. what I mean by this is, best illustrated in the images below.

The cel shading style is supposed to be very simple. Currently, the shading is patchy, and the lines of definition between the shades / paints aren't smooth curves but instead patches, blotches or whatever you wish to call them.

Is this something that can be solved easily within cel shader? Is it the inadequacy of my model itself?

Or is something else at hand which I'm not able to control yet?


* deleted images due to limited online storage space. new renders at the back of this thread posted by me ^_^ *

04 April 2003, 01:49 PM
if you use unreal celshader(free) you can smooth the colour transition between teh zones but the whole point of celshading is to create zones - at least thats the way i look at it

04 April 2003, 02:02 PM
killme --

Yea, its true that cel shading is all about creating zones but I think I didnt explain it properly. What I mean when I say its not smooth is how when you shade the side of a face, you want to create a continuous curve. Taking 1.jpg as an example, there is this small slight break in the smooth curve around the area of the mouth. This shading jutts in toward the mouth. This is ok, but in the nose shading its very rough shading. You can see a distinct stair case kind of shading from the bridge down to the nose , and then it widens again around the nose area.

Instead of a smooth circular shape to shade the shadow the nose is casting, in this case the shading is bumpy line of shading which while on a guy character looks ok, will look very ugly and inelegant on a female character.

Is there a way to work around this problem?

04 April 2003, 02:52 PM

try these out, if you need to search a particular way of doing things then at the bottom of the page, use the search, I entered just "cel" here and got quiet a few hits...

Be sure to be in the Lightwave forum otherwise you will get a stack of hits from all over the forum...

Hope this does the trick

PS. I think your model when you get it sorted with cel-shading will look sweet :thumbsup:

04 April 2003, 03:15 PM
samaritin -- thanks for those links. I did come across some of them in the past. I even tried emailing celshader's author although i have not received a reply from her.

I have heard many different possible explanations as to how to get a smoother line but so far nothing solid. Some folks said to up the tesselation to get a real smooth model. I set it to 8 and watched my box take over itself :)

I was told to try fiddling with the lighting. I guess that would work sometimes but come animation time it would break again. I guess I'll have to wait and hope some more people with experience will shed some light on this.

* deleted images due to limited online storage space. new renders at the back of this thread posted by me ^_^ *

04 April 2003, 04:31 PM
That looks better already, you've captured the anime kinda face, with the v-line jaw and the styling of the hair...

Be interesting to see the animation when you get the cel-shading to your liking, are you going to be using motion designer for the hair to get it swaying around ???

04 April 2003, 04:52 PM
well the way i see it providing the nose shape is ok which seems to be from teh shadows on the non cel shaded image try adjusting the postions of the zones so that the upper shadowed area around the nose isn't in the darker band liek the that should be in shadow as the cel shadows are based of the normal renders one so with careful postitioning of the zone should get the result you want

hope that made some sence

04 April 2003, 01:39 AM
samartin -- Motion designer remains a mystery to me. I tried many many clothing tutorials including the one on Newtek's site:

and the clothing just comes out wrong. I would expect it to perform as unpredictably for the hair. I would like to use it for hair though. The thought of making multiple bones and weight maps for every tendril of hair sounds a bit daunting, let alone animating it.

So far whenever I try the motion designer the clothing is either dancing too much even when there is no wind set up, or behaving in odd ways. Its really got me almost ready to throw in the towel. Would you happen to know some about MD?

killme -- I'm a bit confused about what you said... Could you elaborate? @_@x


04 April 2003, 02:00 AM
i could be wrong but it seem to me this is the way cel shaders work

your model surface can go from fully illuminated to zero illumination so you got alot of degrees of shadowing on your model

depending on the amount of lighting hitting any particular area on your model places it on a scale in your cel shader between teh darkest and lightest

you go more shadow on your nose than desired when the cel shading it on so if you make the shadow zone on your shader not go so far into the lighter end of the scale the annoying bits your want rid off should disappear all by themselves might takes afew trials and errors to find teh esbt zone postions but once done it should work for most if not all lighting condistions

that more clear?

04 April 2003, 02:44 AM
Perhaps a picture of your mesh could shed some light on your problems. Lighting also has a LOT to do with the final result. You might find it's a combination of elements that are giving these results.

04 April 2003, 08:43 AM
kill me -- I will play with it some more. When I reduce the shading zones, the image appears very flat, and very little is shaded.

fatsy -- i have been posting the mesh. the 3d jpg that follows the cell jpg is the mesh! o_O Or are you asking me to post something in wireframe?

New renders of a girl. These are... well... this is the problem I was explaining, the girl looks very inelegant because the shading lines aren't smooth. They're very bumpy and its not feminine at all. :(

* deleted images due to limited online storage space. new renders at the back of this thread posted by me ^_^ *

04 April 2003, 09:42 AM
What are you using for ink lines? an incidence gradient? The thin/thickness of your lines will probably not transfer very well if you plan to animate. Have you tried Lightwaves own ink lines? They are pretty clean and easy to use. Your models definitely have that anime look to them, which is a good thing :) I think your on the right track with the shading, it looks pretty good. You may want to reduce the number of color bands for the skin though, I think it will look better, and it will animate well. If you look at japanese anime most times the skin tones are only two shades, because it makes it easy to see the form of the face (aside from it being easier to animate). Don't forget to add specularity to the eyes!

04 April 2003, 11:51 AM
Hey! It's shade01, the master guru of the Unreal Cel Shader tutorial and translation of the Japanese pages! I'm honored to have you visit my little thread discussion!

The inking lines are done using Ms Hachigian's methods at , where applying a fast frensel and a negative value for diffuse to the darkest zone of the cel paint, and then using black as that paint, can help the edge tracer. The fast frensel creates a black zone of paint that when combined with edge tracer appears to be a variable width line. It doesnt look too bad when animating actually, I tried a 360 camera movement around my guy.

It gets a bit funny when the variable width line gets too thick but other than that it's not a major worry right now. Maybe when its' time to animate then I'll address it if it becomes a problem.

The immediate problem is still how to get smooth shading curves as my pink haired lady looks very unlady-like at this point!

I've been remodeling her giving her thicker hair at the back but ultimately its the cel shading.

I tried turning up sub patch from 3 to 4 to see if a smoother model would smooth out those bumpy shading lines. It smoothed out in some areas, became more bumpy in others, and had no change at all in some areas.

Back to the drawing board.....

To address your mention of using two paints instead of three, I think I would like to try and work it out with three. Two paints and rough bumpy shading lines is still rough bumpy shading lines. In manga cel-style coloring uses 4 paints from what I've seen --- Regular color, darker shading <-- these two will be 90% of the surface area, 5% -> extra dark shading for the smallest regions to give your model that extra depth, and 5%-> highlights along the light regions.

I thought it would be awesome if I could reproduce this manga cel-style shading in an animated version. Normally this isnt done because its too much work to get all the paints to be consistent, but since this is 3D... heck throw it all in, no extra sweat off my back for a better depth looking output right?

04 April 2003, 12:19 PM
You should post some wire frames of your model. Any errors in the smoothness of the render is probably in the model since you say you've gone through all the settings. Cheers :)

04 April 2003, 12:51 PM
shade01 -- I'll do that right now. What should I post? The un-subpatched poly model? Subpatch divisions to 1, 2, 3 ?

I'm rendering with subpatch 3.

While I wait for your guidance here are two more renders --

kyosan and tohru kun in the same body position, using motion designer for the clothing. Its again a mystery. Tonight MD is working well. Tomorrow it will have its bad time of the month again.. its random like that....

* deleted images due to limited online storage space. new renders at the back of this thread posted by me ^_^ *

04 April 2003, 07:14 PM
cel shading loking not to bad there

migth i make a suggestion try making her shoulders abit more narrow and give her a hmm larger bust and she will look more feminine

04 April 2003, 10:27 PM
killme -- i'll give it a try later today and see how she turns out.

I don't know if these are helpful but I'm going to post the wireframes of just the face.

I've got 4 pictures here, from raw poly model, to sub patch at levels 1-3. I render at level 3 for all of the images so far. Here goes:

* deleted images due to limited online storage space. new renders at the back of this thread posted by me ^_^ *

04 April 2003, 12:06 AM
Just made a huge reply and had an error when I posted it! :cry:
Anyway, I just said to go with what Shade01 said, by reducing the amount of colour bands you are using.

Also, something I've noticed lately in anime is that the darker bands are not always just a darker version of the main colour. Often (especially on skin) the darker bands are actually shades of slightly different colours.

Also, really fiddle with the lighting, it has a huge effect on the end render. Try using a point light to get different effects on your zones as well. Experiment.

As for your wireframe...
I think we've found the main problem regarding the "bumpiness".
To get the best results when modelling, it is common practice to try as much as possible to follow the "flow" of the shape of the model as possible.
A quick search pulled up this thread...
Facial Topology (

Doing a search on "Edge Loops" will pull up a lot more ;)

Good reading! :thumbsup:

04 April 2003, 05:01 AM
I just got back from the SPACE ( convention in Columbus, Ohio. I did not have access to my email all weekend, which is why I did not reply immediately.

Here's some quick tips:

If the mesh is not perfectly smooth, the shading will not be perfectly smooth. If the mesh has any bumps or dents in it whatsoever, your shading will get distorted and uneven. Try using the Smooth function in Modeler to smooth out the mesh, or freezing it at a SubD level of 1 to even out the points.
To make the female look more feminine: right now her proportions look extremely masculine, especially with her currently powerful neck. Give her a more slender neck and a higher waist. Also consider giving her daintier hands, feet, jaw, mouth and nose; narrow her shoulders, broaden her hips, and consider giving her larger eyes.
Use LightWave Edges for celshaded animations. Fresnel inking looks fine for illustrations, but falls apart during animations.

04 April 2003, 05:03 AM
Yes, I agree with fasty. Your color bands are not rendering smooth because your mesh is not following proper facial topology. Once you do this, I think you will see immediate improvements in your renders.

04 April 2003, 05:53 AM
shade 01, fatsy and celshader -- Thank you for your feedbacks. I've tried readjusting the placement of the vertices on the mesh to be more coutoured. These are my results so far:

* deleted images due to limited online storage space. new renders at the back of this thread posted by me ^_^ *

I do not know if the coutours are enough, the improvement appears to be more in the guy model than in the girl model. I'm not exactly sure why... Maybe its because I'm fussier when critiquing a female model? :)

Anyway, I'll give the smooth funtion a try in a bit and see how it turns out. Stay tuned :)

Second post for the girl model. Apparently I posted too many in this one the first time around.

04 April 2003, 05:55 AM
Here's the girl model's render images:

* deleted images due to limited online storage space. new renders at the back of this thread posted by me ^_^ *

04 April 2003, 07:56 AM
I've gotten some mixed results, all of them need help :(

First I tried the subpatching of my model, at a resolution of 1 and freezing it. The first thing I noticed was the model shrunk a bit, that wasnt a problem, but I noticed that it created even more bumps in some areas of the model. :(

I tried the smooth function under deform, but it didnt smooth out as well as the subpatch at level 1. I suppose I could make it smooth more but at the expense of losing detail.

Here are my results for the smoothing attempt:

* deleted images due to limited online storage space. new renders at the back of this thread posted by me ^_^ *

The next thing I tried was to take away the fast frensel since celshader advised it does not look good during animation.

Wow! It got very ugly from here on out!
I had been using Unreal's edge tracer up to this point. When I took off the fast frensel, the edge tracer only traced the borders, and a lot of the detail dropped out. What I did next was to use Lightwave's inherent edge outlining under each object's properties.

I got all the edges I wanted this way, and more :(
I noticed that these ugly artifacts disappeared when I unchecked "unshared edges", so I thought that maybe the polygons weren't sharing the same vertices and there were holes in the model hence the artifacts. However, when I tried selecting the adjacent polygons around those artifacts, and merging those polygons, everything merged without complaining. Usualy it complains when there is a hole there so in this case I assume there is no hole which brings me back to square one... Next I tried deleting all the polygons one by one and selecting the vertices / points and creating new polygons to fix it. The artifacts remained. After that I tried deleting polygons to see if there was a second polygon hiding under the first one causing this odd behavior. That also was not the case.
what are those artifacts caused by?

Any advice is greatly appreciated on both these brick walls I've run into!

hehe... I'm also still working on the smoothing shading issues. I think I'll try to make the female model more feminine. Shrinking arms and shoulders seems a bit more in my control right now rather than these other strange problems.

04 April 2003, 01:18 PM
Howdy again. Its been quiet around here but I'll post my progress anyway.

I'm still stuck with the artifacts problem from above. I haven't had a solid lead on what's causing it. I tried the smoothing method on the model. I smoothed the faces about 8 times and kept the result. The model almost appears to melt like a block of ice when you smooth that many times. After that I redid the eyes and the nose to retain their character and came up with these results below.

I am aware of the problem for the clothing below, its because the clothing is going into the body, i didnt turn on MD for this render.
These renders are done using the Unreal Edge tracer, I still can't get the Lightwave Edges to work properly, I get those artifacts I mentioned when I turn on the edge tracing for Unshared Edges. And if I keep it off a significant chunk of the lines that are important are missing.

* deleted images due to limited online storage space. new renders at the back of this thread posted by me ^_^ *

04 April 2003, 09:43 PM
Hi tonami,

the line artifacts you see are an old Lightwave Bug - don't render with subpatch 3 - use 4 instead. For me, this takes away those problems...
The other thing that was allready pointed out is, that you should avoid triangles in your mesh. The Lightwave-Subpatch algorithm doesn't handle them too good and they allways give you problems.
If you want your inklines to be smother, try rendering with enhanced low or enhanced medium with adaptive disabled - the antialiasing of the lines is getting much better from that.
For the lighting, I would suggest using distant lights with raytraced shadows and as little of them as possible. Every additional light adds problems. One or two lights are often enough.
The distant lights add less of those round eggshaped cellzones to your rendering.
I would also suggest using only two tones for the skin and don't use just a darker shade - match it by eye, not by value.

Best Regards,

Thomas Helzle

04 April 2003, 10:01 PM
I have had these line- artefacts in the past, they happen once you delete points in your base- mesh while still having morphmaps. Only way to get rid of them, that I know of is by deleting the morphmap and starting over again. It is not really a bug, but the way it works right now, though if it can be fixed, it should be fixed...
Also make sure there are no 2- point- polys, or degenerated polys there (which might also cause this).

04 April 2003, 09:53 AM
Well its been a hella long time since I posted here. These are new renders using the cel-shader. I apologize for the small size of the images, even at this resolution they're large files. I can't quite get the skin to be smooth but these are the story boards for the animation if I ever get that far.

Any comments on how it is proceeding would be appreciataed ^_^

CGTalk Moderation
01 January 2006, 08:00 PM
This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.