FROM SCRATCH: How would you do next gen NPR rendering


FROM SCRATCH: How would you do next gen NPR rendering

    Lets play a game.

Imagine you had blank canvas, and you could design super intuitive NON PHOTO REALISTIC tools for LW
NOT bound by ANYTHING that came before.
The only rules is that it would have to take inspiration from the LOOK of CEL painted animated work

but also from
Watercolors, Comic Book art and Impressionist painting works

So how would it look like? Well show us or Tell us.


I worked hands-on on this problem over a decade ago. 80% of what you might want can be done with 2D image processing operations post-render - without access to any 3D data from the 3D soft. It would also be realtime (e.g. 30+ FPS in 4K UHD) today because gaming GPUs can run 2D image processing operations and video filters ridiculously fast if you have a 1060 GPU or higher.

You need inked edges? There are dozens of ways to detect exterior and interior edges of objects with 2D image processing, and also to re-draw them with dots, dashes, calligraphy brushes… whatever floats your boat.

You need a particular type of stylized shading? Like a painting or a cel-shaded cartoon? You analyze the shading of the render with 2D image processing operations, do math on it - stepping, rounding, log, square root whatever - and often pretty easily get cartoon-style shading from it. There are old After Effects plugins that did this many years ago. They just didn’t take it as far as it could have been taken. More is possible.

You need a canvas type texture or craquelure type effects that are not static? You can do that with fractals or L-tree type lightning-fork structures that are anchored in the shading of the portion of the image or the object they are generated from.

You want cartoon-style relighting of the photoreal shot, whether CG or live-action footage? Detect and analyze the lighting of the render with 2D image processing operations, remove the lighting partially of completely and render different lighting of your choice on it. Having 3D data from the 3D app would make this easier. But it can also done as a post-operation that just works from the rendered 2D image.

Brush strokes and smear/run effects? You can use motion tracking, shading-anchoring or algorithms like SIFT to track whatever brushstrokes you want onto 3D renders or photoreal video footage. Most brushwork just consists of smearing, running or blurring the image in a particular direction with some mathematically generated irregularity anyway. With a bit of trial-and-error, it should be possible to place brushstrokes on the image in such a way that they don’t flicker in and out of view in an animated shot - e.g. a camera floating down a hallway with roman pillars.

All this stuff is possible after the CG render or live-action shoot and doesn’t have to be done in the 3D soft - although it would be nice of course.

For the stuff you just can’t do exactly right with 2D image processing in post, you can update known 3D NPR rendering techniques to become more sophisticated.

Both 2D image processing and NPR rendering R&D have been somewhat neglected the last few years, so what you have seen in various apps like C4D, LW and similar 5 years ago is actually just the tip of what is technically possible.


Aaron Hertzmann - Non-Photorealistic Rendering and the Science of Art


Blender 2.8 will have integrated NPR material nodes on it. it’s pretty promising.

Paul Caggegi works mostly with blender, but he made a lot of experimentation with its materials and compositing mode to achieve a “comic book” feel.
Paul Caggegi - Using Blender to create Manga/Comics - YouTubeit varies with each program and tools integrated, though, and there are some tricks to get a less nuanced color (less light bounces, for example). I don’t know if Substance Painter works with NPR materials, though.

Thing is: NPR still follows SOME basic rendering rules (light bouncing, shadows, you can turn on fresnel depending on what you’re doing, etc.), so it’s very “open” and dependant on your skill at composing the scene