Volumetric light effect with PFlow/krakatoa

[Bobo]

Hi again!

All approaches sounds possible, but I am not convinced you could actually load one billion particles back into PFlow to get them rendered with Afterburn, and there is no other way to control Afterburn from a PRT Loader except by loading into PFlow, so that might be out of the question. In general, the idea behind Afterburn is to use few particles to create a lot of density in spherical regions around them.

Post processing with some form of a Glow filter is definitely possible, but since the first image you posted obviously represents 3D gas spheres around the stars and not simple defocus, it wouldn't be exactly precise.

I would suggest first trying out the Voxel mode of Krakatoa to see how it looks. Try to specify a voxel size which represents a generic units value close to the size of one pixel when projected onto the image, then increase the Filter Radius and see if it will look like anything.

Another thing you could try is the DOF effect of Krakatoa which distributes particles in a circular pattern (circle of confusion) - if a good f-stop and target distance are selected, it might produce similar results. You could then render the original particles and a series of DOF images with varying settings and composite the results together to get a better gradient.

Finally, there is one feature we have on our Wishlist that we could expedite for v1.5.2 that could help you with a setup like this - hierarchical PRT Loaders. Basically it would allow you to have one PRT Loader defining the positions of the stars (you already have that), but it would then pick one or more PRT Loaders as sources for each star, where the cloud loaded by the secondary PRT could be any other PRT sequence, for example one with a nice spherical shape and radial falloff... We used to have a prototype of this feature hidden inside our builds, not sure if it is still there in 1.5.1 but I will try to find out...
Unfortunately, if you have a billion stars already, replacing each star with 1000 more particles would lead to VERY high memory requirements so I am not sure how doable this would be with the numbers you mentioned.

Yet another method if working with less particles than a billion would be generating the additional particles using PFlow - when working on some look development for Twilight:New Moon I tested something like that (we did not use it in the end, but it was a nice learning experience). Here is the idea:
Let's assume that only about a million out of all stars you have need to have this larger gaseous look. If you had a channel describing their radius, you could split these particles to a separate PRT File, load that into PFlow and then for each particle, Spawn a number of particles using the Spawn Test operator. Using a Script Operator or Box #3, define a random vector with a length based on Gaussian Sphere distribution (Box #3 has one) and store in some Vector channel you can save to PRT again. In the process, delete the original particles. This will give you a cloud of particles where each spawned particle has the same original position as its parent particle and a random vector pointing within a small sphere. Now you could load this new PRT in a PRT Loader, add a KCM on top and just provide a Radius value (which could be also in a channel inside the PRT defining the size of the star if you have that already) to multiply the random vector generated in the previous step. Take the original Position, add the random vector multiplied by the Radius and write back to the Position. This gives you a sphere at each star's position, with full control over the size of each (you can add your own multiplier to push the particles closer or away to control the scaling). The multiple particles will actually look like a sphere with a hotter core and falloff to the side.

Last but not least, you could split your PRT particles into smaller pieces (with 10M each), load each into PFlow and use SIMPLE facing particles with a nice radial gradient to produce the look you want. Use Additive mode in Standard Material. Be sure to set the Render operator to Multiple Meshes because 20M faces will surely crash Max, and try rendering in Scanline (which handles accumulation of transparent planes better than mental ray). It will be SLOW but it could be easier to control the look.

Please keep in mind that ALL these ideas are pushing the limits of Max in terms of memory and performance (except for the Post blur approach), so be prepared to wait in every case... ;)

[pgdavid]

Many ideas, thanks !
Color is supposed to be linked to the local density of particles.
I'll test them and post results.
Regarding the performance, i have in fact 511 PRT files containing an average of 2 million particles each.
If i can achieve the effect on one of the subset only, it will be still very nice.

Note : i've not succeded yet to load automatically a bunch of file (for instance 100 of them) with the PRT loader (files are numbered). I need to load the particles in a single time frame and to keep the absolute position of the particles so i can reconstruct the whole scene. Is there a way to have krakatoa load the file sequence automatically without a partition list file (i've created manually the PRT files with some maxscript) ?

[Bobo]

Many ideas, thanks !
Color is supposed to be linked to the local density of particles.
I'll test them and post results.
Regarding the performance, i have in fact 511 PRT files containing an average of 2 million particles each.
If i can achieve the effect on one of the subset only, it will be still very nice.

Note : i've not succeded yet to load automatically a bunch of file (for instance 100 of them) with the PRT loader (files are numbered). I need to load the particles in a single time frame and to keep the absolute position of the particles so i can reconstruct the whole scene. Is there a way to have krakatoa load the file sequence automatically without a partition list file (i've created manually the PRT files with some maxscript) ?

How are the files numbered? Here are some rules:

The PRT Loader uses the last number of the file name (for example, somefile_0000.prt or anotherfile00.prt) to determine the FRAME number. It will replace those digits internally when you move the time slider unless you check the "Single Frames Only" option. In that case, the frame number will be left as is, but no animation/velocities will be loaded.

Then, the PRT Loader also understands the Partition signature which has the form _partNNofMM_ where NN is the partition number and MM is the total number of partitions expected. So if you would rename your files to follow the same scheme, like

somestars_part001of100_0000.prt
somestars_part002of100_0000.prt
somestars_part003of100_0000.prt

etc., the PRT Loader will let you pick just one of them and will propose to load all 100 of them automagically.


As for 2 million particles, the spawn trick could work with that amount. Still, I would try just blurring the EXRs in post first to see if the result would look anything like what you expect. Fusion for example has a Glow node that mixes a portion of the original image with a blurred version of the same image, thus preserving some of the hotness of the latter while adding a glow around it. I am sure Aftereffects has something similar.

[Glacierise]

1 bil particles with afterburn!!! You'll need to pass that to your grandparents so they can one day see the rendering :D

[noouch]

I think this is a good time to go back to basics and additively blend facing billboard particles with a glow map.

[Glacierise]

Dude have you actually tried that? Not much faster then afterburn... :D

@Bobo - very interesting reads, thanks dude!

[noouch]

Dude have you actually tried that? Not much faster then afterburn... :D
I have actually. Not in the million particles range, but it was reasonably fast for around 100,000.

[pgdavid]

I have succeeded in rendering 200 millions. After that, got RAM problems :) i'll have to do some copositing to have my billion particle on screen i suppose.
I've asked the scientist, the colors in the first image are related to local density : red for low density and yellow for high densities (white for extreme).
Playing now with DOF to get some blur effect and trying to get the Z-depth and Density channel in the EXR file to post-prod it (colorize it according to local density). Still learning how to use EXR but looks promising !
Geez, i feel i'm adventuring very far from my web developer work but all that thing is so fascinating.

Thanks for all the tips, bobo is a tips-machine :D
And thanks to your suggestion noouch, "additively blend facing billboard particles with a glow map" : i still have to decode that as a CG noob :) but i'll give it a try, as it seems a basic technique to master.

[Bobo]

Thanks for all the tips, bobo is a tips-machine :D
And thanks to your suggestion noouch, "additively blend facing billboard particles with a glow map" : i still have to decode that as a CG noob :) but i'll give it a try, as it seems a basic technique to master.

The facing particles was something the tip-machine suggested too (see "Last but not least..." in my post!) :)

Basically, you create the PRT Loader, create a PFlow, add a Krakatoa PRT Birth operator and pick the PRT Loader, then add a Shape Facing operator (it both lets you pick a camera to align to and assigns a rectangular shape to the particles for mapping). You can then add a Material operator to the flow to texture map the particles with gradient maps - assign the map to the Diffuse channel and switch to Additive transparency in the Advanced rollout of the Standard Material. Keep the color very low to avoid overbright results where many particles cover the same pixels.

[pgdavid]

Thanks !
Let's go crash max then :)

[Bobo]

Thanks !
Let's go crash max then :)

To avoid the crashing part, be sure to switch the Render operator of PFlow to "Multiple Meshes". And try to keep the particle count below 1M ;)

[pgdavid]

hey ! I'm fighting with EXR saved files. Layers are empty when opened with PS proEXR. Where could it comes from ?

[Bobo]

hey ! I'm fighting with EXR saved files. Layers are empty when opened with PS proEXR. Where could it comes from ?

Probably from the fact we don't support Layers in EXRs? ;)
Krakatoa saves only RGBA to EXR. Nothing else. If you need other kinds of passes, you will have to make them yourself using KCMs and other EXRs. You can output Normals as colors, Tangents as colors, Distances to the camera - either normalized or real-world - for Z-Depth etc., but these are not supported automatically, you have to calculate the data and write as RGB to an EXR. (Better support for passes is on the To Do list for future versions though).

I have no idea where you got the impression from that Krakatoa would output anything but RGBA to EXR...

[pgdavid]

Oh krakatoa god i need thy help again !

I'm still experimenting on that billion stars starfield challenge. I'm progressing step by step but still many holes in my head.

As for the previous post, i've tried the suggestions, but like you said everything except post-production just kills my single computer render-time wise (100 million particles). so i'll stick to pure krakatoa rendering (which rocks) since i got preview pictures to show soon.

I would like to output a simple density map for my universe-in-a-nutsheel krakatoa particle cloud. Not sure i'm even using the correct words.
by density map i mean a RGB image going from white to black where white areas are high density areas and grey ones low density ones. But i would like large zones, not at the local scale of the particles but at a superscale. I would want to use that map for postproduction.
I started looking at the wonderfull magmaflow editor in order to do that but a few hints would help while i'm at it ...
Doable ? Z-depth would be the same process ?
P.S. How do you write so much content about krakatoa, i see posts form you all over the web (a interesting on the grainy look i've just read!) . are you several men in your body ? :)

[Bobo]

I would like to output a simple density map for my universe-in-a-nutsheel krakatoa particle cloud. Not sure i'm even using the correct words.
by density map i mean a RGB image going from white to black where white areas are high density areas and grey ones low density ones. But i would like large zones, not at the local scale of the particles but at a superscale. I would want to use that map for postproduction.


Currently, there is no good way to query directly information about density gradients in a specific point in space. We have plans to add such MagmaFlow Operators in the future, but right now all you can do is accumulate the particles in an image and make their color or density values represent the density in that area. For example, if you are rendering additively and in Voxel mode with relatively large voxels, each voxel will accumulate the colors of all particles that fall within that box region in space, and the resulting color will be then accumulated into the image on top of any other voxels, thus adding to the total color value in those pixels sharing these voxels. So the pixel color will tell you "how much particle density is there along the ray passing through all voxels behind this pixel".

Similarly, you could look at the density of the particle when shading volumetrically since the density is also accumulated into voxels (or pixels when point rendering) and the Alpha channel ends up representing the total density in the volume behind each pixel.

Not sure if these approaches will give you what you want, but you could try. Using very large voxels, you could get a rather rough representation of the density because thousands of particles would fall into a single voxel and its final color or density would be representative of all colors/densities of all particles it contains. If each particle has the same color and density value, a voxel containing 10 particles will be proportionally dimmer / less dense than one containing 1000... Since all voxels have the exact same size, this should work well. The larger the voxels, the lower the spatial resolution of the density map.


I started looking at the wonderfull magmaflow editor in order to do that but a few hints would help while i'm at it ...
Doable ? Z-depth would be the same process ?


Z-depth is relatively easy.
*Add a global KCM that takes the position of the Camera via a Vector Input linked to the Position of the Camera node, take the Position channel, subtract the two, take the Magnitude - this is the distance from the camera to the particle. You could divide this by the Max. value of the Z-Depth range to normalize it, and subtract from 1.0 if you want the close values to be brighter while farther values being darker. Then convert the value to a Vector and pipe into the Emission Channel Output.
*Enable >Use Emission, check Ignore Scene Lights.
*Set the Density to 1.0/0 to make each particle draw with full opacity.
*Switch the Draw Filter to Nearest Neighbor.
*If you would render from the camera's point of view, the particles will be sorted from back to front and will overwrite each other where they share the same pixel, with the closest particle containing a color representative of the distance to the particle ending up in the final image. (You cannot get info about particles that are behind other particles, and you don't want any anti-aliasing because that would modify the color and change the distance info).


P.S. How do you write so much content about krakatoa, i see posts form you all over the web (a interesting on the grainy look i've just read!) . are you several men in your body ? :)

Writing the documentation on the Prime Focus Website is part of my job and I am being paid for it (which helps). The info in my blog and on CGTalk is mostly my private initiative because I am crazy about Krakatoa and like talking about it. Some of the support I provide here is also part of my job though, esp. in case like yours where it lead to a sale ;)

[pgdavid]

Many thanks again !

Many things are part of people's job, but some do it with more fieriness than others :)
Your dedication is not common !

[pgdavid]

Hello again :)

I'm experimenting with magmaflow. I need a custom arithmetic operator : log10
what i need is to set the density channel to the decimal log of its value, so i have a logarithmic density scale representation. Not sure of the synthax to script that. Searching ...

[Bobo]

Hello again :)

I'm experimenting with magmaflow. I need a custom arithmetic operator : log10
what i need is to set the density channel to the decimal log of its value, so i have a logarithmic density scale representation. Not sure of the synthax to script that. Searching ...

Sorry, it is missing. We will add it in the future.
For now, you can use a Curve Operator approximating the Log. curve to push the values down. Since the Curve input is expected to be in the range from 0.0 to 1.0, you will have to normalize the value respectively before feeding it in, and then scale the output of the Curve node back if desired.

[pgdavid]

Thanks, does the job !

[pgdavid]

... or will do !
How do you get the max value of a given channel already ? (here the max density). Tried the normalize operator but with no success.

[Bobo]

... or will do !
How do you get the max value of a given channel already ? (here the max density). Tried the normalize operator but with no success.

Normalize is a Vector Operator, it works on Vectors and makes them length of 1.0. This is obvious from its category ("Vector") and its description in the MagmaFlow Editor. ;)

You need a Divide Operator and a Float input in the second socket with the Maximum value. There is no way to know what the Maximum value really is. You can only guess it (assume it is something like 10.0, or 100.0, or 1000.0 until it looks right), or perform a visual test by reading in the Density, comparing it to a threshold value using a Greater Operator and using the Switch Operator to, say, paint all particles with a value greater than the threshold in Red, while making the rest Black. THis way, you could visualize all particles with density above the threshold in red and change the threshold until no particle is drawn in red, thus finding the approx. max. density experimentally. Or you could dump out the whole system to CSV and run a script to read each line and figure out the max. Density value (a MAXScript Interface for reading PRT data is on the Wishlist, but we have had no time to implement it yet, only the one for writing).
Yet another approach would be to use a KCM to set the Positions of the particles to [0,0,Density] and just look at the Front viewport to see the highest Z relative to the XZ grid to get an idea where the maximum is...

Just be creative. :)

[pgdavid]

Thanks again ! Was just to check if i didn't missed something.
I have now a good picture of krakatoa functionalities and should be quiet for a while :)
I've started expermenting on renders. Below are some very preliminary tests i've done but most of the work is ahead. Thanks for the very detailed introduction though, i hope one day i'll help someone here in return !

first render tests (http://deus.obspm.fr/tests.html)

Best regards.

PGD

eidt : my initial question was stupid, due to my misunderstanding between a single particle 's density value (=channel value) and a single pixel 's density (= resulting intensity because of overlapping particles and additive opacity) in the renders which hare obviously 2 completly different things. Magmaflow operators work at the particle level, not the cloud level.

[Bobo]

Thanks again ! Was just to check if i didn't missed something.
I have now a good picture of what krakatoa can do 'out of the box', and what it cannot, and should be quiet for a while :)
I've started expermenting on renders. Below are some very preliminary tests i've done but most of the work is ahead. Thanks for the very detailed introduction though, i hope one day i'll help someone here in return !

first render tests (http://deus.obspm.fr/tests.html)

Best regards.

PGD


Wow, pretty impressive given the fact you just started using Krakatoa and half of the things you wanted to do and not directly supported ;)

Regarding "out of the box": I hope you are aware of the fact that all customers of Krakatoa are automatically entitled to participate in the Beta programs and thus have the chance to give direct input during the development process. We are about to start a new Beta cycle soon and would love to have you around to provide feedback about what is missing. For example the Log Operator will be added in the next round (since it is trivial), a method for finding the min. and max. of a particle channel is not as trivial but possible (will discuss it with the developers on Monday).

While I like the visibility of this thread on CGTalk, our private Beta forums will provide you with a place to talk to the developers directly and learn about future development plans that cannot be shared with the wide public yet.

Thanks again for the preview video!

[JohnnyRandom]

first render tests (http://deus.obspm.fr/tests.html)



link is broken :sad: I was hoping to see it, seeings how I have been following your discussion with great interest :)

[PsychoSilence]

same here, wanna see the render and kcm/setup ;-p

[CGTalk Moderation]

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.