Pflow - custom particle positions [need advice]

[coren]

I suggest you to read the topic "how to make it faster" in the maxscript reference.

Anyway try to change your for loop with this one:
for thePointData in pos do

(

--give birth to a new particle

pCont.AddParticle()
pCont.particleIndex = pCont.NumParticles()
pCont.particleAge = 0
pCont.particlePosition = thePointData*1000
pCont.particleSpeed = [0, 0, 0]
)

keep in mind that script operator are always slower than normal operators.

[pgdavid]

Thanks for directions !

I've splitted my 3D array in 3 1D arrays.
I'm also using append to try to avoid memory leaks.

Speed is one of my concerns but just to be sure i'm not following a wrong path : a one million elements array in maxscript is something crazy or standard ?

I've worked a bit further in my scripts and the problem comes from the file reading part (PFlow seems ok handling a million particles).
It seems that reading 150 000 positions from a binary files is pushing max to its limits.
the source file is 20 Mo, i would have guessed that was cake for max. I must do something wrong or asking too much, here is the code :


chemin = getOpenFileName caption:"Choose a file"
f = fopen chemin "rb"

/* Read Simulation Header*/
fseek f 4 #seek_cur /* skip fortran header*/
npart = readBEL f
fseek f 8 #seek_cur
procid = readBEL f
fseek f 8 #seek_cur

/* Simulation bounds */
xminproc = readBEF f
xmaxproc = readBEF f
yminproc = readBEF f
ymaxproc = readBEF f
zminproc = readBEF f
zmaxproc = readBEF f

fseek f 8 #seek_cur

/* Particles --------------*/

/* Positions */
maxn = maxPart.value
for n = 1 to maxn do
(
data = readfloat f
append posx data
bloc = readfloat f
append posy data
bloc = readfloat f
append posz data
)

[Bobo]

Pflow can handle a million particles without a problem, I suspect that the intermediate storage of particle data might be the cause for crashes.

Have you thought of moving the scripted utility that reads the binary file into memory to the Birth Script operator? All you would have to do is read a particle position at a time and create it as you are already doing, but without the whole global array.

Also, have you tried increasing the MAXScript Memory (HeapSize) under Customize>Preferences>MAXScript? I would expect MAXScript to do this automatically, but you never know. Btw, is this in 32 or 64 bit Max and what version?

Yet another alternative would be Krakatoa in Evaluation mode.
Krakatoa provides a MAXScript Interface that can save particles to its own PRT file format.
The PRT file format is compressed and can be read very fast using the Krakatoa PFlow Operators. Of course, this might be slightly outside of your technical level, but if you want to try it out, I could give you some guidance.

Here is a short overview of the process:

*In your scripted utility, instead of writing to an array, you simply write to PRT via this interface:
http://software.primefocusworld.com/software/support/krakatoa/krakatoaparticleostream_maxscript_value.php
*You run that script just once to create a PRT representation of your data and you are ready to use Krakatoa.
*In Max, you create a PRT Loader object and pick the new PRT file.
*In PFlow, you add a Krakatoa PRT Loader Birth and pick the PRT Loader from the scene.
*Done.

You can get the evaluation version of Krakatoa from
http://software.primefocusworld.com/software/products/krakatoa/download/

[pgdavid]

Oh thanks. Many tips here.
I'll try this week-end to read the file directly in the birth script event and then evaluate converting my files to the PRT Krakatoa format for faster reading, which seams quite straitforward.

Thanks a lot.

[pgdavid]

Just in case it can be helpfull to other users looking for the same info (and because i often read forum but don't contribute a lot i realise) i tested the 2 methods suggested by Bobo :

I - reading the million-particles file directly in the birth script and creating particles on the fly
It works! Max doesn't crash, and the particles are created.
Cons : Working on the Pflow is quite slow and rendering is very slow and painfull on a duo core windows 64bit system. I'm not used to optimizations though, there must be some ways to speed it i'm sure.

II - using krakatoa PRT file format
Wow.
I just tested. Loading a million particle with the PRT loader object just takes a second. I can inspect in real time my million particles cloud. I'm so impressed i had to write something here :)
I've made a render test. Works like a charm too. Really impressive.
Cons : Krakatoa requires a license. I'll try to convince my client to purchase one.

Pierre-Gérard David

[Bobo]

II - using krakatoa PRT file format
Wow.
I just tested. Loading a million particle with the PRT loader object just takes a second. I can inspect in real time my million particles cloud. I'm so impressed i had to write something here :)
I've made a render test. Works like a charm too. Really impressive.
Cons : Krakatoa requires a license. I'll try to convince my client to purchase one.


You made my day! :)

Some notes on the licensing thing:
*The only limitations of the Evaluation version are that resolutions above 480x360 render with a watermark, network rendering is disabled and some bonus PFlow tools like the very fast Krakatoa Collision and the Krakatoa Geometry Test are disabled. Everything else works and you can use it as much as you want. For free.
*We will provide you with a free evaluation license for 15 days if you submit a request. That could be enough to finish a project, just in case your client cannot be convinced to spend the $$$. Of course I would hope he would, as it pays our salaries ;)
*At that point, we would hope you would be so in love with Krakatoa that you would budget it for future projects or at least suggest it as option to other clients, thus increasing awareness of its awesomeness... :)

Seriously though, I am so glad these approaches work for you.

Also note that with Krakatoa 1.5, this gives you an extra level of flexibility to tweak the appearance of your particles since you can modify any channels using the MagmaFlow editor, deform particles using regular Max modifiers, cull particles in areas defined by geometry volumes and so on. Be sure to read through the documentation for some inspiration:

Deforming, Culling, Recycling of PFlow Partricles:
http://software.primefocusworld.com/software/support/krakatoa/using_prt_loader_birth_and_update.php

Practical use of MagmaFlow:
http://software.primefocusworld.com/software/support/krakatoa/magmaflow_practical_example_gijoe.php

MagmaFlow Examples:
http://software.primefocusworld.com/software/support/krakatoa/particle_channels_editor_examples.php

[LoneRobot]

Hi Bobo,

Clever, wish I had the time to evaluate it. What is the Krakatoa Magaflow Editor UI made with? it looks a bit like Kees' Helium UI.

[Bobo]

Hi Bobo,

Clever, wish I had the time to evaluate it. What is the Krakatoa Magaflow Editor UI made with? it looks a bit like Kees' Helium UI.

MagmaFlow is 100% Helium+MAXScript. Good eye.

[pgdavid]

Hey ! it's me again.

your tool is really awesome :)

Can i request some more guidance ?

1 . I can't find the macroscripts page (dead link) on the prime focus website which is supposed to describe how to install the krakatoa toolbar. How do i do that ?

2. i have created a maxscript wich writes a PRT file from my custom source file. It is a snapshot of my million particles's positions AND velocities at a given time.
Is there a quick way (PFlow/krakatoa combination?) to animate those particles, based on the written velocities ? (to just make them "continue" their movement)

Best regards,

Pierre-Gérard

[Bobo]

Hey ! it's me again.

your tool is really awesome :)

Can i request some more guidance ?

1 . I can't find the macroscripts page (dead link) on the prime focus website which is supposed to describe how to install the krakatoa toolbar. How do i do that ?


Strange, not sure where that page went.
Check out the offline CHM of the Help - it has the page.
http://software.primefocusworld.com/downloads/Krakatoa_1_5_1_38002_CHM.zip



2. i have created a maxscript wich writes a PRT file from my custom source file. It is a snapshot of my million particles's positions AND velocities at a given time.
Is there a quick way (PFlow/krakatoa combination?) to animate those particles, based on the written velocities ? (to just make them "continue" their movement)


Yes, but due to some earlier design decisions, it requires a little jumping through hoops ;)

*When saving the Particles, do not save the Velocity into a "Velocity" channel, instead save into ANY float16[3] channel like a "Mapping10" or a custom channel (you can add new custom channels to the system in the Channels rollout - for example I created one called "MyVelocity"). But for speed and simplicity, let's assume you will dump the velocity into the Mapping10 channel.
*In the more general case when saving an actual scene particle system like PFlow etc, you can add a global KCM in the Global Render Values rollout and add a MagmaFlow that copies all values from the Velocity channel into your channel. When saving the particle frame, your channel in the PRT will be populated with the actual velocity data.

*Now when loading, create a PRT Loader, set to 100% in viewport.
*Pick the single frame you saved.
*Check "Limit To Custom Range" and hit the "Range" button to set start and end frame to the same frame (0).
RESULT: The particles will appear and stay on all frames.

*Add a Krakatoa Channels Modifier.
*Set the Input to Position, set the Output to Position
*Add a Mapping10 Channel Input.
*Select the Mapping10 Input and the Position Input and hit + on the Numeric Pad to add the two together.
RESULT: The particles will now appear offset significantly since the Speed is stored in units per second.

*Add a Script Input and enter "currenttime.frame/FrameRate" in the script field.
*Select both the Mapping10 Channel Input and the Script Input and hit * on the Numeric Pad to add a Multiply Operator.
RESULT: Now the Velocity is scaled to the correct frame rate and moving the time slider/rendering will produce particles moving along the same velocity vector over time.

*At this point we can add a second KCM and set it to Mapping10 Input -> Velocity Output.
*Switch the PRT Loader to "Display Velocities" instead of "Large Dots" and you will see the actual velocities.


Why do we have to do all this?

Early on before there was MagmaFlow, the design decision was made that loading a single frame with valid velocities and displaying it over multiple frames CANNOT produce motion blur. So if you save the Velocity channel as usual to frame 0000 and either check "Load Single Frame Only" or "Limit To Custom Range", the particles will render on every frame, but the Velocity will be internally reset to [0,0,0] because in fact they cannot be moving :)

Now that MagmaFlow could move them by integrating the position and velocity data over time, this design decision comes to bite us in the butts. But, and there is a but, since we support arbitrary channels, there is usually some workaround like the one above to save them ;)

Hope this helps.

[pgdavid]

Wow, thanks a lot for the details !
I'll try it today.

Pierre-Gérard

[pgdavid]

It works like a charm.
Very last question : now that technically all is good i'm playing with my new toy to make it look nice. I can't seem to be able to apply a glow effect on the particle cloud. Is there a workaround ?
(and i'll shut up for ever :) )

[Bobo]

It works like a charm.
Very last question : now that technically all is good i'm playing with my new toy to make it look nice. I can't seem to be able to apply a glow effect on the particle cloud. Is there a workaround ?
(and i'll shut up for ever :) )

First of all, if you are rendering to EXR (as you should), the Krakatoa output is floating point and you should be able to use it in a post-application like AfterEffects, Fusion or Nuke to apply subtle and not-so-subtle effects incl. Glow.

Also, using the build-in Depth Of Field it might be possible to make point rendered particles appear larger than a pixel which might produce similar results.
http://software.primefocusworld.com/software/support/krakatoa/depth_of_field.php

Also, rendering additively can also produce rather glowing effects (once again, since the output is non-clamped floating point colors, you can scale the result to get what you want).

Then again, rendering the same particles once as a point cloud and once as voxels could give you two passes to combine detail and a more diffused glow version. Or render a second points pass at a lower resolution, then scale up in comp and blend together...

Of course, I am not sure what sort of effect you are after, so these are just shots in the dark. But in general using effects in post is what I would suggest...

[pgdavid]

Many thanks for all !
Never used EXR but seems a nice tool for post-production, wich is a good option.
Maybe i could also use a volumetric rendering plugin.
I 'm going to start a new topic on the effect i would like to achieve since it's not purely krakatoa related.
Note : i've pretty much conviced my client to purchase a copy of krakatoa :) wich is the best tool for handling my large amount of particles.

[Bobo]

Many thanks for all !
Never used EXR but seems a nice tool for post-production, wich is a good option.
Maybe i could also use a volumetric rendering plugin.
I 'm going to start a new topic on the effect i would like to achieve since it's not purely krakatoa related.
Note : i've pretty much conviced my client to purchase a copy of krakatoa :) wich is the best tool for handling my large amount of particles.

Never used anything but EXR since I started working in VFX 6 years ago, it is the way to go.
I answered to your new topic, and we received the order for Krakatoa license, thank you very much! :)

[CGTalk Moderation]

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.