Handling 32M Point Cloud on Sparta

Become a member of the CGSociety

Connect, Share, and Learn with our Large Growing CG Art Community. It's Free!

THREAD CLOSED
 
Thread Tools Search this Thread Display Modes
Old 05 May 2013   #1
Handling 32M Point Cloud on Sparta

Hi guys :D

I updated Sparta to Rev.32 to answer the request “I wish I could treat 1M point cloud per single frame” from users.



This time I’ve optimized all processes (Importing, Exporting, Visualizing, Deforming…) to accelerate computing speed.

And now, Sparta became about 20 times faster than before. e.g. The process between importing cache to visualizing points became over 50 times faster than before. And, the real time sculpting process became about 5 times faster.

Of course, a memory efficiency was improved to control the increased point cloud, too. e.g. the cache in this video is 32M points by 150 frame, the memory became about 200 times efficient.

In addition, there are 9 improvement about a usability enhancement with 6 bug fix. I thinks Sparta became more useful and stable than before.

http://spartaproject.com/
__________________
Lyouta Koduka / Founder of Sparta Project
 
Old 05 May 2013   #2
Looks like a cool app. Before I download and try it, I was looking for information on point cloud formats that you support in your online manual but I guess it isn't complete yet.
 
Old 05 May 2013   #3
Very impressive optimisations. I think you're really onto something here! Fascinating looking tool!
__________________
"Even the Christmas vacation will be darkened by New Zealand scripts…"
~ J.R.R. Tolkien, The Letters of J.R.R. Tolkien, Letter 34
 
Old 05 May 2013   #4
out of curiosity, is the point cloud storage 'out of core' for a single frame ? ie. can I have a 32M pcloud on a single frame not eating all the RAM ? Cheers
 
Old 05 May 2013   #5
To noizFACTORY

Originally Posted by noizFACTORY: Looks like a cool app. Before I download and try it, I was looking for information on point cloud formats that you support in your online manual but I guess it isn't complete yet.


Sparta only support Realflow BIN cache as a file format now. and you can export point cloud by Realflow Plugin distributed from Next Limit Technologies for free.

http://spartaproject.com/platform/

I'm updating the website of Sparta project to more simple and easy to understand :)
__________________
Lyouta Koduka / Founder of Sparta Project
 
Old 05 May 2013   #6
To ctrlstudio

Originally Posted by ctrlstudio: out of curiosity, is the point cloud storage 'out of core' for a single frame ? ie. can I have a 32M pcloud on a single frame not eating all the RAM ? Cheers


You can download sample file which is 1 frame 11M points in Sparta.
Note: That took over 30 sec for downloading by my environment.

And, Spara used 15 GB RAM for 11M point cloud.
So in 32GB case, you need 58GB... :D

__________________
Lyouta Koduka / Founder of Sparta Project
 
Old 05 May 2013   #7
ok, so no 'out of core' stuff.

and what you store in your pcloud that an 11M takes up to 15GB of RAM ? I use pclouds for rendering and even considering memory alignments they take 1/3 of your space.

btw, interactivity seems very good, I guess you're using some efficient nearest neighbors search for brush painting and such, - keep up the good work !

cheers
 
Old 05 May 2013   #8
To ctrlstudio

Originally Posted by ctrlstudio: ok, so no 'out of core' stuff.

and what you store in your pcloud that an 11M takes up to 15GB of RAM ? I use pclouds for rendering and even considering memory alignments they take 1/3 of your space.

btw, interactivity seems very good, I guess you're using some efficient nearest neighbors search for brush painting and such, - keep up the good work !

cheers

Yeah, I'm considering that the real time computing speed is important :)
Sparta store all pcloud to RAM, I want to avoid the file I/O, because accessing to HDD is so slow.

In addition, I think RAM is cheaper than CPU now.
My friend who is FX guy bought 64GB RAM to make large scale FumeFX simulation.
__________________
Lyouta Koduka / Founder of Sparta Project
 
Old 05 May 2013   #9
Originally Posted by lyouta: Yeah, I'm considering that the real time computing speed is important
Sparta store all pcloud to RAM, I want to avoid the file I/O, because accessing to HDD is so slow.
Yep, but with 'out-of-core' I did not mean raw access to HDD but a true cache mechanism, of course putting it to real time is some real effort. For offline stuff I use this one. http://www.tabellion.org/et/paper11/OutOfCorePBGI.pdf. You may already have 'linearized' (morton coded) your pcloud if you use some approximate nearest neighbor for brush searches, which is a pre-requisite for the linked approach above.

cheers
 
Old 05 May 2013   #10
To ctrlstudio

Originally Posted by ctrlstudio: Yep, but with 'out-of-core' I did not mean raw access to HDD but a true cache mechanism, of course putting it to real time is some real effort. For offline stuff I use this one. http://www.tabellion.org/et/paper11/OutOfCorePBGI.pdf. You may already have 'linearized' (morton coded) your pcloud if you use some approximate nearest neighbor for brush searches, which is a pre-requisite for the linked approach above.

cheers


Well, you mean the building a tree structure of pcloud for a nearest neighbor search on the out of RAM.

Yeah, it actually will be a problem when I treat over 10M points in one frame.

I've finished to optimize Sparta for 1M in one frame now, but I feel there is a limit for current way which is storing all pcloud to RAM.

Thank you for a hints to realize next algorism :)
__________________
Lyouta Koduka / Founder of Sparta Project
 
Old 05 May 2013   #11
Quote: Well, you mean the building a tree structure of pcloud for a nearest neighbor search on the out of RAM.
Yeah, it actually will be a problem when I treat over 10M points in one frame.
Look if you use an octree as your data structure for pclouds you can refer to the paper I linked above to turn it out-of-core. Done that you should look for a nearest neighbor search that uses octrees and you're half the way done. I can suggest you the STANN library as a reference for fast approximate searches. The only thing you'll need then to spend time on is to implement out-of-core sorting as you need to linearize the whole pcloud to be able to use the above approaches.

You seem well receptive that I'm sure you'll end up with some good quality software, good luck !

cheers
 
Old 05 May 2013   #12
Thread automatically closed

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.
__________________
CGTalk Policy/Legalities
Note that as CGTalk Members, you agree to the terms and conditions of using this website.
 
Thread Closed share thread



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
CGSociety
Society of Digital Artists
www.cgsociety.org

Powered by vBulletin
Copyright ©2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump
Miscellaneous

All times are GMT. The time now is 06:05 AM.


Powered by vBulletin
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.