Appending text with dotNET

Become a member of the CGSociety

Connect, Share, and Learn with our Large Growing CG Art Community. It's Free!

THREAD CLOSED
 
Thread Tools Search this Thread Display Modes
  03 March 2013
Appending text with dotNET

Hello!

I'm looking for a fast way to write a lot of text lines in a file (several files actually).

Using maxscript's Filestream takes around 40 seconds for 32 files with over 720k lines total. This is a too long and I'm looking to optimize it by writing the text through dotNET instead.

I'm a noob at .NET, but after scouring the internet I managed to get it working by appending all my text lines in an array and using WriteAllLines
contents = #()
for v in vertices do append contents ("v " + v.x as string + " " + v.y as string + " " + v.z as string)
for vn in normals do append contents ("vn " + vn.x as string + " " + vn.y as string + " " + vn.z as string)
for f in faces do append contents ("f " + (f.x as integer) as string + "//" + (f.x as integer) as string + " " + (f.y as integer) as string + "//" + (f.y as integer) as string + " " + (f.z as integer) as string + "//" + (f.z as integer) as string)
(dotNetClass "System.IO.File").WriteAllLines myfile contents
This takes 25 seconds, which is still too much.

Options:
1. write each line directly in the file
This should speed up the process by eliminating the need to append an array. I also learned from experience that maxscript can get slow when using huge arrays, and the ones I'm working with end up having over 100k text elements.

2. Write floats and integers without converting to text
I've seen that the StreamWriter class has methods to write the text representation of values, which might also speed up the process.

I am clueless on how to implement these 2 ideas. Could anyone please help with examples on how to use this StreamWriter class?
 
  03 March 2013
If you're looking for speed, and do not require human readability, storing binary values is the way to go. Not to mention your file sizes will be much much smaller.

Consider "v0.111111 0.222222 0.333333" - 27 bytes of ascii text
vs a binary representation which would be 12 bytes.

You may even get decent results with the BinStream type in MAXScript.
 
  03 March 2013
Thanks for your reply!

I'm well aware of the advantages of binary formats Maxscript's BinStream gives more than decent results - I can write the same data mentioned before in a little over 1 second.

But for now I am forced to use a text format, and dotNET is my last resort (other than compiling my own plugin). If I can get it under 10-15 seconds it will be good enough.
 
  03 March 2013
Originally Posted by Chipicao: Hello!

I'm looking for a fast way to write a lot of text lines in a file (several files actually).

Using maxscript's Filestream takes around 40 seconds for 32 files with over 720k lines total.
...
This takes 25 seconds, which is still too much.
...
working with end up having over 100k text elements.
...


what 25-40 seconds are talking about?
here is sample for 1,000,000 lines and as you can see it takes 7s:

 num = 1000000
 fn saveTXTFile num:num = 
 (
 	ss = createfile (getdir #temp + @"\longtext.txt")
 	
 	for k=1 to num do format "% >> 0x%\n" k (formattedprint k format:"08x") to:ss
 		
 	flush ss
 	close ss
 )
 t1 = timestamp()
 m1 = heapfree
 
 saveTXTFile()
 
 format "time:% memory:%\n" (timestamp() - t1) (m1 - heapfree)
 

maxscript file stream is c++ written and it's pretty fast. using c++ (.net) you can speed up only number to string converting and string formatting operations.
 
  03 March 2013
You would also be wise to cache the format strings:

	local fmt = "% >> 0x%\n"
	local fmtNum = "08x"
 	for k=1 to num do format fmt k (formattedprint k format:fmtNum) to:ss


This will save only 5%-10% in time, but considerably less memory.

The reason your code might be taking so long is because you are using heavy string concatenation. This creates a new string for every operation. Use format as Denis demonstrated.
 
  03 March 2013
@denis 40 seconds for this, ran 32 times (32 files) for a total of 720000 lines
myfile = openFile (modelpath + meshname + ".oft") mode:"wt"

for v in vertices do format "v % % %\n" v.x v.y v.z to:OBJfile
for vn in normals do format "vn % % %\n" vn.x vn.y vn.z to:OBJfile
for f in faces do format "f %//% %//% %//%\n" (f.x as integer) (f.x as integer) (f.y as integer) (f.y as integer) (f.z as integer) (f.z as integer) to:OBJfile
close OBJfile

I also ran your code, and here's the result: time:35044 memory:20916104L

My PC is no slow poke: Core i5 2500K@4.2Ghz, 16GB DDR3@2133, intel SSD
But I believe what you say, so there must be something wrong on my side...

@lo as you can see I was already using format, but that's not the issue. There must be somethign wrong with my max or PC :(


EDIT: ran denis' code in max 2012, results are: time:3201 memory:10296616L
My code that took 40 seconds in max 2013 now runs in 2.5 seconds

So either there's something wrong with my 2013 installation, or this another 2013 fk-up.

Thanks again for all your input guys, and sorry for the trouble.

Last edited by Chipicao : 03 March 2013 at 05:43 PM.
 
  03 March 2013
Could it be that your heap size is very small and you're forcing max to allocate and collect lots of memory throughout the process?

Try setting a very large heapsize and see if it makes a difference.
 
  03 March 2013
Sorry, I edited. There's something wrong with my max 2013. I switched to 2012 and it's blazing fast.
I had already tried with a larger heapsize but results were the same.
 
  03 March 2013
Thread automatically closed

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.
__________________
CGTalk Policy/Legalities
Note that as CGTalk Members, you agree to the terms and conditions of using this website.
 
Thread Closed share thread



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
CGSociety
Society of Digital Artists
www.cgsociety.org

Powered by vBulletin
Copyright 2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump
Miscellaneous

All times are GMT. The time now is 02:15 AM.


Powered by vBulletin
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.