Looking for compositing workflow tips (AE)

Become a member of the CGSociety

Connect, Share, and Learn with our Large Growing CG Art Community. It's Free!

 
Thread Tools Search this Thread Display Modes
Old 04 April 2014   #1
Looking for compositing workflow tips (AE)

Hi,

so this might not be the best place to ask, but Im too lazy to register at yet another forum and I guess at least a couple of you must have to deal with the same issues as me...:

Recently Im doin more and more compositing work and Im having a hard time finding a suitable workflow for it.
We do all our compositing with After Effects CC, but filehandling alone is such a pain.
Recently we did a lot of stereoscopic compositing.
We had to use quite large files (16 bit tif sequencences in 4400*1020 px) and previewing those sequences without any effects applied alone was terribly slow.
Which in the end led to a lot of "shoot and miss" errors, where we had not enought time to preview a sequence before final rendering and ended up with stupid errors that could have been avoided.

I know of the possibility to render out proxies and I just gave it a try.
But rendering only one pass (RGB p.E.) for a sequence of 770 frames would take an hour alone, so doing this for all layers seems like no good idea.

Can you maybe share some of your workflow tips?

Like:

1. What file formats should I use for proxies?
2. Can I somehow automate the process of proxy creation?
3. What file format do you use for rendering out image sequences?
4. Any other tips for a faster performance inside After Effects?

Im also considering rendering out to a local folder of my Workstation first and only copy the files to the server after Im done with compositing, since I also feel like the server is slowing down things considerably...
 
Old 04 April 2014   #2
1. What file formats should I use for proxies?
I use png or tiff a lot, but any lossless image format is actually OK. Always check taht you don't lose any channel data or bit depth.
2. Can I somehow automate the process of proxy creation?
You can setup watched folders for Adobe Media Encoder.
3. What file format do you use for rendering out image sequences?
Any lossless format, never ever JPeg, never ever compressed Tiffs.
4. Any other tips for a faster performance inside After Effects?
Decrease the preview resolution.
You probably don't have a monitor sized 4000+ pixels anyway so half or even quarter resolution is OK with images taht large.
Get an internal SSD for image caching, buy as much RAM as possible, store playback data on a RAID or internal disk/SSD(unless you have a very fast network).

Last edited by scrimski : 04 April 2014 at 09:14 AM.
 
Old 04 April 2014   #3
I use png or tiff a lot, but any lossless image format is actually OK. Always check taht you don't lose any channel data or bit depth.


Well, I thought the idea of proxies was to reduce file size and thus increase playback performance? But of course jpegs wouldnt be suitable to do any serious comping like color correction or DoF, anything that relies on higher bit depth...

So I guess I would either have to
A) scale down the proxies?
B) do everything I need high bit depth before creating proxies from the layers?

I already realized that the server indeed is a big bottleneck right now, after copying the files to a local folder, creating proxies and performance while playing back/ doing ram previews increased 2-3 times.

Thats why decreasing preview resolution didnt really improve performance in the past.

My boss was already looking at a new server, but at 25k for the investment, it would be preferable to find some workarounds first...
 
Old 04 April 2014   #4
Watch folders are probably the way to go.
I have never worked with those before, could you help me get a grip on:

-how the watch folders are beeing watched? For example: Im rendering out a Tiff Sequence to a folder, which is defined as a watch folder: Does the media encoder start encoding right away once the folder has new files in it? How often does it check for updatet files in the watchfolder? Wouldnt make much sense if it started encoding everytime the farm has finished a couple of more frames...

-Most of the times I have a shot folder and inside a folder for each layer or render element.
Does the watch folder include all subfolders or do I have to specify them separately?
 
Old 04 April 2014   #5
Just checked and it's not possible with Adobe Media Encoder. Maybe Squeeze or another app can handle this. One possible workaround would be to create a reference quicktime movie and copy this into a watched folder, this can be setup as a menu item using Apple Script or Automator(on a Mac).
 
Old 04 April 2014   #6
I used to just copy the renders locally, and set those as my proxy. So when I'm working in proxy mode it just reads all the source footage locally, and when I render, it reads it from the network. Pretty simple to setup.
__________________
portfolio
 
Old 04 April 2014   #7
Originally Posted by scrimski: Any lossless format, never ever JPeg, never ever compressed Tiffs.


Why not compressed tiffs? they are as lossless as uncompressed ones afaik.
 
Old 04 April 2014   #8
W
hy not compressed tiffs? they are as lossless as uncompressed ones afaik.
Yes, but the CPU will need to uncompress them while reading.
 
Old 04 April 2014   #9
Should probably steer clear of tiffs anyway - the huge amount of differing compression formats makes it the most error prone still frame format... but in saying that, the compressed/uncompressed argument basically comes down to this, do you save more time in the transfer of frame to application than you lose in CPU cycles to uncompress the frame? - so over a network its possibly best to use some lossless compressed format, if they are local uncompressed is probably best.

Depending on whether after effects can load OpenEXR tiles or scanlines you might get some read speed improvements in switching to that (although in my experience AE and exrs have been pretty poopy, but havent used AE CC). And certainly if you are dealing with frames with a lot of 'empty space' like elements, rendering exrs with data regions will accelerate frame loading hugely.

If your network is your main bottleneck, a really simple solution is to sync your rendered frames locally. Theres a lot of free solutions out there, at home I use 'free file sync' to mirror my network rendered frames onto a drive on my compositing machine. Once its copied the first time, you are just copying a frame when they spit off the queue so its pretty much instantanious (and also gives you a backup of frames should something go wrong).

Another suggestion would be to switch to Nuke, nuke loves big frames, especially region rendered scanline EXR's
__________________
--------------------------------
animationsbychris.com
 
Old 04 April 2014   #10
Yeah i dont use tiffs for animations anyway, its a great format for print though.

I agree on everything Chris said.
 
Old 04 April 2014   #11
Thanks for all the input guys, I really want to improve our compositing workflow, because its one of the most frustrating things not having time to do ram previews in AE and thus preventing stupid errors that could have been easily fixed.

@Chris:
You said to stay clear of Tifs, whats the alternative then?
TGA?
Because Exrs, as you said, are pretty poopy in AE.
We did consider switching to nuke at some point (especially since Adobe introduced its new pricing policy), but we just still have too many projects that rely on AE, that we need to update regularily, so wed either have to convert them all for Nuke or keep paying for AE...

Ill research the sync option aswell as the compressed tif way and Ill see what gives me the bigest performance boost.
Right now I really have the feeling, that file size is the biggest bottleneck.
40Mb per frame (Stereoscopic full HD, 16 bit) vs 20Mb/s transferring speed...you do the math...

I also posted my wishlist on the chaosgroup forum: Being able to create low res jpg Proxies during rendering would get rid of all the problems at once...
 
Old 04 April 2014   #12
Did you take a look at the ProExr Plugin? Might be well worth the money.
 
Old 04 April 2014   #13
The updated ProEXR plugin is included in After Effects, but I didnt check it yet. Ill look into that aswell.
But I still think the transfer times for the files will be the biggest bottleneck and EXRs wont be smaller in filesize, especially if you save all elements at once.
 
Old 04 April 2014   #14
png's are pretty good, they support 16bit from memory, and compression. Targas wont give you 16bit, but do support compression, and are a great 8bit format.

Yeah I would imagine loading those massive frames over the network is probably 90% of your issue. I mean, they are still big but should come in pretty quick coming in off a local hard drive (or even better an SSD) - you could also look at a cheap raid solution maybe?

I'm running a comp at the moment here with 35meg frames, 6 layers off the SSD and its not giving me too much grief
__________________
--------------------------------
animationsbychris.com
 
Old 04 April 2014   #15
I didnt have time to test all the options yet, I just installed free file sync and it seesm to be a good idea to start from here - once Im done deciding with wich file format to go.

Any more opinions on that?
 
reply share thread



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
CGSociety
Society of Digital Artists
www.cgsociety.org

Powered by vBulletin
Copyright 2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump
Miscellaneous

All times are GMT. The time now is 11:12 PM.


Powered by vBulletin
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.