Pipeline setup

Become a member of the CGSociety

Connect, Share, and Learn with our Large Growing CG Art Community. It's Free!

Thread Tools Search this Thread Display Modes
Old 05 May 2013   #1
Pipeline setup

Hi guys,

I would like to ask you for your advice. I tried searching the forum but I didn't really find an answer. I work at a small studio, 3 people at the moment working mainly in commercials. The main packages we use are C4D, Maya and After Effects and Photoshop. To this point we never really had rules for naming convention or directory structure. But now it seems we will be growing in number of employees aswel in the size of projects and the necessity of an organized workflow is apparent.

At this point all the project folders are local. So everyone has its own way of setting up the project directory and naming the files. Textures and assets are local aswel. If we work with multiple people on a project we copy the files over to each others computer back and forth. This way it is obviously very difficult to keep track of the latest version of the files and reconnecting the textures is extremely time consuming.

So I was wondering how you guys handle these things. So to sum up some questions:
  • How do you handle and control versions amongst various local computers. I looked into version control software but it seems to focus on programming and not so much on animation. They don't seem to have an easy to use interface either.

  • We have a small nas server. A qnap ts-219 to be precise. Would it be fast enough to store all the textures and assets and possibly write the renders to it. So basically try to centralize the assets as much as possible. But I'm worried the server isn't fast enough.

  • What naming conventions and directory structure do you use. I read about creating the structures through scripts. How can we approach that. We work with os x. I know the basics of python but I'm by no means a programmer. Naming convention is of course very personal but I would love to hear your opinions about it.

I'm sorry for the long post but I've been researching the whole day to get ideas for a structured workflow. I'm not expecting you to present your workflow on a silver platter but I'm just interested in your ideas. I figured we won't be the only studio who runs into this.

Anyways thanks in advance!
Old 05 May 2013   #2
It should be enough, it has a 1-gigabit LAN port. Just connect it to a switch for all the machines to share. At most files will be tens of megabytes, or perhaps in the hundreds for large textures/ image sequences. But you can work smart, and pull the files to work locally, then at the end of the day, or at the end of a complete task just send the files back to be centrally stored.

I would definitely recommend investing sometime in using software like SVN in conjunction with a tool like Shotgun - you can use the Shotgun API to have all SVN logs updated on Shotgun, which is rather helpful.

As for a central storage system, you could probably even get away with using a cheap PC with lots of internal storage running Linux to act as a super-fast file-storage device. Just stick a dual-gigabit Ethernet card in the back, and have that run to a switch.

As for the pipeline; I have done research, and still am researching for my own pipeline/ workflow. The most important aspect (in my opinion,) is to just try and get everything as standardized as possible; definitely use scripting when possible to automate tasks such as folder structure creation. This shouldn't be too hard though, a few lines of code just to automate the process for a select number of scenarios I.e. creating a master file structure for a project, creating a new scene folder structure, etcetera.

Spend a bit of time with your colleagues understanding how you's work, what formats are used, what sort of data is being passed around, and then from that you's can develop an overview of how you's are generating data, where it is going, and in what format. That will form the basis for your pipeline design. From that you's can make standardized folder structures, standardization for format interchanges (very important,) and overall just understand to a better degree how you guys can make it easier to slot in new staff, new applications, and processes.

One thing that I always try to focus on when developing a new workflow is; how do I keep it open for creativity, but closed for standardization?

Boxing yourself into a super-strict workflow won't help, it will stifle creativity - you may work differently to your colleague, so it really needs to be about the data that is moving around, not the actual creative process. You may already know that, but I found out the hard way.

Take a look at these images:

The first two diagrams were my first attempt at designing a pipeline for a project I was working on - way too complex, and more of a hindrance.

The second diagram though, is much simpler and basically focuses on a very abstracted version of the pipeline, just covering the general flow, and file-formats. I need to do more work on it, but it is rather helpful starting point.

I dabble...
Old 05 May 2013   #3

Thanks a lot for the complete reply. I really appreciate it. I agree that developing the pipeline will take time and trial and error.

I've looked into Svn but it is still a bit unclear to me. It is all gibberish to me at this point. Do reckon it is difficult to set up? I will have to investigate a lot more. Same as for setting up a pc with linux to use as a file servers. Never even touched linux in my life.

The thing is that this stuff is a whole new world to me. I'm a 3d artist and not a system operator. It just takes time to learn all the new stuff and set up a consistent pipeline.

Anyway thanks again and I'd love to hear more opinions on the matter.
Old 05 May 2013   #4
An open-source alternative to Shotgun is Tactic (http://www.southpawtech.com/tactic/). I haven't used it, but it seems quite full featured.

As somebody that doesn't know software coding either, I can only advise keep neat, predictable folder structures and strict naming conventions. The approach I tend to prefer is two main directories; Development and Published. The structures of these are mirrors, but Development will have the WIP files while Published receives only final textures, models etc. This has been a feature of places I've worked, that also tend to be light on tech-staff, so a lot of asset management was left to artists themselves rather than automation. Effectively, keeping final assets away from in progress assets means artists can work without fear of overwriting something being sourced for render. When publishing, define the necessary stages (eg modelling, UVed, textured, shaded, rigged) and publish as that stage is met, so that work can be referenced in and updated as completed. Define standard abbreviations for names and enforce those strongly (most programs give you enough info that you can cobble together a quick script in the script editor if you need an action performed on multiple similar objects, but it will rely on strong naming conventions).

As DanielWray alluded to, try to keep the system adapting to the artists needs and not the artists adapting to system. If the system doesn't do what people need it to, they'll find a way around it and it will break. Test a lot of things too before you decide on stuff. The last project I did in Maya used file-referencing, which is awesome, but constraints wouldn't work (this may be fixed now). We had to get a third-party plugin to correct a fundamental flaw. On a more recent project, if we were rendering to and sourcing textures from the same server, everybody would slow down to a crawl so we had to get another server going. Test everything

And if you can, hire a programmer to create you some tools. For a project on a big enough scale the time you save by automating what you can can save you days or weeks or just let you focus on what really needs your attention.
Visual Effects Artist (of sorts)
Old 05 May 2013   #5
If you're looking for a non-scripted solution what a lot of small shops will do is set up a Default directory structure that you can just duplicate and rename to whatever you need.

Within the default structure you tend to have things like:
  • dailies
    • YYMMDD
  • input (stuff from client)
    • YYMMDD_whatTheySent
  • output (stuff you send to client)
    • YYMMDD_whatYouSend
  • reference
  • assets
  • temp
  • SHOT### (or sequence, or however you divide up the project)
    • 3D
      • V###
    • 2D
    • Other ways of dividing up the working files
    • Renders/Comps
      • V001
      • V002
And so on. The above is just an example and everyone tends to work a little differently. A naming convention is really useful, saves you time looking for things.

Some people divide the work up not into things like 3D/2D/AE/Maya but into people Wil/Sofia/Jack which works in a small office reasonably well too.

Just some thoughts.
Critcal feedback example #62: "Well instead of the Stalinist purges and the divorce and the investigation ... it could be about losing a balloon."
Old 05 May 2013   #6
Thread automatically closed

This thread has been automatically closed as it remained inactive for 12 months. If you wish to continue the discussion, please create a new thread in the appropriate forum.
CGTalk Policy/Legalities
Note that as CGTalk Members, you agree to the terms and conditions of using this website.
Thread Closed share thread

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Society of Digital Artists

Powered by vBulletin
Copyright 2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump

All times are GMT. The time now is 03:10 AM.

Powered by vBulletin
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.