Recommended Sticky - Do not export compressed from AE

[dell]

Great advice,


Thanks

[jussing]

Hear, hear.

I'll shamelessly plug my Quicktime compression tut here:

Quicktime (Pro) Codec Settings (http://forums.cgsociety.org/showthread.php?f=76&t=180825)

Of course there are endless possibilities depending on needs, but these are the settings I use for downloadable/streamable demo reels etc.

- Jonas

[spacefrog]

i would change that "always render uncompressed" to "always render to a lossless codec"
- here are a few ( but not a complete list ! ):

if it's QT:

* use PNG,TGA or TIFF,or if installed "Techsmith's Ensharpen"

if it's AVI :

* HuffYUV 2.1.1... (great and very fast, has probs over 2GB !)

* AlparySoft .... (google it! ) Great , free Beta

* MorganMM JPG2000 has lossless setting

* Techsmith Codec... initially for screencapture, but has good compression on flat /cel-shadedlike clips, very slow when moving playhead (eg. in your editing suite)

* PicVideo (www.jpg.com) has "Losless JPEG" and "Wavelet 2000" Codecs


As for Virtualdub - which doesn't read WMV and MPG2 - look for "Virtualdub-MPEG2" - or simply google for "Avery Lee" and "fccHandler" which does all these things ;-)

[beenyweenies]

i would change that "always render uncompressed" to "always render to a lossless codec"
- here are a few ( but not a complete list ! ):


Sorry Spacefrog, but I completely disagree with your posted suggestions. The point in rendering to an uncompressed format is to have a FULL QUALITY, archive master of your work. None of the codec choices you list are even remotely suitable for this task:

- Techsmith Ensharpen - This codec CAN perform similar to the animation codec when no gradient information is present (for example, desktop screen captures). But for most motion graphics and ALL video, this codec is not a good choice at all. It is also non-standard, meaning most apps will not read it properly and, should you need to share the file with other studios or artists there is a 98% chance they will not have this codec.

HuffyYUV, AlparySoft, etc - None of these are good codecs for any portion of the video production process, much less archival or masters. The main reason is that they are completely non-standard and probably 5% of all computer users, even in this industry, will even have them installed. People need to accept that, while there are 8,000 boutique codecs out there, the vast majority are NOT suitable for video production! Go ask 20 studios if they use these codecs, and I bet you not one of them will say yes.

- JPEG2000 - Is a decent option, but again isn't standard yet and many apps have issues working with this codec, making it a poor choice due to its limitations. This may change in the coming years, but for now I wouldn't use it for anything.

- VirtualDub MPEG2 - Mpeg2 is massively compressed and is probably one of the worst choices available for your masters.


A lot of people out there who are used to ripping DVDs and the like have become accustomed to these whacky codecs and workflows, but they MUST be abandonded if you plan to be a video artist. In professional video production, stability and quality are the two chief concerns and these boutique codecs offer neither. I assure you, the first time you pass off a file to a client in DivX, HuffyYUV etc. you will get a look that makes it clear what they are thinking of you - "This person is an amateur."

[Mylenium]

i would change that "always render uncompressed" to "always render to a lossless codec"

Nope, absolutely not. Your thinking is crooked in that it would only have real value if we are strictly sticking to an 8bpc RGB/ 10bpc YUV workflow. But hey, this is 2007, who does that anymore? Even so, it becomes a question of which quantization and color space transforms CoDecs employ and neitehr of the choices you provided gives any clear and predictable results in that regard. The only reliable formats that can truly count as "lossless" are

a) uncompressed image files that
b) maintain the native color space,
c) do not impose wrong Gamma profiles,
d) can hold additional color management information/ color profiles and
e) ideally support larger bit-depths.

From within AE this limits it pretty much to TIF, PSD and OpenEXR. You are mostly proposing non-standardized formats and procedures which is absolutely wrong and terrible to do. Can you even remotely assume that let's say HuffYUV will have a version compatible with the then-successor of Windows Vista in 6 or 7 years? You can't and therefore anyone would be ill-advised to follow your procedures.

Mylenium

[spacefrog]

I thought you were talking about rendering here and not archiving,
the talk was about multiformatconversions too....not about delivering the things to a different studio or whatever....

if you keep all your AFX rendered output for 7 years... well then you should better find formats that you will be able to use than, and if you need to keep an old rusty XP workstation...

i always thought only the source material and footage and the projectfiles are worth getting archived...and of AFX output mostly as an intermediate ... at least in the long term....

PNG, TIFF (not JPG o'course), TGA are 100% losless - all can be encapsuled in Quicktime...

you are right huffYUV does colorspaceconversions,

MJPEG2000 lossless does wavelet encoding and does it lossless...


I mentioned VirtualDubMP2, cause it's listed in the initial post as tool to do conversions,
Of course it's not a codec it only was mentioned as a tool,MP2 is lossless of course

but i see - uncompressed everything is far better - and i'm a whacky DVD ripping loser ;-)

[beenyweenies]

i always thought only the source material and footage and the projectfiles are worth getting archived...and of AFX output mostly as an intermediate ... at least in the long term....
That's mostly how I see it too. When I'm talking about Masters and Archives, I am including the renders out of After Effects that become source material for the edit, etc. since they must be of the highest quality.

PNG, TIFF (not JPG o'course), TGA are 100% losless - all can be encapsuled in Quicktime...
I agree, but the issue presented here is that "lossless" does not mean "no loss." There isn't really such a thing as 100% lossless. That would mean they 100% throw away only some information, which doesn't make much sense. Lossless codecs throw away as little information as possible, but they ARE still throwing information away.

It really comes down to the source material you are using. An apt comparison is GIF vs. JPEG. GIFs are great for encoding blocky images, like solid colors or aliased text. Encode a photo or anything with gradients in GIF format and they will look horrible and the file will still be huge. Solids in JPEG format inherit a lot of pixelation and don't look as clean as they could, but JPEG works great on photos and other gradated material.

In the video world, lossless codecs are like GIFs - they work well on solid blocks of color and mildly gradated material, but are throwing info away in favor of file size. Uncompressed works great on video footage.

A good test is to create a white solid and output a one minute version of it in both uncompressed and Animation codec. They will look identical, but the Animation codec version will be far smaller. Now run some video footage through the same test. The file sizes will be almost the same, but the Animation codec footage will have tossed valuable data in the process.



but i see - uncompressed everything is far better - and i'm a whacky DVD ripping loser ;-)

Uncompressed everything IS far better, I assure you. It's the way all modern studios work. As for being a "wacky DVD ripping loser" I hope my comment upstream didn't offend you and I SURE didn't mean to imply anyone is a loser. It just seems that a lot of people here who have codec problems originated in that crowd, given the tools they were familiar with such as VirtualDub. It was an observation, not an assault.

[spacefrog]

I agree, but the issue presented here is that "lossless" does not mean "no loss." There isn't really such a thing as 100% lossless. That would mean they 100% throw away only some information, which doesn't make much sense. Lossless codecs throw away as little information as possible, but they ARE still throwing information away

this is wrong !

lossless is lossless - only case that they are NOT lossless is if they do some lossy colorspaceconversion ( eg. YUV kind of codecs) during the compression - if they don't do this - and use a lossless compression algorithm - they are 100% lossless. Of course if your output uses 32 bit per colorchannel you got to use a format that supports this...

eg. PNG is 100% lossless - so is TGA and most TIFF compression options ( LZW etc...)
and many other lossless codecs are 100% lossles - how do you expect ZIP,RAR and so on being able to compress programmcode - where a single switched bit would cause a crash ?

JPG or MPEG on the other hand is LOSSY- cause it is using FFT (Fast Fourier Transformation) and than data reduction in the frequency band -> which means throwing away least significant data

this is Shannon's Information Theory at work - Information Entropy - if you get 1000 zeros in a row, don't store 1000 zeros - but store the value zero and the count of zeros instead - and you got a compressionratio of 1000:2 - easy isn't it ?

and NO data loss happens.... every bit of output is the same as the input
You can get GOOD compression ratio's without missing a single bit of information....

and i never was talking about things like the crappy old Quicktime "Animation" codec - holy lord ....
and the thing talking of me being a loser - it was rather a joke ....
but it seems because there are these kind of people out there ( using DVD ripping all codecs inclusive...) who do not understand what codecs do to the material, people here in this thread seem to look at codecs as being some kind of sickness in general....

Of course i understand that in a big studioenvironments , the usage of codecs without coordination is a bad thing - but ´since i'm a freelancer doing many things for myself on my workstation and delivering end-products to the customer or a third-party on the end of the project most of the time only - performance is far better when you do not have to shuffle 100's of Gigs arround evertime you render little changes...

throw severall nested layers into aftereffects - each layer loads its content from disk -> it'S far better to have small filesizes there if you havent got a whole RAID dedicated to every layer...

[beenyweenies]

eg. PNG is 100% lossless - so is TGA and most TIFF compression options ( LZW etc...)
and many other lossless codecs are 100% lossles - how do you expect ZIP,RAR and so on being able to compress programmcode - where a single switched bit would cause a crash ?
My post was HORRIBLY worded, I was distracted with work when I wrote it. I didn't mean to imply that there was significant visual loss of information with lossless codecs, I only meant what you have said here (much more elegantly than I). The point of lossless codecs is to toss information where it can without degredation to the image, but there IS loss of information happening. On this issue we are in agreement.
and i never was talking about things like the crappy old Quicktime "Animation" codec - holy lord ....
The Animation codec is just as good as any of the other lossless formats from everything I've seen and experienced. It shows no visual loss whatsoever in "difference mode" tests.

But, as stated in my original post, quality loss is not the only issue here, there is still workflow and interopability. For example, while PNG is lossless and files are fairly small (I use it for 3D renders all the time), it is not a GREAT option simply because moving it between Mac and PC causes gamma shifts thanks to its seemingly broken method of using embedded color profiles. This is a deal killer should you EVER need to share the file between platforms. TIFF sequences are great, and are a recommended file format for visual effects work and transferring throughout the production pipeline. They can also be pulled into Quicktime pro to create a TIFF encoded Quicktime, which is much easier to handle than 10,000 frames.

I understand that you are a freelancer and work your own way until you ship out a file to the client. Ultimately, that is the motive for MOST people posting here who use a litany of odd codecs and workflows. The problem is, are you always going to be a freelancer? Are you freelancing for studios that will expect to receive all of your source materials at the end of the job, and may be dismayed to find the files in various non-standard codecs, etc.? And finally, when the majority of studios gravitate towards certain file types and workflows, it can be assumed that there is a good reason for that, primarily the combined decades of experience driving their decision. That experience should be heeded in my opinion, regardless of where, when or how you work.

[jussing]

I just want to say I have on many short film and commercial projects, including all my own animations and showreels, used Quicktime Animation at 100% as master, and have been very happy with it. I have once had an issue with scrambled frames, but setting the keyframe count to 1 instead of the default 15 resolved it, so that's what I work with now.

On my latest 15 minute short film project, which I am onlining as we speak, I am using a TIFF sequence as master, but nearing the end, I have discovered that my external disk drive simply can't contain that many files! I know, I know, ZIP them. But that means I won't be able to read them directly off the disk as a file sequence.

- Jonas

[thedesolateone]

what about canopus procoder? would that cound as a good converter software?

[beenyweenies]

what about canopus procoder? would that cound as a good converter software?

Procoder is a highly recommended conversion tool. I use it frequently.

[avinashlobo]

The point of lossless codecs is to toss information where it can without degredation to the image, but there IS loss of information happening. On this issue we are in agreement.I think you have this wrong. In fact, I don't think spacefrog agrees with you at all. That's not the point of lossless at all. The term 'lossless' can only be used when the exact, original data can be reproduced from the compressed version. Bit for bit. Which translates to ZERO data loss/degradation.

[jussing]

Well, I'd say "lossless" was "no loss", too. But I'm open to new ideas. :)

Beenyweenies, what do you mean when you say "without degredation" at the same time as "there is loss of information"?

Exactly what information is lost if there is no degretation?

- Jonas

[beenyweenies]

I think you have this wrong. In fact, I don't think spacefrog agrees with you at all. That's not the point of lossless at all. The term 'lossless' can only be used when the exact, original data can be reproduced from the compressed version. Bit for bit. Which translates to ZERO data loss/degradation.
Beenyweenies, what do you mean when you say "without degredation" at the same time as "there is loss of information"?

avinashlobo, you are assuming that data loss and visual degredation are tied together, which is the wrong way to think with regards to codecs - their entire existence is to remove data where possible while degrading the image as little as possible. Lossless codecs are no exception, the only difference is they will not toss information that will visually degrade the image (as I explain below). You are also overlooking one major thing - Lossless codecs actually claim to be visually lossless, not 100% bit-for-bit dupes of the original file. That would be an uncompressed file, NOT lossless.

Think about it this way. If you render a one minute HD-sized video of nothing but a black solid out to the Quicktime Animation codec, the file will be very small. Render one minute of video footage at HD size to the Animation codec, and the file will be utterly huge.

The reason is that, with the black solid video, the codec can toss pretty much everything except the information needed to create a single black pixel, which is then used to describe the entire frame since it's made up entirely of black pixels with no variation. Render the same movie but with three different color solids, and the codec only needs to visually describe three different colored pixels. With the video footage render, almost every pixel is unique (especially from frame to frame) and therefore the codec cannot toss information without altering pixels (resulting in visual loss of quality), so the file is very large as a result.

Therefore, it IS tossing information, but only where it can without visually degrading the image. If you compare the Animation codec version to the original, they are visually identical (and placing one over the other in "Difference" mode within AE would confirm this) BUT information was tossed. Were it not for lossless codecs operating in this way there would be no need for the lossless algo, only uncompressed and lossy codecs!

VISUAL LOSS is the keyword here - in the technical documentation lossless codecs only claim to be visually lossless, which is a massive distinction from a bit-for-bit duplicate of the original.

[jussing]

Ah, now I see the misunderstanding as clear as day. :)

avinashlobo, you are assuming that data loss and visual degredation are tied together, which is the wrong way to think with regards to codecs
Well, that is EXACTLY the way to think about data compression (also with regards to codecs).

Beenyweenies, you are incredibly clever about codecs, but you've got the wrong idea about the meaning of "lossless compression".

"Lossless" means that all original data can be restored. Just like when you compress to a ZIP file. -You get a smaller file, but you can unpack ALL ORIGINAL DATA without loss.

When talking about video codecs, the data is pixel data. And if a file is compressed with NO VISUAL DEGREDATION, that means ALL PIXEL DATA was restored. And that means NO LOSS.

Your example with the black image video file is a perfect example. A "lossless" compression WILL result in a much smaller file, but that doesn't mean any information was lost, it just means the information was re-organized in a more optimal way, resulting in a smaller file, which is exactly the purpose of lossless compression.

Ie, instead of saying "black pixel" 1,000 times, it's shorter to say "1,000 black pixels". That's lossless image compression in a nutshell.

- Jonas

[avinashlobo]

Yeah, Jonas is right. Lossless works like a zip or rar file. There is NO tossing of data. To put it in a very, very crude way - there is substitution of repetitive data with smaller data (refer to Jonas' example in his post). When required, the exact, original data can be reproduced from the substitute data. And it is perfectly exact. There is no shifting of bits or any alteration to the original data.

What you, Brendon, are describing is, in fact, high-quality LOSSY compression, where there is no apparent VISUAL degradation, but there is still data degradation. For example, save a file out as a maximum quality JPEG and keep it next to the original uncompressed version. The JPEG will be a fraction of the size, but there will be no VISUAL distinction between the two, unless you're gifted with superhuman eyes.

[beenyweenies]

Sorry, but you guys are both stuck on the idea that lossless is some bit-for-bit exact replica, when it just isn't. The point that data loss and visual loss aren't tied together is also getting lost in the mix here.

As was mentioned, if you replace 1,000 instances of the words "black pixel" with a single instance of "1,000 black pixels" you have just thrown out 999 entries and thus reduced the size of the file without impacting the resulting image. It is tossing data, 999 entries to be exact, otherwise the file would not be smaller. It's not like codecs are some magic suitcase that stuff the exact same bits into a smaller package, they merely consolidate the information and toss the redundant stuff. The resulting file is not a bit-for-bit dupe, it has been completely reorganized and redefined, the codec just knows how to read this new file and produce the same visual result. Because of this, the RAR/ZIP analogy is incorrect. It is more akin to a vector-based artwork, in which the program stores vector data rather than pixel data to reduce file size.

Visual loss is the shifting of pixel values to brute-force match them to the definition "black pixel" and thus enable the codec to roll them into the compression scheme, "1,000 black pixels." Since you can throw away data without resorting to this, DATA loss (or "restructuring" if you prefer) and VISUAL loss are not tied together as I stated.

[jussing]

Hi again. :)

Sorry, but you guys are both stuck on the idea that lossless is some bit-for-bit exact replica No, obviously the COMPRESSED file is not a bit-for-bit replica, we never said that.



It is tossing data, 999 entries to be exact, otherwise the file would not be smaller. No, since the INFORMATION (data) about the 1,000 black pixels is still 100% intact, nothing is tossed or lost, it's just being re-described in a shorter way, which is exactly the way ZIP/RAR etc. compression works.

Tossing data would be if you had a white pixel in there, and you "left it out", and kept saying "1,000 black pixels" instead of "500 black pixels, 1 white pixel, 499 black pixels". THAT would be toss & loss.

"Loss" in terms of compression does NOT mean "a smaller file size", because obviously that's what "compression" means. "Loss" only refers to if you lose something in the compression or not, ie, if you are unable to restore all original data or not. If not, you have LOST data, and the compression is LOSSY.

Lossless data compression is a class of data compression algorithms that allows the exact original data to be reconstructed from the compressed data
http://en.wikipedia.org/wiki/Lossless_data_compression

Cheers,
- Jonas

[beenyweenies]

No, obviously the COMPRESSED file is not a bit-for-bit replica, we never said that.Actually it WAS said upstream that lossless is the same file, bit-for-bit, that's why I mentioned this.
No, since the INFORMATION (data) about the 1,000 black pixels is still 100% intact, nothing is tossed or lost, it's just being re-described in a shorter way, which is exactly the way ZIP/RAR etc. compression works.When you remove data from a file, that data is lost EVEN IF it is being accurately redescribed elsewhere. NO ONE is arguing whether or not the resulting video is visually identical to the original, but data is being removed and then redescribed in short-hand. How is that not data loss? Honestly, I don't really care to debate the semantics of whether removing data from a file is "loss" if there is no negative impact, all that matters (as far as this thread is concerned) is what conditions should be met when choosing one codec over another.

Lossless codecs rely on being able to toss/redefine data in order to keep file sizes small which is the ONLY value in using a lossless codec instead of uncompressed. Lossless codecs can perform very well but ONLY if appropriate source material is being used.

[jussing]

Aye, it's semantics indeed. But the definition is clear, check the Wikipedia article. Or any paper on lossless compression algorithms. :)

When you remove data from a file, that data is lost EVEN IF it is being accurately redescribed elsewhere.No, that's exactly what's false. :) If you re-describe it elsewhere in the same file, the data is, by definition, not lost. How can data be lost when it's still there??

In the example of "1000 pixels", the data of the 999 instances of "black pixels" is not lost at all, it's right there in the phrase "1000 pixels". Reducing it to "1 black pixel" would be data loss, because then the information about the 999 pixels is entirely gone.

If your definition of "loss" was true, then the very term "lossless compression" would be a contradiction in terms, because a "lossless" file wouldn't be compressed at all. (because you say if the file is smaller, something is lost. Which goes against the very idea of compression like ZIP)

COMPRESSED means smaller file size
LOSSY means when you unpack the file, you don't get the same result as to begin with (JPG, MPG, DIVX etc)
LOSSLESS means when you unpack the file, you get the same data you started with (ZIP, RAR etc.)
I quote Wikipedia again:
Lossless data compression is a class of data compression algorithms that allows the exact original data to be reconstructed from the compressed data
And that's the truth. :)

Cheers,
- Jonas

[avinashlobo]

I can see where you're coming from. You're saying the losslessly compressed file does not contain the same data as the original uncompressed file. And you're right!

What WE are saying is that the losslessly compressed file DOES contain the same data as the original uncompressed file. And we're right too!

I suppose this is really a matter of perception; depending on how you want to view the compressed file.

But coming to the crux of the matter, a properly written lossless codec performs EXACTLY the same as an uncompressed file. This cannot be disputed.

The zip/rar analogy holds as well because the behaviour is the same if the appropriate front-end is provided. This can be best described when you consider Windows XP's ZIP folders. All the decompression/recompression happens behind-the-scenes when you edit the zip folder, but the contents (while still being compressed) will match the original data exactly

This is where we're coming from by saying that there is NO loss because while there is the lossless wrapper around the original file, the required data is decompressed/decoded on the fly, edited and then recompressed and put back in the lossless wrapper without any interaction required.

[beenyweenies]

There is NO loss because while there is the lossless wrapper around the original file, the required data is decompressed/decoded on the fly, edited and then recompressed and put back in the lossless wrapper without any interaction required.Not to keep arguing but this is wrong. Lossless codecs are not some magic suitcase that can stuff 1,000 lbs. of s...t into a 500lb sack. There is no lossless "wrapper" and there is no "original file" once it has been compressed to lossless. The video information fed into a lossless codec is permanently, irrevocably altered and redefined to a smaller, compacted state. On decompression the codec CAN translate this information into an expanded state so that the video player understands it, but your original file no longer exists, it has been redefined. In the case of zip files, the decompressor can translate the compacted file back into a 1:1 duplicate, but NOT because the original file was "stored" inside the codec - it merely expands the compacted definitions back into full definitions and creates a new file based on this.

As for this whole argument about whether some form of data tossing/loss is happening, have either of you even considered the notion that the Animation codec, for example, has a quality slider ranging from "best" on down? Think about it. The codec in its "best" state delivers a 1:1 quality because it only tosses/redefines what data it can without affecting quality. At the lower settings, it begins tossing more data, resulting in quality loss. As such, my point stands that the codec is selectively removing data. In its best form it is just limiting the degree to which it does this.

For the purposes of this thread, all that matters is for people to understand that lossless codecs rely on certain types of video data being fed to them to work well. People often complain that Animation codec files are huge, and my response was that this particular codec performs very well when the appropriate subject matter is fed into it. Give it a video file and the file will be remarkably huge. Give it the type of subject matter it was designed to efficiently compress (such as blocky graphics) and the file will be very small while still looking great. What is at issue here is when to use which codec, depending on the circumstances. Upstream someone claimed that lossless files do not throw away any data, and by my definition they DO, that is how the file size is lowered - by removing entries and replacing them with more compact definitions. When people understand how the codec works, they will know when to use it vs. uncompressed and that was the only point of my comments upstream.

I suppose we can end this by agreeing on the following - lossless set to "best" quality is visually identical to the original file, although data has been removed and consolidated in the interest of reducing the file size. Agreed?

[jussing]

Hi again. I'm sorry, I like continuing this. I respect you very much Beenies, and I certainly don't want to continue till we rip each other's heads off. I might stop soon. But for now, I'll have another go. Sorry. :)

Let me begin again by saying that if your definition of lossless was true, there would be no such thing as "lossless compression", because "compression" by your definition means "tossing" and "losing" data. So compression, by your definition, can't be lossless.... explain that?

Lossless codecs are not some magic suitcase that can stuff 1,000 lbs. of s...t into a 500lb sack.Whenever possible, that's exactly what it is, yes. JUST LIKE ZIP. It's a means of discribing THE SAME DATA in a shorter way, so no loss is happening.

You can also ZIP your video files to a smaller file size. That's also lossless compression, the exact same principle. The difference between ZIP and, for instance, Quicktime Animation at 100% is that the Animation algorithm is specialized for image data, so you will get better results than with ZIP.

And with that, I'm beginning to understand your misunderstanding... I have a feeling I know what you're going to say to the ZIP example: You're going to say that it can't be compared, because with ZIP you unpack the original EXACT file, right? Whereas with animation codec you never return to the original file?

Well, that's because the original IMAGE DATA is not a file. IMAGE DATA is just a series of pixels, no matter if you put it in a gif, png, targa, in your mind, or on post-it notes. THE PIXEL DATA is the original master. The uncompressed (huge) file is alraedy just a container, it only REPRESENTS the master (proof: you can convert it to another uncompressed image format, which gives you completely different bit sequence, but the exact same image data - that's also lossless, even though all the bits are different). Put that file through lossless compression, and you get a much smaller file, which is also just a container. Play that file, or load it into compositing or editing, and you restore THE ORIGINAL PIXEL DATA on your screen, compositing app or whereever. The original file? No, you don't restore that, but who cares, your source data is pixels, not a file. If you can restore your original pixels, you have had NO LOSS. If you had loss, there would be "damage" to your image (compression artifacts).

:)

It works like this:

Master data -> Compression algorithm -> Compressed file -> Uncompressed algorithm -> Master data

With ZIP and program files, that would be:

QUAKE.EXE -> ZIP algorithm -> QUAKE.ZIP file (smaller) -> UNZIP algorithm -> QUAKE.EXE (perfect match to original file)

And, with video data:

ORIGINAL PIXEL DATA (in any container!) -> Lossless codec -> VIDEO FILE (smaller than if uncompressed) -> Decoding (playback) -> ORIGINAL PIXEL DATA

Of course, as with the ZIP example, lossless compression only offers a limited effect, depending on the data. As you said, a live video signal will be much less compressed that a blank image or one with large blocks of solid color. And that is paramount in the understanding of lossless compression. It's called entropy (http://en.wikipedia.org/wiki/Information_entropy#Data_compression) - a certain amount of data can only be compressed so much without loss. That's why lossless compression doesn't have a constant bitrate, it's inevitably impossible.

As for this whole argument about whether data LOSS is happening, have either of you even considered the notion that the Animation codec, for example, has a quality slider ranging from "best" on down? Yes, the animation codec is only lossless at 100%.

Again, I'll quote Wikipedia (http://en.wikipedia.org/wiki/Lossless_data_compression):
Lossless data compression is a class of data compression algorithms that allows the exact original data to be reconstructed from the compressed data.


And, with video compression, data = pixel data

Cheers mate,
- Jonas

[jussing]

OK, reading through the whole thing again it strikes me that we are actually very close to total agreement. The only thing we disagree about is, as I was in on in my last post, whether or not the SOURCE DATA for video compression is a file or abstract pixel data.

My case is, obviously, that it's pixel data. To prove that I present two claims:
(as in the above post) You can convert one uncompressed master file (or file sequence) to another uncompressed format, and still have a master. Ie, even if the bit sequence is totally different (100% data loss, according to you), you still have a master just as perfect as the original.
So, surely, the original file's bits doesn't matter, what matters is pixel data and nothing else.
You don't even need a master file to create a lossless compressed file, you can create it directly from your 3D, compositing or editing package.
I rest my case. It's been fun. :)

Cheers mate, :cheers:
- Jonas

[beenyweenies]

Fair enough. In closing here's my final point - lossless codecs can reproduce images identical to the original, but behind the scenes the image data has been consolidated and redundant information is removed to reduce the file size. I call this "tossing data" which I still find an accurate description of what is happening given that redundant data is literally removed and replaced with compacted descriptions. Unnecessary data being removed and the resulting file being smaller is "tossing data," I see no other way to describe it.
In the case of video compression, the original video data is lost for good in the sense that only a compacted description of the video exists within the file. Without the codec that understands how to expand this data, the file cannot be viewed because it is not a complete video file. Data HAS been lost in favor of a new description that utterly requires the decompressor to interpret it.

In the context of how this discussion started, I only intended to point out that video footage files will not benefit from lossless compression anywhere near as much as solid graphics. This was backed up almost verbatim in the Wikipedia thread you linked to, and is all that really matters here.

Anyway, I'm sure we all learned something new in these exchanges, certainly it was fun debating this issue but I doubt future visitors to this sticky will agree! ;) Peace

[thedesolateone]

thanks everyone :) peace out..

[ha-dou-ken]

Damn...that was intense...:applause:

[anell]

Hi,
i have one problem,when i export the video,i don`t know where is the wrong,maybe in the settings when i m going to render,but after this everything look like as though trembling.


Thanks in advance!

[mverta]

Jonas was right. Lossless codecs compress the data, but don't throw away anything. The advantage is they save space, the disadvantage is they have to compress/uncompress every time they're accessed. This happens so quickly, though, you almost don't notice. Go ahead and render a completely uncompressed frame, then render the same frame in .png. Open them both up, with one in difference mode on top of the other. They will precisely cancel each other out, because they are precisely identical. In fact, you can apply any kind of wacky post effect you want on them, and they will respond in identical ways.

Common misconceptions, though.


_Mike

[Cromfel]

I would like to propose adding Dr.DivX into the compressor listing.

http://labs.divx.com/LatestWinDrDivX (Open source)

Or even:

http://www.divx.com/divx/windows/converter/ (Trial for 16 days or 14,99$ lic)

I suppose the difference between these two is rather minimal. Nevertheles good applications for processing the uncompressed data.

[sheepfilms]

I recently found this site which has very thorough tests of all the most common Quicktime codecs, including examples of compression artifacts on lossy codecs after both one and ten generations (!)

If you're interested in the uncompressed/visually lossless codecs and their relative sizes click on the 4:4:4 link at the top of the page

http://codecs.onerivermedia.com/

[vilsevalle]

i cant export to avi,mov .etc i can only export to macromedia flash and sfw or something like that how do i fix this`?

[ftaswin]

I have a different opinion:

never render uncompressed movie straight out of after effects. If your project is long enough to render and deadline is tight enough ALWAYS render out image sequence.

I never want to bet AE is not going to give me bad frames (have u used reelsmart motion blur in AE?). This way any bad frames can be re-produce without wasting time and other things...

Then you can bring the sequence back, line it up with the audio and render out (I normally do) H.264 quicktime in no time at all... (even uncompressed)

Again, just personal opinion.... :P


Ft

[shy-guy]

Hey there. I´ve being reading a looot of threads lately about codecs and compression and stuff like that, cause im finally finishing an animated short i´ve been doing and want to know whats the best posible way to render it,and i just want to say that the discussion between beenyweenies and jussing has been one of the most inspiring and informative things i´ve ever read on cg talk,seriously, i understood so many stuff that has troubled me for so much time that its no funny,and the way they behave and debated each other arguments back and forth without resorting to sarcasm and insults was imo admirable.

Thanks to both of you for taking the time to write those explanations,really. :)

Peace.

[thethule]

So....which one should we be using? I know the answer is in there somewhere. :scream:

[CCGSean]

Really great information. thx

I agree with thethule, which one is the one we should be using ?

[beenyweenies]

So....which one should we be using? I know the answer is in there somewhere. :scream:

For virtually all situations, rendering to the Quicktime Animation codec is advised. Uncompressed Quicktime or AVI is also okay except that the files will be needlessly huge compared to Animation codec, with the exact same visual result. The reasons for this are discussed in great detail upstream if you want more info.

You could also render to a PNG frame sequence if you frequently suffer from failed renders or bad frames (also discussed upstream).

Enjoy!

[abielmuren]

Hi!

I was wondering if you could help me to solve a tricky mistery, at least for me, I have a none compressed file in HD(1080p) I would like to distribute to a couple of friends and maybe other people, but I don't know what kind of codec or tools I need to make it playable for almost averyone who has mac or pc without the lossing of so much quality, Maybe I going to burn the video as a file, and if necessary, add the right tools to be seen.

What is what you suggest that I should use? I heard something about H.264, MPG4, Procoder, Rhozet Carbon Coder, Cinema Craft Encoder,among others, but it is harder to take a good decision, maybe burning it with 2 or 3 formats.

[Sefyu]

I would like to add another encoding tool to the list:

MeGUI (http://sourceforge.net/projects/megui/)
Although it's range of supported output codecs(MPEG-4), it is very, very comprehensive, you can adjust every setting you could ever dream of.
It support the output of Xvid, x264, and the wavelet codec Snow, and the audio codecs that it outputs are AAC, AC3, MP3 and Vorbis.

I could go on about this tool for ages, but try it out, you will be pleasantly surprised if you like loads of settings.

[Sybexmed]

Alright we havent really came up with a solution here to render out... I have done AVI to h.264 and .MOV to h.264 and also wmv, and have seen significant quality loss. I have seen some remarkable footage on youtube with great quality. I think im doing something wrong. My work flow is, out of the 7D straight into AE as .MOV do all of my editing, then render out as .MOV then bring into vegas and render out as wmv or h.264. wmv having better quality. Moreover, I hear AVI to FLV is better but i get some horrible quality doing that with all settings set to best. Can anyone share there workflow using HD footage?

[Sefyu]

For HD with acceptable image quality, you can't go around large filesizes. I do it like this:

I export uncompressed out of AE, then I fire up MeGUI and do a 3-pass h264 encoding at best settings with a bitrate of at least 8000kbps for 720p footage. Then I encode the audio to mp3 of aac(can also be done in MeGUI) and mux the audio and video together.

[tammo]

I always render to single PNG files from my 3D-programs.

Then i do my postwork in AfterEffects and export to a single Quicktime File with the PNG codec.
It's a standard QuickTime codec, it's lossless, much smaller then uncompressed and holds an alpha channel.

This is also the file that i upload to clients if they continue to work with my Animations.

When i need smaller files because i don't have so much time to upload i use quicktime-PhotoJPG 95% Quality setting.
This is slightly lossy but not detectable with human eyes.

Quicktime PhotoJPG 95% is also the codes that the stock footage supplier Artbeats uses on all of their files.

[Sybexmed]

What do you mean 3-pass h.264?

You're right, you cant get around large file sizes with HD footage, i usually render out in chunks of 3 or 4 at 25-35GB each then bring into vegas for my h.264 conversion. For some reason AE crashes when i render out anything larger. Does anyone know if Avids DNxHD is any good? I hear its the equivalent to Prores for us pc users.

[Sefyu]

Here's a good explanation of it:

http://www.afterdawn.com/glossary/term.cfm/multipass

[scrimski]

Does anyone know if Avids DNxHD is any good? I hear its the equivalent to Prores for us pc users.
Yes it is any good especially for editing but not a common codec so you better stick with H264.
And it's the equivalent for FCP's ProRes for Avid users on Mac OS too.

[Age180]

So what if I have the problem of rendering then re-rendering then re-re-rendering..

By this I mean I use Sony Vegas to edit on my PC (I know I'm sorry) and I film with a Canon Rebel t2i in HD (large .mov files) .. Now I want to apply some color correction and effects in AE so should I render the raw footage FIRST before editing in Vegas .. (files would be 30 Gig's +) or should I create my video then apply the effects after the fact.. ?

And if the second option is better I know have the problem of rendering from mov. to .wmv. in Vegas then bringing the wmv. in AE to render back to a mov.?

I skimmed through most of these replies and nothing caught my eye so forgive me if this has been answered already I'm just not sure what to do...

Thanks

[jussing]

By this I mean I use Sony Vegas to edit on my PC (I know I'm sorry) and I film with a Canon Rebel t2i in HD (large .mov files) .. Now I want to apply some color correction and effects in AE so should I render the raw footage FIRST before editing in Vegas .. (files would be 30 Gig's +) or should I create my video then apply the effects after the fact.. ?
Editing on a home PC should never involve huge, uncompressed files, because it slows down your system enormously, and thereby your creative real-time workflow. But the T2i's MOV files are already heavily compressed, so you might be able to use them as they are. Alternatively, you may create your own "offline" material, by converting all the mov files to smaller resolutions with more compression (a frame-by-frame JPG compression is also better for editing than the motion based MPG, H264 etc).

Then you have to re-create a lossless master in After Effects ("onlining"), to do your VFX and grading.

If you worked with the original files in Vegas, you should be able to export a lossless master from there, without any quality loss. Just make sure there's no de-interlacing or anything being applied to the clips in Vegas. Double-check in After Effects that your Vegas render is a perfect match to the original files.

If you used compressed files to edit in Vegas, you'll have to re-create the film in After Effects.

Typically I do this manually clip by clip, because I've always been to lazy to check out the alternatives, and I never trusted a Vegas-AE solution to be stabile... but that's maybe just me being old fashioned. And it feels like a stupid thing to do, because it's so mechanical that it SHOULD be automated. I think you can automate it easily from Premiere to AE (because they're both Adobe), but I don't think there's an easy way from Vegas to AE. There's a script out there called Automatic Duck, which should do something along those lines, but I've never used it...

Good luck!

- Jonas

[CHRiTTeR]

Nope, absolutely not. Your thinking is crooked in that it would only have real value if we are strictly sticking to an 8bpc RGB/ 10bpc YUV workflow. But hey, this is 2007, who does that anymore? Even so, it becomes a question of which quantization and color space transforms CoDecs employ and neitehr of the choices you provided gives any clear and predictable results in that regard. The only reliable formats that can truly count as "lossless" are

a) uncompressed image files that
b) maintain the native color space,
c) do not impose wrong Gamma profiles,
d) can hold additional color management information/ color profiles and
e) ideally support larger bit-depths.

From within AE this limits it pretty much to TIF, PSD and OpenEXR. You are mostly proposing non-standardized formats and procedures which is absolutely wrong and terrible to do. Can you even remotely assume that let's say HuffYUV will have a version compatible with the then-successor of Windows Vista in 6 or 7 years? You can't and therefore anyone would be ill-advised to follow your procedures.

Mylenium

You do realise that every good lossless image format out there uses some kind of compression, yes?

The point of a lossless codec is that it doesnt degrade the image quality in any way. Yes, it is verry silly to use uncompressed formats/codecs.