Recommended Sticky - Do not export compressed from AE

Become a member of the CGSociety

Connect, Share, and Learn with our Large Growing CG Art Community. It's Free!

REPLY TO THREAD
 
Thread Tools Search this Thread Display Modes
Old 11 November 2007   #16
Originally Posted by avinashlobo: I think you have this wrong. In fact, I don't think spacefrog agrees with you at all. That's not the point of lossless at all. The term 'lossless' can only be used when the exact, original data can be reproduced from the compressed version. Bit for bit. Which translates to ZERO data loss/degradation.

Originally Posted by jussing: Beenyweenies, what do you mean when you say "without degredation" at the same time as "there is loss of information"?


avinashlobo, you are assuming that data loss and visual degredation are tied together, which is the wrong way to think with regards to codecs - their entire existence is to remove data where possible while degrading the image as little as possible. Lossless codecs are no exception, the only difference is they will not toss information that will visually degrade the image (as I explain below). You are also overlooking one major thing - Lossless codecs actually claim to be visually lossless, not 100% bit-for-bit dupes of the original file. That would be an uncompressed file, NOT lossless.

Think about it this way. If you render a one minute HD-sized video of nothing but a black solid out to the Quicktime Animation codec, the file will be very small. Render one minute of video footage at HD size to the Animation codec, and the file will be utterly huge.

The reason is that, with the black solid video, the codec can toss pretty much everything except the information needed to create a single black pixel, which is then used to describe the entire frame since it's made up entirely of black pixels with no variation. Render the same movie but with three different color solids, and the codec only needs to visually describe three different colored pixels. With the video footage render, almost every pixel is unique (especially from frame to frame) and therefore the codec cannot toss information without altering pixels (resulting in visual loss of quality), so the file is very large as a result.

Therefore, it IS tossing information, but only where it can without visually degrading the image. If you compare the Animation codec version to the original, they are visually identical (and placing one over the other in "Difference" mode within AE would confirm this) BUT information was tossed. Were it not for lossless codecs operating in this way there would be no need for the lossless algo, only uncompressed and lossy codecs!

VISUAL LOSS is the keyword here - in the technical documentation lossless codecs only claim to be visually lossless, which is a massive distinction from a bit-for-bit duplicate of the original.

Last edited by beenyweenies : 11 November 2007 at 07:01 AM.
 
Old 11 November 2007   #17
Ah, now I see the misunderstanding as clear as day.

Originally Posted by beenyweenies: avinashlobo, you are assuming that data loss and visual degredation are tied together, which is the wrong way to think with regards to codecs

Well, that is EXACTLY the way to think about data compression (also with regards to codecs).

Beenyweenies, you are incredibly clever about codecs, but you've got the wrong idea about the meaning of "lossless compression".

"Lossless" means that all original data can be restored. Just like when you compress to a ZIP file. -You get a smaller file, but you can unpack ALL ORIGINAL DATA without loss.

When talking about video codecs, the data is pixel data. And if a file is compressed with NO VISUAL DEGREDATION, that means ALL PIXEL DATA was restored. And that means NO LOSS.

Your example with the black image video file is a perfect example. A "lossless" compression WILL result in a much smaller file, but that doesn't mean any information was lost, it just means the information was re-organized in a more optimal way, resulting in a smaller file, which is exactly the purpose of lossless compression.

Ie, instead of saying "black pixel" 1,000 times, it's shorter to say "1,000 black pixels". That's lossless image compression in a nutshell.

- Jonas
__________________
 
Old 11 November 2007   #18
Yeah, Jonas is right. Lossless works like a zip or rar file. There is NO tossing of data. To put it in a very, very crude way - there is substitution of repetitive data with smaller data (refer to Jonas' example in his post). When required, the exact, original data can be reproduced from the substitute data. And it is perfectly exact. There is no shifting of bits or any alteration to the original data.

What you, Brendon, are describing is, in fact, high-quality LOSSY compression, where there is no apparent VISUAL degradation, but there is still data degradation. For example, save a file out as a maximum quality JPEG and keep it next to the original uncompressed version. The JPEG will be a fraction of the size, but there will be no VISUAL distinction between the two, unless you're gifted with superhuman eyes.
 
Old 11 November 2007   #19
Sorry, but you guys are both stuck on the idea that lossless is some bit-for-bit exact replica, when it just isn't. The point that data loss and visual loss aren't tied together is also getting lost in the mix here.

As was mentioned, if you replace 1,000 instances of the words "black pixel" with a single instance of "1,000 black pixels" you have just thrown out 999 entries and thus reduced the size of the file without impacting the resulting image. It is tossing data, 999 entries to be exact, otherwise the file would not be smaller. It's not like codecs are some magic suitcase that stuff the exact same bits into a smaller package, they merely consolidate the information and toss the redundant stuff. The resulting file is not a bit-for-bit dupe, it has been completely reorganized and redefined, the codec just knows how to read this new file and produce the same visual result. Because of this, the RAR/ZIP analogy is incorrect. It is more akin to a vector-based artwork, in which the program stores vector data rather than pixel data to reduce file size.

Visual loss is the shifting of pixel values to brute-force match them to the definition "black pixel" and thus enable the codec to roll them into the compression scheme, "1,000 black pixels." Since you can throw away data without resorting to this, DATA loss (or "restructuring" if you prefer) and VISUAL loss are not tied together as I stated.
 
Old 11 November 2007   #20
Hi again.

Originally Posted by beenyweenies: Sorry, but you guys are both stuck on the idea that lossless is some bit-for-bit exact replica
No, obviously the COMPRESSED file is not a bit-for-bit replica, we never said that.



Originally Posted by beenyweenies: It is tossing data, 999 entries to be exact, otherwise the file would not be smaller.
No, since the INFORMATION (data) about the 1,000 black pixels is still 100% intact, nothing is tossed or lost, it's just being re-described in a shorter way, which is exactly the way ZIP/RAR etc. compression works.

Tossing data would be if you had a white pixel in there, and you "left it out", and kept saying "1,000 black pixels" instead of "500 black pixels, 1 white pixel, 499 black pixels". THAT would be toss & loss.

"Loss" in terms of compression does NOT mean "a smaller file size", because obviously that's what "compression" means. "Loss" only refers to if you lose something in the compression or not, ie, if you are unable to restore all original data or not. If not, you have LOST data, and the compression is LOSSY.

Quote: Lossless data compression is a class of data compression algorithms that allows the exact original data to be reconstructed from the compressed data

http://en.wikipedia.org/wiki/Lossless_data_compression

Cheers,
- Jonas
__________________
 
Old 11 November 2007   #21
Originally Posted by jussing: No, obviously the COMPRESSED file is not a bit-for-bit replica, we never said that.
Actually it WAS said upstream that lossless is the same file, bit-for-bit, that's why I mentioned this.
Originally Posted by jussing: No, since the INFORMATION (data) about the 1,000 black pixels is still 100% intact, nothing is tossed or lost, it's just being re-described in a shorter way, which is exactly the way ZIP/RAR etc. compression works.
When you remove data from a file, that data is lost EVEN IF it is being accurately redescribed elsewhere. NO ONE is arguing whether or not the resulting video is visually identical to the original, but data is being removed and then redescribed in short-hand. How is that not data loss? Honestly, I don't really care to debate the semantics of whether removing data from a file is "loss" if there is no negative impact, all that matters (as far as this thread is concerned) is what conditions should be met when choosing one codec over another.

Lossless codecs rely on being able to toss/redefine data in order to keep file sizes small which is the ONLY value in using a lossless codec instead of uncompressed. Lossless codecs can perform very well but ONLY if appropriate source material is being used.
 
Old 11 November 2007   #22
Aye, it's semantics indeed. But the definition is clear, check the Wikipedia article. Or any paper on lossless compression algorithms.

Originally Posted by beenyweenies: When you remove data from a file, that data is lost EVEN IF it is being accurately redescribed elsewhere.
No, that's exactly what's false. If you re-describe it elsewhere in the same file, the data is, by definition, not lost. How can data be lost when it's still there??

In the example of "1000 pixels", the data of the 999 instances of "black pixels" is not lost at all, it's right there in the phrase "1000 pixels". Reducing it to "1 black pixel" would be data loss, because then the information about the 999 pixels is entirely gone.

If your definition of "loss" was true, then the very term "lossless compression" would be a contradiction in terms, because a "lossless" file wouldn't be compressed at all. (because you say if the file is smaller, something is lost. Which goes against the very idea of compression like ZIP)
  • COMPRESSED means smaller file size
  • LOSSY means when you unpack the file, you don't get the same result as to begin with (JPG, MPG, DIVX etc)
  • LOSSLESS means when you unpack the file, you get the same data you started with (ZIP, RAR etc.)
I quote Wikipedia again:
Quote: Lossless data compression is a class of data compression algorithms that allows the exact original data to be reconstructed from the compressed data

And that's the truth.

Cheers,
- Jonas
__________________
 
Old 11 November 2007   #23
I can see where you're coming from. You're saying the losslessly compressed file does not contain the same data as the original uncompressed file. And you're right!

What WE are saying is that the losslessly compressed file DOES contain the same data as the original uncompressed file. And we're right too!

I suppose this is really a matter of perception; depending on how you want to view the compressed file.

But coming to the crux of the matter, a properly written lossless codec performs EXACTLY the same as an uncompressed file. This cannot be disputed.

The zip/rar analogy holds as well because the behaviour is the same if the appropriate front-end is provided. This can be best described when you consider Windows XP's ZIP folders. All the decompression/recompression happens behind-the-scenes when you edit the zip folder, but the contents (while still being compressed) will match the original data exactly

This is where we're coming from by saying that there is NO loss because while there is the lossless wrapper around the original file, the required data is decompressed/decoded on the fly, edited and then recompressed and put back in the lossless wrapper without any interaction required.
 
Old 11 November 2007   #24
Originally Posted by avinashlobo: There is NO loss because while there is the lossless wrapper around the original file, the required data is decompressed/decoded on the fly, edited and then recompressed and put back in the lossless wrapper without any interaction required.
Not to keep arguing but this is wrong. Lossless codecs are not some magic suitcase that can stuff 1,000 lbs. of s...t into a 500lb sack. There is no lossless "wrapper" and there is no "original file" once it has been compressed to lossless. The video information fed into a lossless codec is permanently, irrevocably altered and redefined to a smaller, compacted state. On decompression the codec CAN translate this information into an expanded state so that the video player understands it, but your original file no longer exists, it has been redefined. In the case of zip files, the decompressor can translate the compacted file back into a 1:1 duplicate, but NOT because the original file was "stored" inside the codec - it merely expands the compacted definitions back into full definitions and creates a new file based on this.

As for this whole argument about whether some form of data tossing/loss is happening, have either of you even considered the notion that the Animation codec, for example, has a quality slider ranging from "best" on down? Think about it. The codec in its "best" state delivers a 1:1 quality because it only tosses/redefines what data it can without affecting quality. At the lower settings, it begins tossing more data, resulting in quality loss. As such, my point stands that the codec is selectively removing data. In its best form it is just limiting the degree to which it does this.

For the purposes of this thread, all that matters is for people to understand that lossless codecs rely on certain types of video data being fed to them to work well. People often complain that Animation codec files are huge, and my response was that this particular codec performs very well when the appropriate subject matter is fed into it. Give it a video file and the file will be remarkably huge. Give it the type of subject matter it was designed to efficiently compress (such as blocky graphics) and the file will be very small while still looking great. What is at issue here is when to use which codec, depending on the circumstances. Upstream someone claimed that lossless files do not throw away any data, and by my definition they DO, that is how the file size is lowered - by removing entries and replacing them with more compact definitions. When people understand how the codec works, they will know when to use it vs. uncompressed and that was the only point of my comments upstream.

I suppose we can end this by agreeing on the following - lossless set to "best" quality is visually identical to the original file, although data has been removed and consolidated in the interest of reducing the file size. Agreed?

Last edited by beenyweenies : 11 November 2007 at 05:27 PM.
 
Old 11 November 2007   #25
Hi again. I'm sorry, I like continuing this. I respect you very much Beenies, and I certainly don't want to continue till we rip each other's heads off. I might stop soon. But for now, I'll have another go. Sorry.

Let me begin again by saying that if your definition of lossless was true, there would be no such thing as "lossless compression", because "compression" by your definition means "tossing" and "losing" data. So compression, by your definition, can't be lossless.... explain that?

Originally Posted by beenyweenies: Lossless codecs are not some magic suitcase that can stuff 1,000 lbs. of s...t into a 500lb sack.
Whenever possible, that's exactly what it is, yes. JUST LIKE ZIP. It's a means of discribing THE SAME DATA in a shorter way, so no loss is happening.

You can also ZIP your video files to a smaller file size. That's also lossless compression, the exact same principle. The difference between ZIP and, for instance, Quicktime Animation at 100% is that the Animation algorithm is specialized for image data, so you will get better results than with ZIP.

And with that, I'm beginning to understand your misunderstanding... I have a feeling I know what you're going to say to the ZIP example: You're going to say that it can't be compared, because with ZIP you unpack the original EXACT file, right? Whereas with animation codec you never return to the original file?

Well, that's because the original IMAGE DATA is not a file. IMAGE DATA is just a series of pixels, no matter if you put it in a gif, png, targa, in your mind, or on post-it notes. THE PIXEL DATA is the original master. The uncompressed (huge) file is alraedy just a container, it only REPRESENTS the master (proof: you can convert it to another uncompressed image format, which gives you completely different bit sequence, but the exact same image data - that's also lossless, even though all the bits are different). Put that file through lossless compression, and you get a much smaller file, which is also just a container. Play that file, or load it into compositing or editing, and you restore THE ORIGINAL PIXEL DATA on your screen, compositing app or whereever. The original file? No, you don't restore that, but who cares, your source data is pixels, not a file. If you can restore your original pixels, you have had NO LOSS. If you had loss, there would be "damage" to your image (compression artifacts).



It works like this:

Master data -> Compression algorithm -> Compressed file -> Uncompressed algorithm -> Master data

With ZIP and program files, that would be:

QUAKE.EXE -> ZIP algorithm -> QUAKE.ZIP file (smaller) -> UNZIP algorithm -> QUAKE.EXE (perfect match to original file)

And, with video data:

ORIGINAL PIXEL DATA (in any container!) -> Lossless codec -> VIDEO FILE (smaller than if uncompressed) -> Decoding (playback) -> ORIGINAL PIXEL DATA

Of course, as with the ZIP example, lossless compression only offers a limited effect, depending on the data. As you said, a live video signal will be much less compressed that a blank image or one with large blocks of solid color. And that is paramount in the understanding of lossless compression. It's called entropy - a certain amount of data can only be compressed so much without loss. That's why lossless compression doesn't have a constant bitrate, it's inevitably impossible.

Originally Posted by beenyweenies: As for this whole argument about whether data LOSS is happening, have either of you even considered the notion that the Animation codec, for example, has a quality slider ranging from "best" on down?
Yes, the animation codec is only lossless at 100%.

Again, I'll quote Wikipedia :
Quote: Lossless data compression is a class of data compression algorithms that allows the exact original data to be reconstructed from the compressed data.


And, with video compression, data = pixel data

Cheers mate,
- Jonas
__________________

Last edited by jussing : 11 November 2007 at 05:57 PM.
 
Old 11 November 2007   #26
OK, reading through the whole thing again it strikes me that we are actually very close to total agreement. The only thing we disagree about is, as I was in on in my last post, whether or not the SOURCE DATA for video compression is a file or abstract pixel data.

My case is, obviously, that it's pixel data. To prove that I present two claims:
  1. (as in the above post) You can convert one uncompressed master file (or file sequence) to another uncompressed format, and still have a master. Ie, even if the bit sequence is totally different (100% data loss, according to you), you still have a master just as perfect as the original.
    So, surely, the original file's bits doesn't matter, what matters is pixel data and nothing else.
  2. You don't even need a master file to create a lossless compressed file, you can create it directly from your 3D, compositing or editing package.
I rest my case. It's been fun.

Cheers mate, :cheers:
- Jonas
__________________
 
Old 11 November 2007   #27
Fair enough. In closing here's my final point - lossless codecs can reproduce images identical to the original, but behind the scenes the image data has been consolidated and redundant information is removed to reduce the file size. I call this "tossing data" which I still find an accurate description of what is happening given that redundant data is literally removed and replaced with compacted descriptions. Unnecessary data being removed and the resulting file being smaller is "tossing data," I see no other way to describe it.
In the case of video compression, the original video data is lost for good in the sense that only a compacted description of the video exists within the file. Without the codec that understands how to expand this data, the file cannot be viewed because it is not a complete video file. Data HAS been lost in favor of a new description that utterly requires the decompressor to interpret it.

In the context of how this discussion started, I only intended to point out that video footage files will not benefit from lossless compression anywhere near as much as solid graphics. This was backed up almost verbatim in the Wikipedia thread you linked to, and is all that really matters here.

Anyway, I'm sure we all learned something new in these exchanges, certainly it was fun debating this issue but I doubt future visitors to this sticky will agree! Peace
 
Old 11 November 2007   #28
thanks everyone peace out..
 
Old 01 January 2008   #29
Damn...that was intense...
 
Old 03 March 2008   #30
export in AE

Hi,
i have one problem,when i export the video,i don`t know where is the wrong,maybe in the settings when i m going to render,but after this everything look like as though trembling.


Thanks in advance!
 
reply share thread



Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
CGSociety
Society of Digital Artists
www.cgsociety.org

Powered by vBulletin
Copyright 2000 - 2006,
Jelsoft Enterprises Ltd.
Minimize Ads
Forum Jump
Miscellaneous

All times are GMT. The time now is 06:56 PM.


Powered by vBulletin
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.