System Elements
-
Discussion of MJPEG video compression
A discussion of MJPEG / Photo jpeg video compression in terms of rendered text quality,
color space subsampling, and a pile of codec particulars/oddnesses.
- 02/22/12 Update: Please note that MJPEG is a deprecated codec. We use and recommend Photo JPEG instead. We have not performed the same careful tests with it yet, as described below, however we expect it to behave similarly, and believe the following information is still relevant and useful. -
-
I have posted this page for several reasons:
-
- First, I recently spent a substantial amount of time trying to figure out what the @&##^%#! was going on, and I would like that effort to benefit more than just myself!
- Second, this information is directly relevant to the production of content for TechnoFrolics' FrameGlide + Spin Browser video explorer, and thus this page has direct application to outside parties involved in such activities.
- Third, there is the commonly held belief that MJPEG/JPEG compression, while excellent in many regards, is lousy at handling sharp changes in color/contrast etc., with text being particularly susceptible to visible degradation. There is truth to this, particularly at very high compression settings, but as you will see below, this view covers only a fraction of the story.
As a caveat before beginning, let me note that I am by no means a compression expert at the algorithm level. Nevertheless, if you are a practicing video/multimedia producer, and have a slightly technical bent, I think you will find the below of significant interest.
David Durlach
December 5th, 2007Table of Contents
A clear example of the issue
In the image below, there are four rows, each labeled as to its contents.
[You may immediately object to the fact that the top row is labeled "No Compression", given that the web image itself is JPEG compressed. However, it is visually indistinguishable from the original BMP file, having been compressed with Photoshop CS2's Maximum ("12") compression setting. Furthermore, a 4.4 meg zip file containing the truly uncompressed source image, along with many more foreground/background text color combinations than are shown below, and the JPEG compression artifacts evidenced therein, may be downloaded here. I strongly encourage so downloading.
Also, note that the second column photo, coming from a relatively low quality digital camera, is noticeably JPEG "smeared" even in the "Uncompressed" version. That photo should thus be substantially ignored in terms of the discussion below. (In retrospect I should have tried to find a better shot, but as its presence is sort of random anyway, I did not think it worth redoing everything just for that.)]
You will notice that rows 2 and 3 above are essentially identical in file size, each being JPEG compressed by about 10-1 compared to the original uncompressed source. Yet in row 2, the blue text on a black background looks horrible, while in row 3 it looks fine and is essentially indistinguishable from the original uncompressed version in row 1. Why is this given the files sizes are the same? The answer is that in row 2, color subsampling was performed (I believe 4:2:2), while in row 3, to the best of my knowledge, there was no subsampling at all (i.e., "4:4:4"). (It is also interesting to note that the color subsampling has no apparent detrimental affect on the B&W text/background.)
Finally, just for reference, row 4 shows extreme compression (way too much for most projects) of over 50-1. Here the "standard" problems of JPEG compression show up, including color "smearing" even in the black & white text, and the photo for the first time shows noticeable degradation (even beyond its original smearing). Note that this row was generated by the lowest possible JPEG quality setting (0) in Adobe After Effects (AE) V7.0, exporting a "JPEG Sequence".
To summarize, if you see problems with text degradation in JPEG compressed images, unless you have compressed a HUGE amount, the problem is almost certainly not intrinsic to the JPEG codec, but rather to a less-than-ideal color subsampling scheme. Much additional information, including what common codecs perform color subsampling, follows.
Technical Information, Background, and Links
Background
MJPEG, or Motion JPEG, as the reader may know, is simply a data structure wrapping individual frames which are each themselves JPEG-compressed. (Do not confuse MJPEG with MPEG). Thus, there are (for example) both Quicktime and AVI movies whose codec (Compression/Decompression algorithm) is MJPEG.
MJPEG/JPEG is only loosely defined as a compression scheme - there is much variability in how JPEG frame compression is actually implemented in any particular case. Additionally, the single "Quality" setting in many application's MJPEG/JPEG compression dialog box, be it a percentage value 0-100, or a "Low", "Medium", "High", "Best" choice, is not well defined in terms of what effect it will have on (each step of) the (loosely-defined) algorithm. This state of affairs has very real-world impact as we have already seen above.
I focus on MJPEG (as opposed to simply JPEG) for several reasons. First, the creators of popular MJPEG compressors have often made different algorithm choices than those creating popular single frame JPEG compressors. Second, Quicktime MJPEG files are what TechnoFrolics' FrameGlide + Spin Browser VE requires. Finally, file size when rendering hours of video is frequently a lot more important that when creating just a few dozen still images.
Finally, note that the factor by which one can compress an image without visible artifacts will vary greatly from image to image.
The compression algorithm
MJPEG/JPEG compression has a number of relatively separate steps to the algorithm. More detail than you probably want can be found in Wikipedia here,with further discussion of color space conversion and sub-sampling in the "Fundamentals of embedded video, Part 2"article written by Analog Devices engineers. A much simpler and less mathematical page, with images, and duplicating some of the information presented in our discussion (page was discovered after most here was already written), but focused on JPEG (as opposed to MJPEG), can be found at Optimization of JPEG compression settings.
The concepts that must be understood for our discussion, described in much more detail in the above first two links, are:
Phase 1: Perform (or not), color subsampling. This is where adjacent pixels "share" a single color designator in order to reduce file size. There are two key points to understand about this: First and most important, this is is a discrete image-degrading step - in other words, there is (typically anyway) no "slider" to continuously adjust the quality loss resulting from this step. Second, that the exact algorithm for such color subsampling is not necessarily well defined (though for some JPEG standards it may be).
Phase 2: Perform an operation similar to a two dimensional Fourier transform, and then throw away a certain fraction of the high frequency components. This step is where the image is converted into "high and low frequency components", where high frequencies correspond to sharp color transitions and fine detail, and low frequency components correspond to the larger, softer scene elements. What is key to understand here is that the typical "Quality Slider" in an application's MJPEG/JPEG compression dialog box determines, in a continuously adjustable way, how much of the high frequency info is discarded. (However, the quality slider may also affect the color subsampling scheme!)
Operation of actual, real-world codecs, applications, etc. (Quicktime, Adobe After Effects, etc.)
Video
- To begin, a great resource, covering all kinds of codecs, is http://codecs.onerivermedia.com.
- Quicktime (Note that the below applies to all versions of Quicktime I have tried on a Mac and PC, both current and old. However, I by no means did anything like an exhaustive test over all platforms and versions.)
- MJPEG-A and MJPEG-B appear to do color subsampling (4:2:2 from One River Media) at all quality settings including 100%. Thus, for certain colored text/background choices, they are problematic even at the highest quality setting.
- Photo JPEG appears to do no color subsampling at 100%, but does do subsampling (4:2:2 from One River Media) at any quality less than 100% (confirmed by One River Media). This is problematic because, for certain colored text/background choices, you must make the files much larger than they would otherwise need to be (because in order to get no subsampling, you must choose a much-higher-than-needed "Quality" setting).
- Quicktime Pro (around $30) has the great feature that it can open all kinds of files (JPEG image sequences, AVI files, etc.) and "Save" (not "Export") them back out as a single Quicktime MJPEG-style movie without recompressing the frames. This is invaluable, and indeed is one of the few or only ways to get high quality MJPEG Quicktime 4:4:4 movie files.
- Adobe After Effects V7.0
- AE export setting of "JPEG Sequence" appears to do no color subsampling (or if it does, it probably kicks in only at very high-compression/low-quality export settings).
- In summary, the only way of which I am currently aware to get good quality colored text within a Quicktime MJPEG movie, without unnecessarily ballooning the file (through the Photo JPEG 100% setting), is to import JPEG-compressed frames created in another application. Specifically, use Quicktime Pro to open either an AVI file that uses an MJPEG codec with no (or explicitly settable) color subsampling (they exist), or an AE (or otherwise) generated JPEG image sequence, and then "Save As" to a Quicktime movie.
Stills
- Photoshop CS2, when saving a JPEG, appears (at least in the higher range of the quality setting) to do no color subsampling.
- Windows XP Picture Viewer appears to apply a slight Gaussian blur (or something similar) following decompression, which can be confirmed by opening up the un-blurred image in Photoshop and then applying said blur (perhaps 0.3 pixel radius).
- The Optimization of JPEG compression settingslink provided earlier has significant additional information, including listing programs that allow choosing the color subsampling explicitly (for still images), the comment that Photoshop (version unspecified) switches color subsampling based on Quality setting of slider, etc.
- First, I recently spent a substantial amount of time trying to figure out what the @&##^%#! was going on, and I would like that effort to benefit more than just myself!