The CDF folks at Goddard Space Flight Center have identified a security vulnerability — a buffer overflow to be specific — that can enable the execution of arbitrary code on your machine if you open a particular malformed file. If you’re accessing CDF files via MATLAB, you can download a security patch from NASA GSFC.

Thank you. That is all.

UPDATE: You can also download an update directly from The MathWorks.

read more from this topic.....

This dispatch is a bit of a valediction for me. Since early 2000, I’ve been one of the software engineers on the Image and Scientific Data Formats team at The MathWorks. I’ve learned a lot about an area of technical computing that rarely gets the limelight, which is too bad since file format support is the sine qua none for modern computing. As with any real-world discipline, communication and sharing data are the bases of getting anything done. Along the way I’ve also gained a lot of skills creating code, designing systems, and managing projects. And I’ve worked with some wonderful people. It’s been a really great experience, but an offer that was too good to refuse came along.

So now I’m a Senior Software Engineer in the Image Processing Group at The MathWorks. I still work in the same group with the same great people; only the projects have changed. Instead of programming file format interfaces, I’ll be working on software architecture and optimization. It’s definitely a growth opportunity for me, and I get to keep using a lot of the skills that I’ve gained over the last eight years.

But when you change jobs without changing offices, sometimes there’s a bit of overlap. And my interests haven’t changed radically; I just have more. So perhaps it’s not surprising that I’m still writing about JPEG here. Anyway . . . on with the show.

The most popular page on this web site covers the JPEG landscape. Believe it . . . or not. I’m not complaining. I just find it amusing that on a web site that touches on travel, photography, and (sometimes) software engineering, the most popular pages are about either the technical aspects of JPEG file formats or high dynamic range imaging. I guess that’s the price I pay for writing about the dozens of things that interest me.

Well, the JPEG family article has been gathering some really good comments. Most recently “pixpush” commented on a proprietary extension to classic JPEG that supports high dynamic range (HDR) and wide gamut imagery. And then he/she mused that JPEG would be even better if it could somehow support RAW data.

It would be great, but it’s never going to happen.*

Classic JPEG — the original JPEG that makes up all of our images — is what we might call “venerable.” There’s nothing really wrong with it. In fact, it’s very, very capable. But it’s an old dog with only so many tricks left in it. Unfortunately, the following things needed for RAW support are not part of its bag of tricks**:

• Lossless compression, which you absolutely need for so-called RAW imagery
• More than 8 bits per color component, since most cameras’ A2D converters use 10+ bits
• Wider gamuts than sRGB, which would require some combination of the following: converting to and from something other than YCbCr, using signed data, or somehow specifying the colorspace

It’s possible to put classic JPEG through its paces to do this, probably using the ill-supported lossless codec and extensions in new JPEG markers. (JPEG is a stream-oriented format — unlike TIFF — so you have to parse the stream for “markers” to find where new parts begin, making it hard to jump to “interesting” parts of the file.) But once you start making classic JPEG jump through those flaming hoops, you might as well go with one of the newer versions.

It’s unlikely that JPEG will “die out” in my lifetime. As long as there is data in a format it’s never really dead. (Unless, of course, no one knows what it means or the media dies.) But what format would I choose to replace it?

First, I’ll answer the question of what formats I like:

• TIFF. As long there are file systems that look like the ones we have today — files as sequential collections of bytes — the almost infinite extensibility of the Tagged Image File Format will be useful. You can put almost any kind of metadata into it now, and it’s user extensible (more or less). It supports a limitless number of samples per pixel, any bit depth you’d like, many colorspaces, ICC profiles, and a flotilla of compression modes. It’s also the basis of some vary capable formats such as DNG, and its data layout is used in EXIF, HD Photo/JPEG-XR, and other formats. TIFF and cockroaches will inherit the earth.
• DNG. Okay, so it’s more of a TIFF-based platform for describing RAW imagery than a traditional file format. You need to know how to interpret the format contents in order to get a viewable image, perhaps using a program like Adobe Camera RAW. Consequently, it’s possible for two applications to render the image quite differently. This is a very un-JPEG-like idea, but it brings back the flexibility and creativity of real-world negatives.
• JPEG-XR/HD Photo. I’ve written about this format before. It’s the heir-apparent to classic JPEG. And that’s not just because it’s from Microsoft.
• DICOM. Okay, okay. It has a lot of flaws. I mean, it can change byte order (endianness) in the same file . . . more than once. That’s messed up. To truly understand why it’s a good format, you’d have to be a trained professional, like me or federation president Barry Fife.
• HDF5. If you absolutely must store gigabytes of data using datatypes that you define, arbitrary metadata, and multiple datasets organized in a hierarchical file structure, this is your format. Of course, you’ll need to use an API to access your data, but you’re payin’ the cost to be the boss.

No one format that will replace JPEG, but I fully expect that a small collection of semi-standardized formats (JPEG-XR, DNG, TIFF) are going to fill the ever-growing image space that it doesn’t support well. And we haven’t even touched on HDR. There isn’t a standard HDR format yet, and there’s a lot to work left to do. (I’m really curious to see whether the “standard” HDR image format will include a preferred tone mapping method or whether it will just be a platform for imagery like DNG.)

Let’s see where the future takes file formats and me. . . . Stay tuned.

* — Except maybe as a joke or programming assignment.

** — Can you mix dog and cat metaphors like that?

read more from this topic.....

The discussion in the comments of my recent article about HD Photo (a.k.a. JPEG-XR) got me thinking about all of the different beasts that go by the name “JPEG.”

JPEG: What most of us consider to be “JPEG” is just one of many processes for image encoding and decoding defined within the same specification. The process that makes up 99.99999% of all of the JPEGs ever created is “JPEG Baseline (Process 1)” for 8-bit lossy compression. (That’s just my estimate, which is probably low. It’s probably better to say “almost 100% of all JPEGs.”)

This process divides an image into a bunch of 8×8 blocks, uses the discrete cosine transform (DCT) to move the data into the frequency domain, and compresses the data by (among other things) removing some of the high frequency data that the human visual system usually can’t detect. You can think about it as abridging a novel by taking out a few sentences per paragraph. Unfortunately, if the quality settings are too low, it’s really easy to notice that something has gone missing; or if a scene has a lot of information — such as one with lots of fine detail — there will be blocky artifacts where there the detail should be.

While the removal of high-frequency detail is inherently lossy, even with a maximum quality setting, the original JPEG standard specified a separate lossless mode not based on the DCT. Images compressed this way can be completely retrieved from the compressed data. This is important when you need to preserve all of the data within in image or when adding artifacts can have devastating consequences. “Is that a nodule in the patient’s chest X-ray or a JPEG compression artifact? I guess we’d better do a biopsy just in case….” In fact, the lossless modes for JPEG are really only used within DICOM files, the format used for digital imaging and communications in medicine.

Old school JPEG also supports 12 and 16 bits of data in each channel of a pixel. For color images, this is the difference between about 17 million colors for an 8-bit image, 68 billion colors for a 12-bit image, and 281 trillion colors when using 16 bits. Once again, only those medical imaging people use the extra bit depths, and they just use the gray colors.

JPEG-LS was supposed to be a better lossless format but never really got going. The promises of JPEG 2000 probably had a lot to do with this.

JPEG 2000 is (1) a wavelet-based compression method, (2) a scheme for encoding wavelet-compressed images into randomly accessable “codestreams”, and (3) a file format for encapsulating compressed codestreams. Because it uses a discrete wavelet transform (DWT) the results are generally better than the older JPEG format when comparing images with the same compression ratio.

Images in JPEG 2000 can have an arbitrary bit depth (1 - 32 bps), and different planes can have different bit depths. (For example the luminance channel of a YCbCr image can have a high bit depth to support HDR imagery.) Certain portions of an image can have higher spatial resolution or be encoded at a different compression level. JPEG 2000 has both lossy and lossless components as part of the baseline. Several colorspaces are supported, including bi-level, grayscale, sRGB, YCbCr, and indexed imagery. Hyperspectral and n-sample images are supported using a somewhat convoluted “multi-component” schema. Images can also include alpha channels for transparency. A really amazing thing about JPEG 2000 is that its possible to reorder the parts of the codestream to change how the data is accessed (e.g. access regions faster v. access different resolutions faster) without decompressing and recompressing the data, which can be expensive.

The JPEG 2000 file format uses about 20 hierarchical “boxes” to nest metadata about the compressed codestreams. While the file format is technically unnecessary to read and process a JPEG 2000 image, the extra formatting facilitates random data access, long-term cataloguing and IP management, and efficient transmission. JPEG 2000 files can also contain a limited subset of ICC color profiles. EXIF metadata support is not part of the JPEG 2000 standard, although it can appear as a private metadata field.

JPEG 2000 was touted as the format to replace the 1991 JPEG standard, but this didn’t happen for several reasons. Perhaps most important, the algorithms at the heart of JPEG 2000 require a lot of processing power, making it slower for desktop computers than rendering old-school JPEG and prohibitive for many embedded devices. As of 2007, few Web browsers have built-in support for it, and consumer-level digital cameras don’t produce imagery in the format. In 2007, Adobe Photoshop CS3 stopped including the JPEG 2000 export module in a typical installation.

But because of the smaller file size, flexibility, and more pleasing artifact appearance, the medical and remote sensing communities have adopted it. Both NITF and DICOM have incorporated JPEG 2000 data into their files. NITF is the friendly format used for “national imagery.” I will let you Google that so the NSA can start tracking you.

JPEG-XR is the name that Microsoft’s HD Photo format might have if it’s standardized, which I sincerely hope it will be. JPEG-XR uses a principal components photo core transform (PCT) which I know absolutely nothing about but which promises equivalent performance to JPEG 2000 with lower computational complexity — which means you can put it on a consumer device more easily — and much better size-versus-quality performance compared to the original JPEG format. It also supports more bit depths, high dynamic range imagery, lossy and lossless encoding/decoding using the same algorithm, and wide gamut color; uses a linear light gamma making it possibly suitable to replace RAW formats or enable post-CRT workflows; and can store bucketloads of metadata including EXIF and XMP.

JPEG-Plus. And then there’s JPEG+, which you might reasonably call JPEG - 20% because it’s essentially the same as the original DCT-based JPEG with a modest file-size performance improvement and some claims about better visual appearance. I’m not holding my breath for it; but given the 29+ processes that made up the original JPEG standard, what’s an extra one that no one will implement?

Update: For posterity, the PCT stands for “Photo Core Transform” not “Principal Component Transform”. Thomas Richter said this about it on sci.image.processing:

The transform is an overlapped 4×4 block transform that is related to a traditional DCT scheme, or at least approximates it closely. The encoding is a simple adaptive huffman with a move-to-front list defining the scanning order, and an inter-block prediction for the DC and the lowest-frequency AC path of the transformation.

Some parts are really close to H264 I-frame compression, i.e. the idea to use a pyramidal transformation scheme and transform low-passes again (here with the same, in H264 with a simpler transformation).

The good part is that lossy and lossless use the same transformation. The bad part is that the quantizer is the same for all frequencies, meaning there is no CSF adaption, and the entropy coder back-end is not state of the art.

read more from this topic.....

For my 9-5, I wrangle file formats, making it possible for people to read images into MATLAB and then do something useful with them. I’m always on the look out for what’s new, and there’s always something new. (I only joke about employing a legion of developers in Bangalore to develop new formats so that I can stay employed.) Lately I’ve been looking at Microsoft’s HD Photo format (a.k.a. Windows Media Photo, or WMP, format).

Once upon a time you could count on Microsoft’s image file formats to suck. Just take BMP as an example: a more-or-less unpublished format that changed between revisions of Windows and which you were supposed to access primarily through Windows API calls. File I/O people (like me) needed to have the bits of information that we could crib together handy so that we could figure out what datatype a given piece of data should have because the file didn’t carry that information in it. Nor was there a data dictionary to programmitcally look it up. They couldn’t have designed it worse if they had tried; but they clearly didn’t do any (good) design on the format. My professional opinion: totally sucked.

But perhaps they’ve learned the error of their ways with their new HD Photo format, which they have submitted for standardization. It appears that they’ve talked to a lot of digital image users about their needs and given a lot of thought to the format. I’m about halfway through the WinHEC 2007: HD Photo Implementation Guidelines presentation, which certainly says all the right things.

My only worry is that they’re saying so many of the right things that they’re possibly heading toward a couple of bad outcomes. Let’s call these the unintended consequences of designing a “good” format:

1. The “Kitchen Sink Problem” (a.k.a. DICOM) — Everybody gets what they want, including things of questionable utility like CMYK + alpha and 5-6-5 or 10-10-10 RGB encoding. The problem is that format readers and writers have to decide what to support or use somebody else’s (possibly license-encumbered) code to read a simple file. The imperfect solution to this — which is what DICOM, NITF, and TIFF have done — is to create a mechanism for specifying compliance level.
2. The “Infinite Configurability Problem” (a.k.a. JPEG-2000) — As a file I/O developer, you either have to take everything that comes your way or you have to tell pick what you think is important and hope that you pick the right set of features. HD Photo supports settings for chroma subsampling, overlap processing, frequency or spatial decoding priority, alpha interleaved by pixel or as a separate component, spatial transformations, gamut-management as part of the format, etc. These all have their place, but it might be a bit much. And JPEG-2000 has shown us that when you make a format too smart, it takes a long time to get adopted.

This brings me ’round to a thought I’ve had a lot recently. Image file formats have become platforms for working with data rather than containers for communicating images. The important thing is to get the data down on disk as quickly as possible and then change the interpretation of the pixels later. It looks like HD Photo might be self-contained, but some formats (like DNG or various HDR formats, including HD Photo) don’t include all of the information about how the image should look. If you add Adobe’s special sauce while reading the image in the DNG file, you’ll get a beautifully rendered image. If you use another vendor’s tools or provide other raw conversion settings, you will likely get an image that looks rather different. Similarly, there is no one correct rendering of an HDR image, and I don’t know of any formats that specify a preferred way of tone mapping.

It’s a brave new world of image file formats. We file format developers live in interesting times.

read more from this topic.....