First off, I’m not surprised. Like any device that transmits and receives wirelessly, the signals from pumps and CGMs are interceptable. Furthermore, like any other device that communicates with limited access control—you just need to know (or sniff out or be able to guess) the six or seven digit code that’s used to connect with another device—they’re essentially open. From there it’s all just figuring out the protocols and the format of the data as it’s passed around. As someone who spent about ten years working with and occasionally reverse-engineering formats, I can tell you, it’s all just a matter of trial and error and careful observation. (If I were a hacker, my handle would be “gluX0se.”)

So, in a world where relatively few people have these medical devices—unlike, say, mobile phones or bluetooth devices—the device manufacturers essentially did the easy thing, which was to assume we use our medical devices in a trustable world where people don’t mess with medical devices. (BTW, who knew there was a free Vulnerability Management for Dummies e-book?)

There’s been a lot of unease in the community about the way that the information was presented to the press and the way that some outlets sensationalized it (e.g., “Black Hat: Lethal Hack and wireless attack on insulin pumps to kill people”). It’s hard not to agree with a lot of the criticism there. But I can’t criticize looking for security holes in medical devices. Nor can I fault the impulse to hack into own’s own medical device—even one that keeps people alive—or to help other people hack their devices. Not all hacking is scary villainy, but this incident certainly exposes some problems.

Using the AP to share this information leaves a bad taste in my mouth, but presenting the findings at the Black Hat Conference seems like the most appropriate way to publicly disclose this research. (And it is, in my mind, legitimate personal security research that should be shared openly.) I would have preferred that Radcliffe work more closely with the device manufacturers leading up to the announcement. (I’m assuming that he did not.)

On the other hand, just presenting the findings to the device manufacturers—as some would have liked—violates the hacker ethos, both the black hat and white hat versions. Part of hacking—the part that I can get down with—is when motivated hobbyists exploit technology to solve a problem (real or imagined). I have thought many times how great it would be to sniff the unprotected data that’s transmitted by my pump/CGM and skip the middleman of uploading data to a web site. I’ve even gone so far as to seek out the information that Radcliffe presented, but it wasn’t available at the time.

Device manufacturers limit our access to our own medical data and tightly control the way that we can interact with our devices. It’s understandable given the limitations put on them by the FDA, their own desire to help (not harm) customers/patients, and their lawyers’ desire to limit risk exposure. It does mean, though, that the enormous potential for third-party, patient-focused tools goes untapped. Those tools could benefit so much from being able to present data the way that their users want to see them: A dashboard light in a car, a desktop computer widget that display CGM values, a mobile app that records all of the data for later use, a device that calls parents of children with diabetes when something happens, an awesome mood ring displaying BG, etc.

I suspect (and once again I’m assuming here) that Radcliffe was intrigued by the rather obvious possibilities of unprotected communication, and that’s getting lost in the whole “malicious people ruining diabetics’ lives” reporting. I fear the notoriety this incident is garnering is going to scare manufacturers into closing exploitable security holes without providing a secure, replacement method for getting at all of that data. And that’s a shame.

Real low on the world’s priority list is a patent-pending image compression algorithm and format that attempts to replace JPEG. (I’ve written why before, more than once.) But just in case I’m wrong — after all, JPEG-XR hasn’t really taken off like I thought it would three years ago — here’s a link to JPEGmini.

The makers say via Twitter that they’ll be at the same conference that I’m at right now. I will report more when I have details. (I sure hope these aren’t the same folks that I tried to talk out of adding a new JPEG format some years ago. Gosh, that would be awkward.)

While formats never really retire*, there’s a steady stream of rookies. Sometimes a format totally destroys the competition: PDF, JPEG, GIF, etc. (Being first helps, as does being in the right place at the right time.) Other times a new file format results from an actual deficiency for one community in an existing family of widely-used formats. Those formats — such as DNG, JPEG 2000, etc. — have rather more difficulty overcoming the inertia of the majority of data users’ workflows despite their superior qualities.

For example, DNG never really took off the way I had hoped. My Nikon D300′s RAW file is still NEF. As are all Nikon RAW files. And I’m not convinced that there are enough applications that support DNG in my workflow (beyond the obvious Adobe applications) for me to consider converting my .nef files to DNG on import. It’s a funny chicken and egg problem.

Add to this menagerie two new video file formats.

I don’t have a lot of video experience. Still photography was always more accessible and interesting to me, though I have to confess that I’ve been greatly enjoying editing the video from our trip to Australia. iMovie is surprisingly good at what it does, and the video coming out of my point-and-shoot camera is acceptable for reminiscing. I still like the story that a still photograph can tell, but video fits that niche that I always used to fill with babbling during my slide shows.

Anyway, I digress.

I don’t have a lot of video file format experience. Undoubtedly it’s more complicated than I know, but the sense I got was that there are a few widely used file formats — AVI, MPEG, Quicktime — with a variety of audio and video compression codecs, chroma subsampling settings, and bit depths thrown in to complicate what would otherwise be a very simple landscape.

Enter the consumer HD video revolution — partly thanks to a new generation of dSLR cameras — and it seems like we’re on the cusp of another explosion of proprietary file formats. Add in the demands of professional workflows, and you get two new file formats.

Just as it did with DNG for still cameras, Adobe is proposing CinemaDNG as an open file format for storing RAW files from digital video cameras.

Storing, retrieving, and manipulating the RAW pixels in a video frame only goes so far. Eventually those frames are edited, cut and combined with audio tracks. Those frames and audio are mixed with other assets, such as subtitles, alternate audio tracks, time codes, and other metadata. Finally all of these assets are combined with a desired output intent to create a digital or film copy for cinema projection, a television broadcast, a DVD, streaming video, etc.

The Entertainment Technology Center at the University of Southern California (ETC) has worked with industry players to develop an interoperable master format (IMF) that encapsulates audio, video, and effects assets together with metadata and output profiles into a package. Basically IMF is the file-level portion of a digital asset management (DAM) solution.

The details of this encapsulating master format are quite numerous, but the following might be of interest to people who need to contemplate support for reading and writing the imagery portions of IMF. The format is evolving, but as of version 0.82a these were true.

• IMF is pretty permissive with respect to image dimensions, audio sampling frequencies, bit depths, and so on. There are a lot of “shoulds” in the spec.
• “Essence files” contain the video and audio assets.
• Essence files must use ISO or SMPTE standard formats. That’s good news. I hate the reinvention of the wheel.
• Frame rates must be constant.
• There are some required standard and nonstandard resolutions and frame rates.
• Non-1:1 pixel aspect ratios are OK.
• 8- and 10-bit samples must be supported, and I/O drivers should support 12- and 16-bit imagery, too.
• 4:4:4 and 4:2:2 chroma sampling is allowed.
• RGB-709, YCbCr-709, YCbCr-601, and CIE XYZ are supported color spaces.
• 3-D/stereoscopic imagery must be supported.
• Compression is recommended, especially visually/perceptually lossless methods (but not necessarily mathematically reversible).
• Compression must be industry standard and open. In fact, it probably should look a lot like JPEG-2000.
• Uncompressed data will look a lot like DPX or SMPTE 384M.

Once again this is just the tip of the iceberg of the details are in the draft document. If you like these or don’t agree with them or if you have other suggestions — such as specifying a particular set of options and metadata settings as a “baseline” — do download the spec yourself and comment.

* — For an example of a moribund format, consider PICT from Apple.

Click for larger . . .

This envelope came in the mail yesterday. I don’t know who put me onto this mailing list, but I’m pretty sure it’s related to the work I’ve done over the last few years supporting the NITF file format, whose users are an interesting lot. They don’t really like to talk about what they do or what they keep in their files: secret stuff mostly.

I’m not one to judge. I’ll just say that I’m very glad that I was also responsible for adding support for the DICOM medical imaging format to MATLAB.

You can divide the world of image file formats into different kingdoms based on the their structure. There is some overlap between these categories, but for the most part image formats are (1) tag/record-based, (2) structure-like, (3) marker/stream-based, (4) textual, (5) card-like, (6) raw, or (7) opaque.

TIFF, DNG, and DICOM are examples of tag/record-based formats. A unique tag identifies the entity in the file and its meaning. For example, a particular hexadecimal tag might indicate that this is the “photometric interpretation” record. The datatype of this record either explicitly appears after the tag or appears in a data dictionary that’s known to the application developer. Almost always, these records explicitly tell the length of their data, which makes it easy to skip to the tag location of the next record.

Microsoft was (for a time) very fond of making structure-like formats. In these formats, the file looks a lot like the memory representation of a C/C++ data structure. These formats are easy to describe and easy to read if you have the structure definition; simply fread() the data into a variable and reference the data members by name. The problems should be pretty clear. You need to be using a programming language that supports C structs. And you need to know the layout of the struct. And once you define the layout of the struct, it’s fixed. (Well, not exactly. Microsoft changed the data layout in its BMP family of formats with every release of Windows, and used a “magic” value to tell readers which struct to use.) All told, it’s a very brittle kind of format.

JPEG is the prototypical — but certainly not the only — marker-based format. Markers are special combinations of bytes that, like a tag, tell what the data is that’s coming next in the stream. But, very much like struct-based formats and very unlike tagged formats, the data that follows the marker can be heterogeneous. In JPEG, the data that appears after the SOF (Start of Frame) marker is a record, while the data that follows an RSTn marker is just a stream of compressed bytes. The SOI and EOI (Start/End of image) markers don’t even have any bytes that follow them. In marker-based formats, semantics and syntax are rather carelessly jumbled together.

It’s very difficult to quickly parse marker-based formats, because often markers don’t specify how much data appears before the next marker. These are very much “streams” of bytes that you’re forced to read until you come to the next marker. Consequently the number and appearance of markers is very limited and this limitation ripples through to the data that they contain. JPEG markers all begin with the 0xFF byte followed by another byte, which taken together specify which marker it is. Consequently, the appearance of an 0xFF byte in the data of a marker has to be escaped by a NULL byte so that it’s not mistaken for the next marker.

Textual formats, such as XML, have the benefit of being self-describing and readable by both humans and machines. Their main drawbacks are the inflated size of the data they contain (even when represented in a semi-binary CDATA hunk) and the inability to quickly skip through them with binary I/O routines.

FITS is a fairly prototypical “card-like” format. As the name implies, these are fixed-length records like one might have encountered on a punch card. For example in format with 120-character records, the first n characters are reserved for the “variable name” part of the equation, while the remaining 120-n characters are the textual representation of the value of the record. They are frequently text-only for the descriptive part of the format with a binary payload at the end. These are easy to read, but a pain to parse, since the “right hand side” values often have to be interpreted.

Raw and opaque formats aren’t very easy to describe because they’re so varied. In a “raw” format (and there are dozens or hundreds . . . possibly more) all of the bytes are jumbled together in a payload-only file. A separate file may have a header that describes the data and helps a reader/parser make sense of the payload. Or not. These are almost always completely free of any helpful description within the file.

This shouldn’t be confused with opaque files, such as HDF, CDF, or netCDF. These formats are completely defined by their API, which for all intents and purposes, you have to use to access the data within the file. This allows for a lot of richness in handling the data contents, which can be organized in highly optimized ways. The downside is that you’re limited in how you can interact with your data to mechanisms someone else has defined. And data permanence can suffer, since if the tool chain changes (or goes out of existence) you don’t really have a way to get at your data.

Practically, each format style has it’s pros and cons. But tagged formats (which might incorporate features of the record style) are the most durable and easiest for third-parties to work with.

* — Cue awesome “CSI” + “Law and Order” + “House” mashup daydream. *DOINK DOINK*

My naïve idea was to copy the JPEG-encoded to a temporary file and then read that file using the Independent JPEG Group‘s libjpeg library. That’s what I did with JPEG data encapsulated in DICOM. This is far too simple an approach for NITF, resulting in incomplete images. Here’s why:

• NITF breaks most images into multiple tiles.
• Each tile is independently compressed into its own image stream.
• NITF uses “block masking,” which prevents storing unimportant tiles.

The idea makes sense on one level. If you’re going to send an image over a low-bandwidth or low-fidelity channel, you want to limit the amount of data that you send, and you want to avoid an all-or-nothing situation during image transmission or reception. But it’s a total pain in the ass for application developers.

Add to this the fact that JPEG is a marker-based format that isn’t very self-describing, and you have a tricky parsing situation.*

Here’s the basic idea behind getting imagery out of a NITF file if it’s been JPEG compressed. (I assume that you already know how to parse a NITF file — see MIL-STD-188-198A if you don’t — and that you have a JPEG codec that you can use to decode the data.)

1. The first two bytes of the compressed stream should be the standard JPEG SOI marker (0xFF 0xD8). This is your sanity check.
2. The next two bytes should be the APP6 marker (0xFF 0xE6). The payload of this marker contains a bunch of useful information about tile sizes and counts, bit depths, etc. Some of this is redundant with what’s inside the NITF file.
3. The remainder of the NITF file should be a bunch of JPEG codestreams delimited by SOI and EOI (0xFF 0xD9) markers. Each delimited stream is one tile in the image; and it’s a completely standalone JPEG stream. It can be extracted to its own file (if necessary) and decompressed. Tiles are stored across the image horizontally and then down.
4. If there’s no block masking, it suffices to read each tile in turn and store it in the appropriate region in the output image.
5. If the NITF file does use block masking, use the values in the BMRnBNDm attribute of the image subheader to find the locations of the blocks that contain actual image data. The masked out blocks will have 0xFFFFFFFF values. The other values — there’s one for each tile — are 0-based offsets pointing to the SOI marker that starts each tile, relative to the start of the JPEG compressed data.

And that’s it. After coding, you should probably test out your parser on the sample NITF files provided by the NGA.

* – You can divide the world of image file formats into different buckets based on the their structure. There is some overlap between these categories, but for the most part image formats are (1) tag/record-based, (2) structure-like, (3) marker/stream-based, (4) textual, (5) card-like, (6) raw, or (7) opaque. I’m going to write more about this in the next post.

Thank you. That is all.

UPDATE: You can also download an update directly from The MathWorks.

So now I’m a Senior Software Engineer in the Image Processing Group at The MathWorks. I still work in the same group with the same great people; only the projects have changed. Instead of programming file format interfaces, I’ll be working on software architecture and optimization. It’s definitely a growth opportunity for me, and I get to keep using a lot of the skills that I’ve gained over the last eight years.

But when you change jobs without changing offices, sometimes there’s a bit of overlap. And my interests haven’t changed radically; I just have more. So perhaps it’s not surprising that I’m still writing about JPEG here. Anyway . . . on with the show.

The most popular page on this web site covers the JPEG landscape. Believe it . . . or not. I’m not complaining. I just find it amusing that on a web site that touches on travel, photography, and (sometimes) software engineering, the most popular pages are about either the technical aspects of JPEG file formats or high dynamic range imaging. I guess that’s the price I pay for writing about the dozens of things that interest me.

Well, the JPEG family article has been gathering some really good comments. Most recently “pixpush” commented on a proprietary extension to classic JPEG that supports high dynamic range (HDR) and wide gamut imagery. And then he/she mused that JPEG would be even better if it could somehow support RAW data.

It would be great, but it’s never going to happen.*

Classic JPEG — the original JPEG that makes up all of our images — is what we might call “venerable.” There’s nothing really wrong with it. In fact, it’s very, very capable. But it’s an old dog with only so many tricks left in it. Unfortunately, the following things needed for RAW support are not part of its bag of tricks**:

• Lossless compression, which you absolutely need for so-called RAW imagery
• More than 8 bits per color component, since most cameras’ A2D converters use 10+ bits
• Wider gamuts than sRGB, which would require some combination of the following: converting to and from something other than YCbCr, using signed data, or somehow specifying the colorspace

It’s possible to put classic JPEG through its paces to do this, probably using the ill-supported lossless codec and extensions in new JPEG markers. (JPEG is a stream-oriented format — unlike TIFF — so you have to parse the stream for “markers” to find where new parts begin, making it hard to jump to “interesting” parts of the file.) But once you start making classic JPEG jump through those flaming hoops, you might as well go with one of the newer versions.

It’s unlikely that JPEG will “die out” in my lifetime. As long as there is data in a format it’s never really dead. (Unless, of course, no one knows what it means or the media dies.) But what format would I choose to replace it?

First, I’ll answer the question of what formats I like:

• TIFF. As long there are file systems that look like the ones we have today — files as sequential collections of bytes — the almost infinite extensibility of the Tagged Image File Format will be useful. You can put almost any kind of metadata into it now, and it’s user extensible (more or less). It supports a limitless number of samples per pixel, any bit depth you’d like, many colorspaces, ICC profiles, and a flotilla of compression modes. It’s also the basis of some vary capable formats such as DNG, and its data layout is used in EXIF, HD Photo/JPEG-XR, and other formats. TIFF and cockroaches will inherit the earth.
• DNG. Okay, so it’s more of a TIFF-based platform for describing RAW imagery than a traditional file format. You need to know how to interpret the format contents in order to get a viewable image, perhaps using a program like Adobe Camera RAW. Consequently, it’s possible for two applications to render the image quite differently. This is a very un-JPEG-like idea, but it brings back the flexibility and creativity of real-world negatives.
• JPEG-XR/HD Photo. I’ve written about this format before. It’s the heir-apparent to classic JPEG. And that’s not just because it’s from Microsoft.
• DICOM. Okay, okay. It has a lot of flaws. I mean, it can change byte order (endianness) in the same file . . . more than once. That’s messed up. To truly understand why it’s a good format, you’d have to be a trained professional, like me or federation president Barry Fife.
• HDF5. If you absolutely must store gigabytes of data using datatypes that you define, arbitrary metadata, and multiple datasets organized in a hierarchical file structure, this is your format. Of course, you’ll need to use an API to access your data, but you’re payin’ the cost to be the boss.

No one format that will replace JPEG, but I fully expect that a small collection of semi-standardized formats (JPEG-XR, DNG, TIFF) are going to fill the ever-growing image space that it doesn’t support well. And we haven’t even touched on HDR. There isn’t a standard HDR format yet, and there’s a lot to work left to do. (I’m really curious to see whether the “standard” HDR image format will include a preferred tone mapping method or whether it will just be a platform for imagery like DNG.)

Let’s see where the future takes file formats and me. . . . Stay tuned.

* — Except maybe as a joke or programming assignment.

** — Can you mix dog and cat metaphors like that?

JPEG: What most of us consider to be “JPEG” is just one of many processes for image encoding and decoding defined within the same specification. The process that makes up 99.99999% of all of the JPEGs ever created is “JPEG Baseline (Process 1)” for 8-bit lossy compression. (That’s just my estimate, which is probably low. It’s probably better to say “almost 100% of all JPEGs.”)

This process divides an image into a bunch of 8×8 blocks, uses the discrete cosine transform (DCT) to move the data into the frequency domain, and compresses the data by (among other things) removing some of the high frequency data that the human visual system usually can’t detect. You can think about it as abridging a novel by taking out a few sentences per paragraph. Unfortunately, if the quality settings are too low, it’s really easy to notice that something has gone missing; or if a scene has a lot of information — such as one with lots of fine detail — there will be blocky artifacts where there the detail should be.

While the removal of high-frequency detail is inherently lossy, even with a maximum quality setting, the original JPEG standard specified a separate lossless mode not based on the DCT. Images compressed this way can be completely retrieved from the compressed data. This is important when you need to preserve all of the data within in image or when adding artifacts can have devastating consequences. “Is that a nodule in the patient’s chest X-ray or a JPEG compression artifact? I guess we’d better do a biopsy just in case….” In fact, the lossless modes for JPEG are really only used within DICOM files, the format used for digital imaging and communications in medicine.

Old school JPEG also supports 12 and 16 bits of data in each channel of a pixel. For color images, this is the difference between about 17 million colors for an 8-bit image, 68 billion colors for a 12-bit image, and 281 trillion colors when using 16 bits. Once again, only those medical imaging people use the extra bit depths, and they just use the gray colors.

JPEG-LS was supposed to be a better lossless format but never really got going. The promises of JPEG 2000 probably had a lot to do with this.

JPEG 2000 is (1) a wavelet-based compression method, (2) a scheme for encoding wavelet-compressed images into randomly accessable “codestreams”, and (3) a file format for encapsulating compressed codestreams. Because it uses a discrete wavelet transform (DWT) the results are generally better than the older JPEG format when comparing images with the same compression ratio.

Images in JPEG 2000 can have an arbitrary bit depth (1 – 32 bps), and different planes can have different bit depths. (For example the luminance channel of a YCbCr image can have a high bit depth to support HDR imagery.) Certain portions of an image can have higher spatial resolution or be encoded at a different compression level. JPEG 2000 has both lossy and lossless components as part of the baseline. Several colorspaces are supported, including bi-level, grayscale, sRGB, YCbCr, and indexed imagery. Hyperspectral and n-sample images are supported using a somewhat convoluted “multi-component” schema. Images can also include alpha channels for transparency. A really amazing thing about JPEG 2000 is that its possible to reorder the parts of the codestream to change how the data is accessed (e.g. access regions faster v. access different resolutions faster) without decompressing and recompressing the data, which can be expensive.

The JPEG 2000 file format uses about 20 hierarchical “boxes” to nest metadata about the compressed codestreams. While the file format is technically unnecessary to read and process a JPEG 2000 image, the extra formatting facilitates random data access, long-term cataloguing and IP management, and efficient transmission. JPEG 2000 files can also contain a limited subset of ICC color profiles. EXIF metadata support is not part of the JPEG 2000 standard, although it can appear as a private metadata field.

JPEG 2000 was touted as the format to replace the 1991 JPEG standard, but this didn’t happen for several reasons. Perhaps most important, the algorithms at the heart of JPEG 2000 require a lot of processing power, making it slower for desktop computers than rendering old-school JPEG and prohibitive for many embedded devices. As of 2007, few Web browsers have built-in support for it, and consumer-level digital cameras don’t produce imagery in the format. In 2007, Adobe Photoshop CS3 stopped including the JPEG 2000 export module in a typical installation.

But because of the smaller file size, flexibility, and more pleasing artifact appearance, the medical and remote sensing communities have adopted it. Both NITF and DICOM have incorporated JPEG 2000 data into their files. NITF is the friendly format used for “national imagery.” I will let you Google that so the NSA can start tracking you.

JPEG-XR is the name that Microsoft’s HD Photo format might have if it’s standardized, which I sincerely hope it will be. JPEG-XR uses a principal components photo core transform (PCT) which I know absolutely nothing about but which promises equivalent performance to JPEG 2000 with lower computational complexity — which means you can put it on a consumer device more easily — and much better size-versus-quality performance compared to the original JPEG format. It also supports more bit depths, high dynamic range imagery, lossy and lossless encoding/decoding using the same algorithm, and wide gamut color; uses a linear light gamma making it possibly suitable to replace RAW formats or enable post-CRT workflows; and can store bucketloads of metadata including EXIF and XMP.

JPEG-Plus. And then there’s JPEG+, which you might reasonably call JPEG – 20% because it’s essentially the same as the original DCT-based JPEG with a modest file-size performance improvement and some claims about better visual appearance. I’m not holding my breath for it; but given the 29+ processes that made up the original JPEG standard, what’s an extra one that no one will implement?

Update: For posterity, the PCT stands for “Photo Core Transform” not “Principal Component Transform”. Thomas Richter said this about it on sci.image.processing:

The transform is an overlapped 4×4 block transform that is related to a traditional DCT scheme, or at least approximates it closely. The encoding is a simple adaptive huffman with a move-to-front list defining the scanning order, and an inter-block prediction for the DC and the lowest-frequency AC path of the transformation.

Some parts are really close to H264 I-frame compression, i.e. the idea to use a pyramidal transformation scheme and transform low-passes again (here with the same, in H264 with a simpler transformation).

The good part is that lossy and lossless use the same transformation. The bad part is that the quantizer is the same for all frequencies, meaning there is no CSF adaption, and the entropy coder back-end is not state of the art.

Update 3-February-2009: JPEG-XR has advanced to “draft standard balloting,” which means it’s very likely it will become a standard (unless everyone hates it, of course). More info…

Once upon a time you could count on Microsoft’s image file formats to suck. Just take BMP as an example: a more-or-less unpublished format that changed between revisions of Windows and which you were supposed to access primarily through Windows API calls. File I/O people (like me) needed to have the bits of information that we could crib together handy so that we could figure out what datatype a given piece of data should have because the file didn’t carry that information in it. Nor was there a data dictionary to programmitcally look it up. They couldn’t have designed it worse if they had tried; but they clearly didn’t do any (good) design on the format. My professional opinion: totally sucked.

But perhaps they’ve learned the error of their ways with their new HD Photo format, which they have submitted for standardization. It appears that they’ve talked to a lot of digital image users about their needs and given a lot of thought to the format. I’m about halfway through the WinHEC 2007: HD Photo Implementation Guidelines presentation, which certainly says all the right things.

My only worry is that they’re saying so many of the right things that they’re possibly heading toward a couple of bad outcomes. Let’s call these the unintended consequences of designing a “good” format:

1. The “Kitchen Sink Problem” (a.k.a. DICOM) — Everybody gets what they want, including things of questionable utility like CMYK + alpha and 5-6-5 or 10-10-10 RGB encoding. The problem is that format readers and writers have to decide what to support or use somebody else’s (possibly license-encumbered) code to read a simple file. The imperfect solution to this — which is what DICOM, NITF, and TIFF have done — is to create a mechanism for specifying compliance level.
2. The “Infinite Configurability Problem” (a.k.a. JPEG-2000) — As a file I/O developer, you either have to take everything that comes your way or you have to tell pick what you think is important and hope that you pick the right set of features. HD Photo supports settings for chroma subsampling, overlap processing, frequency or spatial decoding priority, alpha interleaved by pixel or as a separate component, spatial transformations, gamut-management as part of the format, etc. These all have their place, but it might be a bit much. And JPEG-2000 has shown us that when you make a format too smart, it takes a long time to get adopted.

This brings me ’round to a thought I’ve had a lot recently. Image file formats have become platforms for working with data rather than containers for communicating images. The important thing is to get the data down on disk as quickly as possible and then change the interpretation of the pixels later. It looks like HD Photo might be self-contained, but some formats (like DNG or various HDR formats, including HD Photo) don’t include all of the information about how the image should look. If you add Adobe’s special sauce while reading the image in the DNG file, you’ll get a beautifully rendered image. If you use another vendor’s tools or provide other raw conversion settings, you will likely get an image that looks rather different. Similarly, there is no one correct rendering of an HDR image, and I don’t know of any formats that specify a preferred way of tone mapping.

It’s a brave new world of image file formats. We file format developers live in interesting times.