Category Archives: Fodder for Techno-weenies

Artificial Pancreas Workshop slides

Posted on November 11, 2010 by Jeff Mather

Here are some slides from yesterday’s NIH/FDA Artificial Pancreas Workshop. Click any image for a larger version.


Public Workshop on Artificial Pancreas (AP)

Slides from John Mastrototaro, Ph.D., VP Global R&D Medtronic Diabetes


Product opportunities to integrate toward AP


Predictive low-glucose suspend and resume


FDA questions for AP vendors


Percentage of glucose values in target range using AP

Slides from Henry Anhalt, DO, Medical Director Artificial Pancreas, Animas Corp.


AP development roadmap


Closed-loop (AP) versus open-loop (traditional) blood glucose


Model predictive control


Possible methods for improving pharmacokinetics/pharmacodynamics


What’s involved in moving the AP forward

Posted in Diabetes, Fodder for Techno-weenies, NaBloPoMo, NaBloPoMo 2010 | Leave a comment

Artificial Pancreas Workshop

Posted on November 10, 2010 by Jeff Mather

Man, I wish there were times that I didn’t have to work. Don’t get me wrong; I love my job and get a lot of satisfaction from the work that I do and from being with my coworkers. The MathWorks is a great company, and I wouldn’t want to work anywhere else. But sometimes it would be nice if I could just take the day off to attend or follow one of the diabetes events that people discuss on Twitter as it happens: EASD, FDA meetings, Epatcon, ADA scientific sessions, and so on.

Today it was the FDA and NIH’s public workshop on the artificial pancreas project (APP). From the little bits of the live webcast that I heard during lunch, the project is engaged in numerous clinical trials at several locations using multiple different technologies. It would seem that there’s also a lot of room for debate between the doctors, researchers, and FDA regulators about how to interpret the results of the trials and what the standards for safety and efficacy really are.

So many things seem so promising. Some pumps can predict low blood glucose and then suspend background insulin delivery, preventing hypoglycemia from occurring. Other solutions involve a “closed loop” between the insulin pump and the continuous glucose monitor (CGM), constantly adjusting the amount of insulin delivery in response to changing levels of blood sugar. Stopping lows is one thing; preventing highs seems much more difficult.

Everybody said the right things: a solution should prevent lows, minimize the intensity and duration of post-meal blood glucose spikes, be easy to use by people with diabetes, and make our lives better. Of course, for the doctors and regulators, making our lives better means reducing the number of complications and the risk of death from hypoglycemia. Those are important, but I think that most of us with diabetes think that “making our lives better” involves feeling like we have a greater degree of influence over our blood glucose, making our daily choices and actions easier.

But I was surprised by how much disagreement there was between the doctors on the research and regulatory sides. How many hours of in-clinic trials do you need to do before you can start out-patient trials? How much can your rely on software and hardware simulation (“in silica” v. “in vivo”)? If CGMs are reliable enough to use in the project, why aren’t they reliable enough to gather the data needed to validate the research findings? What level of higher blood glucose is acceptable if a pump prevents 90+% of all lows? Can you look at outcomes instead of just A1c numbers?

Those are all great questions, but answering them to the FDA’s satisfaction is going to take a lot of time. It’s good that the FDA purports to take an impartial look at the technology, free from conflicts of interest. But in my mind, they seem like they’re really dragging their feet when it comes to patient-operated technology. It’s hard for me to see so many promising technologies and wonder how long we’re going to have to wait before the products are available on the market. Patricia Beaston, one of the regulators, was very skeptical of integrating medical devices with mobile devices. (It’s enough to make someone like me say things like, “We don’t need your stinking FDA approval.”)

The variability — one might say “inaccuracy” — of CGM sensors and blood glucose meters seems to be the limiting factor in the current technology. The algorithms appear fairly robust, but the data they receive is frequently suspect. As one of the speakers said, the CGM sensors are quite accurate near the calibration points, but not so much as you get farther away. (Those less accurate zones seem likely to be the ones where critical decisions about insulin delivery need to happen.) I wish the FDA would apply the same amount of scrutiny and rigor to existing sensors and monitors that they’re applying to the rest of the artificial pancreas project.

Expect to see some more thoughts on the APP here in the coming days.

Posted in Diabetes, Fodder for Techno-weenies, Health Care, NaBloPoMo, NaBloPoMo 2010 | Leave a comment

Performance Programming Links

Posted on October 22, 2010 by Jeff Mather

Faithful readers: This dispatch falls into the bucket of “stuff that’s about work and not diabetes.” (Not that diabetes isn’t its own kind of work.) This is what I do for my 9-5. It’s the stuff I do between thinking about diabetes, my next bike outing, where I want to travel someday, and all those other things. Don’t worry, we’ll get back to all that “good stuff” soon enough; but this post is for the lurking coworkers and people searching on the Google.

Here are a few performance-related articles and presentations that I stumbled upon recently:

Igor Ostrovsky at Microsoft wrote a very useful description of various processor cache effects. Here are the notes I took while reading it:

  • Array access is slower than math
  • Data alignment can determine how many cache lines are touched
  • Keep modifications/accesses within the same cache level
  • Be aware of instruction level dependencies and parallelism opportunities
  • Avoid touching too many memory locations from the same cache line, causes contention
  • There’s a trade-off between the size of what you’re putting in the cache and the number of elements you touch in it. — What does that mean, Jeff? Not exactly sure, but cache assoc. is usually not a huge deal compared to other issues.
  • Because cores have their own caches and because memory needs to remain consistent, avoid letting two threads modify data on the same cache line.
  • Even when you think you know what you’re doing, there’s other crazy stuff going on.

Isn’t that last one the sad truth about performance optimization?

Igor also has an excellent article about branch prediction. Basically, if you structure your code so that if branches are predictably true (or predictably false) the CPU can start walking down that code path until it’s proved wrong. But if it’s close to random, you’ll see performance hits.

Joe Duffy, also at Microsoft, debunks the “premature optimization is evil” myth. Joe summarizes the article himself quite well, so I’ll just quote him.

In this short article, we’ll look at some important principles that are counter to what many people erroneously believe this [“avoid premature optimization”] statement to be saying. To save you time and suspense, I will summarize the main conclusions: I do not advocate contorting oneself in order to achieve a perceived minor performance gain. Even the best performance architects, when following their intuition, are wrong 9 times out of 10 about what matters. (Or maybe 97 times out of 100, based on Knuth’s numbers.) What I do advocate is thoughtful and intentional performance tradeoffs being made as every line of code is written. Always understand the order of magnitude that matters, why it matters, and where it matters. And measure regularly! I am a big believer in statistics, so if a programmer sitting in his or her office writing code thinks just a little bit more about the performance implications of every line of code that is written, he or she will save an entire team that time and then some down the road. Given the choice between two ways of writing a line of code, both with similar readability, writability, and maintainability properties, and yet interestingly different performance profiles, don’t be a bozo: choose the performant approach. Eschew redundant work, and poorly written code. And lastly, avoid gratuitously abstract, generalized, and allocation-heavy code, when slimmer, more precise code will do the trick.

Follow these suggestions and your code will just about always win in both maintainability and performance.

If you do anything that falls under the labels of “multicore,” “multithreaded,” or “multi-processor,” then you should definitely add Multicore Info and Intel’s software blogs to your RSS feed aggregator.

One recent Intel offering is a recording of “Introducing Intel Parallel Building Blocks”. The Intel folks discusses Intel Cilk Plus, Threading Building Blocks (TBB), and Array Building Blocks. Here are some notes from the 45 minute presentation:

  • Intel Cilk Plus is a lower-level way to expose parallelism and the potential for vectorization to C++ code. It currently only works with the Intel compiler.
  • TBB is awesome. That’s me saying that. I use it to easily exploit data parallelism in our existing codebase. (Because I use it already, I didn’t take many notes on it. Sorry.)
  • Array Building Block (ArBB) provide the highest level of abstraction presenting the ability to specify array data containers (1-D to N-D, including nested containers and [soon] sparse data) and then do vector operations on the data in the containers. Sounds familiar.
  • Cilk Plus includes reducers to prevent data races, has #pragma simd to empower the compiler to make vector assumptions, and has a mechanism for specifying a range of vector data.
  • Parallel Inspector and Parallel Amplifier are alleged to help squeeze performance out of code that uses TBB.
  • ArBB uses a two-part compilation process, one of which is a runtime JIT.
  • There are collisions between the schedulers in Intel MKL and Cilk (and TBB, etc.). Beware.

And finally, there’s a recording of a webinar about Intel Parallel Advisor that may be of interest. It’s next in my queue of things to watch.

Posted in Computing, Fodder for Techno-weenies, Software Engineering, Worthy Feeds | Leave a comment

Insulin and Holy Grails

Posted on September 29, 2010 by Jeff Mather

I seem to have something like the anti-Midas touch going on today at work. Instead of turning everything I touch into pure gold, it turns into a freaking neutron bomb, blowing away my MATLAB session when I run the test suite. I’m taking some existing image processing code and using it in a multithreaded harness I wrote a while back. The goal is to easily exploit data parallelism for the newish generation of multicore CPUs. You can see some of the resulting performance wins — though I didn’t do all of the work, and it doesn’t all use this framework.

Anyway, I’ve managed to kill MATLAB more times today than I’d care to admit because of some threading issues in my sandbox. My sanity needs a break while I get myself into a frame of mind to look for threading issues. (That pretty much comes down to asking yourself, “What’s not happening here? What other threads might be giving this data ‘the bad touch?’”) While I let my brain settle down, why not put the final touches on this post about insulin?

Insulin is our frenemy, but what do we really know about it? Here are some recent articles about insulin.

  • Mind the GAPP: Insulin Therapy Woes. An industry group discovered “interesting” things about insulin — such as, 1/3 of people with diabetes (PWDs) skip insulin doses or don’t follow their doctor’s Rx exactly — and along the way manages to get some of us patients a bit indignant.
  • Insulin and Bone Fractures [PDF]. A paper presented at EASD last week exposes the correlation between insulin and bone fractures. It seems that insulin can cause us to be clumsy and fall when we are driving the magic bus have hypoglycemia. Other than being more inclined to feel the effects of gravity, our bones are otherwise unchanged.
  • Novo says cancer risk with insulin drops over time. Evidently there’s some link between insulin and cancer, but the incidence decreases the longer one is on insulin. “Furthermore, the marked drop in risk with long-term insulin use indicate that other factors than insulin may play an important role.” Huh. . . .
  • Sanofi-Aventis Seeks the Holy Grail of Insulin. Um. Okay. This article, which is written for people trying to decide which bio-pharma stocks to buy, has a pretty good recap of the three new fast-acting insulin analogues that may be coming to market within the next few years: Mannkind’s Afrezza inhaled insulin, Biodel’s Linjeta injected insulin, and Halozyme’s rHuPH20 insulin additive/friend/thing. There’s a fair amount of doubt in the minds of PWDs about inhaled insulin — especially since recent research “demonstrates that inhaled insulin rapidly forms amyloid in the lungs causing a significant reduction in pulmonary air flow.” Yikes. And while I know almost nothing about it, Provasic rHuPH20, according to the Seeking α article, uses “an enzyme called hyalronidase that locally degrades body components under the skin to enable quicker absorption of a drug (like insulin) into the blood.” I think I’ll pass. Linjeta sounds cool: faster than Novolog, Humalog, or Apidra; undergoing safety and efficacy tests for injection and insulin pump usage; not likely to “disrupt tissue structure;” etc. But about that “holy grail” comment. . . . I have to disagree with the idea that an insulin to mix with Lantus so that PWDs can have one meal-time-slash-any-time insulin is really equivalent to the Cup of Christ. Maybe for the drug companies, since some type 2 PWDs will have something new and expensive to replace their 70/30 insulin once it goes off patent, but c’mon. Let’s keep some perspective. Srsly!
  • New insulin could cut need for daily injections. Er, “some” daily injections. That is, insulin degludec — if effective and safe — might be able to replace your daily Lantus injection with a thrice-weekly injection. “Researchers found that the variability of treatment effects was lower among patients given insulin degludec than standard treatment.” And “an accompanying study showed that insulin degludec injected three times a week was as effective at lowering HbA1c levels as existing type-2 diabetes therapy.”
  • Biopharmaceutical Companies Developing a Record 235 Medicines to Treat Diabetes and Related Conditions [PDF]. In case one of the four insulins above doesn’t get your juices flowing, here are 231 other diabetes medications for you.
  • The implant that can free you from insulin jabs. First off, I didn’t know there was a link between insulin injections and migraines. The article is kind of a hot mess, but it also picks up the Grail theme, saying that the “holy grail” of diabetes is an “artificial pancreas,” but not the one you’re probably thinking of. No, this one, which is undergoing preclinical trials, “is an implant placed above the hip containing a supply of insulin kept in place by a barrier of gel that reacts to glucose.” More glucose, more insulin. Easy, peasy. Right?
  • Lilly R&D Chief Pursues Late-Stage Drug License Deals. Yup, like Sanofi-Aventis, Lilly needs more of our diabetes-related total healthcare dollars, too.

None of this solves the basic question at hand of my inadvertently destroying MATLAB, but I think I’m ready to get back to it. Except, there’s a cookie here staring at me. If I eat it that means an insulin bolus, but that will make exercise more difficult. Choices. Choices. If only there were an awesome, Grail-looking chalice here to drink and/or dunk my cookie into.

Posted in Diabetes, Fodder for Techno-weenies | 2 Comments

Reading a Freestyle Blood Glucose Meter with MATLAB

Posted on September 10, 2010 by Jeff Mather

Here’s some MATLAB code that will read data from a Freestyle blood glucose meter that is attached to your computer via serial cable. (Be sure to replace the “smart” quotes with appropriate single quotes. And the “…”, too. Sorry.)



% Jeff Mather - 3 October 2001
% Connect to meter.
s = serial('com1', 'baudrate', 19200, ...
           'FlowControl', 'Software');
fopen(s);

% Initiate communication.
fprintf(s, 'mem');
fprintf(s, 'log250');

% Read diagnostics.
fgetl(s);   % First line is empty.
meter_ID     = fgetl(s);
software_ver = fgetl(s);
current_time = fgetl(s);
num_readings = str2num(fgetl(s));

for p = 1:num_readings

    reading = fgetl(s);
    sprintf(reading)
    
    pause(0.25);
    
end

fclose(s);
Posted in Computing, Data-betes, Diabetes, Fodder for Techno-weenies, MATLAB | Leave a comment

The Ultimate Blood Glucose Meter

Posted on September 8, 2010 by Jeff Mather

Let’s talk gadgets.

Usually I can hold off getting things with buttons and displays. I’m a guy whose “smartphone” is an unlocked multi-band phone from the middle of the last decade that we took to India and that now mostly sits in my bag uncharged unused. It has a camera and Bluetooth, and I can play Monopoly on it (using a Java virtual machine) but that’s pretty much it. Fancy MacBook Pro laptop (from 2006)? Yes. 4th-gen iPod (ca. 2007)? Also yes. iPad/iPhone/Android/everything else? Not so much. After that decision to get an overpriced HP iPAQ hx4700 with a short shelf-life in early 2005, I’ve been reluctant to get many expensive, shiny things.

I’m definitely not opposed to gadgets, though. After thinking about the experience of recording my diabetes events at the end of last month, I’ve decided that I would like one shiny, new thing. Unfortunately, what I want — an awesome, new blood glucose meter — doesn’t exist yet. What would such a sugasheen have?

First off, it would be part of an integrated system. It would work with my pump, sending BG readings to it, similar to the way my OneTouch Ultra Link does with my Minimed Paradigm pump. It would also accept sensor data from a CGMS.

It would be easy to get data off of it. We’re talking as simple as docking an MP3 player, preferably using a standard USB connector. The data should be open and sharable. I hate that Medtronic’s software lets me download all of my pump’s events — including suspensions, temporary basals and bolus wizard suggestions — but then stores them in a password-protected Microsoft Access database that I can’t open because they won’t share the password or let me change it. And unlike the Freestyle connection cable, Medtronic won’t provide the API calls that I could use as an application developer to retrieve the data from my pump/meter. It’s my data. They don’t own it. It’s my life I’m trying to improve. Meter manufacturers should help me do that, right?

It would be open. There are lots of ways to slice and dice and visualize data if you have access to it. I should be able review my data, get averages, display events, and plot my activity in a billion different ways: time series, modal day, statistical box plots, histograms, even pie charts. Basically, if you can think of something to do with BG data, it should be possible to get the data and do it. That doesn’t have to happen on the meter itself, but it could. Wouldn’t that be sweet?

It would have lots of metadata. My Freestyle meter didn’t have any metadata at all. My OneTouch has two axes: before/after meals and a limited set of comments. But I find the events don’t often describe the reading I’ve just taken. “Not enough food” and “too much food” might describe the experience of people with type 2 or those unfortunate souls still on NPH, but I find myself wanting something like “miscalculated carbs” and “infusion set change.” I would love to be able to define my own events — especially a “WTF?!” comment — and be able to tag particular readings for follow-up with my endocrinologist or her nurse.

I would love to be able to type short, free-form notes to myself about my BG and bolus events: “Bolus wizard said 5.0 units, but I took 4.0 because of exercise” or “Had some ketones.” We’re talking Twitter-length here (140 characters) not Anna Karenina. Since a lot of diabetes is tweaking things, the perfect meter software would let me remember what I’ve tried and try it again or change it up next time.

As long as we’re talking about notes and readings and metadata, let’s talk sharing. I would love to be able to e-mail a portion or all of my log to my healthcare team for consultation. There’s no need to post my results to Twitter, but then again, why not?

Switching to the nuts-and-bolts, getting-the-BG-reading level, let’s try to think beyond blood and consumables: The ultimate BG meter would take blood and test strips out of the picture. Strips are expensive and have to be disposed. Pricking fingers isn’t exactly painful for me (I know it is for some people) but it is a bit 20th century. How about something that uses subcutaneous light scattering or UV “tattoos” that change depending on BG levels? However it might work, we’re talking about something noninvasive that doesn’t require another trip to the pharmacy every month in order to buy something I’m going to bleed on and (responsibly) throw away.

And, of course, it would be very accurate. None of this ±10% nonsense. We’re occasionally making life-or-death decisions with those BG readings, especially when deciding how to treat a high BG. The values should be precise and accurate enough that we can’t overcorrect because we’re in the margin of error.

It would also be possible to use this meter at night. LEDs are cheap. Use them. Put one in the front so that I can see my finger and another behind the display so that I can read the results w/o blinding myself at 4:00AM.

And for the love of g-d, if you’re going to require batteries, use normal ones that you find at a gas station on the NJ Turnpike or at the Eiffel Tower gift shop. AAAs would be nice.

Since blood glucose self-monitoring is the first step in a decision-making process, the ideal BG meter would help me with the rest of the process. Get a BG reading, add in the name and amount of the thing I’m going to eat, and voilà the carbs show up for the bolus wizard. I should be able enter in foods that I frequently eat as well as downloading nutritional information for those place we won’t tell our nutritionist about. If I’m going for a bicycle ride, the decision-support part of the system could remind me what works and suggest some snacks or insulin pre-gaming that I need to do based on my own history. It could even point my attention to the fact that I’ve been going high around 4:00 most afternoons for the last month and suggest that I bring it up with my healthcare team.

Finally, it would be at least as small as my current meter. If it has to be a separate device, it should fit into a stylish case that won’t get nasty after a year of constant use. But . . .

Look at this:

iPod Touch

Look closer!

iPod Touch

The data ports on the iPod and iPhone are perfect for attaching something small that can communicate with the software on the device. The ultimate BG meter would just attach to an iPhone or iPod Touch (or Android device), adding the BG reading capability to a platform that is perfect for everything else. Basically, I want a smartphone with “an app for that,” where “that” is diabetes.*

BG database with flexible journaling? There’s an app for that.

BG results analysis? There’s an app for that, too.

Bolus wizard support using a food database? E-mailing results? Decision support? Apps for them, of course.

These devices already have a backlight for those late-night readings and wireless connectivity to integrate CGMs and insulin pumps.

And we already carry them around. Or at least I would go buy one now if this were available. I’d probably buy two.


* — Look at what Apple has already done with the point-of-sale (POS) system in their own stores. “Running an Apple supply chain? There’s an app for that.”

Posted in Computing, Data-betes, Diabetes, Fodder for Techno-weenies | 6 Comments

CGM

Posted on September 8, 2010 by Jeff Mather

This is the third part of a triptych of diabetes posts. The first part references the time just before my diagnosis. The second part covers the most recent year of my post-diagnosis life. Here I cover some recent developments and provide a view of the near future.

What follows is one of the few things I remember from the classical Greek class I took in my senior year at Grinnell: “᾽ͅΟμοι κακαδαιμων. Ἐγενετω τυφλως. Ω Ζεω, χαλαπως ἐστι ὁ βιος.” That pretty much means “Oh the misfortune! I have become blind. Oh Zeus, life is so hard!” As someone who fears going blind, that story in the primer had a bit of an impact. Another thing I clearly remember is the inscription I put at the top of my final exam: “᾽ͅΕγαρ μα πραχοιμι κακος.” (I hope that I don’t do badly.) Athena didn’t help me as much as I’d hoped.

What does this have to do with diabetes, or with continuous glucose monitors (CGM) for that matter? Not much really. I’ve always just loved those sentences, and any time I hear “blind” or “blinded” they instantly spring to mind.

Last week, I wore a “blind” CGM to gather data for my endocrinologist. Yes, for a week — from Thursday to Thursday — I had a small device attached to me recording close approximations of my blood glucose every five minutes. A thin filament was inserted horizontally into my skin — it didn’t hurt, honest! — and the readings were recorded in a little device about the size of two stacked quarters that was attached to the sensor. Everything was held in place by a boat-load of tape. I couldn’t see the data as it was recorded, hence “blind.”

I was told to do what I normally do with two exceptions. #1: Take at least four blood glucose finger-stick readings every day using their special meter. No problem there. I usually test 7-10x day, and I seemed to test extra knowing I had this thing gathering loads of data. #2: Record everything I did on paper. Blood glucose tests. Food. Insulin. Exercise. Infusion set and insulin changes. Everything. I made a photocopy of the log (five pages!) before sending it all back to Medtronic Minimed.

I should have the results back sometime in the next week, but here are my thoughts after a week of wearing a CGM sensor:

  • I hope to finally know what’s happening overnight. Do I go low and rebound? Do I steadily climb throughout the night? Do I wake up with high readings because of dawn phenomena that I can counter with extra insulin?
  • What really happens when I exercise?
  • Does bolusing 10-15 minutes before I eat make any difference for me? I hope to find out.
  • Wearing a CGM is totally something that I could do, and I definitely want to know when I’m going up or down too much.
  • Wearing the sensor didn’t hurt at all, although I’m not sure I could put one on my own back.
  • One strip of tape over the sensor was not enough, and I had to add a full metric tonne more over the week.
  • Logging (or journaling or whatever) is a total pain in the ass, but I did start to notice some trends on my own.
  • “Observations of daily living” (ODLs) should be as easy to make as a blood sugar test or an insulin bolus. It’s currently a pain in the ass to integrate all of the data that people with diabetes use in our heuristics for self-management, but there’s no need for it to be that way.
  • I don’t eat nearly enough fruits and vegetables. I’m glad a dietitian isn’t looking at my logbook.

Hopefully, I’ll be able to glean a lot of useful information out of this week-long endeavor. And if the accuracy is there, I also hope that I can get insurance approval for CGM sensors that will send data to my insulin pump. I’ll keep y’all posted.

Posted in Data-betes, Diabetes, Fodder for Techno-weenies | 5 Comments

New Video File Formats

Posted on August 26, 2010 by Jeff Mather

File formats come, and file formats go. Strike that last part. File formats never really go away. People just stop storing data in them, and vendors stop supporting the formats in their products. Eventually the data is just a bunch of bits that nobody really cares about. (At least that’s how I feel about most of the papers that I wrote in college.)

While formats never really retire*, there’s a steady stream of rookies. Sometimes a format totally destroys the competition: PDF, JPEG, GIF, etc. (Being first helps, as does being in the right place at the right time.) Other times a new file format results from an actual deficiency for one community in an existing family of widely-used formats. Those formats — such as DNG, JPEG 2000, etc. — have rather more difficulty overcoming the inertia of the majority of data users’ workflows despite their superior qualities.

For example, DNG never really took off the way I had hoped. My Nikon D300′s RAW file is still NEF. As are all Nikon RAW files. And I’m not convinced that there are enough applications that support DNG in my workflow (beyond the obvious Adobe applications) for me to consider converting my .nef files to DNG on import. It’s a funny chicken and egg problem.

Add to this menagerie two new video file formats.

I don’t have a lot of video experience. Still photography was always more accessible and interesting to me, though I have to confess that I’ve been greatly enjoying editing the video from our trip to Australia. iMovie is surprisingly good at what it does, and the video coming out of my point-and-shoot camera is acceptable for reminiscing. I still like the story that a still photograph can tell, but video fits that niche that I always used to fill with babbling during my slide shows.

Anyway, I digress.

I don’t have a lot of video file format experience. Undoubtedly it’s more complicated than I know, but the sense I got was that there are a few widely used file formats — AVI, MPEG, Quicktime — with a variety of audio and video compression codecs, chroma subsampling settings, and bit depths thrown in to complicate what would otherwise be a very simple landscape.

Enter the consumer HD video revolution — partly thanks to a new generation of dSLR cameras — and it seems like we’re on the cusp of another explosion of proprietary file formats. Add in the demands of professional workflows, and you get two new file formats.

Just as it did with DNG for still cameras, Adobe is proposing CinemaDNG as an open file format for storing RAW files from digital video cameras.

Storing, retrieving, and manipulating the RAW pixels in a video frame only goes so far. Eventually those frames are edited, cut and combined with audio tracks. Those frames and audio are mixed with other assets, such as subtitles, alternate audio tracks, time codes, and other metadata. Finally all of these assets are combined with a desired output intent to create a digital or film copy for cinema projection, a television broadcast, a DVD, streaming video, etc.

The Entertainment Technology Center at the University of Southern California (ETC) has worked with industry players to develop an interoperable master format (IMF) that encapsulates audio, video, and effects assets together with metadata and output profiles into a package. Basically IMF is the file-level portion of a digital asset management (DAM) solution.

The details of this encapsulating master format are quite numerous, but the following might be of interest to people who need to contemplate support for reading and writing the imagery portions of IMF. The format is evolving, but as of version 0.82a these were true.

  • IMF is pretty permissive with respect to image dimensions, audio sampling frequencies, bit depths, and so on. There are a lot of “shoulds” in the spec.
  • “Essence files” contain the video and audio assets.
  • Essence files must use ISO or SMPTE standard formats. That’s good news. I hate the reinvention of the wheel.
  • Frame rates must be constant.
  • There are some required standard and nonstandard resolutions and frame rates.
  • Non-1:1 pixel aspect ratios are OK.
  • 8- and 10-bit samples must be supported, and I/O drivers should support 12- and 16-bit imagery, too.
  • 4:4:4 and 4:2:2 chroma sampling is allowed.
  • RGB-709, YCbCr-709, YCbCr-601, and CIE XYZ are supported color spaces.
  • 3-D/stereoscopic imagery must be supported.
  • Compression is recommended, especially visually/perceptually lossless methods (but not necessarily mathematically reversible).
  • Compression must be industry standard and open. In fact, it probably should look a lot like JPEG-2000.
  • Uncompressed data will look a lot like DPX or SMPTE 384M.

Once again this is just the tip of the iceberg of the details are in the draft document. If you like these or don’t agree with them or if you have other suggestions — such as specifying a particular set of options and metadata settings as a “baseline” — do download the spec yourself and comment.


* — For an example of a moribund format, consider PICT from Apple.

Posted in Computing, File Formats, Fodder for Techno-weenies, Photography, Video | Leave a comment

Ten Things I Love about Adobe Photoshop CS5

Posted on August 2, 2010 by Jeff Mather

I just recently upgraded from Adobe Photoshop CS to Photoshop CS5. As you might imagine, a lot has changed in four major releases over the last seven years. I know I should have upgraded sooner . . . blah blah blah.

After installing CS5 over the weekend, I gave it a quick go and was immediately pleased. Tonight I used it a bit more, and now I’m even happier. Who knows how many of these are new to CS5? Not me. Anyway, here are ten things I really like:

Screenshot #1 of Adobe Photoshop CS5
Click for larger.

Screenshot #2 of Adobe Photoshop CS5
Click for larger.

  1. The “Adjustments” panel — I can work on multiple layers without being locked into editing one layer at a time. This makes me sooooo happy.
  2. Photo filters — Yeah, so they’re not real photo filters, since they don’t work on color spectra. But who cares? It’s 2010, not 2020. At least now I can easily make a warming filter.
  3. Adjustment presets — A lot of the curve shapes and levels adjustments are now precooked. Just select the right one from the drop-down list.
  4. Nicer panels — The way to move around and resize panels/palettes is just like I’d expect.
  5. More adjustments shiz — Okay, I guess I really like the adjustment layer improvements, because I honestly can’t remember what I was going to say here. I think it might have had something to do with being able to browse the precooked adjustments. Maybe? I don’t know.
  6. New adjustment layers — Easy to get to, right there next to the adjustments I want to make.
  7. Tabs — Tabs for files . . . Up there on the top, right where they belong.
  8. Action menus — The context menus appear on each panel, and the Photoshop engineers have located most of the common actions in the action menus, making them easier to find.
  9. Workspace browser — I like very much that Adobe is customizing the Photoshop experience for various communities via workspace. Click a button and you’ve changed your Photoshop experience. The panels move around, and (perhaps controversially) the menus change. I’ve saved my panel configuration as my own workspace.
  10. Mini Bridge — Do I like this or don’t I? I’ve never used Bridge effectively before. I used the Mini Bridge tonight to find and load a file; it seemed alright.

And of course there’s stuff you can’t see here, stuff buried in the menus. Some of them are brand new features to CS5, and some — like many of the items listed above — are only new to me.

I can’t wait to see what else is new and how it will continue to improve my retouching and editing workflow.


* — A couple years ago I was part of the Photoshop CS4 beta. As part of that, I used the Photomerge feature to create some panoramas from multiple photographs. I loved it then, and I’m excited to try it out with some photos from Australia. Stay tuned.

Posted in Computing, Fodder for Techno-weenies, Photography | Leave a comment

Uluṟu Sunset

Posted on July 4, 2010 by Jeff Mather

Well, I finally made it work. After striking out with the PC — more like deciding I didn’t want to stay up all night on vacation searching for Windows software to make the video — I fired up iMovie HD earlier this evening and (eventually) built the video below from 83 still photos from the 19th of June.

You can read all about our awesome Aussie adventure:

Photos are on the way posted. Diabetes down under, too!

Posted in Australia, Fodder for Techno-weenies, Photography, Travel, Video | Leave a comment

A Menagerie of Image File Formats

Posted on May 18, 2010 by Jeff Mather

This is a follow-up to my recent post on parsing NITF files that contain JPEG data. It’s basically a crash course into the organization of the guts of image file formats. If I were ever asked to be an expert witness in a trial, it would probably be about file formats.* This is the area of my expertise.

You can divide the world of image file formats into different kingdoms based on the their structure. There is some overlap between these categories, but for the most part image formats are (1) tag/record-based, (2) structure-like, (3) marker/stream-based, (4) textual, (5) card-like, (6) raw, or (7) opaque.

TIFF, DNG, and DICOM are examples of tag/record-based formats. A unique tag identifies the entity in the file and its meaning. For example, a particular hexadecimal tag might indicate that this is the “photometric interpretation” record. The datatype of this record either explicitly appears after the tag or appears in a data dictionary that’s known to the application developer. Almost always, these records explicitly tell the length of their data, which makes it easy to skip to the tag location of the next record.

Microsoft was (for a time) very fond of making structure-like formats. In these formats, the file looks a lot like the memory representation of a C/C++ data structure. These formats are easy to describe and easy to read if you have the structure definition; simply fread() the data into a variable and reference the data members by name. The problems should be pretty clear. You need to be using a programming language that supports C structs. And you need to know the layout of the struct. And once you define the layout of the struct, it’s fixed. (Well, not exactly. Microsoft changed the data layout in its BMP family of formats with every release of Windows, and used a “magic” value to tell readers which struct to use.) All told, it’s a very brittle kind of format.

JPEG is the prototypical — but certainly not the only — marker-based format. Markers are special combinations of bytes that, like a tag, tell what the data is that’s coming next in the stream. But, very much like struct-based formats and very unlike tagged formats, the data that follows the marker can be heterogeneous. In JPEG, the data that appears after the SOF (Start of Frame) marker is a record, while the data that follows an RSTn marker is just a stream of compressed bytes. The SOI and EOI (Start/End of image) markers don’t even have any bytes that follow them. In marker-based formats, semantics and syntax are rather carelessly jumbled together.

It’s very difficult to quickly parse marker-based formats, because often markers don’t specify how much data appears before the next marker. These are very much “streams” of bytes that you’re forced to read until you come to the next marker. Consequently the number and appearance of markers is very limited and this limitation ripples through to the data that they contain. JPEG markers all begin with the 0xFF byte followed by another byte, which taken together specify which marker it is. Consequently, the appearance of an 0xFF byte in the data of a marker has to be escaped by a NULL byte so that it’s not mistaken for the next marker.

Textual formats, such as XML, have the benefit of being self-describing and readable by both humans and machines. Their main drawbacks are the inflated size of the data they contain (even when represented in a semi-binary CDATA hunk) and the inability to quickly skip through them with binary I/O routines.

FITS is a fairly prototypical “card-like” format. As the name implies, these are fixed-length records like one might have encountered on a punch card. For example in format with 120-character records, the first n characters are reserved for the “variable name” part of the equation, while the remaining 120-n characters are the textual representation of the value of the record. They are frequently text-only for the descriptive part of the format with a binary payload at the end. These are easy to read, but a pain to parse, since the “right hand side” values often have to be interpreted.

Raw and opaque formats aren’t very easy to describe because they’re so varied. In a “raw” format (and there are dozens or hundreds . . . possibly more) all of the bytes are jumbled together in a payload-only file. A separate file may have a header that describes the data and helps a reader/parser make sense of the payload. Or not. These are almost always completely free of any helpful description within the file.

This shouldn’t be confused with opaque files, such as HDF, CDF, or netCDF. These formats are completely defined by their API, which for all intents and purposes, you have to use to access the data within the file. This allows for a lot of richness in handling the data contents, which can be organized in highly optimized ways. The downside is that you’re limited in how you can interact with your data to mechanisms someone else has defined. And data permanence can suffer, since if the tool chain changes (or goes out of existence) you don’t really have a way to get at your data.

Practically, each format style has it’s pros and cons. But tagged formats (which might incorporate features of the record style) are the most durable and easiest for third-parties to work with.


* — Cue awesome “CSI” + “Law and Order” + “House” mashup daydream. *DOINK DOINK*

Posted in Computing, File Formats, Fodder for Techno-weenies, Life Lessons | Leave a comment

NITF + JPEG

Posted on May 18, 2010 by Jeff Mather

I’ve recently been working with streams of JPEG data inside of NITF files. Given my experience supporting I/O involving DICOM files that contain JPEG-compressed imagery, I was extremely surprised to learn how difficult it is to read JPEG from “National Imagery Transmission Format” files. This post exists to help the next person who needs to read JPEG data embedded in NITF or another file format.

My naïve idea was to copy the JPEG-encoded to a temporary file and then read that file using the Independent JPEG Group‘s libjpeg library. That’s what I did with JPEG data encapsulated in DICOM. This is far too simple an approach for NITF, resulting in incomplete images. Here’s why:

  • NITF breaks most images into multiple tiles.
  • Each tile is independently compressed into its own image stream.
  • NITF uses “block masking,” which prevents storing unimportant tiles.

The idea makes sense on one level. If you’re going to send an image over a low-bandwidth or low-fidelity channel, you want to limit the amount of data that you send, and you want to avoid an all-or-nothing situation during image transmission or reception. But it’s a total pain in the ass for application developers.

Add to this the fact that JPEG is a marker-based format that isn’t very self-describing, and you have a tricky parsing situation.*

Here’s the basic idea behind getting imagery out of a NITF file if it’s been JPEG compressed. (I assume that you already know how to parse a NITF file — see MIL-STD-188-198A if you don’t — and that you have a JPEG codec that you can use to decode the data.)

  1. The first two bytes of the compressed stream should be the standard JPEG SOI marker (0xFF 0xD8). This is your sanity check.
  2. The next two bytes should be the APP6 marker (0xFF 0xE6). The payload of this marker contains a bunch of useful information about tile sizes and counts, bit depths, etc. Some of this is redundant with what’s inside the NITF file.
  3. The remainder of the NITF file should be a bunch of JPEG codestreams delimited by SOI and EOI (0xFF 0xD9) markers. Each delimited stream is one tile in the image; and it’s a completely standalone JPEG stream. It can be extracted to its own file (if necessary) and decompressed. Tiles are stored across the image horizontally and then down.
  4. If there’s no block masking, it suffices to read each tile in turn and store it in the appropriate region in the output image.
  5. If the NITF file does use block masking, use the values in the BMRnBNDm attribute of the image subheader to find the locations of the blocks that contain actual image data. The masked out blocks will have 0xFFFFFFFF values. The other values — there’s one for each tile — are 0-based offsets pointing to the SOI marker that starts each tile, relative to the start of the JPEG compressed data.

And that’s it. After coding, you should probably test out your parser on the sample NITF files provided by the NGA.


* – You can divide the world of image file formats into different buckets based on the their structure. There is some overlap between these categories, but for the most part image formats are (1) tag/record-based, (2) structure-like, (3) marker/stream-based, (4) textual, (5) card-like, (6) raw, or (7) opaque. I’m going to write more about this in the next post.

Posted in Computing, File Formats, Fodder for Techno-weenies, Life Lessons | Leave a comment

Buckshot o’ Links – Software Development Edition

Posted on April 16, 2010 by Jeff Mather

I’m a hoarder. I may not have lived through the Great Depression like my grandmother did, but I seem to have inherited the gene that led her to keep dozens of plastic Cool Whip tubs in her attic “just in case she needed them.” My grandfather kept used bolts and nails for reuse. Me, I keep articles about things I think I should know some day. From smart people on Twitter and e-mail lists and coworkers I gather links to articles, PowerPoint presentations, blog posts, and videos. And they hang out in my browser tabs — forever.

It’s time to clear some of them out. Here are some daytime-themed links. (If I had a Tumblr account, I’d just post there. But I don’t need another website no one reads.)

Parallel Computing
James Reinders of Intel TBB fame estimates that datasets (images, videos, etc.) have grown 10x larger over the last five years. Sequential systems are just too dang slow to process this amount of data. In the near future, “parallel programming” will just become “programming.” (I’m still hoping for better language support so that state synchronization and multicore memory issues are as easy to get right as the sequential aspects of programming are now.)

New memory models would certainly help. A new paper, DMP: Deterministic Shared Memory Multiprocessing by Joseph Devietti, Brandon Lucia, Luis Ceze, and Mark Oskin presents some of the problems with the current memory model and provides one possible solution.

Another post on SoftTalk (sponsored by Intel) describes some of the ways that Intel plans to make parallel programming easier this year: Parallel Studio 2010, a Cilk-based offering for task parallelism, “a data-parallel centric model with safety guarantee,” new SIMD instructions, and new array notations.

You might also want a high-level view of how Intel’s offerings work together.

Herb Sutter, who really knows his stuff, wrote an article for Dr. Dobbs a couple years ago about understanding parallel performance. It’s one of a series of articles about multicore/multithreaded programming, and this helps set expectations about what’s possible and gives pointers on where to start making changes.

Miscellaneous
Visual Studio + Time Machine = IntelliTrace. Don’t just move up and down the call stack; now you can move forward and backward in time, too.

Everything you ever wanted to know about floating-point representation: Floating Point Guide. (Everything, that is, unless you work where I do. Then you just have to go down the hall to get that last 2%. Of course, you’ll be drinking from the fire hose. . . .)

What does Microsoft think are the key trends in software development? Cloud computing, the web as a platform, parallel computing, proliferation of devices (with their own capabilities, IO paradigms, etc.), agile development processes, distributed development. Nothing terribly futuristic here, and comments want to know why “mobility” (i.e., phones) isn’t on the list.

The C++0x “final draft” revision is ready for “final” comments. Here’s your chance to see the significant changes to C++ that will be part of the standard next year (they hope).

Posted in C, Computing, Fodder for Techno-weenies, Hoarding, Software Engineering | Leave a comment

What’s the Right HbA1c Target?

Posted on November 16, 2009 by Jeff Mather

In my most recent post I mentioned the matter of an on-going debate over the proper target for HbA1c values. I went back and listened again to the debate at this year’s EASD conference on the “right” A1c target. (You can find it by starting at the EASD webcast page and searching for “Berger”.)

The talks by Drs. Irl B. Hirsch and Andrea Siebenhofer-Kroitzsch are chock full of information, and I’ll summarize them here. I’ve also taken the liberty of including some of their slides. Any mistakes in recapping these presentation are entirely due to my limited knowledge of epidemiology and shouldn’t reflect on the good doctors arguments. Also, there’s a lot of statistics in these presentations; and while I studied mathematics as an undergraduate, I went the abstract math route and am rubbish with statistical jargon.

First, Dr. Hirsch advocated for a target of 6.5%.

  • An A1c of 6.5% works out to an average of 144 mg/dL or 7.8 mmol.
  • But 6.5% actually has a fairly large overlap with 7.0% if you use the 95% confidence intervals of each.
  • So, there’s not that much of a difference between 6.5 and 7.0%.
  • 6.5% is protective. There’s a small but significant reduction in the risk of complications with an A1c below 7.0%.
  • And there’s an “exaggerated increased risk of hypoglycemia” below 7.0% for type 1.
  • The DCCT (which suggested that there was increased risk of severe hypoglycemia with better A1c) was reported in 1993. It’s a different time now. JDRF continuous glucose monitor (CGM) studies show that risk is way down at all levels of A1c to the same as very high A1c.
  • Lower A1c values are protective for type 2 as well, but treatment is a “moving target.”
  • No matter what you target, the actual is going to be higher (often by as much as 0.5%). — So why not aim lower?
  • Furthermore, doctors have been told “additional action” is required if the A1c is more than 1.0% over the target — so people with diabetes (PWDs) often aren’t getting told to change until well into the danger zone.
  • Does the ACCORD study indicate that trying to achieve normal A1C results in increased mortality? [Yikes!]
  • As in other trials, a higher average A1c in ACCORD is associated with a higher risk of death: +22% for each 1% of A1c.
  • And with intensive treatment, the risk of death continues to decrease below 7.0% — Not so with “standard treatment strategy.” [I believe the "intensive treatment" group aimed at lower A1c.]
  • Moreover, the people with the highest risk of mortality in the study, were the people with the lowest decline in A1c.
  • The causes of death in ACCORD weren’t differentiated, though.
  • Of course, the target value is individualized based on other factors. “Clinical judgement always trumps guidelines.” “If you can’t get there, don’t push it.”

Dr. Andrea Siebenhofer-Kroitzsch used meta-analysis of several different diabetes trials to argue that we can’t tell definitively whether improving glycemic control (as measured by a lower A1c) actually helps.

  • There are many other factors that along with A1c confound what causes improved/worse outcomes. Factors such as social status.
  • So far, there are only seven randomized, controlled trials on A1c (DCCT is by far largest with 1441 patients).
  • The age of people in the DCCT were quite young, so it was hard to see macrovascular complications (such as heart disease) directly from the study.
  • And, the ACCORD study had to be stopped because intensive treatment had a higher mortality for some groups. [Seriously?]
  • Some studies showed more hypoglycemia and more weight gain with aggressive treatment.
  • Other changes are more effective in preventing complications: blood pressure, cholesterol reduction.
  • Don’t confuse intensive self-management with A1c.
  • Tight glycemic control is very beneficial for young people with type 1. Admits that people with type 1 need tight glycemic control because of length of treatment. [But type 2 is occurring much earlier in age, one audience member reminded us.]
  • Lots of open questions about the value of glycemic control.
  • Individualized treatment is more important, almost suggesting letting patients to set their own targets as long as a minimum level of outcomes are met.

Here are my reactions and some questions I had while listening to the session:

  • It’s good to see that the increased use of CGM technology can dramatically reduce the number of severe hypoglycemia episodes. Given what it costs to go to the ER, there’s another argument in favor of covering CGMs and supplies more widely by insurance carriers.
  • The meta-analysis was very . . . meta. Consequently it pushed very hard on the difference between “proof” and “connection.” A lower A1c may suggest better outcomes, but it doesn’t prove that the lower A1c caused it. I get this, but the data are compelling to me.
  • Almost all of Dr. Siebenhofer-Kroitzsch charts suggest favoring more intensive treatment, although that seemed to contradict the core of the presentation. Perhaps I don’t know how to read them correctly? [Mo meta, Mo problems.]
  • Does the fact that clinical trials never met their A1c targets mean that people with diabetes (PWDs) are going to feel even worse about not meeting A1C targets if the target is lowered to 6.5%?

Ultimately, based on the questions and comments from the audience, the debate sounded like a draw. I think I’m in Dr. Hirsch’s camp, and I would ultimately like the lowest A1c I can safely get; but I don’t think I’ve ever really been close to breaking below 6.5%. So maybe Dr. Siebenhofer-Kroitzsch’s camp and the ADA have a point worth considering carefully when looking for a number to cover all of us.

Some of Dr. Hirsch’s slides:

Some of Dr. Siebenhofer-Kroitzsch’s slides:

Posted in Diabetes, Fodder for Techno-weenies, From the Yellow Notepad, NaBloPoMo, NaBloPoMo 2009 | Leave a comment

The Free Advertising Issue

Posted on June 17, 2009 by Jeff Mather

It’s time for another hodge-podge posting. This time, I’ll just give random shout-outs.

Image Engineering: Dietmar Wueller has a great company that makes test equipment, targets, and software. I’ve had the good fortune of working a booth in a trade show next to them (more than once). I trust their expertise completely.

WyoFOTO, LLC: My mother and step-father are photographers and have a little web site of their own.

Trek 2.1 Road Bike: Every month brings a new event to distract me from posting here. In June, it’s my new bike.

Twitter: With millions of users, Twitter doesn’t really need me to get the message out. I think it’s a nice way to follow a small number of people who post links to photography, design, and software engineering articles/blogs/etc. that might interest me. Others use it differently — sometimes I do, too.

Flickr: They don’t need my advertising either.

The Wire: The final three episodes of “The Wire” should arrive in my mailbox today (via Netflix). If you’ve ever been hooked on the show, you’ll know another reason why I’m not posting more.

Posted in Color and Vision, Cycling, Fodder for Techno-weenies, Photography | Leave a comment