Creation of HDR Images in CinePaint

Abstract
The text describes the creation of a High Dynamic Range image from bracketed exposures using a CinePaint tool.

Table of Contents

• Introduction
  • HDR Images
  • About this text
• Theoretical Background
• The CinePaint Tool
  • Exposure Parameters
  • Numerical Parameters
  • "Follow-up Curves"
  • Shifts
  • Workflow
  • The Play of Colors of the Buttons
    • De/Activation of Images
    • The "Init" Button
    • The "Compute Response" Button
    • The "HDR" Button
• Conclusion
• Acknowledgment

Concerning contrast and detailedness, the brightness information coded in HDR images normally exceeds the dynamic range of usual reproduction media as monitor or paper. For instance, if the whitewashed wall is represented by the brightest, i.e. ungrayed white, then the brightest sun will appear just as white ("bright") — the white of the paper or screen, whereas, in terms of measurable radiance intensity the wall may have (in any arbitrary unit) a value of 1, the sun corresponds to a value of 1000! One of the advantages of HDR images is that the decision on which information will be of interest does not have to be made until the moment of looking at it (instead of during the shot). Another advantage is that HDR images provide a basis for improving the contrast at selected spots, or to generate more realistic images of high-contrast scenes by using algorithms modeling the automatic contrast compensation of our sense of sight.

About this text

The radiation emanated or/and reflected by the objects of a scene goes through the aperture and lens system and causes a certain light intensity E at a pixel in the plane of the photo sensors. During the exposure the amount of E is supposed to be constant. Integrated over the exposure time Δt we get from E the light quantum X = E⋅Δt, which defines the exposure result. In a following digitization process X is transfered by an unknown non-linear function z=f(X) into an integer value z (e.g. z=0...255 for 8-bit converters) — values which we read out later as the digital RGB values. The "response function" z=f(X) is a device property, it characterizes the entire digitization process X --> z of a color channel including photo sensor characteristic and possible camera-internal correction algorithms. Even the intermediate step of a first chemical developed photo material, which has been digitized later, can be incorporated here. The task is, for given z-values of an exposure bracketing and known exposure times to reconstruct the function z=f(X)=f(E⋅Δt), and its inverse, and finally to calculate the values of E. The intensity values E constitute the content of a HDR image.

Four assumptions are made:

1. That the response function z=f(X) is strictly monotonic, i.e. that an increasing of the light quantum X always causes an increasing of z; — a reasonable assumption at least within the working range of a photo sensor. Then the inverse function f^{-1}(z) = X exists.
2. That in all exposures of a bracketing series the same response function was active. (Of course, as otherwise no correlation exists between the exposures, but this assumption could be undermined if, depending on the exposure situation, the camera internally uses different correction algorithms.)
3. That all the pixels on the photo sensor of a camera (within a color channel) have identical response functions. Spatially, over the picture, varying z-values can then be interpreted as views at different operating points of the same function z=f(X), only with different arguments X.
4. That the intensity E applied at a fixed pixel is the same in all exposures of a bracketing series (static scene + identical camera position).

  z_ij = f(E_i⋅Δt_j)   for all i=1...N, j=1...P

  g(z_ij) = ln(E_i) + ln(Δt_j)  for all i=1...N, j=1...P,

  g(z) := ln[f^{-1}(z)]

      N  P
  Ω = ∑ ∑ [g(z_ij) - ln(E_i) - ln(Δt_j)]^2 + λ ∑ g''(z)^2
      i  j                                     z

Because the objective function Ω is quadratic in the unknowns g(z_k) and ln(E_i), minimizing of Ω leads to a linear least squares problem, which is solvable with straightforward standard methods like QR-decomposition or singular value decomposition (SVD).

After the function g(z) has been found in this way and so exp(g(z)) = f^{-1}(z), the z-data can be converted via f^{-1}(z) = E⋅Δt into their related intensities E: E_i = f^{-1}(z_ij) / Δt_j.

Three Remarks:

1.) The solutions for E and f^{-1}(z) are unique only so far, up to an arbitrary constant factor α>0, because the equations remain complied, if ln(E) is replaced by ln(E)+ln(α) and g(z) by g(z)+ln(α). To make it unique, the additional condition g(z_mid)=0, i.e. f^{-1}(z_mid)=1 is added, where z_mid means the middle z-value. I.e. it is stipulated that the middle z-value corresponds to the light quantum X=1. This is a priori a mere arbitrary stipulation about the absolute amount of the X- respectively of the E-values, which therefore can have only relative meaning. For absolute radiance data a rescaling would be necessary. Further this normalization happens for each color channel independently, so that a gray tone is postulated at this point. However, especially for camera raw data, a sensor output of (z_mid, z_mid, z_mid) doesn't have to represent a gray tone necessarily, and a subsequent color correction would be needed.

2.) Digital cameras often work with motif-dependent correction algorithms. If the same algorithm is used for all pictures of an exposure bracketing, this is not a problem, one will merely get (perhaps) different response functions depending on the motif. However, it would be problematic if different correction algorithms are used within an exposure bracketing (assumption 2 violated) or on the other hand, if zones within an image would be post-processed differently (assumption 3 violated). Though a result will be achieved then — in the end it is a minimization task — but perhaps not a good one. One should be on the safe side with camera raw data.

3.) The assumed monotonicity of the response function is not enforced in the numerical procedure, it can result also non-monotonic curves. This is often a hint at problematic ("against rules") input data.

The CinePaint Tool

The tool is started in CinePaint via the <Toolbox>-menu "File" -> "New From" -> "Bracketing to HDR". It opens a separate main window, wherein under "File" -> "Open" the images of an exposure bracketing are loaded; all image formats supported by CinePaint are allowed. The images are sorted automatically by average brightness per pixel, the darkest one on top, and thus assumed to be the shortest exposed.

In the input field "smoothing" one can adjust the parameter λ mentioned above. The goal should be to get a monotonic, smooth response function with as small a λ as possible. Here one can experiment a little bit. I got fine results with λ=10...20 for input data I had confidence in. Understood that with a sufficient high λ>200...500 one can enforce smooth, monotonic response functions even for most heavily irregular input data (see section "Follow-up Curves", whose response curves oscillated wildly for λ=20), but of course the substance of such curves tends to zero.

"Follow-up Curves"

  (9,43,122), (15,55,135), ...  (N exemplars)

Currently it is done this way: At first the "follow-up curves" are acquired which means that: For all (z=0)-points in image 1 their corresponding values in image 2 are looked up and the average <z> of all those as well as their deviation Δz is determined. This gives for z=0 the triple (0, <z>, Δz). And so for all z-values 0, 1, ..., z_max in image 1. These averages <z> plotted over z gives the follow-up curve of image 2 to the reference image 1. Of course every other image could be used as a reference image as well. Such curves are shown in the tab "FollowUpCurves", the deviation is represented by vertical error bars. Follow-up curves are a good indicator of the quality of a data base. Ideally their deviation should be minimal and their slope uniform. Below on the left is an example, where the middle image (green curve) was the reference image. (The "pseudo follow-up curve" of the reference image itself is always a straight line and without deviation and is plotted only for orientation purposes.) On the right we see the same exposures, but the blue image was displaced sideways here by an arbitrary 20 pixels against both the others. The deviation of the blue curve is now considerable and the behavior non-monotonic. Clearly, a response function determined from such a data base will look strange and the related HDR image could have some odd color gradients. We also note this: in that situation, had the blue image be chosen as reference image, then both follow-up curves (green and red) would be abysmal instead of just the blue one. By the way, the black line shows the histogram: the number of points in the reference image with the corresponding z-value; what's missing is a scale at the right side.

From the follow-up curves, namely those for the currently chosen reference image, the N tuples mentioned above are selected internally (N vertical slices through a follow-up curve diagram). A valuation of the neighborhood of the pixel thus doesn't take place at present. Here remains room for alternate algorithms.

Shifts

After loading the images and adjustment of the exposure data, take a first look at the quality of the input data in the "FollowUpCurves" diagram; play with different reference images! To avoid frequently switching between the two tabs cards "Images" and "FollowUpCurves" in the next steps, you can get the second panel in a separate window via the menu item "Analyze" -> "FollowUpCurves".

Now, to decide on the method of generating the N tuples. At present this is restricted to selecting a reference image, choosing one for which the follow-up curves take on a favorable shape. Later on, alternate algorithms could be written, but more probably this whole task will be automated, at least for the default choice.

Clicking the "Compute Response" button starts the calculation of three response functions, one for each color channel, which can be evaluated in a window which will open automatically. Any time a non-monotonic characteristic is cause for concern, making one check the choice of the reference image (in the "FollowUpCurves" panel) or try a higher smoothing coefficient λ and recompute the response functions. If no reasonable characteristics can be found with moderate λ ≤ 50, a problem with the input data probably exists. Currently there are four parameters which affect the response functions for a given image data set: 1) the exposure times, 2) the number N of grid points, 3) the smoothing coefficient, 4) the reference image. Because the calculation of response functions is the most time-consuming step, re-calculations after changes in any one of the four are not done automatically, but require clicking the "Compute Response" button.

A click on the "HDR" button computes the resultant HDR image using the last calculated response functions; it appears in a new window as a CinePaint image. The "32-bit IEEE Float" in the window title announces, that indeed a floating point image has been created. With the CinePaint color picker <image menu> -> "Tools" -> "Color Picker" RGB values can be read out, and you will notice that with floating point numbers where. By convention, 1.0 is the brightest representable value in each color channel, in a HDR image, defined values can now occur which are greater than 1.0.

The CinePaint tool called in the <image menu> under "Image" -> "Colors" -> "Gamma-Expose" [3] allows simulation of various exposures with its slide control "expose" and allows you a quasi review of the light situations of the input images, but now continuously. With the slide control "gamma" we can deviate from the linearity (γ=1) of the brightness transmission profile. Values over 1.0 emphasize the dark and the bright domains and move differences in brightness, which had disappeared into the black or white, into the visible, of course at the cost of the contrast of the previous middles. This allows us to make visible e.g. subtleties of the filament together with nuances from dark regions in a single image, as demonstrated by the above example. Of course, more sophisticated manipulations will avoid the graying of the mid-tones, but this is beyond our subject here, where we are generating the data base for such things.

The Play of Colors of the Buttons

By default, newly loaded images are always activated.

The "Init" Button

In summary: The "Init" button is blue if no HDR calculator exists, it is normal colored if an initialized HDR calculator exists and the used images coincide with the currently activated ones, it is yellow if a difference has occurred in activated images; and it is — in which color ever — inactive (grayed out), if a (re)initialization to the present set of activated images is impossible.

The "Compute Response" Button

Hartmut Sbosny    <hartmut.sbosny@gmx.de>
Chemnitz, 2005

Footnotes