Single-Shot High Dynamic Range Imaging with Conventional Camera Hardware

Keigo Hirakawa and Paul M. SimonIntelligent Signal Systems Laboratory, University of Dayton

300 College Park, Dayton, OH 45469{k.hirakawa,simonpam}@notes.udayton.edu

Abstract

A combination of photographic filter placed over the lensand the color filter array on image sensor induces differ-ences in red, green, and blue channel sensitivities. Spec-trally selective single-shot HDR (S4HDR) imaging treatsthis as an exposure bracketing. Optimally exposed regionsof low dynamic range red/green/blue color components aremerged in a principled manner to yield a single HDR colorimage. Though not expected to yield results superior to thetraditional time multiplexing counterparts, the single-shotHDR solution we propose is a robust alternative that can berealized with conventional camera hardware.

1. IntroductionMost conventional single-chip color image sensors today

make use of color filter array (CFA), a spatial multiplex-ing of absorptive red, green, and blue filters placed over anarray of pixel sensors. Due to differences in translucency,more photons are allowed to penetrate through green filterscompared to red or blue filters. These differences becomeeven more noticeable when an additional filter designed toattenuate certain spectra of incoming light is applied to theoptical path of a camera. Whether these properties of CFAand photographic filters can be exploited for the purpose ofhigh dynamic range (HDR) imaging is a question that hasreceived surprisingly little attention in the extant literature.

In this paper, we investigate the plausibility of recoveringa HDR color image from a conventional sensor hardware ina single shot. Appealing to the notion that red, green, andblue components of natural color images are highly cor-related, we prove that color channels of raw sensor com-prise a set of overdetermined observations with diverse ex-posure rates. As suggested in Figure 1, optimally exposedregions of low dynamic range (LDR) red/green/blue colorcomponents are merged in a principled manner to yield oneHDR color image based on rigorous image formation mod-els. This approach will be referred to as spectrally selectivesingle-shot HDR, or “S4HDR” for short.

The work presented here overcomes major obstacles inHDR imaging solutions considered to date. Known as ex-posure bracketing, an HDR image is recoverable from LDRimages that comprise a set of overdetermined observationswith an adequate coverage of the dynamic range[22, 9, 17,11, 7] or with the help of flash[12, 23]. Though exposurebracketing is typically implemented as a “time multiplex-ing” of exposure rates, capturing moving objects pose aparticular challenge; time multiplexing also precludes HDRvideo and “hand held” camera applications. Single-shot al-ternatives have been proposed as a way to capture images ofnonstationary objects. For instance, an exposure mosaick-ing and assorted pixels implement spatially varying pixelexposures [20, 21, 18]. Multisensor and alternative pixelarchitecture solutions have emerged in recent years as well[27, 25, 14, 19, 26]. However, these single-shot HDR so-lutions require a special purpose or modified hardware—amajor drawback that would disfavor a wide adaptation bythe camera manufacturers in practice.

Notwithstanding the above limitations, the S4HDR ap-proach we propose here is an outgrowth of existing workin HDR imaging. Indeed, in the conventional setting, thekey to blending multiple LDR images together to form asingle HDR image is to draw from optimally exposed re-gions of LDR images that are least affected by quantization,noise, and saturation [22, 9, 17, 11, 7]. In this paper, wecarry out similar tasks, but with a particular attention paidto the spectral selectivity. Our work is also closely relatedto assorted pixels, which incorporate spatial mosaicking ofdifferent colors and exposures[20, 21, 18]. Key difference,however, is that we do away with exposure mosaicking be-cause it is already implied in color mosaicking. We alsodraw from demosaicking methods [8, 6, 1, 4] and CFA de-signs [10] that successfully exploit spatial-color correlationstructure. There exists a separate body of work in HDRimaging that focus on camera response curve [22, 17, 2, 24]and tonemapping [13]—these aspects do not enter into thecurrent work because S4HDR is a sensor-level design.

The proposed single-shot HDR imaging solution hasbroad implications in today’s imaging market. Because it

Figure 1. A combination of photographic filter placed over the lens and the color filter array on image sensor induces differences inred, green, and blue channel sensitivities. Spectrally selective single-shot HDR (S4HDR) method treats this as an exposure bracketing.Optimally exposed edge/texture information gathered from LDR red/green/blue color components are merged in a principled manner toyield a monochromatic HDR highpass image. The lowpass image in the saturated regions of the image is projected on a set of “plausible”colors. The final HDR reconstruction is the sum of the lowpass and highpass images.

makes use of existing hardware, HDR capability can be de-ployed to digital still/video/cell phone cameras with a sim-ple firmware update. Cameras with a limited bit depth data-path would enjoy the performance of a more complex hard-ware. Proposed solution is ideal for vehicular vision sys-tems that work with rapidly changing environments and dy-namic scenes. With some optimization, it may even becomepossible in the near future to postprocess already stored,lightly compressed JPEG images to re-enable HDR render-ing. Though the proposed single-shot HDR solution is notexpected to yield results superior to the time multiplexingcounterparts, none of these benefits are available to the ex-isting single-shot or multishot HDR solutions.

2. Motivation

Consider the proposition that color channels of raw sen-sor comprise a set of overdetermined observations with di-verse exposure rates—two requisite conditions for exposurebracketing. CFA-based exposure bracketing is analyzedbased on empirical data and color image models.

2.1. Color Filter Translucency

One can assess the translucency of color filters in a con-ventional image sensor by performing a simple calibrationexperiment. Suppose raw sensor measurements of Gretag-macbeth colorchecker are captured under a simulated sun-light (see Figure 2(b)). Regressing the sample means of thepixel values observed within the same “squares” onto pub-lished reflectance values produces a matrix A ∈ R3×3 thatmaps the sensor RGB to the linear sRGB space. For NikonD90, for example, the color space transformation matrix is

A =

1.001 −0.165 −0.051−0.004 0.336 −0.0950.034 −0.101 0.393

. (1)

Then the channel “exposure” corresponds to e = A−11 ∈R3, where 1 = (1, 1, 1)T is a neutral light in linear sRGB.For example, channel exposure in Nikon DSLR in (1) is

e =(1.8271 3.9605 3.4043

)T.

Although A combines the hardware specific (color filter)and the environment (e.g. illuminant) parameters, the con-tribution of simulated sunlight to e is limited owing to therelative flatness of the sunlight spectrum. Hence the dif-ferences in translucency of red, green, and blue filters arelargely responsible for the unequal diagonal elements of e.We may conclude that the translucency of the green pixelsis at least twice as much as red in the Nikon DSLR settingconsidered above. In practice, channel exposure e is hard-ware specific. Although the authors do not have immediateaccess to the Kodak panchromatic CFA sensors (RGB plusunfiltered)[3], even higher translucency is expected for un-filtered pixels. On the other hand, a camera with relativelybalanced channel exposure e can still attain CFA-based ex-posure bracketing by using an ordinary photographic filter(see next section).

2.2. Photographic Filters

A major advantage to digital photography is the flexibil-ity of postprocessing afforded by the digital format. In par-ticular, color balance can be achieved posthoc by perform-ing color space conversion on the acquired RGB data. Aphotographic filter mounted on the lens serves a similar pur-pose for film cameras by changing the spectrum of the in-

(a) camera with filter (b) colorchecker without filter (c) colorchecker with filter (d) spectral response of 85B filterFigure 2. The influence of photographic filter. (Credit: (d) Helen Bach; reproduced with permission)

coming light. The spectral response of these filters are oftendescribed in terms of Wratten number. A “warming” filterwith Wratten number of 85A and 85B, for example, attenu-ates blue channel by 2/3 stops in order to map 5500K colortemperature to 3400K and 3200K, respectively; a “cooling”filter with Wratten number of 80A or 80B have the oppositeeffect, attenuating the red and green channels by 2 or 1 2/3stops, respectively. See Figure 2 for an example.

Not surprisingly, the usage of photographic filters forthe purpose of color balance is rare in digital photography.However, these filters provide the means to magnify thedifferences in translucencies between red, green, and bluefilters. Mathematically, the channel exposure is now com-puted as e = P−1A−11 where P ∈ R3×3 models the at-tenuation of incoming light that the filter achieves. Accord-ing to our lab measurements, a Nikon D90 equipped withHoya 85A filter indeed attenuated red, green, and blue chan-nels by factor of 1.04, 1.51, and 2.70, respectively. Overall,the translucency of the green pixels is at least twice blue,achieving an even greater leverage for CFA-based exposurebracketing. It is also possible to stack multiple filters to fur-ther amplify the effects of the filter.

2.3. Signal And Observation Model

The information redundancy in color images can be ex-ploited to the maximum effect of HDR imaging. One keyobservation in color image processing that has enabled thelikes of demosaicking and compression is the idea that spa-tially highpass components of red, green, and blue chan-nels are similar[6, 8]. That is, color radiance map Xi =(Ri, Gi, Bi)

T ∈ R3 (where i = (ix, iy) denote spatial in-dex of pixels) is separable into lowpass (superscript LP )and highpass (superscript HP ) components:Ri

Gi

Bi

︸ ︷︷ ︸

Xi

=

RLPiGLPiBLPi

︸ ︷︷ ︸

XLPi

+XHPi

111

︸ ︷︷ ︸

1

.

Here, XHPi is the highpass shared by all RGB channels.

From this perspective, the lowpass represents the underly-ing “baseline” that describes textures and edges; and high-pass encodes the “deviation” from the baseline.

Recall the channel exposure e = A−11. Ignoring the

spatial subsampling in color filter array1 for the moment,we expect the following sensor measurement:

Zi = (ri, gi, bi)T = f

(A−1Xi

)≈ max(min(A−1Xi, zmax), zmin) (2)

= max(min(A−1XLPi , zmax), zmin)︸ ︷︷ ︸

ZLPi

+XHPi Φie︸ ︷︷ ︸ZHP

i

.

Here f(·) is a monotonic sensor response function, whichwe assume to be linear (with slope one) near the middleof the curve, and saturates when under/over-exposed; andΦi is a diagonal matrix whose entries indicate saturation(1=not saturated; 0=saturated). Saturated signals are locallylowpass because saturation occurs in batches of pixels.

3. Spectrally Selective Single-Shot HDRThe basic S4HDR theory is developed first in Section 3.1

by assuming that the tristimulus values are observable at ev-ery pixel location. Section 3.2 examines the demosaickingproblem from the S4HDR perspective, before S4HDR re-covery is then extended to the CFA-based sensor setting inSection 3.3.

3.1. S4HDR Reconstruction Algorithm

Appealing to (2), HDR image recovery from tristimu-lus LDR sensor data Zi seems plausible. Since XHP

i isoverdetermined whileXLP

i is potentially underdetermined,they requires different strategies for reconstruction. S4HDRat a high level accomplishes HDR recovery in four steps:

1. Sensor data Zi is separated into highpass (ZHPi ) andlowpass ZLPi components using convolution filters.

2. Color components ofZHPi are pooled together to yieldan achromatic highpass data XHP

i ((3) and (4) below).

3. Saturated pixels in ZLPi are corrected by borrowingacross spectrums to yield XLP

i ((6) and (8) below).

4. The final HDR image is computed asXi = X

LPi + XHP

i 1.

1The theory holds for alternative sensor configurations—such as theones discussed in [10, 3]—that yield measurements other than the RGB.

(a) saturation (b) variable ti (c) variable uiFigure 3. Variables used in S4HDR to process data in Figure 1. (a) Saturated pixels, indicated by their color (e.g. yellow=green and red).(b) Ideal weights ti suggest the preference to use green pixels to reconstruct edges in underexposed regions (and blue for overexposed). (c)Approximation weights ui used in practice. Due to demosaicking constraints, the red and blue pixels are coupled together (hence purple).

Steps 1 and 4 are trivial. We now detail steps 2 and 3.Let XHP

i denote recovery from the observationZHPi via

XHPi := tTi Z

HPi = XHP

i tTi Φie. (3)

The desired weighting vector ti is an inverse of Φie inthe sense that we want tTi Φie to evaluate to unity. Ow-ing to the fact that the inverse is not unique unless twocolor components are saturated, we have the ability toweigh ti by the importance of individual color components(rHPi , gHPi , bHPi )T = ZHPi . To this end, regions of colorcomponents that are better exposed are given more weight:

ti =πi

πTi Φie, (4)

where πi = (πri , πgi , π

bi)T ∈ [0, 1]3 is fuzzy membership of

pixels in the nonsaturated region [22]: (same for πgi and πbi )

πri =

∣∣∣∣zmax + zmin − 2rizmax + zmin

∣∣∣∣ . (5)

The recovery of the lowpass signal XLPi from ZLPi —a

potentially saturated version of the signalA−1XLPi —is an

underdetermined problem. Since there are many solutionsforXLP

i that map toZLPi , the solution space ofXLPi must

be regularized. One sensible solution among the feasiblesolutions is the “most neutral” tristimulus value:

XLPi,reg =arg min

XLPi

‖Ci‖2 + ‖Di‖2, (6)

subject to Y LPi =(Li, Ci, Di)

T =MXLPi

ΦiZLPi =ΦiA

−1M−1Y LPi ,

where the matrix M ∈ R3×3 transforms RGB tristimulusvalues into luminance (Li) and chrominance (Ci, Di). Lin-ear algebra yields the following closed form solution:

XLPi,reg =AΦiZ

LPi − (7)

As(ATsM

TΨMAs)−1AT

sMTΨMAΦiZ

LPi

where Ψ = diag(0, 1, 1) andAs is a submatrix ofAwhoserows correspond to the saturated pixels. Intuitively, thisa projection of the nonsaturated pixel components onto a

space of feasible colors that approximate neutral light. Theregularization in (6) is consistent with prior work show-ing that pixels with high risk for over-exposure likely cor-respond to neutral colors [15, 16], meaning grayworld as-sumption holds better in the region-of-interest. Human vi-sion is reportedly less sensitive to chrominance in the dark-est regions of the image (i.e. under-expose pixels), also. Inpractice, this scheme succeeds when ‖ΨMAΦiZ

LPi ‖ >

‖ΨMXLPi ‖ (see (7)). To safeguard against the possibility

that transition towards saturation is a gradual one, the finalestimate ofXLP

i will be a convex combination:

XLPi =A(diag(πi)Z

LPi +diag(1− πi)A

−1XLPi,reg), (8)

where πi is the aforementioned fuzzy membership. Thefinal HDR reconstruction is Xi = X

LPi + XHP

i 1.

3.2. Analysis of Demosaicking Problem

Can one produce HDR from a post-demosaicking image?No unfortunately. As will be made clear shortly, demo-saicking forces RGB channels to share the highpass data,hence defeating S4HDR. Based on the theoretical exposi-tion in Section 3.1, it is instructive to understand the limita-tions of existing approaches to demosaicking.2 Section 3.3will develop techniques to overcome these limitations.

The Fourier analysis of a Bayer CFA data is[1, 4, 10]

F{αi}(ω) + F{βi}(ω −(0π

))

+ F{βi}(ω −(π0

)) + F{γi}(ω −

(ππ

))

(9)

where F{·} is a Fourier transform operator, ω = (ωx, ωy)denote two dimensional spatial frequency, andαi

βiγi

=

14

12

14

14 0 − 1

414 − 1

214

︸ ︷︷ ︸

N

Zi. (10)

Owing to the invertibility ofN , the problem of demosaick-ing is equivalent to the recovery of (αi, βi, γi). As illus-trated in Figure 4(a) and noted by [4, 10], the overlapping

2Readers are referred to [1, 4, 10] for further details.

support of the summands in (9) indicate aliasing (e.g., bothF(αi)(ω) and F(βi)(ω−

(π0

)) are nonzero for some value

of ω); and demosaicking performance improves when theregions of overlap is reduced by the bandlimitedness of βiand γi. Indeed, the advantage to representation in (10) isthat the difference images βi and γi enjoy rapid spectral de-cay (i.e. they do not retain XHP

i ) and can serve as a proxyfor chrominance. On the other hand, the “baseband” imageαi can be taken to approximate luminance where XHP

i ispreserved by the relation

XHPi ∝ αHPi =

(14

12

14

)︸ ︷︷ ︸weights

ZHPi (11)

(from first row of (10)). Hence, demosaicking operating onCFA data is expected to recover the highpass signal XHP

i

that is proportional to αHPi in (11). Contrasting the linearcombination ofZHPi in (11) (where the implied weights are( 14 ,

12 ,

14 )) to a more desirable weighting vector (4), we con-

clude that (11) ignores exposure bracketing entirely. Thusdemosaicking output under the ordinary scenario is LDR.

Consider an alternative setup. Most modern digitalcameras perform post-capture, pre-demosaicking “equal-ization” aimed at neutralizing the exposure bracketing byscaling red, green, blue channels by the inverse of expo-sure e. Mathematically, this is equivalent to replacing everyinstance of Zi in (9-11) with diag(e)−1Zi. For example,(10) is updated as follows:αi

βiγi

=N diag(e)−1ZLPi +XHPi N diag(e)−1e︸ ︷︷ ︸

(1,0,0)T

.

This suggests that equalization improves demosaickingperformance—the bandlimitedness assumptions of βi andγi are robust and the risk of aliasing is reduced. Updating(11) also, the combination of equalization and demosaick-ing is expected to recover the highpass signal XHP

i via:

XHPi = αHPi =

(14

12

14

)diag(e)−1︸ ︷︷ ︸

weights

ZHPi = XHPi .

Comparing this to the desired weighting vector (4), how-ever, the linear combination of ZHPi implied by post-capture, pre-demosaicking equalization fails to yield HDRrecovery of XHP

i .

3.3. S4HDR With CFA-Based Sensor

The key observation of Section 3.2 is that the high-pass signal XHP

i recovered from demosaicking is propor-tional to αHPi . Hence we propose a post-capture, pre-demosaicking processing aimed at precisely controlling thelinear combination of ZHPi in αHPi . The output from

(a) ordinary CFA data (b) constraint relaxationFigure 4. 2D Fourier transform of CFA data after equalization. Thered/green/blue shading indicate the frequency support of αi/βi/γichannels, respectively. (a) Ordinary equalization minimizes alias-ing risk and ignores exposure bracketing. (b) Flexibility of allow-ing F{γi} support to grow is exploited for HDR imaging.

demosaicking operating on this preprocessed CFA data isS4HDR reconstruction in the style of (3) and (4).

Denoting by diagonal matrix W the pre-demosaickingscaling of red, green, and blue channels, we update (9-11).For example, demosaicking recovers XHP

i via the relation

XHPi = αHPi =

(14

12

14

)W︸ ︷︷ ︸

uTi

ZHPi . (12)

We may control the linear weightsui indirectly by choosingW intelligently. ConsiderW that satisfy the condition

NWe = (1, 0, τ)T . (13)

Updating (11),αi

βiγi

=NWZLPi +XHPi NWe︸ ︷︷ ︸

(1,0,τ)T

. (14)

Unlike the equalization example in Section 3.2, τ is not re-quired to be zero—γi may now have larger support in thefrequency domain as a result. As suggested by Figure 4(b),this “relaxation” is justifiable because most cameras todayhave exceedingly high spatial resolution compared to whatthe optics can provide. Hence the risk of aliasing betweenF{αi} and F{γi} is acceptably low, even though the alias-ing risk between F{αi} and F{βi} remains high. SolvingforW in (13), the admissible choices ofW are:

Wτ = diag(e)−1 diag((1 + τ, 1− τ, 1 + τ)). (15)

Allowing τ and Wτ to be spatially adaptive, we chooseamong the admissible set in (15) the one that gives more im-portance to the regions of color components that are betterexposed. To this effect, we seek ui that best approximatesthe “ideal weights” in (4) in the following sense:

τi = argminτ

∥∥tTi − ( 14 12

14

)Wτ

∥∥2 . (16)

where ti is as defined previously in (3)3. The closed formsolution to this optimization problem is a projection:

τi =

⟨tTi − ( 14 ,

12 ,

14 ) diag(e)

−1, ( 14 ,−12 ,

14 ) diag(e)

−1⟩⟨

( 14 ,−12 ,

14 ) diag(e)

−1, ( 14 ,−12 ,

14 ) diag(e)

−1⟩ .

By (12), the highpass component of demosaicking outputZHPi is an HDR reconstruction of XHP

i .Lastly, the equalization weights Wi have no significant

effects on the recoverability of XLPi . We simply take the

demosaicking output ZLPi and process according to (8).The final HDR reconstruction is Xi = X

LPi + XHP

i 1.

4. DiscussionsThe space of trade offs explored in HDR imaging is noise

versus range. An image captured with short exposure pre-serves bright areas of the image, but it suffers from randomand quantization noise in the dark regions. A long exposureon the other hand substantially enhances the dark regions,but at the cost of pixel saturation. Thus the quality of single-shot HDR image can be evaluated by comparing the noiseand range performance to a single-shot LDR image.

Reconstructions from Nikon D90 raw sensor data areshown (with compressive tonemapping [13]) in Figure 5.The images in Figure 5(a) and 5(b) were taken without anyphotographic filters (i.e. how cameras normally operate);Figure 5(c) represents recovery from sensor data capturedwith a Hoya 85A filter that attenuates blue intensities. Mov-ing targets (e.g. people and pendulum) cannot be imaged bymultishot HDR solutions[22], as seen in Figure 6. Imper-fections in registration also cause blurred image details.

Despite that images in Figures 5(a) and (b) share thesame input data, S4HDR (Xi = X

LPi +XHP

i 1) succeedsin extending the dynamic range over the ordinary cameraprocessing described in Section 3.2. This is largely due tothe fact that red channels are underexposed compared togreen and blue channels, even without the help of photo-graphic filters. The S4HDR reconstructions in Figure 5(c)enjoy the benefit of the exposure bracketing induced byHoya 85A filters.

S4HDR reconstruction is also robust to noise, eventhough no explicit denoising is involved. The penalty innoise that S4HDR pays for an increased dynamic range isminimal when the best-exposed regions of the RGB colorcomponents (see Figure 3) are combined. To wit, S4HDRreconstructions in Figures 5(b-c) have less noise than theirLDR counterparts in Figure 5(a) (details in Figure 7).

On the down side, the proposed technique shifts thebright regions of the output image towards neutral colors

3Given a CFA data, only one of three elements in weights πi and sat-uration indicator φi are computable. Applying demosaicking to the com-putable elements of πi and φi produces “full color” weights and satura-tion indicator, which can then used in (3) to produce ti.

(a) ordinary (b) S4HDR (c) S4HDR+filterFigure 7. Zoomed and brightened portion of images in Figure 5(top row) for comparing noise.

(a) red/blue vs green saturation (b) SNR vs saturationFigure 9. Simulation results (Image 2). (a) Saturation is color de-pendent. (b) S4HDR outperforms ordinary reconstruction.

slightly. This is expected because XLPi regularization fa-

vors neutral colors—a good “guess,” but not without limi-tations. One way to improve S4HDR in the future is to pre-vent color shift by constructing a dictionary of reflectancevalues present in the nonsaturated parts of the image.

The supplementary material shows extensive resultsfrom simulated experiments. Examples are shown in Fig-ures 8-9. As evidenced by Figure 9(a), green channel sufferfrom pixel saturation, while red and blue channels retain de-tails in the bright regions. As a result, S4HDR preserves farwider dynamic range than what an ordinary camera process-ing can deliver. Quantitative comparisons shown in Figure9(b) are made based on pixelwise SNR [9]. See supplemen-tary materials for details and examples. JPEG images fromNikon D90 accompanying raw sensor data are also shownin supplementary material to verify the consistency betweenFigure 5(a) and commercial camera processing.

In our experiments, we found by practice that 85A and85B filters tended to work well. Since blue sky tends to sat-urate the blue pixels, these filters are suboptimal for captur-ing cloud details in the sky. In the future, we will investigatethe optimal choice of filter to combine with Bayer CFA pat-tern. For this, incorporation of studies in optimal exposurebracketing [17, 9, 11] into the spectrally selective single-shot HDR setting should help further our understanding andallow S4HDR to meet its full potential.

The supplementary material and S4HDR software areavailable at campus.udayton.edu/˜ISSL.

Acknowledgment

This project is funded in part by Texas Instruments.

(a) ordinary reconstruction (b) S4HDR (c) S4HDR with filterFigure 5. Single-shot reconstruction from Nikon DSLR raw sensor data. (a) ordinary reconstruction (equalization+demosaicking); (b)S4HDR recovery from the same input data as (a); (c) S4HDR recovery with 85A filter. Output shown with iCam06 tonemapping[13].

Figure 6. HDR reconstructed from three time-multiplexed LDR images (1 stops apart each)[22]. Moving objects cause severe ghostingartifacts (compare people and pendulum in Figure 5). Slight imperfections in registration contributes to blurred image details (zoom in).

(a) (b) (c) (d) (e)Figure 8. Simulation results (Image 2). (a) Original image. (b) Ordinary reconstruction. (c) S4HDR recovery using same data as (b). (d)S4HDR recovery with a 85A filter. (e) S4HDR recovery with two 85A filters.

References[1] D. Alleysson, S. Susstrunk, and J. Herault. Linear demosaic-

ing inspired by the human visual system. Image Processing,IEEE Transactions on, 14(4):439–449, 2005. 1, 4

[2] A. Chakrabarti, D. Scharstein, and T. Zickler. An empiricalcamera model for internet color vision. In British MachineVision Conference, 2009. 1

[3] J. Compton and J. Hamilton. Image sensor with improvedlight sensitivity. US Patent 20070024031, 2005. 2, 3

[4] E. Dubois. Frequency-domain methods for demosaickingof Bayer-sampled color images. Signal Processing Letters,IEEE, 12(12):847–850, 2005. 1, 4

[5] A. Foi, M. Trimeche, V. Katkovnik, and K. Egiazarian.Practical Poissonian-Gaussian noise modeling and fitting forsingle-image raw-data. Image Processing, IEEE Transac-tions on, 17(10):1737–1754, 2008.

[6] J. Glotzbach, R. Schafer, and K. Illgner. A method of colorfilter array interpolation with alias cancellation properties. InProc. Int. Conf. Image Processing, volume 1, pages 141–144,Oct. 2001. 1, 3

[7] M. Grossberg and S. Nayar. High Dynamic Range from Mul-tiple Images: Which Exposures to Combine? In ICCV Work-shop on Color and Photometric Methods in Computer Vision(CPMCV), Oct 2003. 1

[8] B. Gunturk, Y. Altunbasak, and R. Mersereau. Color planeinterpolation using alternating projections. Image Process-ing, IEEE Transactions on, 11(9):997–1013, 2002. 1, 3

[9] S. Hasinoff, F. Durand, and W. Freeman. Noise-optimal cap-ture for high dynamic range photography. In Computer Vi-sion and Pattern Recognition (CVPR), 2010 IEEE Confer-ence on, pages 553–560. IEEE, 2010. 1, 6

[10] K. Hirakawa and P. Wolfe. Spatio-spectral color filter arraydesign for optimal image recovery. Image Processing, IEEETransactions on, 17(10):1876–1890, 2008. 1, 3, 4

[11] K. Hirakawa and P. Wolfe. Optimal Exposure Control forHigh Dynamic Range Imaging. In IEEE Intl Conf ImageProcessing, 2010. 1, 6

[12] D. Krishnan and R. Fergus. Dark flash photography. ACMTransactions on Graphics, SIGGRAPH 2009 ConferenceProceedings, 28:To appear, 2009. 1

[13] J. Kuang, G. Johnson, and M. Fairchild. iCAM06: A re-fined image appearance model for HDR image rendering.Journal of Visual Communication and Image Representation,18(5):406–414, 2007. 1, 6, 7

[14] X. Liu and A. El Gamal. Photocurrent estimation for a self-reset CMOS image sensor. In Proc. SPIE, volume 4669,pages 304–312. Citeseer, 2002. 1

[15] D. MacAdam. Maximum visual efficiency of colored mate-rials. J. Opt. Soc. Am, 25:316–367, 1935. 4

[16] D. MacAdam. The theory of the maximum visual efficiencyof colored materials. J. Opt. Soc. Am., 25:249–252, 1935. 4

[17] T. D. N. Barakat, A.N. Hone. Minimal-bracketing sets forhigh-dynamic-range image capture. IEEE Transactions onImage Processing, 17(10), 2008. 1, 6

[18] S. Narasimhan and S. Nayar. Enhancing Resolution alongMultiple Imaging Dimensions using Assorted Pixels. IEEETransactions on Pattern Analysis and Machine Intelligence,27(4):518–530, Apr 2005. 1

[19] S. Nayar and V. Branzoi. Adaptive Dynamic Range Imaging:Optical Control of Pixel Exposures over Space and Time. InIEEE International Conference on Computer Vision (ICCV),volume 2, pages 1168–1175, Oct 2003. 1

[20] S. Nayar and T. Mitsunaga. High Dynamic Range Imag-ing: Spatially Varying Pixel Exposures. In IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR), vol-ume 1, pages 472–479, Jun 2000. 1

[21] S. Nayar and S. Narasimhan. Assorted Pixels: Multi-Sampled Imaging With Structural Models. In EuropeanConference on Computer Vision (ECCV), volume IV, pages636–652, May 2002. 1

[22] J. M. P. Debevec. Recovering high dynamic range radiancemaps from photographs. In Proc. SIGGRAPH 1997, Com-puter Graphics, pages 369–378. ACM, 1997. 1, 4, 6, 7

[23] G. Petschnigg, R. Szeliski, M. Agrawala, M. Cohen,H. Hoppe, and K. Toyama. Digital photography with flashand no-flash image pairs. ACM Transactions on Graphics(TOG), 23(3):664–672, 2004. 1

[24] E. Reinhard, G. Ward, S. Pattanaik, and P. Debevec. High dy-namic range imaging: acquisition, display, and image-basedlighting. Elsevier/Morgan Kaufmann, 2006. 1

[25] M. Sayag. Non-linear photosite response in CCD imagers,1991. US Patent 5,055,667. 1

[26] Y. Schechner and S. Nayar. Generalized Mosaicing: HighDynamic Range in a Wide Field of View. International Jour-nal on Computer Vision, 53(3):245–267, Jul 2003. 1

[27] S. Sugawa, N. Akahane, S. Adachi, K. Mori, T. Ishiuchi,and K. Mizobuchi. A 100 dB dynamic range CMOS im-age sensor using a lateral overflow integration capacitor. InSolid-State Circuits Conference, 2005. Digest of TechnicalPapers. ISSCC. 2005 IEEE International, pages 352–603.IEEE, 2005. 1

[28] L. Zhang, X. Wu, and D. Zhang. Color reproduction fromnoisy CFA data of single sensor digital cameras. IEEE Trans-actions on image processing, 16(9):2184–2197, 2007.