high-resolution hyperspectral imaging for cultural heritage
DESCRIPTION
High-resolution Hyperspectral Imaging for Cultural Heritage. Rei Kawakami 1 John Wright 2 Yu-Wing Tai 3 Yasuyuki Matsushita 2 Moshe Ben-Ezra 2 Katsushi Ikeuchi 3 1 University of Tokyo, 2 Microsoft Research Asia (MSRA), 3 Korea Advanced Institute of Science and Technology (KAIST) - PowerPoint PPT PresentationTRANSCRIPT
High-resolution Hyperspectral Imaging for Cultural Heritage
Rei Kawakami1 John Wright2 Yu-Wing Tai3 Yasuyuki Matsushita2 Moshe Ben-Ezra2 Katsushi Ikeuchi3
1University of Tokyo, 2Microsoft Research Asia (MSRA), 3Korea Advanced Institute of Science and Technology (KAIST)
2011 Dunhuang Forum
Giga-pixel Camera
M. Ben-Ezra et al.
Giga-pixel Camera
Large-format lens CCD
Spectrum
200 5000[nm]
100.10.001
700 1000600500400300200 [nm]
Ultraviolet InfraredVisible light
1 meter 3100 meter0.00001
CosmicRays
GammaRays
X-raysUltraViolet
InfraredTV AndRadio Waves
Electric Waves
Electromagnetic Spectrum
Vio
let
Blu
e
Gre
en
Yello
w
Ora
ng
e
Red
0
20
40
6080
100
Solar radiationreachingearth’s surface(Relative Energy)
RGB vs. Spectrum
Applications
Light simulation
Layered surface decomposition
Morimoto et al. CVPR2010: Estimating Optical Properties of Layered Surfaces Using the Spider Model
Why difficult?
Approach
Low-reshyperspectral
High-resRGB
High-resolutionHyperspectral image
Combine
Two-step approach
1. Factorize low-res hyperspectral image into basis functions of spectra and coefficients
2. For each pixel in high-res RGB image, estimate coefficients of the basis functions
Problem formulation
W(Image width)
H(Image height)
S
Goal:
Given:
(Spectral wavelength)
Representation: Basis function
W (Image width)
H (Image height)
S
𝒁
= …
01.00…0
= +x 0 x 1.0 x 0 x 0++
Reflectance vectors
1: Matrix factorization
Sparse
For all pixel (i,j)
Sparse matrix
W (Image width)
H (Image height)
S
= …
00.40…
0.6
𝒀 h𝑠
• At each pixel of , only a few () materials are present
Reflectance matrix
2: Reconstruction
W
H
S Sparse
𝒀 𝑟𝑔𝑏
�̂� (𝑖 , 𝑗 )=argmin‖𝒉‖1
Reconstruction
𝒁 (𝒊 , 𝒋 ,∗ )≈ 𝑨𝒉 (𝒊 , 𝒋 )
𝒁
• At each pixel of , materials should be even much fewer
Simulation experiments
Balloons Beads Sponges Oil painting
Flowers CD Peppers Face
Spectral image database F. Yasuma, T. Mitsunaga, D. Iso and S. K. Nayar.Generalized assorted pixel camera: Postcapture control of resolution,Dynamic range, and spectrum. IEEE Trans. IP, 19(9):2241-2253, 2010
460 nm 550 nm RGB/620 nm 460 nm 550 nm RGB/620 nm
Input images: Balloons and Beads examples
Ground truths
Reconstruction using component substitution method
Reconstruction by the proposed method
430 nm 490 nm 550 nm 610 nm 670 nm
Input images: Sponges examples
Ground truths
Reconstruction by the proposed method
Error images of the proposed method
RGBimage
GroundTruth(430 nm)
Estimated430 nm
Method Balloons Beads Sponges Oil painting
Flowers CD Peppers Face
CSM[2] 13.9 28.5 19.9 12.2 14.4 13.3 13.7 13.1
Global 6.9/4.7 10.5/8.8 15.4/12.3
5.4/3.8 9.8/8.9 10.3/10.0
7.1/5.9 4.7/3.8
Local win 7.0/4.9 10.6/8.9 14.0/10.6
5.7/4.1 7.5/6.3 9.6/9.2 8.8/8.0 10.9/10.5
RGB clust 6.6/4.3 9.7/7.9 13.6/10.0
5.5/4.0 7.8/6.5 9.1/8.6 8.5/7.6 4.7/3.8
Proposed 3.0/3.0 9.2/9.2 3.7/3.7 4.7/4.7 5.4/5.4 8.2/8.2 4.7/4.7 3.3/3.3
RMSE
Balloons Beads Sponges Oil painting
Flowers CD Peppers Face
HS camera
Filter
CMOSLens Aperture
Focus
Translational stage
Real data experiment
Input RGB Input (550nm) Input (620nm)Estimated (550nm) Estimated (620nm)
Summary•Method to reconstruct high-resolution
hyperspectral image from ▫Low-res hyperspectral camera▫High-res RGB camera
•Spatial sparsity of hyperspectral input▫Search for a factorization of the input into
basis functions set of maximally sparse coefficients
Acknowledgement
•This work was in part supported by Microsoft CORE 6 project.