image processing in infrared cameras - meetupfiles.meetup.com/4379272/biv20160602_norvig.pdfboston...

Boston Imaging and Vision Group Infrared Vision

Image Processing in Infrared Cameras

Marc Norvig Principal Engineer FLIR Systems, Inc.

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Hashagen, J. (September 2014). SWIR Applications and Challenges: A Primer. EuroPhotonics http://www.photonics.com/Article.aspx?AID=56646

Reflected IR Thermal IR

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Reflected Infrared

Photos by Nick Spiker https://www.facebook.com/invisiblelightimages/

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Long-Wave (8 – 14 μm)

Photos Courtesy FLIR Systems, Inc.

Thermal Infrared Camera Comparison Mid-Wave (3 – 5 μm)

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Long-Wave (8 – 14 μm) + Room temperature operation

Photos Courtesy FLIR Systems, Inc.

Thermal Infrared Camera Comparison Mid-Wave (3 – 5 μm) – Cyrogenic (77˚K to 155˚K)

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Long-Wave (8 – 14 μm) + Room temperature operation + Compact + Lightweight + Lower power + Lower price + Instant imaging + 20-plus year service interval + Very good radiometric accuracy

Thermal Infrared Camera Comparison Mid-Wave (3 – 5 μm) – Cyrogenic (77˚K to 155˚K) – Size – Weight – Power – Cost – Cooldown time – 10 to 15,000 hour service interval + Best thermal image quality + Best sensitivity + High contrast + Long distance viewing + Higher frame rates

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

offset(x,y) gain(x,y)

Sensor

Temp

Image

Memory

Vshutter (x,y)

OV(T3)

1

GV(T2)

Raw

Image

Data-

+

+

+ -Corrected

Infrared

Image

Shutter

Sensor Corrections

- Mechanical shutter captures “dark image” for subtraction

- Each pixel has a different bias point and sensitivity due to manufacturing process variation

- Corrected using per-pixel offset and gain tables

- All pixels now have uniform response to relative temperatures

- For accurate temperature measurement we need to correct for sensor temperature variation

Corrected Image

Sensor Corrections

Raw Image

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Sosnowski, T. et al. (2010). Processing of the Image from Infrared Focal Plane Array Using FPGA-based System. MIXDES Proceedings of the 17th International Conference, 581-586

Corrected Image

Sensor Corrections

Raw Image

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Budzier, H. & Gerlach, G. (2015). Calibration of uncooled thermal infrared cameras. Journal of Sensors and Sensor Systems,4(1), 187-197

Defective Pixel Replacement

- Some pixels can’t be made to match others (dead, excessive gain, hot, blinking)

- These pixels are marked as defective at factory calibration

- The bad pixels must be replaced with a better value for image viewing

- Approach this as either a denoising problem or an inpainting problem

- Simplest solution is nearest neighbor replacement - Need method to choose between equidistant neighbors

- Spatial filter over a small local neighborhood - If any neighborhood pixel is also defective it must be excluded (remember to renormalize the kernel)

- Simple mean of a few neighbors is surprisingly effective

- Median filter never seems to turn out as well as you want it to

- Adaptive, gradient-based, and patch-based methods preserve image structure - Probably not be worth the computational effort

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Image Enhancement

- Corrected sensor output is 14-bit but display is 8-10 bits

- Typical scenes have high dynamic range (hot car engine on a winter day) - Most of the pixels in this scene will be the same temperature (low contrast)

- Combined bit length reduction and contrast enhancement is required

- Contrast Limited Adaptive Histogram Equalization (CLAHE) and its derivatives have emerged as the best solutions.

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Image Enhancement

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Image Enhancement

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Original Histogram EQ

Image Enhancement

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Original CLAHE

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Visible–Infrared Image Fusion: Registration

- Image-driven registration requires features that appear in both visible and IR images - Finding good joint feature detectors is an open problem

Lui, F. & Seipel, S. (2015). Infrared-visible image registration for augmented reality-based thermographic building diagnostics. Visualization in Engineering,3(16)

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Visible–Infrared Image Fusion: Image Combining

- Combining can occur at various levels - Pixel-level

- Feature-level

- Object-level

- Transform Domain Techniques - Pyramids (Gaussian, Laplacian, Gradient)

- Wavelets, Curvelets

- Spatial Domain Techniques - Weighted blending (global or locally adaptive weights)

- PCA

- Gram-Schmidt

- High-pass filter

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Feature-based Fusion Example

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Infrared Camera 50% Blend Visible Camera

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Vis + IR Features

Infrared Camera Shared Features

Visible Features Infrared Features

Visible Camera

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Infrared Camera 50% Blend

Visible Features Infrared Features

Visible Camera

Grayscale Fused

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Infrared Camera False-Color Fused

Visible Features Infrared Features

Visible Camera

Grayscale Fused

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

High-pass Filter Fusion Example (FLIR MSX®)

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Visible Camera Infrared Camera High-pass filter

Boston Imaging and Vision Group Infrared Vision

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras

Visible Camera Infrared Camera MSX

®

Visible Camera Infrared Camera MSX

®

Boston Imaging and Vision Group Infrared Vision

Thank you

marc.norvig@gmail.com 06/02/2016 Image Processing in Infrared Cameras