Download - Wpafb 2010
TNO Defence, Security and Safety
INVIS Integrated Night Vision Surveillance and Observation System
Lex Toet & Sjaak Kriekaard
TNO Defence, Security and Safety2
Overview
Rationale of image fusion
Color the Night: real-time sample-based natural color fusion
Previous implementations:
– Gecko = Visual + NIR
– Viper = II + LWIR
Latest development : Triclobs = Visual + NIR + LWIR
INVIS system = Triclobs + Synthetic Scene Generator +
Image Enhancement Capabilities
Example nighttime movies
• Intensified imagery: see in the dark, context but no targetsIntensified imagery: see in the dark, context but no targets
• Thermal imagery: see through smoke, targets but no contextThermal imagery: see through smoke, targets but no context
• Fused imagery: Fused imagery: - increased information content - increased information content - best of both worlds: both targets and context- best of both worlds: both targets and context - all weather capability- all weather capability - increased system robustness- increased system robustness - decreased false alarm rate- decreased false alarm rate
Rationale of Image Fusion :
Color-the-night
• deduces a natural color mapping from a multiband night sensor image + daytime reference image
• color mapping can be applied to other multiband images with similar content, resulting in a multiband nighttime image with natural daytime colors
• Method is:• Fast, easy to implement in real-time applications• Results in stable object colors (invariant under panning,
zooming etc).
• Natural colors lead to:• Easier and better interpretation of scene• Faster scene recognition• Better discimination & recognition of materials and objects
TNO Defence, Security and Safety5
Sample Based Color Transfer
color photograph visual NIR
vis + NIR recolored
TNO Defence, Security and Safety6
Color the Night: Applying Daytime Colors to Nighttime Imagery
ThermalImagery
Intensified Imagery
Color Reference Image
Resulting Imagery
TNO Defence, Security and Safety7
Visual Thermal
Grayscalefused
Color fusedimage
TNO Defence, Security and Safety8
Gecko: portable real time Visual + NIR fusion
TNO Defence, Security and Safety9
Visual + NIR Fusion: Gecko principle
TNO Defence, Security and Safety10
Gecko: Visual + NIR
< 700 nm
> 700 nm
TNO Defence, Security and Safety11
Visual Thermal
RG Gecko
NVG photo
Gecko system
TNO Defence, Security and Safety12
Viper: Portable Real time Visual + Thermal Fusion
eyepinhole
TNO Defence, Security and Safety13
Viper system: principle
TNO Defence, Security and Safety14
Image Intensifier Thermal camera
Intevac NightVista E1100 FLIR A10
Viper: portable real time Visual + Thermal fusion
TNO Defence, Security and Safety15
Viper: combined Visual + Thermal Imager
TNO Defence, Security and Safety16
Viper: Visual + Thermal
TNO Defence, Security and Safety17
Visual Thermal
RG Viper
Photo
Viper system
Gecko bands
NV
G >
700nm
NV
G <
700nm
Daytime image
Gecko image
Th
emal
NV
G
Viper bandsViper image
Daytime image
The INVIS Integrated Night Vision Surveillance and Observation System
Dynamic fusion of Visual + NIR + LWIR
Real-time natural color mapping
Real-time image enhancement
Synthetic image generation
TNO Defence, Security and Safety21
INVIS
sensor suitelayout
TNO Defence, Security and Safety22
Photonis Intensified Camera Unit (ICU PP3000L)
XenICs Gobi 384 infrared microbolometer
INVIS sensors
TNO Defence, Security and Safety23
Viewing window
displays
LWIR
Visual
NIR GPS - compass
Data storage
Network connection
TNO Defence, Security and Safety25
INVIS
mounted on all terrainmobileplatform
TNO Defence, Security and Safety26
Visual NIR LWIR
RGB Photo
Color the Night DNR LACE
Color the Night
Visual NIR LWIR
TNO Defence, Security and Safety28
Color the Night
Visual NIR LWIR
TNO Defence, Security and Safety29
Color the Night
Visual NIR LWIR
TNO Defence, Security and Safety30
Color the Night
Visual NIR LWIR
TNO Defence, Security and Safety31
Color the Night
Visual NIR LWIR
TNO Defence, Security and Safety32
Color the Night
Visual NIR LWIR
TNO Defence, Security and Safety33
Color the Night
Visual NIR LWIR
TNO Defence, Security and Safety34
Color the Night
Visual NIR LWIR
Image enhancement techniques
• Temporal noise reduction• Scene-based non-uniformity correction• Super-resolution reconstruction• Local adaptive contrast enhancement• Color-the-Night image fusion
SBNUC DSR
Color-the-Night
LACE
DSR
DSR
Resizing the TIR imageTIR
VISUAL
NIR
Noise reductionby Dynamic Super Resolution
• Advantages• Decreased noise (improve SNR ratio)• Increased detection range• Increased situational awareness
Fixed pattern noise reductionby Scene-Based Non-Uniformity Correction
• Advantages
• Low maintenance (hands-off operation)
• Clean pictures (performance maximized)
• Reduced sensor size / weight / cost (simpler design)
Resolution enhancementby Dynamic Super Resolution
• Advantages• Increased sharpness • Increased situational awareness• Increased recognition and identification
Local Adaptive Contrast Enhancement (LACE)• Advantages
• Increased situational awareness• Increased recognition and identification
in the total dynamic range
Noise reduction
NIR VIS LWIR
Noise reduction for I2
NormalDSR1DSR1+low LACE
NIR image
Noise Reduction and LACE on I2 images
Noise Reduction and LACE on LWIR images
weak LACE vs strong LACE
Original images, RGB fusion Original images, Color-the-Night fusion
Enhanced images, RGB fusion Enhanced images, Color-the-Night fusion
No enhancement DSR1 + SBNUC
DSR1 + SBNUC; LACE before CtN DSR1 + SBNUC; LACE after CtN
Color-the-Night & LACE
No LACE Low LACE before CtN Low LACE after CtN
High LACE before CtN High LACE after CtN
Conclusions on Image Enhancement :
• Temporal Noise reduction works for all imagery• Scene Based Non Uniformity Correction (SBNUC) is useful for LWIR
• Super-resolution reconstruction has added value for LWIR• images are resolution limited
• Super-resolution reconstruction has limited value for I2
• Images are noise limited
• Enhancement algorithms need accurate motion estimation• works for small moving objects• not for scenery dominated by smoke or large moving objects
• LACE can be applied on all imagery• noisy imagery requires noise reduction
Relation between Image Enhancement and Image Fusion
• Enhanced images can be used as input for Color-the-Night fusion
• The results have similar appearance and a significantly improved SNR
• LACE applied after tone mapping keeps the appearance similar
• LACE applied before tone mapping may enhance more details
• but is more expensive and may disturb color mapping!
• Effect of LACE applied before fusion is reduced by color mapping
TNO Defence, Security and Safety50
Synthetic Scene Generation
TNO Defence, Security and Safety51
TNO Defence, Security and Safety52
Corresponding LWIR and synthetic imagesin 1st person perspective
LWIR Synthetic
TNO Defence, Security and Safety53
Corresponding LWIR and synthetic imagesin 3rd person perspective
LWIR Synthetic
TNO Defence, Security and Safety54
LWIR synthetic synthetic
Switching walls on --- off
TNO Defence, Security and Safety55
LWIR synthetic synthetic
Switching vegetation on --- off
TNO Defence, Security and Safety56
LWIR Material view
TNO Defence, Security and Safety57
Visual NIR LWIR
RGB
TNO Defence, Security and Safety58
RGB Photo
Color the Night DNR LACE
TNO Defence, Security and Safety59
RGB Synthetic scene
Color the Night DNR LACE
TNO Defence, Security and Safety60
Visual NIR LWIR
RGB
TNO Defence, Security and Safety61
Photo
Color the Night DNR LACE
RGB
TNO Defence, Security and Safety62
Synthetic scene
Color the Night DNR LACE
RGB
June 2010: Data Collection Experiment on Military Base in the Netherlands
Triclobs on scootmobile
The Targets
Multispectral Pinkus Target
Real-time Scene Display
Visual NIR
LWIR
The End