wpafb 2010
DESCRIPTION
Lecture given at Wright Patterson AFB, Dayton, Ohio, 30 September 2010TRANSCRIPT
![Page 1: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/1.jpg)
TNO Defence, Security and Safety
INVIS Integrated Night Vision Surveillance and Observation System
Lex Toet & Sjaak Kriekaard
![Page 2: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/2.jpg)
TNO Defence, Security and Safety2
Overview
Rationale of image fusion
Color the Night: real-time sample-based natural color fusion
Previous implementations:
– Gecko = Visual + NIR
– Viper = II + LWIR
Latest development : Triclobs = Visual + NIR + LWIR
INVIS system = Triclobs + Synthetic Scene Generator +
Image Enhancement Capabilities
Example nighttime movies
![Page 3: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/3.jpg)
• Intensified imagery: see in the dark, context but no targetsIntensified imagery: see in the dark, context but no targets
• Thermal imagery: see through smoke, targets but no contextThermal imagery: see through smoke, targets but no context
• Fused imagery: Fused imagery: - increased information content - increased information content - best of both worlds: both targets and context- best of both worlds: both targets and context - all weather capability- all weather capability - increased system robustness- increased system robustness - decreased false alarm rate- decreased false alarm rate
Rationale of Image Fusion :
![Page 4: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/4.jpg)
Color-the-night
• deduces a natural color mapping from a multiband night sensor image + daytime reference image
• color mapping can be applied to other multiband images with similar content, resulting in a multiband nighttime image with natural daytime colors
• Method is:• Fast, easy to implement in real-time applications• Results in stable object colors (invariant under panning,
zooming etc).
• Natural colors lead to:• Easier and better interpretation of scene• Faster scene recognition• Better discimination & recognition of materials and objects
![Page 5: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/5.jpg)
TNO Defence, Security and Safety5
Sample Based Color Transfer
color photograph visual NIR
vis + NIR recolored
![Page 6: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/6.jpg)
TNO Defence, Security and Safety6
Color the Night: Applying Daytime Colors to Nighttime Imagery
ThermalImagery
Intensified Imagery
Color Reference Image
Resulting Imagery
![Page 7: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/7.jpg)
TNO Defence, Security and Safety7
Visual Thermal
Grayscalefused
Color fusedimage
![Page 8: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/8.jpg)
TNO Defence, Security and Safety8
Gecko: portable real time Visual + NIR fusion
![Page 9: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/9.jpg)
TNO Defence, Security and Safety9
Visual + NIR Fusion: Gecko principle
![Page 10: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/10.jpg)
TNO Defence, Security and Safety10
Gecko: Visual + NIR
< 700 nm
> 700 nm
![Page 11: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/11.jpg)
TNO Defence, Security and Safety11
Visual Thermal
RG Gecko
NVG photo
Gecko system
![Page 12: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/12.jpg)
TNO Defence, Security and Safety12
Viper: Portable Real time Visual + Thermal Fusion
eyepinhole
![Page 13: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/13.jpg)
TNO Defence, Security and Safety13
Viper system: principle
![Page 14: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/14.jpg)
TNO Defence, Security and Safety14
Image Intensifier Thermal camera
Intevac NightVista E1100 FLIR A10
Viper: portable real time Visual + Thermal fusion
![Page 15: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/15.jpg)
TNO Defence, Security and Safety15
Viper: combined Visual + Thermal Imager
![Page 16: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/16.jpg)
TNO Defence, Security and Safety16
Viper: Visual + Thermal
![Page 17: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/17.jpg)
TNO Defence, Security and Safety17
Visual Thermal
RG Viper
Photo
Viper system
![Page 18: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/18.jpg)
Gecko bands
NV
G >
700nm
NV
G <
700nm
Daytime image
Gecko image
![Page 19: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/19.jpg)
Th
emal
NV
G
Viper bandsViper image
Daytime image
![Page 20: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/20.jpg)
The INVIS Integrated Night Vision Surveillance and Observation System
Dynamic fusion of Visual + NIR + LWIR
Real-time natural color mapping
Real-time image enhancement
Synthetic image generation
![Page 21: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/21.jpg)
TNO Defence, Security and Safety21
INVIS
sensor suitelayout
![Page 22: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/22.jpg)
TNO Defence, Security and Safety22
Photonis Intensified Camera Unit (ICU PP3000L)
XenICs Gobi 384 infrared microbolometer
INVIS sensors
![Page 23: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/23.jpg)
TNO Defence, Security and Safety23
Viewing window
displays
![Page 24: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/24.jpg)
LWIR
Visual
NIR GPS - compass
Data storage
Network connection
![Page 25: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/25.jpg)
TNO Defence, Security and Safety25
INVIS
mounted on all terrainmobileplatform
![Page 26: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/26.jpg)
TNO Defence, Security and Safety26
Visual NIR LWIR
RGB Photo
Color the Night DNR LACE
![Page 27: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/27.jpg)
Color the Night
Visual NIR LWIR
![Page 28: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/28.jpg)
TNO Defence, Security and Safety28
Color the Night
Visual NIR LWIR
![Page 29: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/29.jpg)
TNO Defence, Security and Safety29
Color the Night
Visual NIR LWIR
![Page 30: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/30.jpg)
TNO Defence, Security and Safety30
Color the Night
Visual NIR LWIR
![Page 31: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/31.jpg)
TNO Defence, Security and Safety31
Color the Night
Visual NIR LWIR
![Page 32: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/32.jpg)
TNO Defence, Security and Safety32
Color the Night
Visual NIR LWIR
![Page 33: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/33.jpg)
TNO Defence, Security and Safety33
Color the Night
Visual NIR LWIR
![Page 34: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/34.jpg)
TNO Defence, Security and Safety34
Color the Night
Visual NIR LWIR
![Page 35: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/35.jpg)
Image enhancement techniques
• Temporal noise reduction• Scene-based non-uniformity correction• Super-resolution reconstruction• Local adaptive contrast enhancement• Color-the-Night image fusion
SBNUC DSR
Color-the-Night
LACE
DSR
DSR
Resizing the TIR imageTIR
VISUAL
NIR
![Page 36: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/36.jpg)
Noise reductionby Dynamic Super Resolution
• Advantages• Decreased noise (improve SNR ratio)• Increased detection range• Increased situational awareness
![Page 37: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/37.jpg)
Fixed pattern noise reductionby Scene-Based Non-Uniformity Correction
• Advantages
• Low maintenance (hands-off operation)
• Clean pictures (performance maximized)
• Reduced sensor size / weight / cost (simpler design)
![Page 38: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/38.jpg)
Resolution enhancementby Dynamic Super Resolution
• Advantages• Increased sharpness • Increased situational awareness• Increased recognition and identification
![Page 39: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/39.jpg)
Local Adaptive Contrast Enhancement (LACE)• Advantages
• Increased situational awareness• Increased recognition and identification
in the total dynamic range
![Page 40: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/40.jpg)
Noise reduction
NIR VIS LWIR
![Page 41: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/41.jpg)
Noise reduction for I2
NormalDSR1DSR1+low LACE
NIR image
![Page 42: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/42.jpg)
Noise Reduction and LACE on I2 images
![Page 43: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/43.jpg)
Noise Reduction and LACE on LWIR images
![Page 44: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/44.jpg)
weak LACE vs strong LACE
![Page 45: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/45.jpg)
Original images, RGB fusion Original images, Color-the-Night fusion
Enhanced images, RGB fusion Enhanced images, Color-the-Night fusion
![Page 46: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/46.jpg)
No enhancement DSR1 + SBNUC
DSR1 + SBNUC; LACE before CtN DSR1 + SBNUC; LACE after CtN
![Page 47: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/47.jpg)
Color-the-Night & LACE
No LACE Low LACE before CtN Low LACE after CtN
High LACE before CtN High LACE after CtN
![Page 48: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/48.jpg)
Conclusions on Image Enhancement :
• Temporal Noise reduction works for all imagery• Scene Based Non Uniformity Correction (SBNUC) is useful for LWIR
• Super-resolution reconstruction has added value for LWIR• images are resolution limited
• Super-resolution reconstruction has limited value for I2
• Images are noise limited
• Enhancement algorithms need accurate motion estimation• works for small moving objects• not for scenery dominated by smoke or large moving objects
• LACE can be applied on all imagery• noisy imagery requires noise reduction
![Page 49: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/49.jpg)
Relation between Image Enhancement and Image Fusion
• Enhanced images can be used as input for Color-the-Night fusion
• The results have similar appearance and a significantly improved SNR
• LACE applied after tone mapping keeps the appearance similar
• LACE applied before tone mapping may enhance more details
• but is more expensive and may disturb color mapping!
• Effect of LACE applied before fusion is reduced by color mapping
![Page 50: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/50.jpg)
TNO Defence, Security and Safety50
Synthetic Scene Generation
![Page 51: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/51.jpg)
TNO Defence, Security and Safety51
![Page 52: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/52.jpg)
TNO Defence, Security and Safety52
Corresponding LWIR and synthetic imagesin 1st person perspective
LWIR Synthetic
![Page 53: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/53.jpg)
TNO Defence, Security and Safety53
Corresponding LWIR and synthetic imagesin 3rd person perspective
LWIR Synthetic
![Page 54: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/54.jpg)
TNO Defence, Security and Safety54
LWIR synthetic synthetic
Switching walls on --- off
![Page 55: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/55.jpg)
TNO Defence, Security and Safety55
LWIR synthetic synthetic
Switching vegetation on --- off
![Page 56: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/56.jpg)
TNO Defence, Security and Safety56
LWIR Material view
![Page 57: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/57.jpg)
TNO Defence, Security and Safety57
Visual NIR LWIR
RGB
![Page 58: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/58.jpg)
TNO Defence, Security and Safety58
RGB Photo
Color the Night DNR LACE
![Page 59: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/59.jpg)
TNO Defence, Security and Safety59
RGB Synthetic scene
Color the Night DNR LACE
![Page 60: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/60.jpg)
TNO Defence, Security and Safety60
Visual NIR LWIR
RGB
![Page 61: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/61.jpg)
TNO Defence, Security and Safety61
Photo
Color the Night DNR LACE
RGB
![Page 62: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/62.jpg)
TNO Defence, Security and Safety62
Synthetic scene
Color the Night DNR LACE
RGB
![Page 63: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/63.jpg)
June 2010: Data Collection Experiment on Military Base in the Netherlands
![Page 64: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/64.jpg)
Triclobs on scootmobile
![Page 65: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/65.jpg)
The Targets
![Page 66: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/66.jpg)
Multispectral Pinkus Target
![Page 67: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/67.jpg)
Real-time Scene Display
![Page 68: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/68.jpg)
Visual NIR
LWIR
![Page 69: Wpafb 2010](https://reader030.vdocuments.us/reader030/viewer/2022012900/54b95ea14a79594e738b487f/html5/thumbnails/69.jpg)
The End