nvl sensor fusion test bed
DESCRIPTION
NVL Sensor Fusion Test Bed. March 18, 2004. Introduction. US Army Night Vision & Electronic Sensors Directorate (NVESD) Network of acoustic and image sensors Visible and IR Classification of civilian targets. Motivation & Background. Military targets ~98% - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/1.jpg)
Bala Lakshminarayanan, Mike McCullough
NVL Sensor Fusion Test Bed
March 18, 2004
![Page 2: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/2.jpg)
Bala Lakshminarayanan, Mike McCullough
Introduction
• US Army Night Vision & Electronic Sensors Directorate (NVESD)
• Network of acoustic and image sensors– Visible and IR
• Classification of civilian targets
![Page 3: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/3.jpg)
Bala Lakshminarayanan, Mike McCullough
Motivation & Background
• Military targets ~98%– 3 levels of fusion process, acoustic and
seismic data
• Current civilian classification ~80%– Improve accuracy rating– Both civilian targets and personnel
classification– Image data
![Page 4: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/4.jpg)
Bala Lakshminarayanan, Mike McCullough
Objectives
• Generation of data set of image and acoustic data– Development and fusion of moving target ATR
algorithms
• Establish methods to collect data and its “ground truth”
![Page 5: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/5.jpg)
Bala Lakshminarayanan, Mike McCullough
SFTB Setup
SFTB Base Station
Node 3 Acoustic sensor
Node 1 Acoustic sensor Node 2 Acoustic sensor
Node 3 IR sensor
Node 3 Acoustic sensor
Node 1 IR sensor
Node 1 Acoustic sensor
Node 2 Visual sensor
Node 2 Acoustic sensor
Met station & GPS
Commands through DOS Scripts
GPS
GPS
GPS
Wireless Ethernet connection
![Page 6: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/6.jpg)
Bala Lakshminarayanan, Mike McCullough
Sensors
• Indigo Alpha Thermal camera
• Pulnix TMC-7DSP Color camera
• Knowles BL-1994 Microphone
![Page 7: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/7.jpg)
Bala Lakshminarayanan, Mike McCullough
General Test Conditions (1)
• 3 nodes each with hexagonal acoustical array of 7 microphones and imaging sensor– Nodes 1 & 3 have uncooled IR camera– Node 2 has visible color camera
• Nodes gather information simultaneously for 3 minutes
• Acoustic sensor turns on imaging sensors
• MUSIC algorithm for DOA estimation
![Page 8: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/8.jpg)
Bala Lakshminarayanan, Mike McCullough
General Test Conditions (2)
• Targets moving on gravel & asphalt roads
• Fully exposed– Trees or other vehicles occasionally in the
way
• License plates on the targets are not readable
• Stationary sensors
• Daylight operation (9:30am to 3:30pm)
![Page 9: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/9.jpg)
Bala Lakshminarayanan, Mike McCullough
General Test Conditions (3)
• Target motion– Constant speed– Stops midway
• Constant acceleration, deceleration• Stops for count of ten
• Each target traverses at 5, 10, 15, 20 mph• Start and stop outside FoV of nodes• Creation of different scenarios
![Page 10: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/10.jpg)
Bala Lakshminarayanan, Mike McCullough
SFTB Operation (1)
• Attended mode– Short term data collection / Demo mode
– Collects 4 types of data
– Surveillance, directed, pan scanning
– SFTB_Base.exe, FullSim.exe
• Data collection mode– Pure data collection
– Collects 4 types of data
– Acquire.exe (Video and acoustic), MetEffects.exe
![Page 11: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/11.jpg)
Bala Lakshminarayanan, Mike McCullough
SFTB Operation (2)
• Collected data– Acoustic .dat – Image .arf– Ground Truth .agt
• Filenames depends on sensor, node, scenario and targets
![Page 12: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/12.jpg)
Bala Lakshminarayanan, Mike McCullough
Numbering System
• SSSN00000_0000
• SSS = camera name– IN1 = Indigo IR camera 1– IN2 = Indigo IR camera 2– PX1 = Polinex Visible camera 1– AC1 = Acoustic number 1
• N = node number (1-3)• 00000 = scenario number• _0000 = number indicating vehicle number (1-7)
– Can have multiple numbers multiple targets
![Page 13: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/13.jpg)
Bala Lakshminarayanan, Mike McCullough
Number System Example
• in1200004_0003– In1 indigo thermal camera #1– 2 node 2– 00004 scenario number 4– _0003 target 3
• ac1300001_0056– ac1 acoustic array #1– 3 node 3– 00001 scenario number 1– _0056 both targets 5 and 6
![Page 14: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/14.jpg)
Bala Lakshminarayanan, Mike McCullough
AGT Format
• Very similar to a class in a high level programming language
• Agt{
PrjSect {…}SenSect {
SenUpd {…}}TgtSec{
TgtUpd {…}}
}
![Page 15: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/15.jpg)
Bala Lakshminarayanan, Mike McCullough
AGT format
![Page 16: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/16.jpg)
Bala Lakshminarayanan, Mike McCullough
PrjSect
• Name = “sftb”
• Scenario = “00001_0001”
• Site = “nvis0306”
![Page 17: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/17.jpg)
Bala Lakshminarayanan, Mike McCullough
SenSect
• Denotes the sensor section of the AGT
• Contains all of the SenUpd
• Name = “in11”– Denotes which sensor being used
![Page 18: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/18.jpg)
Bala Lakshminarayanan, Mike McCullough
SenUpd
• LatLong
• Elevation
• Keyword “Frame #1”
• Keyword “AcousticAz: 0”
• Keyword “Nodeld: 1”
• Azimuth 75.9077301025
• Time 2003 160 16 22 34 587
• Fov 40.0 30.0
![Page 19: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/19.jpg)
Bala Lakshminarayanan, Mike McCullough
TgtSect & TgtUpd
• TgtSect is the sector that contains all the target updates
• TgtUpd– Keyword “Frame: #1”– Time 2003 160 16 22 34 587– Tgt
• Range 48.0• TgtType “MAN”• Aspect 46.0• PixLoc 40 63
![Page 20: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/20.jpg)
Bala Lakshminarayanan, Mike McCullough
ARF Info
• Automatic target recognition working group Raster Format
• Contains header, sub headers, footers, 1 or more images
• Supports multiple frames
• Supports 16 different image types in same file
![Page 21: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/21.jpg)
Bala Lakshminarayanan, Mike McCullough
ARF Info
Rows, cols, version, type, # frames, offset…
Colormap, comments
Image file
![Page 22: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/22.jpg)
Bala Lakshminarayanan, Mike McCullough
ImageJ
• NVL used a plugin for converting image from .arf to other formats– Image processing and analysis in java
• Formats – dicom, pgm, jpg, bmp, tiff, raw…
• Operations – FFT, convolution, fractal box count, morphological…
![Page 23: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/23.jpg)
Bala Lakshminarayanan, Mike McCullough
Acoustic Data
• Raw data in acoustic .dat file
• Contains header information for system time (similar to AGT), node #, longitude/latitude (0’s), and bearing (0’s)
• make_wave.py to convert from .dat to .wav– Changes to specify output file needed within
the python script to make it work properly– Script drops .dat extension and adds .wav
![Page 24: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/24.jpg)
Bala Lakshminarayanan, Mike McCullough
Targets
• Honda CRX (target 1)
• Chevy Cavalier (target 2)
![Page 25: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/25.jpg)
Bala Lakshminarayanan, Mike McCullough
Targets
• Toyota pickup (target 3)
• GMC pickup (target 4)
![Page 26: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/26.jpg)
Bala Lakshminarayanan, Mike McCullough
Targets
• Vehicle (?) (target 5)– Names obtained from AGT files that would eventually
contain a TgtType indicating the target
• Toyota 4Runner (target 6)
• Stake body light truck (target 7)– Dave Rankins
![Page 27: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/27.jpg)
Bala Lakshminarayanan, Mike McCullough
Example Data
• Convert arf files into raw using ImageJ
• Modify raw image into PGM– Switch endianness
• Apply image processing techniques to the image– Very hard to distinguish objects due to having a
dark image
![Page 28: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/28.jpg)
Bala Lakshminarayanan, Mike McCullough
Example Sound Clips
• Normalized data using shareware program
• Target 1 – Node 1– Node 2– Node 3
• Target 6– Node 1– Node 2– Node 3
![Page 29: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/29.jpg)
Bala Lakshminarayanan, Mike McCullough
Example Images (1)
• Sensor placement• FoV of a sensor covers atleast half of total FoV
![Page 30: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/30.jpg)
Bala Lakshminarayanan, Mike McCullough
Example Images (2)
![Page 31: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/31.jpg)
Bala Lakshminarayanan, Mike McCullough
Example Images (3)
![Page 32: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/32.jpg)
Bala Lakshminarayanan, Mike McCullough
Example Images (4)
![Page 33: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/33.jpg)
Bala Lakshminarayanan, Mike McCullough
Future Work
• MatLab acoustical analysis
• Segmentation & shape analysis
• Feature selection & extraction
• Fusion
• Target Recognition algorithms
![Page 34: NVL Sensor Fusion Test Bed](https://reader036.vdocuments.us/reader036/viewer/2022070418/5681581b550346895dc57ff8/html5/thumbnails/34.jpg)
Bala Lakshminarayanan, Mike McCullough
Questions?