high resolution 2d imaging and 3d scanning with line sensors
TRANSCRIPT
![Page 1: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/1.jpg)
High Resolution 2D Imaging and 3D Scanningwith Line Sensors
Thesis Defense
Jian Wang
7/27/2018
![Page 2: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/2.jpg)
Imaging – Key driver of the modern society
2
Hubble telescope
Super-res. microscopy
CT LIDAR Cellphone entertainment
![Page 3: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/3.jpg)
Traditional imaging methods are woefully inadequate
• Passive imaging
– Non-visible light imaging (challenge I)
• Active imaging
– 3D scanning under strong ambient light (challenge II)
– 3D scanning under strong global light like in scattering media (challenge III)
3
with high spatial resolutionhigh temporal resolutionlow cost
![Page 4: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/4.jpg)
Challenge I: Passive imaging in non-visible wavebands
4
Wavelength ShorterLonger
Visible
Ultra-VioletInfraredTHz X-ray
![Page 5: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/5.jpg)
5
visible
SWIR
visible
SWIR
SWIRfor seeing through fog
and night vision
MWIRfor high contrast thermal imaging
THzfor surveillance
Images courtesy: http://www.sensorsinc.com/gallery/imageshttps://www.edmundoptics.com/resources/application-notes/imaging/what-is-swir/
Challenge I: Passive imaging in non-visible wavebands
![Page 6: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/6.jpg)
Per-pixel price
Spectral band Detector technology Per-pixel price ($/pix)
mmW/THz Multiple 102 − 104
LWIR HgCdTe < 101
Bolometer 10−2
MWIR InSb/PbSe 10−1
SWIR InGaAs/PbSe 10−1
NIR/VIS/NUV Si < 10−6
MUV Si (thinned) < 10−3
EUV Si-PIN/CdTe 102 − 103
Soft x ray Si (thinned) 10−2
Si-PIN/CdTe 102 − 103
Hard x ray/gamma Multiple 102 − 1046
Gehm, M. E., and D. J. Brady. "Compressive sensing in the EO/IR." Applied optics 54.8 (2015)
![Page 7: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/7.jpg)
Inexpensive alternative designs for 2D imaging in non-visible wavebands
7
Point detector with 2D scanning 1D sensor with 1D scanning Point detector with multiplexing
Point detectorPoint detector 1D sensor
SLM
Duarte, Marco F., et al. "Single-pixel imaging via compressive sampling." IEEE signal processing magazine 25.2 (2008): 83-91.
𝑅𝑔𝑎𝑙𝑣𝑜 = 5000
𝑅𝑆𝐿𝑀 = 104, 10 × compressionImage resolution 𝑁 × 𝑁
![Page 8: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/8.jpg)
Inexpensive alternative designs for 2D imaging in non-visible wavebands
8
Point detector with 2D scanning 1D sensor with 1D scanning Point detector with multiplexing
Point detectorPoint detector 1D sensor
SLM
5000
𝑁2
5000
𝑁
105
𝑁2Frame rate
Duarte, Marco F., et al. "Single-pixel imaging via compressive sampling." IEEE signal processing magazine 25.2 (2008): 83-91.
𝑁 = 1000
![Page 9: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/9.jpg)
Inexpensive alternative designs for 2D imaging in non-visible wavebands
9
Point detector with 2D scanning 1D sensor with 1D scanning Point detector with multiplexing
Point detectorPoint detector 1D sensor
SLM
0.005 5 0.1Frame rate
Duarte, Marco F., et al. "Single-pixel imaging via compressive sampling." IEEE signal processing magazine 25.2 (2008): 83-91.
![Page 10: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/10.jpg)
Inexpensive alternative designs for 2D imaging in non-visible wavebands
10
Point detector with 2D scanning 1D sensor with 1D scanning Point detector with multiplexing
Point detectorPoint detector 1D sensor
SLM
5000
𝑁2
5000
𝑁
105
𝑁2Frame rate
Light throughput Much light loss Much light loss Half light
Duarte, Marco F., et al. "Single-pixel imaging via compressive sampling." IEEE signal processing magazine 25.2 (2008): 83-91.
![Page 11: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/11.jpg)
Challenge II: Active imaging under ambient light
• Sunlight is orders of magnitude stronger than active light source
– Photon noise
11
Microsoft Kinect
Fast but not robustImages courtesy: https://www.youtube.com/watch?v=rI6CU9aRDIo
2D sensor
2D projector
![Page 12: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/12.jpg)
Strategy 1: concentrate and scan
12
Point scanning-based TOF Point scanning-based SL Line scanning-based SL
2D sensor 2D sensorPoint detector
5000
𝑁2
30
𝑁2
30
𝑁Frame rate
𝑁2 𝑁2 𝑁SNR increase
Too slow
![Page 13: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/13.jpg)
Strategy 2: SWIR
13
250 500 750 1000 1250 1500 1750 2000 2250 2500
wavelength
Sola
r in
ten
sity SWIR
Solar energy distribution
2D is too costly
![Page 14: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/14.jpg)
Challenge III: Active imaging in scattering medium
14
visible
SWIR
Problem with imaging in scattering medium
Use of SWIRPoint scanning-based selective imaging:
confocal imaging
Light source
Aperture
Focalplane
Aperture
Point detector
Strategies:
Too slow 2D Too costly
Marvin, Minsky. "Microscopy apparatus." U.S. Patent No. 3,013,467. 19 Dec. 1961.
![Page 15: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/15.jpg)
Traditional imaging methods are woefully inadequate
• Passive imaging
– Non-visible light imaging (Challenge I)
• Active imaging
– 3D scanning under strong ambient light (Challenge II)
– 3D scanning under strong global light like in scattering media (Challenge III)
15
Point detector based 2D sensor based
• Low cost• Robust • Too slow
• High cost• Not reliable in harsh envir.• High ST resolution
![Page 16: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/16.jpg)
Contributions
• Passive imaging
– Non-visible light imaging
• Active imaging
– 3D scanning under strong ambient light
– 3D scanning under strong global light like in scattering media
16
Point detector based 2D sensor based
• Low cost• Robust • Too slow
• High cost• Not reliable in harsh envir.• High ST resolution
Line sensor based
• Low cost• Robust • High ST resolution
![Page 17: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/17.jpg)
17
Optics Mechanics ComputationLine sensor
• Benefits• Affordable price / High resolution / High-speed readout / Bigger pixel / Have space on top/bottom /
Frame transfer…
• Sensing modality• Visible light / SWIR / CW-TOF / SPAD / DVS / PSD…
![Page 18: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/18.jpg)
18
LiSens2D imaging architectureJian Wang, Mohit Gupta, and Aswin C. Sankaranarayanan. "LiSens-A scalable architecture for video compressive sensing." ICCP 2015.
DualSL3D scanning architectureJian Wang, Aswin C. Sankaranarayanan, Mohit Gupta, and Srinivasa G. Narasimhan. "Dual structured light 3d using a 1d sensor." ECCV 2016.
TriLCRobust proximity sensorJian Wang, Joe Bartels, William Whittaker, Aswin C. Sankaranarayanan, and SrinivasaG. Narasimhan. "Programmable Triangulation Light Curtains." ECCV 2018.
High spatial temporal resolution with line sensors
![Page 19: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/19.jpg)
19
LiSens2D imaging architectureJian Wang, Mohit Gupta, and Aswin C. Sankaranarayanan. "LiSens-A scalable architecture for video compressive sensing." ICCP 2015.
DualSL3D scanning architectureJian Wang, Aswin C. Sankaranarayanan, Mohit Gupta, and Srinivasa G. Narasimhan. "Dual structured light 3d using a 1d sensor." ECCV 2016.
TriLCRobust proximity sensorJian Wang, Joe Bartels, William Whittaker, Aswin C. Sankaranarayanan, and SrinivasaG. Narasimhan. "Programmable Triangulation Light Curtains." ECCV 2018 .
High spatial temporal resolution with line sensors
![Page 20: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/20.jpg)
Task: Imaging beyond visible light
• Choice 1: 2D camera Too costly!
• Choice 2: camera based on single pixel
– Option 1: single pixel + 2D galvomirror
– Option 2 (better): single pixel + spatial light modulator, Single Pixel Camera
20
![Page 21: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/21.jpg)
Task: Imaging beyond visible light
• Choice 1: 2D camera Too costly!
• Choice 2: camera based on single pixel
– Option 1: single pixel + 2D galvomirror
– Option 2 (better): single pixel + spatial light modulator, Single Pixel Camera
21
objective lens
relay lens
~ ~ ~digital micromirror device(DMD)
single pixel
ADC
![Page 22: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/22.jpg)
Task: Imaging beyond visible light
• Choice 1: 2D camera Too costly!
• Choice 2: camera based on single pixel
– Option 1: single pixel + 2D galvomirror
– Option 2 (better): single pixel + spatial light modulator, Single Pixel Camera
22
Best result: 128×128 at video rate
[Sankaranarayanan et al, Video Compressive Sensing for Spatial Multiplexing Cameras using Motion-Flow Models, arXiv:1503.02727]
![Page 23: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/23.jpg)
• DMD is slow
• Calculation
– Mega-pixel video @ 10fps, 106 × 10 = 107 pixels’ value
– 10x compressive sensing is used
– Need measurement rate: 107
10= 106
• measurement rate: # of measurements per second
– DMD can only flip 104 times per second
23
Why cannot SPC capture mega-pixel video?
![Page 24: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/24.jpg)
Proposed: Replace the single pixel by multiple pixels
24
Measurement rate
𝑅𝐴𝐷𝐶
# of pixels, 𝐹
Optimal # of pixels =𝑅𝐴𝐷𝐶
𝑅𝐷𝑀𝐷
typically only 1000s of pixels
![Page 25: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/25.jpg)
Line-Sensor-based compressive camera (LiSens)
25
objective lens
relay lens
digital micromirror device(DMD)
~ ~ ~
line sensor
1. Use a linear array of pixels (a line-sensor)
![Page 26: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/26.jpg)
Line-Sensor-based compressive camera (LiSens)
26
objective lens
relay lens
digital micromirror device(DMD)
~ ~ ~
line sensor
1. Use a linear array of pixels (a line-sensor)2. Add a cylindrical lens
cylindrical lens
![Page 27: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/27.jpg)
27
![Page 28: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/28.jpg)
Cylindrical lens
28
![Page 29: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/29.jpg)
Each line-sensor pixel integrates one row of the scene
29z
y
z
x
Objective lens Cylindrical lens
Lin
e-
sen
so
rLin
e-
sen
so
r
Objective lens Cylindrical lens
![Page 30: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/30.jpg)
30
Each line-sensor pixel integrates one row of the scene
![Page 31: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/31.jpg)
31
![Page 32: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/32.jpg)
LiSens
32
(a) Image on DMD (d) Image on sensor plane(b) Code on DMD (c) Coded image
![Page 33: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/33.jpg)
Hardware prototype
33
DMD
SPC
objective lens
relay lens
cylindrical lensline-sensor
![Page 34: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/34.jpg)
34
Image resolution: 1024 x 768Capture duration: 880 msCompression ratio: 1x
Outdoor Outdoor
![Page 35: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/35.jpg)
1024 x 768 pixels9 fps
8x compression
35
![Page 36: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/36.jpg)
36
OutdoorOutdoor
![Page 37: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/37.jpg)
Build two LiSenses in both sides of the DMD
• No light loss
• Joint deblurring
37
By row-multiplexing LiSens By column-multiplexing LiSens
![Page 38: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/38.jpg)
Build two LiSenses in both sides of the DMD
• Joint deblurring
38
By both LiSensesBy row-multiplexing LiSens By column-multiplexing LiSens
![Page 39: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/39.jpg)
Summary of LiSens
• A 2D imaging architecture based on line sensor
– Min. # of pixels to achieve max. possible measurement rate
– Cost effective
– High spatial and temporal resolution
39
![Page 40: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/40.jpg)
40
LiSens2D imaging architectureJian Wang, Mohit Gupta, and Aswin C. Sankaranarayanan. "LiSens-A scalable architecture for video compressive sensing." ICCP 2015.
DualSL3D scanning architectureJian Wang, Aswin C. Sankaranarayanan, Mohit Gupta, and Srinivasa G. Narasimhan. "Dual structured light 3d using a 1d sensor." ECCV 2016.
TriLCRobust proximity sensorJian Wang, Joe Bartels, William Whittaker, Aswin C. Sankaranarayanan, and SrinivasaG. Narasimhan. "Programmable Triangulation Light Curtains." ECCV 2018 .
High spatial temporal resolution with line sensors
![Page 41: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/41.jpg)
High Cost of 2D Sensors
41
Structured Light 3D Imaging Using 1D Sensor?
SWIRSee through
scattering media
DVS sensorHighly efficient
3D sensor
- Yes
![Page 42: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/42.jpg)
CameraProjector
Light plane
Scene
Conventional structured light
42
Ray-plane triangulation
Camera’s ray Projector’s plane
![Page 43: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/43.jpg)
Integrated plane
Scene
Cylindrical lensCameraProjector
Proposed: Dual structured light (DualSL)
43
Projector’s ray Camera’s plane
No moving parts
∑ ∑ ∑ ∑∑ ∑
∑
Ray-plane triangulation
![Page 44: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/44.jpg)
Camera
Data acquired bythe 1D sensor
Projector
44
DualSL using Gray codes
Scene
Cylindrical lens
![Page 45: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/45.jpg)
Camera
Data acquired bythe 1D sensor
Projector
# of rows of projector
# of pixels of 1D sensor
45
DualSL using Gray codes
Scene
Cylindrical lens
![Page 46: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/46.jpg)
Camera Projector
Data acquired bythe 1D sensor
46
DualSL using Gray codes
Scene
Cylindrical lens
![Page 47: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/47.jpg)
Camera Projector
Data acquired bythe 1D sensor
47
DualSL using Gray codes
Scene
Cylindrical lens
![Page 48: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/48.jpg)
… 950
1050
1150
mm
Depth mapData acquired by
the 1D sensor
48
3D visualizations
![Page 49: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/49.jpg)
Objective lens Relay lens
Cylindrical lens
Line-sensor
Projector
2D camera helper
Hardware prototype
49
DualSL
Traditional SL
![Page 50: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/50.jpg)
Objective lens
Relay lensCylindrical lens
Line-sensorProjector
2D camera helper
50
![Page 51: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/51.jpg)
3D visualizations of DualSL results
51
![Page 52: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/52.jpg)
DualSLwith 1D sensor
Target sceneTraditional SL
with 2D sensor
mm950
1050
1150
Difference map
RMSE = 1.49mmmm
0
5
10
52
![Page 53: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/53.jpg)
53
Target scene
DualSLwith 1D sensor
Traditional SLwith 2D sensor
mm950
1050
1150
Difference map
RMSE = 2.4mmmm
0
5
10
![Page 54: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/54.jpg)
54
DualSLwith 1D sensor
Traditional SLwith 2D sensor
mm950
1050
1150
Difference map
RMSE = 3.7mmmm
0
5
10
Target scene
Concave shiny bowl
![Page 55: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/55.jpg)
55
DualSLwith 1D sensor
Traditional SLwith 2D sensor
mm950
1050
1150
Difference map
RMSE = 2.18mmmm
0
5
10
Target scene
Translucent wax
![Page 56: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/56.jpg)
Analysis: Acquisition Speed
56
# Camera Pixels
# P
roj.
Ro
ws
# Camera Pixels
# C
amer
aR
ow
s
DualSL Traditional SL
Same acquisition speed
![Page 57: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/57.jpg)
Summary of DualSL
• Structured light 3D scanning architecture with line sensor
– Cost effective
• Same performance as traditional SL
– Temporal resolution
• Better performance
– Spatial resolution
– Under ambient light
– Under global light
60
![Page 58: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/58.jpg)
61
LiSens2D imaging architectureJian Wang, Mohit Gupta, and Aswin C. Sankaranarayanan. "LiSens-A scalable architecture for video compressive sensing." ICCP 2015.
DualSL3D scanning architectureJian Wang, Aswin C. Sankaranarayanan, Mohit Gupta, and Srinivasa G. Narasimhan. "Dual structured light 3d using a 1d sensor." ECCV 2016.
TriLCRobust proximity sensorJian Wang, Joe Bartels, William Whittaker, Aswin C. Sankaranarayanan, and SrinivasaG. Narasimhan. "Programmable Triangulation Light Curtains." ECCV 2018 .
High spatial temporal resolution with line sensors
![Page 59: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/59.jpg)
Task: detect object at specific depth (proximity sensor)
62
• Workable under strong sunlight
• Cost effective• Energy efficient• Computationally efficient
![Page 60: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/60.jpg)
Task: see through scattering media
63
• High spatial-temporal resolution
• Cost effective• Energy efficient• Computationally efficient
With smoke
![Page 61: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/61.jpg)
Proposed: light curtain device
64
Line Sensor
Line Laser
![Page 62: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/62.jpg)
Applications
65
Light Curtains for Robots Light Curtains for Vehicles
![Page 63: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/63.jpg)
Vehicle lane monitoring
66
![Page 64: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/64.jpg)
Optical schematic
67
laser
collimation lens
galvo
cylindrical lensor Powell lens
illumination module
line sensor
lens
galvo
sensor module
Top view
![Page 65: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/65.jpg)
Hardware prototype
68
Illumination SystemImaging System
Power & Control
Camera*Helper
*For Calibration and Visualization Only
![Page 66: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/66.jpg)
Hardware prototype
69
Illumination System
Galvomirror
Powell LensLaser
Imaging System
Galvomirror
S-Mount Lens
1D Camera
![Page 67: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/67.jpg)
Cosine light curtain
Scene Imaged CurtainLight Curtain
Z
X
6m
-2m 2m
4m
70
![Page 68: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/68.jpg)
Volume light curtain
Scene Imaged Curtain
Z
X
6m
2m
-1m 1m
Light Curtain
71
![Page 69: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/69.jpg)
Tilted curtain outdoors in cloudy day
Scene Imaged Curtain
Z
X
3m
6m
-2m 2m
Light Curtain
72
![Page 70: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/70.jpg)
Planar curtain outdoors under strong sunlight
Scene Imaged Curtain
Z
X
5m
-2m 2m
Light Curtain
73
![Page 71: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/71.jpg)
Scene Imaged Curtain
Backup detection – Vertical curtain
Z
X
8m
-1m
4m
Light Curtain
74
![Page 72: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/72.jpg)
Scene Imaged Curtain
П – Curtain for vehicle lane monitoring
Z
X
10m
-1m 1m
4m
Light Curtain
75
![Page 73: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/73.jpg)
Scattering media (fog, smoke…)
76
Traditional imaging
Pixel
Multi-scattered light
Single-scattered light Pixel
Light curtain imaging
Signal
Noise
Only multi-scattered light
![Page 74: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/74.jpg)
Planar light curtain in smoke
Scene Imaged Curtain
Z
X
3m
-1m 1m
Light Curtain
77
![Page 75: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/75.jpg)
Performance – 100klux
35m25m15m5mScene
78
![Page 76: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/76.jpg)
Depth Map under strong sunlight
79
Depth (meters)
Scene Depth Map
Sunny – 100klux
![Page 77: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/77.jpg)
Thickness
80
camera pixel laser
Thicknessby Triangulation
uncertainty
Top view
![Page 78: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/78.jpg)
Thickness
81
Scene Light curtain dataLight curtainwith thickness
Z
X
4m
-1m 1m
![Page 79: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/79.jpg)
Thickness
82
Uncertainty
Depth 𝑑
Triangulation∝ 𝑑2
CW Time-of-flight∝ 1/𝑓 𝑓 – modulation freq.
![Page 80: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/80.jpg)
Combine with time-of-fight sensor
83
z0
![Page 81: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/81.jpg)
Combine with time-of-fight sensor
84
z
depth by TOF
0
For 4-correlations TOF
Thickness before after
z0
For 2-correlations TOF
![Page 82: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/82.jpg)
Combine with time-of-fight sensor
85
0 1
By triangulation Phase Triangulation + Phase
![Page 83: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/83.jpg)
Combine with time-of-fight sensor
86
0 1
By triangulation Phase with wrapping
Triangulation + Phase
![Page 84: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/84.jpg)
Light fall-off
87
dep
th
exposure
![Page 85: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/85.jpg)
Depth-adaptive energy
88
dep
th
exposure0 5
Constant Depth-adaptive
![Page 86: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/86.jpg)
Depth-adaptive energy
89
dep
th
Laser power0 12
Constant Depth-adaptive
![Page 87: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/87.jpg)
Summary of TriLC device
• The depth of interest is pre-specified, so the device can be optimized accordingly
– Cost, energy, computation - low
– Workable under sunlight, in thick smoke/fog – ‘high’
– Flexible: shape, thickness
– Fast
• ‘Limitation’: only depth of interest
90
![Page 88: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/88.jpg)
91
LiSens2D imaging architectureJian Wang, Mohit Gupta, and Aswin C. Sankaranarayanan. "LiSens-A scalable architecture for video compressive sensing." ICCP 2015.
DualSL3D scanning architectureJian Wang, Aswin C. Sankaranarayanan, Mohit Gupta, and Srinivasa G. Narasimhan. "Dual structured light 3d using a 1d sensor." ECCV 2016.
TriLCRobust proximity sensorJian Wang, Joe Bartels, William Whittaker, Aswin C. Sankaranarayanan, and SrinivasaG. Narasimhan. "Programmable Triangulation Light Curtains." ECCV 2018.
High spatial temporal resolution with line sensors
![Page 89: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/89.jpg)
Conclusion remarks
• Traditional methods – Passive imaging in non-visible wavebands
– Active imaging in strong ambient light and global light
– Point detector based: cheap, robust but too slow
– 2D sensor based: fast but costly, not reliable
• This thesis: High resolution 2D imaging and 3D scanning with line sensors– Cheap, robust and fast
– LiSens: 2D imaging architecture
– DualSL: 3D scanning architecture
– TriLC: robust proximity sensor and see-thru-smoke imager92
![Page 90: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/90.jpg)
Thanks the committee
• Vijayakumar Bhagavatula
• Mohit Gupta
• Aswin Sankaranarayanan (advisor)
• Srinivasa Narasimhan (advisor)
93
![Page 91: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/91.jpg)
94
![Page 92: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/92.jpg)
Future work
• TriLC with multiple line lasers
– Eye-safety distance is same as light curtain usage distance
– Interference problem is solved
• Triangulation gating + temporal gating for seeing through scattering media
– Spatial cue and temporal cue
• Outdoor real-time photometric stereo
– One rolling-shutter camera and two aligned line light and the Sun
• Incorporating machine learning
– LiSens with fast reconstruction
– DualSL with less patterns
– TriLC with adaptively assigning lines
95
![Page 93: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/93.jpg)
LiSens
![Page 94: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/94.jpg)
97
1024 x 768 pixels104 fps
70x compression
Laser dots movement
![Page 95: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/95.jpg)
Line sensor vs. low-res 2D sensor array
98
line sensor
objective lens
relay lens
digital micromirror device(DMD)~ ~ ~
2D sensor
vs.
![Page 96: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/96.jpg)
If misalignment…
99
2D sensorLine-sensor
DMD DMD
![Page 97: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/97.jpg)
Frame transferNo light loss during readout
100
Simultaneous exposure and readoutWith frame transfer
Sequential exposure and readout
Without frame transfer
![Page 98: High Resolution 2D Imaging and 3D Scanning with Line Sensors](https://reader036.vdocuments.us/reader036/viewer/2022081503/62a259ae3e52b959704ffcf6/html5/thumbnails/98.jpg)
Laser Safety
101
0 2 4 6 8 10
z (m)
0
1
2
3
4
Po
we
rd
en
sity
( W/ m
2) allowed power density
actual power density
0 2 4 6 8 10
z (m)
-1
0
1
2
3
Po
wer
de
nsit
y( W
/ m2) allowed power density
actual power density
0 2 4 6 8 10
z (m)
-3
-2
-1
0
1
2
3
Po
wer
de
nsit
y(W
/m2) allowed power density
actual power density
Single pulse test Consecutive rows test Long time run (8 hrs) test
0.72m