20131102-towards ecv-輔大
Post on 04-Jun-2018
228 Views
Preview:
TRANSCRIPT
-
8/13/2019 20131102-Towards ECV-
1/66
Wang, Yuan-Kai()Electrical Engineering,Fu Jen University ()Email: ykwang@mail.fju.edu.tw,URL: http://www.ykwang.twSlideshare: Search Towards Embedded Computer Vision
Towards
EmbeddedComputer Vision
-
8/13/2019 20131102-Towards ECV-
2/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
2
MIT Oxygen
http://oxygen.lcs.mit.edu/ 1999
-
8/13/2019 20131102-Towards ECV-
3/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
3
The Aware Home
http://awarehome.imtc.gatech.edu/
-
8/13/2019 20131102-Towards ECV-
4/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Ambient Intelligence v.s.
Object Intelligence
Embedded system + Sensors
4
Wearable ComputingSmart Home, Smart
-
8/13/2019 20131102-Towards ECV-
5/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
55
Embedded Systems
"Without" Sensors
MIC ITIS
-
8/13/2019 20131102-Towards ECV-
6/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
66
Embedded Systems
"With" Sensors
GPS
Navigation
Wii
Roomba
Exoskeleton
-
8/13/2019 20131102-Towards ECV-
7/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
7
Driver assist
Front view
Night vision
Blind corner
Blind spot
Rear view
Embedded Systems
"With" Image Sensors
-
8/13/2019 20131102-Towards ECV-
8/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
88
Embedded Systems
"With" Image Sensors
Google Glass
-
8/13/2019 20131102-Towards ECV-
9/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Smart Watch
9
-
8/13/2019 20131102-Towards ECV-
10/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Microsoft
SenseCam
Embedded Systems
"With" Image Sensors
10
-
8/13/2019 20131102-Towards ECV-
11/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Observation
Miniaturization
High Resolution
3D Capability
11
-
8/13/2019 20131102-Towards ECV-
12/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
12
Embedded Computer Vision
! Embedded System + Camera+ Computer Vision Algorithm
Image
Processing
Image
Recognition
Image
Capturing
-
8/13/2019 20131102-Towards ECV-
13/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
13
Embedded Computer Vision
! Embeddedcompute vision" Implements computer vision algorithms
on low-cost, low-power,
constrained hardware! Constrainedhardware" Low-speed CPU" Low capacity memory" No floating-point (FPU)" Low-resolution image sensor
-
8/13/2019 20131102-Towards ECV-
14/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
A BResearch Status (1/3)
14
-
8/13/2019 20131102-Towards ECV-
15/66
T d E b dd d C Vi i 1616
-
8/13/2019 20131102-Towards ECV-
16/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
1616
Research Status (3/3)
DARPA
Grand
Challenge
Google
CarGoogle
Taxi
T d E b dd d C t Vi i 17
-
8/13/2019 20131102-Towards ECV-
17/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
17
International Activities (1/2)!
Special conferences" IEEE Int. Workshop on Embedded ComputerVision
" IEEE Int. Conf. Distributed Smart Cameras! Special journal issues
" IEEE Journal of Selected Topics in SignalProcessing, vol. 2, no. 4, Aug. 2008
" EURASIP Journal on Embedded Systems! Short courses in important CV conferences
" CVPR07&08: Distributed vision processing insmart camera networks" ESC07: Embedded CV and smart cameras" ICASSP09: Distributed processing in smart
cameras
T d E b dd d C t Vi i 18
-
8/13/2019 20131102-Towards ECV-
18/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
18
International Activities (2/2)
! Research projects and Lab." Princeton Univ./Georgia Tech.:
Embedded Systems Lab., Wayne Wolf" Stanford Univ.
Wireless sensor networks Lab." UCLA, CMU, MIT" Delft Univ. of Technology
SmartCam Project" Graz Univ. of Technology
T d E b dd d C t Vi i 19
-
8/13/2019 20131102-Towards ECV-
19/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
19
Three Ways for ECV
! CPU (Central Processing Unit)" ARM, PowerPC
! DSP (Digital Signal Processor)" TI, ADI, NXP
! FPGA (Field Programmable Gate Array)" Altera, Xilinx
Towards Embedded Computer Vision p 20
-
8/13/2019 20131102-Towards ECV-
20/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
20
CPU
! Stanford MeshEye" ARM 7 (55MHz), ZigBee node" 3 image sensors" 30x30 grayscale x 2" 640x480 color x 1
Object detection
(http://wsnl.stanford.edu/smartcam.html)
"MeshEye:a hybrid-resolutionsmart camera mote for applicationsin distributed intelligentsurveillance", IPSN-SPOTS, 2007
Towards Embedded Computer Vision p 21
-
8/13/2019 20131102-Towards ECV-
21/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
21
DSP
! "Visual surveillance on DSP-basedembedded platform," Graz Univ. ofTechnology, 2008(Phd. dissertation)" TI C6414 (600MHz)" Applications: Adaboost face detection, vehicle
detection, license plate detection
Towards Embedded Computer Vision p 22
-
8/13/2019 20131102-Towards ECV-
22/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
22
FPGA (2/3)
"Hardware, Design andImplementation Issues on a
FPGA-Based Smart Camera,"ICDSC, 2007
Towards Embedded Computer Vision p 23
-
8/13/2019 20131102-Towards ECV-
23/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
23
Hybrid: CPU+DSP
! "Distributed Embedded Smart Camerasfor Surveillance Applications,"IEEE Computer, 2006.
Developed for traffic surveillance
Towards Embedded Computer Vision p 24
-
8/13/2019 20131102-Towards ECV-
24/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Hybrid: CPU+DSP by SoC
! TI OMAP
24
Towards Embedded Computer Vision p 25
-
8/13/2019 20131102-Towards ECV-
25/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
25
Hybrid: CPU+FPGA
! UCLA Cyclops" MCU: Atmega128" CPLD: image processing" ECV app.: Hand gesture recognition
http://research.cens.ucla.edu/projects/2007/
Multiscaled_Actuated_Sensing/
Cyclops/
Towards Embedded Computer Vision p 26
-
8/13/2019 20131102-Towards ECV-
26/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Hybrid: CPU+FPGA by SoC
! Xilinx Zynq
26
Towards Embedded Computer Vision p. 27
-
8/13/2019 20131102-Towards ECV-
27/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
27
Challenges for ECV
! Algorithm refinement" Parallel computation" Function partition, Multi-threading
" Stream processing" Memory hierarchy optimization
! Hardware design" Pipeline, SIMD, board design
! Optimized programming skills" Fixed-point arithmetic" Memory management" Intrinsic commands
Towards Embedded Computer Vision p. 28
-
8/13/2019 20131102-Towards ECV-
28/66
Towards Embedded Computer Vision p.
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
28
Our Experiences
Entertainment Robot (CPU)Vision Sensor (CPU)Video Surveillance (FPGA)
Wearable Vision (DSP) Towards Embedded Computer Vision p. 29
-
8/13/2019 20131102-Towards ECV-
29/66
p p
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Case 1Entertainment
RobotCPU
Towards Embedded Computer Vision p. 30
-
8/13/2019 20131102-Towards ECV-
30/66
p p
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Sony AIBO
Towards Embedded Computer Vision p. 31
-
8/13/2019 20131102-Towards ECV-
31/66
p p
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Rolling Dice
Towards Embedded Computer Vision p. 32
-
8/13/2019 20131102-Towards ECV-
32/66
p p
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Face Detection
Towards Embedded Computer Vision p. 33
-
8/13/2019 20131102-Towards ECV-
33/66
p p
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Face Recognition
Towards Embedded Computer Vision p. 34
-
8/13/2019 20131102-Towards ECV-
34/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Facial Expression Recognition
Towards Embedded Computer Vision p. 35
-
8/13/2019 20131102-Towards ECV-
35/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Case 2Vision Sensor
CPU
Towards Embedded Computer Vision p. 36
-
8/13/2019 20131102-Towards ECV-
36/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
FJUCam
Powerconsumption:1W
Power sources:5V DC current5V Cell Battery
Weight: 35gmSize:6 x 4.5 x 5 (cm)
(W x H x D)
Three Modules1. Main board, 2. Lens module
3. Storage module
Towards Embedded Computer Vision p. 37
-
8/13/2019 20131102-Towards ECV-
37/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Face Detection
FJUCam
Face Detection
Image
Display
Vision Sensor Network
Towards Embedded Computer Vision p. 38
-
8/13/2019 20131102-Towards ECV-
38/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Cyclops Robot
! Color tracking
Towards Embedded Computer Vision p. 39
-
8/13/2019 20131102-Towards ECV-
39/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Case 3Video Surveillance
FPGA
-
8/13/2019 20131102-Towards ECV-
40/66
Towards Embedded Computer Vision p. 41
-
8/13/2019 20131102-Towards ECV-
41/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Background Subtraction
Current
Frame
-
Background
Image Update
Post Processing
Motion Object Image
Background Image
1+kB
kB
1+kP
1+kM
),(),(
),(
1
1
yxByxP
yxM
kk
k
!=
+
+
),()1(),(
),(
1
1
yxPyxB
yxB
kk
k
+
+
!+= ""
Speed up by (1) Circuit design, (2) Paralization
Towards Embedded Computer Vision p. 42
-
8/13/2019 20131102-Towards ECV-
42/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
FPGA Speed Up
! Comparisons" PC: 2.8GHz CPU, C implementation" FPGA can speed up 500 times
2.8G
25M
51
10
Towards Embedded Computer Vision p. 43
-
8/13/2019 20131102-Towards ECV-
43/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Case 4Wearable Vision
X-EYE
ARM + DSP
Towards Embedded Computer Vision p. 44
-
8/13/2019 20131102-Towards ECV-
44/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Goal
44
Towards Embedded Computer Vision p. 45
-
8/13/2019 20131102-Towards ECV-
45/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Demo Video
Towards Embedded Computer Vision p. 46
-
8/13/2019 20131102-Towards ECV-
46/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Development
SD USB-WIFI
SB-RS232
SB
BeagleBoard
CameraUSBHub
Towards Embedded Computer Vision p. 47
-
8/13/2019 20131102-Towards ECV-
47/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Hardware
4747
Peripheral I/O
USB HostJTAGDVI-D video outS-Video outSD/MMC+Stereo in/outRS-232 serial1Alternate powerUSB 2.0 HS OTG
USER
OMAP3530 Processor600MHz Cortex-A8
NEON+VFPv316KB/16KB L1256KB L2
430MHz C64x+ DSP32K/32K L1
48K L1D32K L2
Power VR SGX GPU64K on-chip RAMPOP Memory256MB LPDDR RAM256MB NAND flash
RESET
7.6 cm Towards Embedded Computer Vision p. 48
-
8/13/2019 20131102-Towards ECV-
48/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
System
2010.04.25
48 Towards Embedded Computer Vision p. 49
-
8/13/2019 20131102-Towards ECV-
49/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Software
Towards Embedded Computer Vision p. 50
-
8/13/2019 20131102-Towards ECV-
50/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
FJUCam2
Towards Embedded Computer Vision p. 51
-
8/13/2019 20131102-Towards ECV-
51/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Next : UAV
Towards Embedded Computer Vision p. 52
-
8/13/2019 20131102-Towards ECV-
52/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Next : Sleep Monitoring
Towards Embedded Computer Vision p. 53
-
8/13/2019 20131102-Towards ECV-
53/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Next : Vehicular
Object Detection
Towards Embedded Computer Vision p. 54
-
8/13/2019 20131102-Towards ECV-
54/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Heterogeneous Multicore
! Xilinx Zynq: ARM+FPGA
Towards Embedded Computer Vision p. 55
-
8/13/2019 20131102-Towards ECV-
55/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Future ofEmbedded Vision
Towards Embedded Computer Vision p. 56
-
8/13/2019 20131102-Towards ECV-
56/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Summary
! Embedded CPU is not appropriate forECV" Pros" Great flexibility" Fast development
" Cons"
No float-point arithmatics"Interrupt-based architecture
! Smart phone with 4/8 cores is OK? Towards Embedded Computer Vision p. 57
-
8/13/2019 20131102-Towards ECV-
57/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
I Think!
!Embeddedcompute vision"Low-cost, low-power,
minimal hardware
! High-Performancecomputer vision" Fast speed without cost,
power, and hardware
constraintsFrom contradictionto convergence ?!
ConstrainedResource
AbundantResource
Yes by multicore
Contradiction
Towards Embedded Computer Vision p. 58
-
8/13/2019 20131102-Towards ECV-
58/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Challenges (1/2)
! Algorithm decomposition" Function decomposition" Partition serial part and parallel part" Data flow analysis and
data dependency analysis" Parallelism" Loop unrolling"
Multithreading" Pipeline
Towards Embedded Computer Vision p. 59
-
8/13/2019 20131102-Towards ECV-
59/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Challenges (2/2)
! Performance analysis method" For efficiency improvement! Implementation efforts" Choose a good embedded platform for
computer vision" Software issues" Hardware issues
"Programming skills" Multi-threading
Towards Embedded Computer Vision p. 60
-
8/13/2019 20131102-Towards ECV-
60/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Future Research Directions
HeterogeneousMulticore
ARM+ DSP
ARM +FPGA
ARM +GPGPU
Towards Embedded Computer Vision p. 61
-
8/13/2019 20131102-Towards ECV-
61/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
I Think Again!
! Is there a development platform forembedded computer vision with" Built-in camera" AE, AF, AWB
" High-performance processor" Heterogeneous multicore
" Software development environment:"
C, Matlab" OpenCV" Sample codes" Labs
Towards Embedded Computer Vision p. 62
-
8/13/2019 20131102-Towards ECV-
62/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
Embedded Vision Alliance
Towards Embedded Computer Vision p. 63
-
8/13/2019 20131102-Towards ECV-
63/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
! Ramesh RaskarMIT Platform
Towards Embedded Computer Vision p. 64
-
8/13/2019 20131102-Towards ECV-
64/66
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai
! Andrew AdamsStanford Platform
-
8/13/2019 20131102-Towards ECV-
65/66
Towards Embedded Computer Vision p. 66
-
8/13/2019 20131102-Towards ECV-
66/66
!Creative Commons - 3.0 ! (1) (2)!
top related