folien ss2009 machine vision
TRANSCRIPT
-
8/3/2019 Folien SS2009 Machine Vision
1/163
Machine VisionMarkus Vincze
Automation and Control InstituteTechnische Universitt Wien
-
8/3/2019 Folien SS2009 Machine Vision
2/163
2
Contents
Motivation: many projects Terminology: from computer to machine vision Components: from light to images Machine Vision Segmentation Edge detection Grouping
Application: XPERO
robot learning
-
8/3/2019 Folien SS2009 Machine Vision
3/163
3
Industrial Projects FESTO Checkbox
(inspection of parts and part recpgnition)
AVL
Engine
Videoscope
(temperature measurement inthe engine during operation) IAEA/FAO Seperation ofmale and female Tsetse fliesto fight malaria Rauscher Inspection oftampoos, quality control
-
8/3/2019 Folien SS2009 Machine Vision
4/163
4
Prototypes
Holzer
Training-Optimisation-System
(TOS)
-
8/3/2019 Folien SS2009 Machine Vision
5/163
5
Research Projects CARAK Measuring 3D shape of retina FlexPaint Spray painting any industrial part
FibreScope
automatic
finding
of bore
holes
REDUX follow seam of carbon mats (EADS)
-
8/3/2019 Folien SS2009 Machine Vision
6/1636
Research Projects RobVision Navigation in ship ActIPret Interpretation of ahuman handling objects MOVEMENT reliable vision indoors Kognitives Sehen understanding to see,hide and seek XPERO Learning by experimentation
-
8/3/2019 Folien SS2009 Machine Vision
7/1637
3D (6 DoF) Object TrackingRobVision
EU Project 1998 -
2001
Model-based: Lines, ellipses estimate object pose
Robustness: integration of model cue and image cues Real time, 10 Hz
-
8/3/2019 Folien SS2009 Machine Vision
8/1638
RobVision: Vision Based NavigationRobVision
-
8/3/2019 Folien SS2009 Machine Vision
9/1639
Video: Navigation in Mock-up
RobVision
-
8/3/2019 Folien SS2009 Machine Vision
10/16310
Robot Navigation in Office
Vision for
Robotics
(V4R Tracking
Tool)
-
8/3/2019 Folien SS2009 Machine Vision
11/16311
Object Recognition and Tracking
Interest
points
and local
description
of the
objects appearance Code book as compact storage of many objects Recognition: relation of interest points to object centre
-
8/3/2019 Folien SS2009 Machine Vision
12/16312
Contents
Motivation: many projects Terminology: from computer to machine vision Components: from light to images Machine Vision Segmentation Edge detection Grouping
Application: XPERO
robot learning
-
8/3/2019 Folien SS2009 Machine Vision
13/16313
Computer Vision Computer Vision is a subfield of AI concerned withprocessing of images from the real world. Purpose:program a computer to"understand" a scene or features Methods:detection, segmentation,tracking, pose estimation, mapping to 3Dmodel, recognition of objects (e.g., human faces) Achieved bymeans of pattern recognition,statistical learning, projective geometry, imageprocessing, graph
theory
and other fields.
[1]
Dana H. Ballard, Christopher M. Brown, (1982) Computer Vision(2nd edition), Prentice Hall.
-
8/3/2019 Folien SS2009 Machine Vision
14/16314
Pattern Recognition
"the act of taking in raw data and taking an
action based on the category of the data" [1] Methods:statistics, machine learning, ... Problem:how to determine category of data
[1]
Richard O. Duda, Peter E. Hart, David G. Stork (2001) Patternclassification(2nd edition), Wiley, New York.
-
8/3/2019 Folien SS2009 Machine Vision
15/16315
Image Processing (Bildverarbeitung) Image to image processing [1] Subarea of computer / machine vision
[1]
Rafael C. Gonzales, Richard E. Woods, (2002) Digital ImageProcessing(2nd edition), Prentice Hall.
Fig.: Recognising number plates; 7 Spektra of NASA LANDSAT
-
8/3/2019 Folien SS2009 Machine Vision
16/16316
Machine Vision
=
application of computer
vision
to factory
automation.
A MV system is a computer that makes decisionsbased on the analysis of digital images.
Light
Lighting
ObjectSensor
Image
DataResultsControl
Media Reflection
Processing
Figure: Components of a machine vision system.
-
8/3/2019 Folien SS2009 Machine Vision
17/16317
Machine Vision Problem 1:narrow aplications Problem 2:understanding?
Lesson learned:more
options
to control
Lesson learned:consider complete systemLight
Lighting
ObjectSensor
Image
DataResultsControl
Media Reflection
Processing
Figure: Components of a machine vision system.
Light
Lighting
ObjectSensor
Image
DataResultsControl
Media Reflection
Processing
-
8/3/2019 Folien SS2009 Machine Vision
18/16318
Information searched for in Images
Determination of
Geometry: form, size Position and orientation (pose)
Properties
of materials: colour,
texture, irregularities (errors) Detect and recognise objects
Faces, persons, activities, ... Number plates, tables, mugs,industrial
parts, ...
-
8/3/2019 Folien SS2009 Machine Vision
19/16319
Contents
Motivation: many projects Terminology: from computer to machine vision Components: from light to images Machine Vision Segmentation Edge detection Grouping Stereo images Application: MOVEMENT robot navigation
-
8/3/2019 Folien SS2009 Machine Vision
20/163
20
From Light Source to Image
Many different influencing factors
Complete
modeling
is
impossible, yet
Solutions only under very constrained conditions
-
8/3/2019 Folien SS2009 Machine Vision
21/163
21
Electromagnetic Spectrum
[Encarta]
-
8/3/2019 Folien SS2009 Machine Vision
22/163
22
Radiometry Photometry
Radiometryis the
science of the measurement ofillumination
(light,
electromagnetic radiation)
Photometrieincludes
the
aspect of how lightappears to the humanobserver Radiometric units matched to human eye
Human average
eye
normed in trials of CIE(CommissionInternationale de
lEclairage)
-
8/3/2019 Folien SS2009 Machine Vision
23/163
23
Standard Observer
Fig.: CIE StandardObserver curve [B1.19]Fig.: Sensitivity of humancolour vision [Padgham
1975]
-
8/3/2019 Folien SS2009 Machine Vision
24/163
24
Day and Night Sight
Fig.: Spectral intensity function
(efficiency) of the human eye as
defined in CIE for night sight (1951) and sum daylight [B1.17]
-
8/3/2019 Folien SS2009 Machine Vision
25/163
25
Radiometric
und Photometric
Units
Lm Lumen
W Watt = J/s
Cd Candela
Sr Steradiant
= m2/m2
Lx Lux
Einheiten:
-
8/3/2019 Folien SS2009 Machine Vision
26/163
26
Radiant Flux Luminous Flux
(Strahlungsfluss)
Amount of radiation (photons) per second in Watt [W]
(Lichtstrom)
Amount of radiation perceived by standardobserver in Lumen [lm]
Fig.: Spectral sensitivity ofhuman eye defeined by CIE for
day light, 1931, 1964
-
8/3/2019 Folien SS2009 Machine Vision
27/163
27
Radiant Flux Luminous Flux
is
the
integral over
the entire spectrum S
weighs
the
radiant
flux according to the intensity function of thestandard
observer
eF
=0
)/( dFSF ee =750
380
)/(683 dFSF ev
vF
Example: Source
wit
radiant
flux
of 1W at a wave length
of 555nm
emits 683 LumenQuestion: How many Lumens emits an infra red 0.7 mW LED?
-
8/3/2019 Folien SS2009 Machine Vision
28/163
28
Radiant Inten. Luminous Intensity
(Strahlstrke)
Radiated power per unit of solid angle [W/sr]
(Lichtstrke)
Measured in the SIbasic unit Candela [cd]]/[1/1][1
222
mmrAsr =
Fig.: Definition of the solid angle orsteradians [sr] [B1.8]
Solid angle
(Raumwinkel) ofunit
sphere: 4
-
8/3/2019 Folien SS2009 Machine Vision
29/163
29
Irradiance Illuminance
(Bestrahlungsstrke)
Amount of radiation falling onto the innersurface of a unit sphere [W/m]
(Beleuchtungsstrke)
Visible illumination
Unit: Lux
[lux]
eE vE
Fig.: Definition of unitLux [B1.21]
-
8/3/2019 Folien SS2009 Machine Vision
30/163
30
Examples for IlluminanceLux Description
100,000 Sun light, noon32,000 Cloudy, noon
2,000 Cloudy, 1 hour after sun set
600 Illumination in supermarket450 Average office illumination
175 Street light, at night
10 Candel in 20 cm distance, pocket lamp0.3 Bright moon light, clear sky
-
8/3/2019 Folien SS2009 Machine Vision
31/163
31
Radiance Luminance
(Strahldichte)
Emitted intensity perunit area (e.g. from object) [W/m/sr]
(Leuchtdichte)
Response of humaneye or sensor toradiance [cd/m]
eL vL
Fig.: Definition of luminance [B1.24]
-
8/3/2019 Folien SS2009 Machine Vision
32/163
-
8/3/2019 Folien SS2009 Machine Vision
33/163
33
Fore-shortening
Fig.: Reduction of intensity [lux] withdistance and angle [B1.22].
Fig.: Effect of fore-shortening ishandled using the cosine
of tilt angle[H10.3].
-
8/3/2019 Folien SS2009 Machine Vision
34/163
34
Characteritics of Surfaces
Lambertian surface
Appears equally bright from all viewing directions
All incident
light is
reflected
Specular surface Mirror, all incident light is reflected Albedo Coefficent between 0 and 1 indicating how much
light a surface
reflects
relative to
ideal surface with no absorption Lambertian surface: albedo = .
-
8/3/2019 Folien SS2009 Machine Vision
35/163
35
Dichromatic Reflection Model Combines Lambertian and specular model
Fig.: Body and surface reflection.
Ee()
-
8/3/2019 Folien SS2009 Machine Vision
36/163
36
Reflection Models
Fig.: Summary of reflection models [Bajscy]
-
8/3/2019 Folien SS2009 Machine Vision
37/163
37
Dichromatic Reflection Model In general, radiance: Dichromatic model: Lambertian geometry function Gb independent of
viewing angle
Fig.: Actual, typical reflectivity pattern [B3.7]
)()( SEL ee =
bbeess GSGSS )(),()()( +=
-
8/3/2019 Folien SS2009 Machine Vision
38/163
38
Geometry of a Scene
Fig.: Angles relative toobject normal [H10.7].
Fig.: Definition of polarandazimuth angles [H10.6].
-
8/3/2019 Folien SS2009 Machine Vision
39/163
39
Image Formation
42
cos
4
=
f
DLE ee
Fig.: Relation between image irradianceEe and radiance of the surfaceLe tacking into account the size of regions in the image corresponding
to the patch on the object surface [H10.4].
Assumption:
is
small const.
Application:shape from shading shape of surface
Ee
Le .... radiance
f z
-
8/3/2019 Folien SS2009 Machine Vision
40/163
40
Orthographic Projection Normal- or parallel projection: x = X, y = Y Simplification when object is far away
-
8/3/2019 Folien SS2009 Machine Vision
41/163
41
Perspective Projection
Lochmodell (pin-hole model) einer Kamera
Fig.: Basic model of imaging process.
xz
fu =
yz
f
v =
Fig.: Pinhole model.
-
8/3/2019 Folien SS2009 Machine Vision
42/163
42
Pose of Camera in 3D Space Homogenous matrix:
ppRp += ic
=
10
pRAo
c
o
cAFig.: Camera and object coordinate system.
Pose estimation: possible from 3 (or more)points or lines of known object size
-
8/3/2019 Folien SS2009 Machine Vision
43/163
43
Lenses
Gaussian
lens
equations: and
Focal length:gbf
111+=
g
b
G
B=
BG
gBf
+=
Fig.: The lens law [DBS].
-
8/3/2019 Folien SS2009 Machine Vision
44/163
44
Field of View, FOV
small
focal
length wide FOV:
Depth of View Is obtained by Small aperture
Short focus length Large object distanceFig.: Field of view angle
[DBS].
=
f
B
2
atan2 max
-
8/3/2019 Folien SS2009 Machine Vision
45/163
45
Image Blur
Blur
circle
(Unschrfekreis)
d ... Aperture
opening
Power of alens
= 1/f [diopters]
babdu ='
Fig.: Diameter d of blur circle depending onaperture [DBS].
-
8/3/2019 Folien SS2009 Machine Vision
46/163
46
IlluminationDark
field
Bright field
diffuse: bell or
ring light
Through
light transparent object
Cast
schadow
Figs.: lighting options [DBS]
-
8/3/2019 Folien SS2009 Machine Vision
47/163
47
CCD Flchensensor
-
8/3/2019 Folien SS2009 Machine Vision
48/163
48
Farbkameras
3-Chip Kamera
1-Chip Kamera
Bayer
Farbmaske
[DCO]
[DCO]
-
8/3/2019 Folien SS2009 Machine Vision
49/163
49
Aliasing
[DCO]
Artefacte bei Farbbergngen Farb- und Intensittskante an verschiedenen Orten
Bildaufnahme bei
-
8/3/2019 Folien SS2009 Machine Vision
50/163
50
Bildaufnahme bei
Bewegung
Interlaced
Kamera: 2 Aufnahmen bei
je 25
(30) Hz
Abhilfe: Full-frame Kamera Ganzes Bild mit 25/50 Hz
odd field
even field
original
imageinterlaced image
of moving object
[DCO]
-
8/3/2019 Folien SS2009 Machine Vision
51/163
51
Images
Information in images: Intensity, Colour, multi-
spectral images, depth/distance/disparity, ...
Fig.: Intensity image (left) and range/depth image extracted fromthree intensity images [PointGrey]
-
8/3/2019 Folien SS2009 Machine Vision
52/163
52
Resolution, Image Pyramid
Fig.: Bildpyramide: Bilder wie oben nur mit gleicher Gre [G. Sandini]
Fig.: Bild bei verschiedenen Auflsungen (120x120, 60x60, 30x30)
-
8/3/2019 Folien SS2009 Machine Vision
53/163
53
Space-variant Images
Fig.: Images where resolution and field of view changes so that each imagecontains the same number of pixels
Fig.: Superimposing the above images [G. Sandini]
-
8/3/2019 Folien SS2009 Machine Vision
54/163
54
Log-polar Images, Fovea
Fig.: Log-polar pixel tessellation [G. Sandini]Fig.: Transformationof log-polar image
into a uniform,
rectangular pixeltesselation[G. Sandini]
[IBIDEM retina]
Advantage: similar tocharacteristics of human eye Efficient coding, e.g. video conferencing (64 kPixel)
-
8/3/2019 Folien SS2009 Machine Vision
55/163
55
Log-polar Images: ApplicationsFig.: Concentriccircles in log-polar
image project tovertical line, spokesto (nearly) horizontallines
Fig.: When focusingon the vanishing point
the radial lines (floorand ceiling to walls)project to horizontallines [Peters, Bishay,
1996].
-
8/3/2019 Folien SS2009 Machine Vision
56/163
56
Contents
Motivation: many projects Terminology: from computer to machine vision Components: from light to images Machine Vision Segmentation
Edge detection
Grouping
Application: XPERO
robot learning
-
8/3/2019 Folien SS2009 Machine Vision
57/163
57
Segmentation Partitioning the image into its constituent parts Constituent parts depend on the task Detect object, object class, foreground/background Stop whenobject of
interest is
found
-
8/3/2019 Folien SS2009 Machine Vision
58/163
58
Segmentation Partitioning the image into its constituent parts Constituent parts depend on the task Detect object, object class, foreground/background Stop whenobject of
interest is
found
-
8/3/2019 Folien SS2009 Machine Vision
59/163
59
Gaussian Convolution
2
22
22
1),(
yx
eyxG
+
=
Fig.: Original image =1, 5x5 =2, 9x9 =4, 15x15
Better smoothingthan mean
-
8/3/2019 Folien SS2009 Machine Vision
60/163
60
Salt and Pepper Noise
1% of Pixels either black or weight
Smoothing: outliers are spread but not eliminated
Fig.: Salt and Pepper Gauss filter=8, 5x5 Original image
-
8/3/2019 Folien SS2009 Machine Vision
61/163
61
Median Filter
Fig.: Median filter 3x3 Median filter 7x7 3x Median filter 3x3
Salt and Pepper (5%) Eliminates outliers (up to 50%)
-
8/3/2019 Folien SS2009 Machine Vision
62/163
62
Edge Detection 1D
Perfectedge
Perfectspreadedge
Edge withnoise
=1
Edge with noiseafter filtering
(Gauss)
Signal 1D
= Input
First
derivative= gradient
Second
derivative(Laplaceoperator)
-
8/3/2019 Folien SS2009 Machine Vision
63/163
63
Edge Detection 2D Fast: Roberts cross Strength:Angle:
Fig.: Original image Edge strength Binary image, threshold:80
,22
yx GGG += yx GGG +=4/3)/arctan( = xy GG xIGx /= yIGy /=
-
8/3/2019 Folien SS2009 Machine Vision
64/163
64
Edge Detection SobelAngle:
Fig.: Edge strength
)/arctan( xy GG=
Fig.: Original imageand after Sobel
Prewitt:
-
8/3/2019 Folien SS2009 Machine Vision
65/163
65
Edge Detection Canny Processing steps:
1. Gaussian smoothing2. Edge detection (e.g. Sobel)3. Ridge following with hystereses (t1 >t2
)
Optimised, standard method Good compromise Thin, 1 pixel edge (ridge) Smoothing can eliminate detail
Fig.:= 1, t1=255, t2=1
-
8/3/2019 Folien SS2009 Machine Vision
66/163
66
Canny - Examples Properties
Y-Effect: 3 edges meeting in a point are not connected Adaptive: detail and edge elements, but image dependent
Fig.:= 1, t1
=255, t2
=1 = 1, t1
=255, t2
=220 = 2, t1
=255, t2
=1
-
8/3/2019 Folien SS2009 Machine Vision
67/163
67
Laplacian of Gaussian - LoG Laplacian:
2
2
2
2
),(y
I
x
IyxL
+
=
2
22
22
22
4 21
1),(
yx
eyx
yxLoG
+
+=
Fig.: LoG or Mexican hat Example filter 9x9, =1.4
Fig.: Example of masks
-
8/3/2019 Folien SS2009 Machine Vision
68/163
68
LoG Example
Edges
are
zero
crossings
Ideal case: closed curves
Fig.: LoG with= 1 All zero crossings
-
8/3/2019 Folien SS2009 Machine Vision
69/163
69
LoG Edge Detection Derivative (differencing) amplifies noise ! Better results if scaled correctly
Fig.: zero crossings at= 2 = 1, strong zerocrossings (difference toneighbours >40)
Zero crossings at= 3
-
8/3/2019 Folien SS2009 Machine Vision
70/163
70
Contents
Motivation: many
projects
Terminology: from computer to machine vision Components: from light to images Machine Vision Segmentation
Edge detection
Region detection Grouping Application: XPERO robot learning
-
8/3/2019 Folien SS2009 Machine Vision
71/163
71
Histogram
Fig.: Original image t1=150 t1=130, t2=150 Regions (blobs)
Statistics over many pixel
Robust, fastIntensity
Numberofpixel
Intensity
t1
t1
t2
Numberofpixel
-
8/3/2019 Folien SS2009 Machine Vision
72/163
72
Thresholds - Examples
Fig.: Original image Histogram Binary image for t1=80 and t1=120
Fig.: Original image Histogram Binary image for t1=120
-
8/3/2019 Folien SS2009 Machine Vision
73/163
73
Finding Local Thresholds Statistics in neighborhood (e.g. 7x7) of a pixel
t = (mean value), or t = med (median) t = ( + med) / 2, or t = C (C .. constant)
Fig.: Original image Window 7x7, C=4 Window 140x140, C=8
-
8/3/2019 Folien SS2009 Machine Vision
74/163
74
Contents
Motivation: many
projects
Terminology: from computer to machine vision Components: from light to images Machine Vision Segmentation
Edge detection
Region detection Grouping Application: XPERO robot learning
Object Recognition Approaches
-
8/3/2019 Folien SS2009 Machine Vision
75/163
75
Object Recognition - Approaches Model based CAD Model of objects
Geometric features Finding features and their relationships Appearance based Interest points or entire object Gestalt Principles Regard structure in data and features
Perceptual grouping
Object Recognition Approaches
-
8/3/2019 Folien SS2009 Machine Vision
76/163
76
Object Recognition - Approaches Model based CAD Model of objects
Geometric features Finding features and their relationships Appearance based Interest points or entire object Gestalt Principles Regard structure in data and features
Perceptual grouping
-
8/3/2019 Folien SS2009 Machine Vision
77/163
77
Relation to the State of the Art Object Recognition = re-cognising a known object
Point features
and signatures
(e.g., SIFT [Lowe04])
Object categorisation = recognising objects belongingto a category
(bottle, animal)
Codebooks of features (blob size, colour, circle, e.g. [Pinz07,Skocaj07, Poggio07])
Object
detection
= find (relevant) entities
(to task)
Attention, grouping, figure-ground segmentation: task dependent methods (e.g., [Itti06] [Zillich07]) Perceived affordances = features related to actions Learned feature clusters, 2D laser data (e.g. [Kuipers06,07])
-
8/3/2019 Folien SS2009 Machine Vision
78/163
78
Perceptual Grouping Idea: exploit all the known structure in the data Learn what are objects (what they mean foroverall system), rather than specific objects Uses Gestalt principles
built-in knowledge
levels of abstraction
P t l G i
-
8/3/2019 Folien SS2009 Machine Vision
79/163
79
Perceptual Grouping
State of the Art
Recognition by components: A theory of
human image understanding [Biederman 1987,Dickinson, Bergevin, Biederman 1997]
Function depends on part relationship
36 geons
3D Object Recognition from single 2D images[Lowe 1987] Integration of regions and contours for objectrecognition [Schlter 2000]
-
8/3/2019 Folien SS2009 Machine Vision
80/163
80
A ComputationalStructure forPreattentive PerceptualOrganization[Sarkar 1994]
-
8/3/2019 Folien SS2009 Machine Vision
81/163
81
Principles of Perceptual Grouping
Perceptual organization
to form groupings andstructures in image
Proximity
Parallelism Collinearity Probabilistic ranking 13 principles [Wilson ]
[Lowe 87]
-
8/3/2019 Folien SS2009 Machine Vision
82/163
82
Model-based Recognition
[Lowe 87]
ModelImage Edges
Groupings Reprojections
-
8/3/2019 Folien SS2009 Machine Vision
83/163
83
Grouping, Objects and Semantics
Pixels
Object
Shortcut
Pixels
Object
Features
Proto Objects
Gestalts
Shortcut
Classical object recognition
Use
Vision(+phys.knowl.)
Vision(+model)
Cognitive system
-
8/3/2019 Folien SS2009 Machine Vision
84/163
84
Detection from Grouping Abstraction fromfeatures
to
Gestalts
Hierarchical grouping Learning is possiblefrom all features Problem:scaleability of feature search
Approach:
Incremental indexing [Zillich 2007]
-
8/3/2019 Folien SS2009 Machine Vision
85/163
85
Non-Accidentalness
Helmholtz
principle: non-accidental groupings
of features are perceived as Gestalts
The less likely randomness, the more likely an
underlying cause
Low likelihood of a grouping = highsignificance
-
8/3/2019 Folien SS2009 Machine Vision
86/163
86
Poisson Process Possible outcomes of a trial are 0 and 1 Trials are independent Expected number of occurrences in given
interval = Probability that there areexactly koccurrences
is
= Probability Mass/Distribution Function, discrete k
http://upload.wikimedia.org/wikipedia/commons/c/c1/Poisson_distribution_PMF.pnghttp://upload.wikimedia.org/wikipedia/commons/c/c1/Poisson_distribution_PMF.png -
8/3/2019 Folien SS2009 Machine Vision
87/163
87
Poisson Process in 2D
Randomly distributed points
-
8/3/2019 Folien SS2009 Machine Vision
88/163
88
Poisson Process in 2D
Adding some points
-
8/3/2019 Folien SS2009 Machine Vision
89/163
89
Poisson Process in 2D
How non-accidental are k points in the circle?
Interval: two dimensional or area
-
8/3/2019 Folien SS2009 Machine Vision
90/163
90
Ellipse Detection
Joint work
of
Michael Zillich and Jiri Matas, CMP, Prague Exploit local information where possible Take any edge segmented image, e.g. Canny Minimal number of parameters
-
8/3/2019 Folien SS2009 Machine Vision
91/163
-
8/3/2019 Folien SS2009 Machine Vision
92/163
92
Approach1. Connected edgels
2. Local shape: fit circular arcs3.
Geometric constraints: convexity
4. Fit Ellipses
Canny Edges
-
8/3/2019 Folien SS2009 Machine Vision
93/163
93
Canny Edges
Canny Edges
-
8/3/2019 Folien SS2009 Machine Vision
94/163
94
Canny Edges
-
8/3/2019 Folien SS2009 Machine Vision
95/163
95
Approach1.
Connected edgels
2. Local shape: fit circular arcs
3.
Geometric constraints: convexity
4. Fit Ellipses
-
8/3/2019 Folien SS2009 Machine Vision
96/163
96
ArcsComparing three methods
SPLIT
GROW
RANSAC
-
8/3/2019 Folien SS2009 Machine Vision
97/163
97
SPLIT (Rosin, West 1995)
-
8/3/2019 Folien SS2009 Machine Vision
98/163
98
SPLIT
Problem:
When to stopsplitting?
-
8/3/2019 Folien SS2009 Machine Vision
99/163
99
GROW
1. Locally fit cirle2.
Grow
when
radius
similar
-
8/3/2019 Folien SS2009 Machine Vision
100/163
100
GROW
Problem:
Sensitive to end
points
-
8/3/2019 Folien SS2009 Machine Vision
101/163
101
RANSAC
1.
Grow
with
random
seeds
2. Take MDL optimum
-
8/3/2019 Folien SS2009 Machine Vision
102/163
102
Approach1.
Connected edgels
2. Local shape: fit circular arcs3. Geometric constraints: convexity
4. Fit Ellipses
-
8/3/2019 Folien SS2009 Machine Vision
103/163
103
Convex Arcs
convex non-convex
-
8/3/2019 Folien SS2009 Machine Vision
104/163
104
ConvexityInner side
of arc
Beyond endpoint of arc
-
8/3/2019 Folien SS2009 Machine Vision
105/163
105
Image Space VotingVote = Intersection of arc tangents/normals
-
8/3/2019 Folien SS2009 Machine Vision
106/163
106
Voted Arcs
Limited length Unlimited length
Depends strongly on length of tangent/normal
-
8/3/2019 Folien SS2009 Machine Vision
107/163
107
Run Time Complexity
Testing all pairs: O(n2
) Image space voting: O(n)
-
8/3/2019 Folien SS2009 Machine Vision
108/163
108
Growing Groups
Extend group of arcs to form ellipse
hypothesis
Exhaustive search Greedy search, heuristics: Co-curvilinearity
Ellipse centers Fit quality
-
8/3/2019 Folien SS2009 Machine Vision
109/163
109
Exhaustive vs. Greedy
17 groups 1 group
Greedy Search Heuristics
-
8/3/2019 Folien SS2009 Machine Vision
110/163
110
Greedy Search Heuristics
Co-curvilinearity
Yuen et. al.,1989:3 points on ellipse (P, Q, R)+ tangents
centre (C)
Ellipse fit, relative support
-
8/3/2019 Folien SS2009 Machine Vision
111/163
111
True Positive Rate (TPR)
-
8/3/2019 Folien SS2009 Machine Vision
112/163
112
Runtime
-
8/3/2019 Folien SS2009 Machine Vision
113/163
113
Approach1.
Connected edgels
2. Local shape: fit circular arcs3.
Geometric constraints: convexity
4. Fit Ellipses
-
B2AC [Fitzgibbon, Pilu, Fisher 1999], OpenCV
All Ellipses
-
8/3/2019 Folien SS2009 Machine Vision
114/163
114
All Ellipses
Good Ellipses
-
8/3/2019 Folien SS2009 Machine Vision
115/163
115
Good Ellipses
-
8/3/2019 Folien SS2009 Machine Vision
116/163
116
-
8/3/2019 Folien SS2009 Machine Vision
117/163
117
TPR vs. Runtime
Office scene in different sizes320x240 - 1280x960
-
8/3/2019 Folien SS2009 Machine Vision
118/163
118
More Structure
Lines, arcs, ellipses, parallelities,
continuations, junctions, ... higher level shapes, object, symbolic?
Vocabulary sentences (Prolog) C Perceptual grouping + symbolic reasoning
-
8/3/2019 Folien SS2009 Machine Vision
119/163
119
Hierarchical Perceptual Grouping1.
Connected edgels
2. Local shape: fit lines or circular arcs3.
Hierarchical
grouping
L- and T-junctions Collinear lines, parallel lines Ellipses Rectangle, flaps; cylinder
-
8/3/2019 Folien SS2009 Machine Vision
120/163
120
Detection from Grouping Abstraction fromfeatures
to
Gestalts
Hierarchical grouping Learning is possiblefrom all features Problem:scaleability of feature search
Approach:
Incremental indexing [Zillich 2007]
Rectangle
-
8/3/2019 Folien SS2009 Machine Vision
121/163
121
Rectangle
rect(A,B,C,D) :-
u(A,B,C), u(C,D,A)
line(A,B) :- left(A,X), right(B,X).rect(A,B,C,D) :- line(A,B), line(B,C), line(C,D), line(D,A)
-
8/3/2019 Folien SS2009 Machine Vision
122/163
122
Flap
-
8/3/2019 Folien SS2009 Machine Vision
123/163
123
Book Scene
3847 edges
First 2 rectangles All rectangles
Original image
-
8/3/2019 Folien SS2009 Machine Vision
124/163
124
Office Scene
-
8/3/2019 Folien SS2009 Machine Vision
125/163
125
Shelves scene
-
8/3/2019 Folien SS2009 Machine Vision
126/163
126
Cylindercyl(E,A,B) :-
tangent_l(E,A), tangent_r(E,B), parallel(A,B)
-
8/3/2019 Folien SS2009 Machine Vision
127/163
127
Cup Scene
Original image Edges First cylinder
-
8/3/2019 Folien SS2009 Machine Vision
128/163
128
Kitchen Scene
Original image First 3 cylinders
-
8/3/2019 Folien SS2009 Machine Vision
129/163
129
Illusion Kanizsa Square
-
8/3/2019 Folien SS2009 Machine Vision
130/163
130
Problem
Fig.: fast detection difficult, longer time detection
http://c/thesis/rectangles/incvote.sh -
8/3/2019 Folien SS2009 Machine Vision
131/163
131
Grouping and Incremental Indexing Incremental growing of search lines given aprocessing-time Image as index reduction of search space to imagespace Incremental detection of Gestalts (e.g., closures,proto-object)
Searchlines after 100ms Searchlines after 500ms
ExamplesExamples forfor ClosuresClosures andand CubesCubes
-
8/3/2019 Folien SS2009 Machine Vision
132/163
132
Results ofincremental indexing Specify onlyprocessing-time No thresholds Closures detected after 160, 240 and 420 ms
Cubesdetected
from
Eddy
-
8/3/2019 Folien SS2009 Machine Vision
133/163
133
VS2 Example
Vision System 2
tool
for
grouping
-
8/3/2019 Folien SS2009 Machine Vision
134/163
134
Finding Closures
Closed
Polygon
Any size Smaller ones
better Better ratio ofedgel
support
over perimeter
Ontology: Adjacency and Rule
-
8/3/2019 Folien SS2009 Machine Vision
135/163
135
Base
Adjacency map from segments/closures polygon relationships, e.g.,
2 (blue) is_StablySupported_by
1 (violet),
[barycenter
within x-limits]
3 (green) is_InstablySupported_by
2 (blue),
[barycenter too much right]4 (brown) is_right_of 1 (violet),...
Relationships
are
input
to reasoning engine
Finds concepts according to rules
2 rules
for
stack, one
for
arch, one
for
transitivity
Processing time typically less than a second(Java)
1
2
3
4
RealReal--WorldWorld EExamplexample
-
8/3/2019 Folien SS2009 Machine Vision
136/163
136
Segmentation
Some arches found:
Best from functionalp.o.v.Correct, yet irrelevant
from functionalp.o.v.
-
8/3/2019 Folien SS2009 Machine Vision
137/163
137
Scenario - Exit Test on door in office
378 segments, 86 closures,
35 stacks
Example
of two
columns:
Original image closures best arch window arch
Exit in XPERO Setup
-
8/3/2019 Folien SS2009 Machine Vision
138/163
138
p
Onotology operates
on
segments andclosures
Results
comparable
Exit
-
arch
Closing the loop Arch as relevant entity Learn object concepts from interaction
Occlusion Handling and Depth
-
8/3/2019 Folien SS2009 Machine Vision
139/163
139
Ordering
Perceptual Grouping Clustering based onGestalt principles
Improved Tracking
Interest point based Lucas-Kanade algorithm
Delaunay
representation
Local affine assumption
Spatial Reasoning
Occlusion Detection Occlusion Reasonin
Occlusion Handling
-
8/3/2019 Folien SS2009 Machine Vision
140/163
140
g
Only closures
T-junctions
indicate occlusion Occlusion indicates
depth ordering of closures More occlusion more closures ordered
Future work
Exploitation of motion Estimate plane normals
-
8/3/2019 Folien SS2009 Machine Vision
141/163
141
Conclusion Same method to detect different shapes Exploit local information (smoothness, ...) Avoid local decisions and early pruning ofhypotheses (thresholds) Only threshold: good ellipses Use ranking
Perceptual grouping = well-defined pruning ofhypotheses reduce search space Image Space Indexing: O(n2) O(n)
-
8/3/2019 Folien SS2009 Machine Vision
142/163
142
Further Work Difficult to find a good local quality measure Global measure: combinatorial searchproblem Interaction with scene,mobile observer:
false positives vs. stable percepts
Used to initialise tracking
^
http://c/tmp/office_movie.shhttp://c/tmp/office_movie.sh -
8/3/2019 Folien SS2009 Machine Vision
143/163
143
Detection and Tracking
143Edges
-
8/3/2019 Folien SS2009 Machine Vision
144/163
144144
Arcs
Detection and Tracking
-
8/3/2019 Folien SS2009 Machine Vision
145/163
145145
Detection
Detection and Tracking
-
8/3/2019 Folien SS2009 Machine Vision
146/163
146146Tracking
Detection and Tracking
-
8/3/2019 Folien SS2009 Machine Vision
147/163
147
Detection and Tracking
-
8/3/2019 Folien SS2009 Machine Vision
148/163
148
Contents
Motivation: many
projects
Terminology: from computer to machine vision
Components: from
light to images
Machine Vision Segmentation
Edge detection
Region detection Grouping Application: XPERO robot learning
MatchObservations and
BRS
Execute Observe
BRS
TUWUVR
Hypothesisformation
AUPBRS
AUP
UVR
BRS
TUW
ntation
-
8/3/2019 Folien SS2009 Machine Vision
149/163
149
Innateknowledge
Evaluation of
experiment(s)
Design of
experiment(s)
Learning/Gaining Insights/
KnowledgeBase
A priori/existingknowledge
TUW UVR
ULJ
intelligent data collection/
experimental feedback(may not involve the generation
of new hypotheses)
Predictions
Educated Guess(time series analysis,reasoning by analogy)
feedback loopon the
actualexecution
of anexperiment
Observation of
experiment(s)
Planning andexecution ofexperiment(s)
TUW UVR
Plan Predict
UVR BRS
by analogy
ULJAUP
BRSULJ
BRSUVR
ULJTUW
TUW
UVR
BRSAUP
Anarchitecturefor
BRS
BRSUVR
BRSULJ
UVR
UVR
AUPULJ
ULJ
ULJ
UVR
AUP
BRSAUP
ULJTUW
TUW
TUW
TUW
UVR
learnin
gbyex
perimen
intelligent datacollection: reportlocation of robot
and object(s)
Showcase
-
8/3/2019 Folien SS2009 Machine Vision
150/163
150
Robot needs to know Where am I and Where are relevantentities (objects)
Robot pose
Egomotion relativeto environment Odometry, vision
Object pose Basic features todescribe
objects
Blobs: centre, size Sha e: cube, ...
Goal: observe object and agents
-
8/3/2019 Folien SS2009 Machine Vision
151/163
151
j
g
Robot observes experiments Start: overhead camera Goal: robot view 3rd 1st person
perspective Blobs shapes
Object Pose: Proto-object
-
8/3/2019 Folien SS2009 Machine Vision
152/163
152
Detection
Objective: detect
entities
relevant to experiment
Goal: abstract features to groups or objects Attributes such as size, location, shape Detection from grouping (VS3) Incremental indexing
Closures, Cube, cylinder, cone
Occlusion handling Tracking and depth ordering Ontology to label
Tracking and Grouping
-
8/3/2019 Folien SS2009 Machine Vision
153/163
153
Tracking and Grouping
Hypothesis
generation
from
tracked
Gestalts
Tracking and Grouping
-
8/3/2019 Folien SS2009 Machine Vision
154/163
154
ac g a d G oup g
Combining
infor-
mation over time Tracking through
degenerate views
Detecting Proto-Objects
-
8/3/2019 Folien SS2009 Machine Vision
155/163
155
Cube, Cylinder and Cone
Rule based grouping of Gestalts to obtain
hypotheses
Masking of proto-objects by the use of geometricfeatures from the hypothesis
(a) edge image
(b) closures (c) proto-objects
Detecting proto-objects from an edge image
XPERO - Ontology to Label Objects
-
8/3/2019 Folien SS2009 Machine Vision
156/163
156
Tackling last gap betweenfeatures (proto-objects) andsemantically meaningful(e.g.,in the sense of function)objects
Affords thoughts on thedefining substrate of objectconcepts spatial configuration of parts
Needs abstraction techniques arbitrary parts can take a roleina concept (e.g., being thesupport of another part)
Conclusion Machine Vision
-
8/3/2019 Folien SS2009 Machine Vision
157/163
157
System approach to solve a task (Passive) understanding scene is too hard Solving a particular task in a scenario is much easierto resolve Understanding features is under way E.g., structure, grouping, interest points, patches Segmentation in itself is impossible it depends on task Cameras cheap but processing difficult
Machine Vision Lessons Learned
-
8/3/2019 Folien SS2009 Machine Vision
158/163
158
Design vision system for task(s)
50% of solution
is
good illumination
Move out into the real world
Databases
and videos
are
nice
But reality is richer Avoid thresholds robustness is parameter-free A lot can be done with a camera and a PC
EU Project robots@home
-
8/3/2019 Folien SS2009 Machine Vision
159/163
159
j @
Mile stones 2007: Start 2008: Learn one room, touch screen 2009: Learn entire apartment 2010: Navigate in 4 apartments
Classify objects according to function (table, sofa, ...)
Using
stereo and
Time-of-flight cameras
Recognising Furniture
-
8/3/2019 Folien SS2009 Machine Vision
160/163
160
g g Using vanishing point and 3D Group using Gestalt principles surfaces
Some Detected Furniture
-
8/3/2019 Folien SS2009 Machine Vision
161/163
161
Thesis Topics
-
8/3/2019 Folien SS2009 Machine Vision
162/163
162
p
Ontology
for
furniture
Any-door detector and trying it in many homes
Shape
and shade
(Make3D)
Poke object and predict what will happen
Thank you
-
8/3/2019 Folien SS2009 Machine Vision
163/163
y