illumination independent marker tracking using cross-ratio invariance
DESCRIPTION
VRST2014 presentation Marker tracking is used in numerous applications. Depending on the context and its constraints, tracking accuracy can be a crucial component of the application. In this paper, we firstly highlight that the tracking accuracy depends on the illumination, which is usually not controlled in most applications. Particularly, we show how corner detection can shift of several pixels when light power or background context change, even if the camera and the marker are static in the scene. Then, we propose a method, based on the cross ratio invariance, that allows to re-estimate the corner extraction so that the cross ratio of the marker model corresponds to the one computed from the extracted corners in the image. Finally, we show on real data that our approach improves the tracking accuracy, particularly along the camera depth axis, up to several millimeters, depending on the marker depth.TRANSCRIPT
Illumination Independent Marker Tracking using
Cross-Ratio Invariance
Vincent Agnus Stéphane Nicolau Luc Soler
Institut de Recherche Contre les Cancers de l'Appareil Digestif
ACM Symposium on Virtual Reality Software and Technology(VRST2014)
November 11, 2014
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 1 / 18
Outline
Causes & E�ects of light condition on corner detection
How cross-ratio can be used to correct corner shifting
Method & validation
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 2 / 18
Motivation
Corner Detection : Illumination ConditionsExample
Light conditions in�uence the estimation of corner position
Outcomes : Errors
Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18
Motivation
Corner Detection : Illumination ConditionsExample
Light conditions in�uence the estimation of corner position
Outcomes : Errors
Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18
Motivation
Corner Detection : Illumination ConditionsExample
Light conditions in�uence the estimation of corner position
Outcomes : Errors
Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18
Motivation
Corner Detection : Illumination ConditionsExample
Light conditions in�uence the estimation of corner position
Outcomes : Errors
Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18
Motivation
Corner Detection : Illumination ConditionsExample
Light conditions in�uence the estimation of corner position
Outcomes : Errors
Image : 2 pixels ⇒ World : 3 mm depth ⇒ medical applications
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 3 / 18
Motivation
Medical applications using markers
A low cost and accurate guidance system for
laparoscopic surgery : Validation on an
abdominal phantom [Nicolau VRST2005]
A low cost simulator to practice ultrasound
image interpretation and probe manipulation :
Design and �rst evaluation [Nicolau ISVRI2011]
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 4 / 18
Motivation
Medical applications using markers
A low cost and accurate guidance system for
laparoscopic surgery : Validation on an
abdominal phantom [Nicolau VRST2005]
A low cost simulator to practice ultrasound
image interpretation and probe manipulation :
Design and �rst evaluation [Nicolau ISVRI2011]
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 4 / 18
Error on tag position
⇒ Larger position errors for :
Needle tip
UltraSound virtual plane
Motivation
Corner DetectionShift under light conditions
Shifting Causes : Mitsunaga and Nayar 1999
non-linear mapping between scene radiance and measured brightness
measured brightness rasterisation (spatial and intensity) on CDD
White
Black
Scene Radiance
state of the art
Studies : marker detection , shifting ⇒ Biased images
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18
Motivation
Corner DetectionShift under light conditions
Shifting Causes : Mitsunaga and Nayar 1999
non-linear mapping between scene radiance and measured brightness
measured brightness rasterisation (spatial and intensity) on CDD
White
Black
Scene Radiance Measured Brightness
RadiometricResponse
state of the art
Studies : marker detection , shifting ⇒ Biased images
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18
Motivation
Corner DetectionShift under light conditions
Shifting Causes : Mitsunaga and Nayar 1999
non-linear mapping between scene radiance and measured brightness
measured brightness rasterisation (spatial and intensity) on CDD
White
Black
Scene Radiance Measured Brightness
RadiometricResponse
state of the art
Studies : marker detection , shifting ⇒ Biased images
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18
Motivation
Corner DetectionShift under light conditions
Shifting Causes : Mitsunaga and Nayar 1999
non-linear mapping between scene radiance and measured brightness
measured brightness rasterisation (spatial and intensity) on CDD
White
Black
Scene Radiance Measured Brightness
Rasterisation Spatial & IntensityRadiometric
Response
state of the art
Studies : marker detection , shifting ⇒ Biased images
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 5 / 18
Proposed Method
Key Idea
Light Condition
underexposure : black region ↗, white region ↘overexposure : black region ↘, white region ↗
⇒ error on estimated position of the marker
How we can detect and correct this phenomena ?
Our method
use the marker geometry : inner & outer corners
check that white/black region proportion are preserved
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 6 / 18
Proposed Method
Corner Detection ShiftDetection & Correction
Cross Ratio (CR)
A projective invariant feature de�nedfor collinear points sets
A B C D
A′ B′ C ′ D′
CR =(AB/AC)
(DB/DC)=
(A′B′/A′C ′)
(D′B′/D′C ′)
Cross Ratio on markers
ABC
D
A BC D
CR should be constant :position & light
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 7 / 18
Proposed Method
Corner Detection ShiftDetection & Correction
Cross Ratio (CR)
A projective invariant feature de�nedfor collinear points sets
A B C D
A′ B′ C ′ D′
CR =(AB/AC)
(DB/DC)=
(A′B′/A′C ′)
(D′B′/D′C ′)
Cross Ratio on markers
ABC
D
A BC D
CR should be constant :position & light
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 7 / 18
Proposed Method
Theoretical Cross Ratio vs Measured Cross Ratio
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18
Proposed Method
Theoretical Cross Ratio vs Measured Cross Ratio
CRmodel
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18
Proposed Method
Theoretical Cross Ratio vs Measured Cross Ratio
CRmodel
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18
Proposed Method
Theoretical Cross Ratio vs Measured Cross Ratio
CRmodel
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 8 / 18
Proposed Method
Cross Ratio : selected points
Cross Ratio : Our Study
A BC D
4+4+2 lines for CR
Cross Ratio : inner corners of themarker (future)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 9 / 18
Proposed Method
Cross Ratio : selected points
Cross Ratio : Our Study
A BC D
4+4+2 lines for CR
Cross Ratio : inner corners of themarker (future)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 9 / 18
Proposed Method
Cross Ratio : selected points
Cross Ratio : Our Study
A BC D
4+4+2 lines for CR
Cross Ratio : inner corners of themarker (future)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 9 / 18
Proposed Method
Correction of corner shifts
How cross ratio invariance can be used to �x corners shifting ?
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 10 / 18
Proposed Method
Correction of corner shifts
Idea
Displace the corners such that CR(displaced) = CRmodel
Method
H
(+dx,+dy)
(+dx,−dy)
p3
p1 p5 p6p2
p7p14
p15
p4p9
p12 p11 p16
p8p10p13
p′1
H(p1)
p′i = H(pi) + (±dx,±dy)
(dx, dy) = argmindx,dy
∑(a,b,c,d)∈I
(CR(p′a, p′b, p′c, p′d)− CRmodel)
2
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 11 / 18
Proposed Method
Correction of corner shifts
Idea
Displace the corners such that CR(displaced) = CRmodel
Method
H
(+dx,+dy)
(+dx,−dy)
p3
p1 p5 p6p2
p7p14
p15
p4p9
p12 p11 p16
p8p10p13
p′1
H(p1)
p′i = H(pi) + (±dx,±dy)
(dx, dy) = argmindx,dy
∑(a,b,c,d)∈I
(CR(p′a, p′b, p′c, p′d)− CRmodel)
2
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 11 / 18
Proposed Method
Correction of corner shifts
New Corner Positions
H(+dx,+dy)
(+dx,−dy)
p3
p1 p5 p6p2
p7p14
p15
p4p9
p12 p11p16
p8p10p13
p′1H(p1)
correct(pi) = H−1(H(pi) + (±dx,±dy)︸ ︷︷ ︸
p′i
)
corner in original image corrected cornerVincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 12 / 18
Proposed Method
Optimization Process
Minimization
argmindx,dy
∑(a,b,c,d)∈I
(CR(p′a, p′b, p′c, p′d)− CRmodel)
2
Levenberg-Marquardt algorithm
not optimal : jacobian not provided, estimated from aforward-di�erence approximation
5 ms for each image : SciPy , (Core [email protected]), 10 lines for CR
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 13 / 18
Validation
ValidationProtocol
Metric
Distance camera ↔ Marker (before/after correction)vs a reference distance (chessboard)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18
Validation
ValidationProtocol
Metric
Distance camera ↔ Marker (before/after correction)vs a reference distance (chessboard)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18
Validation
ValidationProtocol
Metric
Distance camera ↔ Marker (before/after correction)vs a reference distance (chessboard)
Pi
O
Πcheck
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18
Validation
ValidationProtocol
Metric
Distance camera ↔ Marker (before/after correction)vs a reference distance (chessboard)
Pi
P proji
O
Πcheck
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 14 / 18
Validation
ValidationExperiments
Chessboarddistance
cross ratio
Before correction
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18
Validation
ValidationExperiments
cross ratio
After correction
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18
Validation
ValidationExperiments
After correction
distance
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18
Validation
ValidationExperiments
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 15 / 18
Logitech c© HD ProWebcam C9101920×1080
Microsoft c© Lifecam 1280×720 Logitech c© Procam 640×480
Validation
ValidationExperiments
3 cameras × 5 orientations (range 40-60cm) × 8 light conditions3 groups : illumination condition.over-exposure generate more errors.
without correction with correction
Under-exposure 1.15 ± 0.22 mm 0.42 ± 0.19 mmNormal exposure 0.90 ± 0.32 mm 0.67 ± 0.22 mmOver-exposure 2.60 ± 0.70 mm 0.71 ± 0.25 mm
Table: average errors of the marker depth estimation
Cross Ratio :
Can detect extreme light condition
Can correct depth estimation (<1 mm)
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 16 / 18
Discussion
Discussion & Perspectives
Discussion
estimated homography H → corner correction
assumption detected corners are collinear
assumption of isotropic in�uence of light after homographicrecti�cation
Future Works
use inner corners of markers
minimization : analytical formulation ?
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 17 / 18
Q & A
Thank you for your attention
Vincent Agnus (IRCAD) Illumination Independent Marker Tracking using Cross-Ratio Invariance 18 / 18