compsci 773 - university of auckland · march 2, 2010 compsci 773 s1t 3 compsci 773 • the report...

1

COMPSCI 773• Lecturers: Dr Patrice Delmas (303.389) Lecturers: Dr Patrice Delmas (303.389)

A. Prof. Georgy Gimel’farb (303.391) • Lecture time: W 1pm-2pm F 12pm-2pm• Marking:

– 40% Final exam – 60% Lab work (30% group work, 20% individual assessment, 10%

oral assessment)oral assessment)• 3 assignments

March 2, 2010 1COMPSCI 773 S1T

COMPSCI 7731st assignment: camera calibration synchronisation of USB 1 assignment: camera calibration, synchronisation of USB

cameras for image acquisition, image pairs rectifications –03.04.09

2nd assignment: image matching / stereo matching algorithms implementation, depth map construction, 3D visualisation (OpenGL/CUDA, optional) – 08.05.09

3rd assignment (06.06.09):– 2+3D Face mapping/recognition (tentative)– Whole system testing (live demo)


2

COMPSCI 773• Individual assessment: (either or both)Individual assessment: (either or both)

• Oral interview (with Patrice)• Research paper presentation during final 3 lectures (time permitting)• Overall involvement in group work

• Assignment reports (what’s in):• A group report:

• Who did whatWho did what• What solution has been chosen and why

• An individual report• Detailing parts each student did• Presenting OWN solution and results (if any)


COMPSCI 773• The report should look like a research report with referencesThe report should look like a research report with references

– Justified explanations on the chosen solutions, graphs, resultsand – Critical assessment of the outcomes

• Programming• Windows C, C++ • OpenGl, CUDA, Gtkp• You are allowed to use external libraries but you have to state it• You may be asked to pass your code to other groups for the next

assignment• If you use another group’s codes you MUST state it in your report


3

The project: Advanced biometrics: 2D/3D mapping/face recognition

• Part 1:– Camera calibration– Image acquisition, camera synchronisation (GUI)– Calibration assessment– Rectification

• Part 2:– Improve calibration accuracy/rectification– Matching/stereo matching– Camera synchronisationCamera synchronisation– Database acquisition

• Part 3: Face authentication– Goal: Identify faces from images using 2/3D data– 2D/3D Statistical analysis– Live face recognition

• Each group will have to do BOTH partsMarch 2, 2010 5COMPSCI 773 S1T

What is available and what you will have to do

US: 3D USB 2 0 web-camerasUS: 3D USB 2.0 web cameras• Calibration object• 3D scanner• PCs-Windows

YOU: A calibration object (if you do not like ours)• Code in C, C++ (avoid Java) real-time• Create a proper GUI integrating the different parts of the project• Setup drivers if necessarySetup drivers if necessary• A very strong team/personal effort

OUTCOME• You will undertake work at the top-edge of today’s research• You will gain a unique experience of Applied Computer Vision


4

What is expectedProjects at the edge of research development: neither easy nor simpleProjects at the edge of research development: neither easy nor simple• A great deal of work is required but: (1) you will learn a lot; (2) this

could count as work experience in an expending IT area; (3) you can show what you are worth without having to fear too much about the final exam

• Still 2 students failed because of exam in 2005, none since but some ended up with B-/C+ final grade

• The exam will encompass all that will be lectured + project-related questions• It is not a good idea to concentrate on a very restricted part of the project as

this will penalise you by the end• If you like the projects you can continue towards the same directions for

COMPSCI 780 / MSc studies• If you think COMPSCI 773 is too hard for you we can offer a COMPSCI 780

project along the same directions


What you should know or learn very quickly

• C C++ programming• C, C++ programming• Confidence in mathematical skills (linear algebra)• Basics of Image Processing• To learn quickly (by yourself)• GUI design (just the basics)• A bit of CUDA for matching computation• Camera control and synchronization• The rest we will teach you…March 2, 2010 8COMPSCI 773 S1T

5

Course contents• Introductory lecture • Image matching

• Real-time image processing• Image matching

• Stereo image matching• 2D and 3D vision

geometry• Camera calibration• Stereo calibration

processing• Rectification of stereo

pairs• Colour discrimination • Features classification:

PCA-LDA• Segmenting binary images• Feature extraction• 3D scene description /

understanding

• Higher level statisticalapproaches

• Course overview and final demo

March 2, 2010 COMPSCI 773 S1T 9

Project details: Part 1• Acquire synchronised still streams from 3D webcams• Acquire synchronised still streams from 3D webcams

– Acquisition of face images• Calibrate cameras• Triangulate corresponding points to obtain their 3D

positions• Rectification• Demo


6

Project details: Part 2• Improve calibration/rectification accuracy• Matching / stereo matching

– Implement a simple window-based correlation-based matching algorithm

• Acquire face database with 3D webcam• Extract faces from images• Normalize database images• Analyse 2D database images• Compare new faces versus faces within databasep• Create a GUI to interact with database• Perform a real-time demo


Project details: Part 3• Matching / stereo matchingMatching / stereo matching

– Implement a complex matching algorithm• Analyse 3D database images• Compare new faces versus faces within database• Repeat with 2+3D data• Perform a real-time demo• Write a conference submission like report• Write a conference submission-like report


7

A Possible Tentative FrameworkRGB.ppm or video stream

Morphological filtering

Hand colour segmentation

Clustering -labelling

Hand normalisation

PCAei ei+

1

... en

w1 w2 ... wnInput pose

e e e

Trained informationRepresentative weightings


Classificationei ei+1 ... en

w11 w21 ... wn1

w12 w22 ... wn2

w13 w23 ... wn3

w14 w24 ... wn4

p g g

Class 1: OpenClass 2: GoClass 3: StopClass 4: Left

Mean image

Classified pose: Class 1

Questions ?• First thing firstFirst thing first

– Get access to the 773 open space-level 3 (one swipe card per group)– Create groups

• Next– I will do the first 3 weeks of the semester; Georgy - the 6 weeks following,

Patrice will do the last 3 weeks– I am very keen to help and answer questions:

• Better ask from Wednesday after lectures to Friday• Better ask from Wednesday after lectures to Friday• This is a research: I am keen to learn from you!• My advice: read research articles, respect assignments requirements,

but feel very welcome to explore alternative solutions


CD1 celine, 23/02/2009

8

3D vision at CS3D vision at CS--UoAUoA

January 05, 2010 Intelligent Vision Systems: Dept. of Computer Science

15

People– Dr. P. Delmas, A/Prof. G. Gimel’farb– 6 PhDs and 4 MSc students– 2 postgraduate courses 773, 760

Di it l h t t & t ti l t• Digital photogrammetry & computational stereo– Camera calibration; forming epipolar stereo pairs– Fast 3D reconstruction: our own algorithms (faster, more

accurate)– symmetric dynamic programming stereo (SDPS);

symmetric concurrent stereo matching (SCSM)– Interactive editing of 3D models to form 3D surfaces – 3D face reconstruction and animation, 2/3D face

recognition– RT 3D vision (from acquisition to features then to fast

prototyping and modelling)


9

3D vision Lab (city campus)• New Lab located on the 5th floor CS building

– Lab supervisor: PhD candidate A. GastelumF d d b CS F S t ff h f d FRST (300k NZ$)– Funded by CS, FoS staff research fund, FRST (300k NZ$)

• 3D acquisition capabilities:– 3D scanner (sub-mm accuracy on a 0.5m3 volume)– 3D stereo vision (active, passive, still or video up to 80 fps)

• Synchronised USB video cameras (80fps)• Synchronised Firewire video cameras (30fps)

– 3D display panel (Nov 09)November 30 2009• Projects

– 3D face animation (RT acquisition, 3d face features extraction, automatic prototyping, High res multi-layer SPH model)

– Real-time 3D face features– Stereo algorithms benchmarking– 3D characterisation of solids deformation

November 30, 2009


Photogrammetry Laboratory(2)

• Optical ‘Range’.Integrated– Integrated.

– Multiple systems view a common scene.

• Stereo bench.– Sideways for face

capture!• Projector for structured

Example benchmark Data:

3D scanner

• Projector for structured lighting.

• Commercial 3D scanner.• Solutionix Rexcan 400.

Depth map Perspective visualisation


10

3D Shape Reconstructionfrom stereo images (SDPS)

Binocular stereo


2D face feature extraction (1996-2002)

(Geodesic) Active Contours for face feature extraction

C. Petitjean

With M. LievinMarch 2, 2010 COMPSCI 773 S1T 20

11

3D faceRendering using 2D images (2004)Idea: map 3d skin texture to CT-scan skull surfaceApplication: dental scanning

+3D model

3D face

2D image

Mapping program


Automatic adjustment of the 3D generic model to patient features

AAM (active appearance models)AAM (active appearance models) • Set of points describing face features• Generic point location trained on a database of faces:

• Using Gabor jets (bunch of filters of varying amplitude, frequency and orientation) computed on manually selected face features

• Node points respective locationNode points respective location• Combines shape and texture in a PCA space• Searches a new image iteratively by using the texture error (pixel

wise) to drive the model parameters• Converge well if initialize close to solution


12

AAM results (automatic extraction of face features)

Interface between Raw Data and Generic Model (2005)

2 modes: semi-automatic, automaticAutomatic: AAM on colour image, correspondences to 3D values on depth map, mapping of fiducial points

Model with

p p, pp g pto animated 3D model

animation system

Correspondences made and mapped via RBF with a final nearest point map and texture projection

Depth map

Results in a custom face with animation system in place


13

Static face reconstruction Static face reconstruction evaluation (2004)evaluation (2004)

• Evaluated approaches:– Active/passive binocular stereo:

Active structured lightingActive structured lighting– Active photometric stereo

• 12 algorithms in total. Ground truth data: 3D scanner• Aims:

– Evaluate effectiveness• Accuracy, time complexity

– Database of faces.

25

• Alternative test set• Focus on stereo vision algorithms

– Determine best approach for dynamic 3D reconstruction system


Images

Stereo pair Stereo pair withcolour code projection


Ground truth

14

Statistical resultsStatistical resultsBP BP +

GradBP + Strip

DPM DPM + Grad

DPM + Strip

CM CM + Grad

CM + Strip

SAD SDPSGC SAD +Grad.

SAD +Strip

SDPS +Grad.

SDPS +Strip

GC + Grad.

GC + Strip

Grad. Strip Grad. StripGrad. Strip73% 77% 89% 88% 89% 92% 79% 84% 92%

77% 83% 92% 80% 85% 92% 89% 90% 93%

27Ground truth

Gray codeFCV Four Path Shapelet

77% 83% 92% 80% 85% 92% 89% 90% 93%

69% 54% 71% 97%

Results


Depth map without structured lighting Depth map with structured lighting

Conclusion: SDPS + gray code is very close to ground truth• 3D face database is currently built. Pb: Processing data• Specific study on face feature accuracy

15

3D face animation using markers and stereovision (2006-2007)

• Stereo web-cameras perform marker based motion capture to obtain rigid head motion and non rigid capture to obtain rigid head motion and non-rigid facial expression motion.

• Acquired 3D points are then mapped onto a 3D face model with a virtual muscle animation system that can create face expressions.M l i ki ti d t l • Muscle inverse kinematics updates muscle contraction parameters based on marker motion to create a virtual character's performance.


Marker based expression modelling Marker based expression modelling systemsystem

• Data driven:– Stereo web-cameras, face

kmarkers.– Head motion -rigid – Expressions - non-rigid.

• Tracked 3D points – 3D face model mapping.

Vi t l l i ti – Virtual muscle animation system

• Muscle inverse kinematics (IK) – Jacobian Transpose


16

Video ResultsVideo Results

31


More results


17

Results


Results-video


18

Dynamic face reconstructionDynamic face reconstruction• Reconstruction at video rates.

– Design using static reconstruction conclusions:• ‘One shot’ active illumination + Symmetric Dynamic Programming One shot active illumination Symmetric Dynamic Programming

(SDPS) • Project pattern every other frame to get a clean texture.

Monochromatic Basler video cameras + colour web camera to obtain colour information.


Active binocular stereoActive binocular stereo• Strip colour pattern: much higher accuracy

– SAD stereo algorithm: Depth map

• Pattern colour should avoid skin tones.

Strip pattern SAD - without pattern

SAD - with strip pattern

93%80%


19

Video resultsVideo results


Marker-based Emotion Modelling


Muscle positions Face texture Underlying 3D mesh 17 muscles: Animation – tracking 3D points-markers

20

Happiness – easiest to reproduce

Surprise


3d face features extraction2007-2010

• Segmentation of face features from 2D images has Segmentation of face features from 2D images has been researched extensively in the past

• 2D approaches are error-prone in presence of challenging illumination conditions

• 3D scanners emphasize depth accuracy over texture resolutiontexture resolution

• Recent research: Multimodal approaches outperform 2D and 3D techniques alone in face biometrics and recognition


21

• Texture and curvature information can be used to segment lips from 3D scans

3D face data: texture and geometry

to segment lips from 3D scans3D Face

Texture Curvature


Results (II): the lips (examples)

• The computed “anatomical” lip contour approximates the texture defined contour well.


22

Results (III): the lips

• Method works on self acquired data (! more noise, less prominent ridges)


Results (IV): the eyes


23

Image Analysis in Soil Science(in collaboration with NZ, France, Mexico, Germany)

• Aim: (low cost) quantitative study of preferential flow Aim: (low cost) quantitative study of preferential flow using dye solution infiltration– Field and lab work: calibration and column images– Image processing for quantitative reconstruction of the

infiltration pattern – Relate concentration of dye to gray-level of images

Produce 3D maps of preferential flows


– Produce 3D maps of preferential flows

C. Duwig, P. Delmas, K. Müller, B. Prado, K. Ren, H. Morin, A. Woodward: Quantifying fluorescent tracer distribution in allophanic soils to image solute transport. European Journal of Soil Science, 2007

Estimating Soil Properties Using Image Processing Techniques

Optical scan of soil core thin section


Electronic microscopy of soil core thin section

Dyed Soil Core Section

24

Results

Image of a soil core cut vertically under black light illumination for 30 seconds Segmented image:

White: area with dyeBlack: area without dye


Black: area without dyeNext steps:• Reconstruct volume of flow pathways for a soil core• Analyse variations over several soil cores• Model the soil flow pathways

Dye concentration and flow pathways for a given soil core cut

Image Analysis in Soil Science• 2D slice and 3D reconstruction2D slice and 3D reconstruction


25

X-ray tomography

Brand : X-ray CT scanner RX solutionsSpatial resolution : 5 µm3 (specimen diameter 4mm)

to 220 µ m3 (specimen diameter 210 mm)

Experimental set up

to 220 µ m (specimen diameter 210 mm)X-ray generator: multi-energy (40 to 230kV) and

different spot size (from 7 to 50 µm)Detector: large field of view (195.07 x 243.84 mm2)

Type: X-ray flat panel detectorMatrix size: 1920-1536 pixels2Data size: 14bit-16384 grey levelsRotation stage: position and velocity controlled,

annular shape and high capacity (70 kg) Reconstruction by proprietary software RX-actExperiment conducted on an Andosol, depth 30cmSoil core dimension: Height 60 mm, diameter 55 mmDistance Source Detecteur : SDD = 752 mmDistance Source Objet : SOD = 347 mmPixel resolution: 58.6 µmAcquisition time: 45 minsMeaningful reconstructed slices: 1300

X-ray shot #240 out of 1000

COMPSCI 773 S1T 49


26

We use the CT data to build a particle system

ResultsSPH modelling results

We defined 1500 particles to trace the fluid. In the image we show part of the solution, the blue particles are the trace particles, red and yellow particles are the solid particles. The yellow particles shows a selected set of cuts where we are obtaining the flux behaviour, we do the same procedure at different deeps.

for the soil, the soil particle system consist in 496789 particles.


Medical Image Analysis(in collaboration with University of Louisville, USA)

• Global image registration using the learned prior appearance Global image registration using the learned prior appearance model (a Markov-Gibbs random field with multiple pair-wise cliques)

A. El-Baz, G. Gimel’farb: CVPR 2008, Anchorage, Alaska, USA


Kidney image Gibbs energies for 5,000 neighbours76 out of 5,000 cliques in the learned prior

27

90.2% 62.6%Overlap between kidneys aligned with our and the mutual information (MI)based approaches


Kidney: our registration vs. MI-based registration

based app oac es

96.8% 54.2%Overlap between lungs aligned with our and the mutual information (MI)based approaches


Lungs: our registration vs. MI-based registration

based app oac es

28

Recent Publications on Medical Image Analysis

A. El-Baz, G. Gimel’farb, R. Falk, M. Abo El-Ghar, Automatic A. El Baz, G. Gimel farb, R. Falk, M. Abo El Ghar, Automatic Analysis of 3D Low Dose CT Images for Early Diagnosis of Lung Cancer, Pattern Recognition, vol. 42, no. 6, pp. 1041-1051, 2009

A. El-Baz, M. Casanova, G. Gimel’farb, M. Mott, A. Switala: An MRI-Based Diagnostic Framework for Early Diagnosis of Dyslexia, International Journal of Computer Assisted Radiology and Surgery vol 3 no 3 4 pp 181 189 September 2008


Surgery, vol. 3, no. 3-4, pp. 181-189, September 2008A. El-Baz, G. Gimel’farb, Image Segmentation with a Parametric

Deformable Model using Shape and Appearance Priors, CVPR 2008, Anchorage, Alaska, June 23-28, 2008

Image Description by Primitive Elements(in collaboration with INRIA Sophia Antipolis, France)

Generalized multi-marked Generalized multi marked random point process– Library of marks

(primitive objects) for image description

– Limited inter-object interactions


– Efficient jump-diffusion random processes for fast convergence

F.Lafarge, G.Gimel’farb, and X.Descombes: Geometric Feature Extraction by a Multi-Marked Point Process. IEEE Trans. Pattern Analysis and Machine Intelligence, 2009

29



30

Image Matching under Large Occlusions and Contrast Variations

• Conventional matching: errors for objects under large occlusions Conventional matching: errors for objects under large occlusions and non-uniform variations of contrast

• Symmetric correlation with soft masking of outliers

Actual template Best match


Occluded template Soft masks of occlusions

Occluded template Soft mask Changed background Soft mask

• Non-uniform contrast transformations to account for changing illumination directions (e.g. from frontal 0o to oblique 75o)


Template 0o Image 75o Template transformation by polynomial model by QP adaptation

31

Unsupervised Texture Segmentation (in collaboration with Dresden University of Technology, Germany) • Multi-component Markov-Gibbs model with analytic potential estimates


Initial collage Region map found Restored 1st texture Restored 2nd texture

Shape Prior Modelling(in collaboration with Dresden University of Technology, Germany)

• Markov-Gibbs “texture-like” model of geometric shapes with analytic potential estimatesp


Training set of 7-region shapes Simulated set of the shapes learned

32

MGRFs with Higher-order Cliques(in collaboration with University of Louisville, Louisville, Kentucky, USA)


Greyscale image “Zebra” Energy image: 2nd-order cliques



Energy image: 2nd/3rd-order cliques 2nd/3rd/4th-order cliques

33



Energy image: 2nd/3rd/4th/5th-order cliques

Texture Synthesis by Bunch Sampling


34

Texture Synthesis by Bunch Sampling


Brain MRI Segmentation(in collaboration with University of Louisville, Louisville, Kentucky, USA)


MRI brain slice Multi-modal segmentation Ground truth provided by a

radiologist

35

Questions?

compsci 773 - university of auckland · march 2, 2010 compsci 773 s1t 3 compsci 773 • the report...

Documents