Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Image Processing forVideo-rate Scanning Probe Microscopy

Martin Moene Interface Physics Leiden University The Netherlands

graphic by Prof.Dr. Richard Berndt, Kiel University

50 x 49 nm 300 K Au(110)

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Scanning Probe Microscopy

• 1981 Scanning Tunneling Microscope (STM) [1]

• 1986 Atomic Force Microscope (AFM)• Other variants…

graphic by Prof. Dr. Richard Berndt, Kiel University

[1] G. Binnig, H. Rohrer, C. Gerber, and E. Weibel, Phys. Rev. Lett. 49, 57 (1982).

20 x 13 nm 300 K Si(111)

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

40s per Image

1024 x 1024

90 x 90 nm

Si(111)

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

27 IMAGES per second (64 x 64 pixels2)

[2] M.J. Rost, L. Crama, P. Schakel, E. van Tol et al.; Rev. Sci. Instrum. 76 (2005) 053710

Zoom

Rotate

Pan

27 Hz

r e a l t i m

e

80 Hz

Au(110) HOPG

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Feedback

Drivers

Scan Generator

ADCs

LeidenProbeMicroscopy.com

STM Head

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Stabilizing and Comparing Images

thermal drift

50 x 49 nm 300 K Au(110)

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

pixels

AU

Apply Image Stabilisation to:

• Stay Focused• Enable Quantitative Analysis (comparing images)

A tool for both• Image Stabilisation and• Quantitative Analysis

heightlin

e

1st Solution: Normalized Cross-correlation (NCC)

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

What is Cross-correlation (CC) ?

Simplifiednano wire orsingle-atom row0

20

40

60

80

100

120

140

x

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

What is Cross-correlation (CC) ?

Simplified crystal surface

0

20

40

60

80

100

120

140

x

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

0

20

40

60

80

100

120

140

where f is the image and the sum is over x under the window containing the feature t positioned at c: x = c..c+w

Cross-correlation

CC(c) = x f(x) t(x − c)0

10000

20000

30000

40000

50000

60000

70000

c

What is Cross-correlation (CC) ?

“error”

CC depends on offset and amplitude

x

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

- 60

- 40

- 20

0

20

40

60

80

100

120

Mean subtracted

Better Correlate Signal Form

Cross-correlation

CC(c) = x f’(x) t’(x − c)- 50000

- 40000

- 30000

- 20000

- 10000

0

10000

20000

30000

40000

where f is the image and the sum is over x under the window containing the feature t positioned at c: x = c..c+w

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Normalized Cross-correlation

[-1,+1]

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Symmetric Computation

CC(c) = N-1x=0 f(c + x − N/2) t(x)

The usual notation to compute symmetrically around the column at hand

Values required that are outside the signal

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Values required that are outside the image

Boundary ConditionsC

C(c

) =

N-1 x

=0 f

(c +

x −

N/2

) t(

x)

Constant0 0 0 0 00 0 0 0 00 0 1 2 30 0 2 3 40 0 3 4 5

Reflect5 4 3 4 54 3 2 3 43 2 1 2 34 3 2 3 45 4 3 4 5

Extend1 1 1 2 31 1 1 2 31 1 1 2 32 2 2 3 43 3 3 4 5

Periodic3 4 2 3 44 5 3 4 52 3 1 2 33 4 2 3 44 5 3 4 5

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

NCC Application 1: determine shift vector

template

dy

dx

image

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

0

0. 2

0. 4

0. 6

0. 8

1

1. 2

poi nt s

NCC

Normal

0

0. 2

0. 4

0. 6

0. 8

1

1. 2

poi nt s

NCC

Sl ant ed

0

0. 2

0. 4

0. 6

0. 8

1

1. 2

poi nt s

NCC

Normal Sl ant ed

NCC Application 2: compare images

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

• Qualitative: locate ‘a’ at global peak• Quantitative: ‘a’-s can be found at 1• Quantitative: ‘o’-s can be found at 0.7

NCC Application 3: locate feature

1

0.7

templateimage

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Several Ways to Normalise Cross-correlation

[3] J. Martin and J.L. Crowley. Experimental comparison of correlation techniques. In Proc. International Conf. on Intelligent Autonomous Systems, 1995.

energy

zero-mean

image mean under template

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Numerator computed via FFT as a convolution with the template reversed

Fast NCC Implementation [4]

[4] J.P. Lewis. Fast normalized cross-correlation. In Vision Interface, pages 120–123, 1995. [5] H.Huang, D.Dabiri and M.Gharib. On errors of digital particle image velocimetry.

Meas. Sci. Technol. 8 (1997) 1427-1440.

• FFT requires size 2N, pad with zeros• FFT is periodic, prevent errors by padding larger area [5]

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Fast NCC Implementation

Denominator computed from table containing the integral(running sum) of the image square over the search area.

image energy under template

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Fast NCC Implementation: Integral Image

Using the integral image representation one can compute the value of any rectangular sum in constant time.

For example the integral sum inside rectangle D we can compute as:

ii(4) + ii(1) — ii(2) — ii(3)

[6] P. Viola and M. Jones. Robust real-time object detection.Second International Workshop on Statistical and Computational Theories of Vision, 2001.

Def: The integral image at location (x,y), is the sum of the pixel values above and to the left of (x,y), inclusive.

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Results: Timing *)

While Measuring, Registrate (Preliminary)• Decimate image to 64 x 64 pixels2

• Apply Gaussian sub-pixel interpolation [7]

• Background subtraction plus fast NCC: 14 ms

While Analysing, Registrate and Correlate• Spatial Domain NCC: 40 minutes• Fast NCC: 300 ms

*) timing for images of 512 x 512 pixels2 on a PC with an AMD Athlon at 2.8 GHz

[7] J. Bolinder. On the accuracy of a digital particle image velocimetry system. 1999.

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Results: Stabilisation

Au(110)

300 K

39 x 38 nm

26 sec/frame

Au(110)

300 K

52 x 55 nm

3.8 sec/frame

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Summary:• NCC enables finding features• NCC enables quantitatively comparing features &

images• NCC enables tracking to compensate for drift,

there is room for improvement

Future improvement: Lucas-Kanade [8]

• Spatial intensity gradient• Taylor series expansion, iteration• Gaussian Filter ( resolution)• Pyramid of images at different

resolution

[8] B. Lucas and T. Kanade, An iterative image registration technique with an application to stereo vision, in Proc. Imaging Understanding Workshop, 1981, pp. 121—130.

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Recognizing FeaturesCoalescence of Vacancy Islands on Cu(100)

Paul Ruijgrok

200 x 200 nm 300 K Cu(100)

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Finding the Vacancy Islands

Paul Ruijgrok

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Leveling the Image

Accuracy:• Data based number of bins• Fit (part of) Gaussian curve

Paul Ruijgrok

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Finding the Vacancy Islands: threshold

Paul Ruijgrok

hthreshold = h0 + sa0 ,

s: 0.1…0.9

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Detecting the Island Edges

Paul Ruijgrok

erosion

Island A Erosion E(A,N4) ∂A = A—E(A,N4)

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

-3

-2

-1

0

1

2

3

4

5

-2 -1 0 1 2 3 4 5

y

x

P1(-1,0)

P2(0,1)

P3(1,2)

P4(2,3)

P5(4,1)

-3

-2

-1

0

1

2

3

4

5

-2 -1 0 1 2 3 4 5

b

a

P1

P2

P3

P4

P5

42 2

2

2

22

2 1 1 1 1 1

11111

111

1

11

11

11

1

111111

11

1 1

1 11

11

1111

1

11

Paul Ruijgrok

yi = axi + b

or

b = -xia+ yi

Transform points to curves in parameter

space

a = 1, b = 1 y = x +

1

[9] Duda, R. O. and P. E. Hart, "Use of the Hough Transformation to Detect Lines and Curves in Pictures," Comm. ACM, Vol. 15, pp. 11–15 (January, 1972).

Finding the Vacancy Lines

Hough

Transform

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Finding the Vacancy Lines

Hough Transform

• Slope-intercept representation:unbounded parameters

• Want grid of limited size:

• ρ = x cos(θ) + y sin(θ) , or

• ρ = C cos(θ + δ)

Paul Ruijgrok

rho

theta

x

y

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Summary

Paul Ruijgrok

422

22

22

2 1111111

1111

11

1

11

11

11

1

111111

11

11

111

11

1111

1

Thanks to DIPimage team, Delft University of Technology. DIPimage: a scientific image processing toolbox for MATLAB.

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Thanks To:

www.LeidenProbeMicroscopy.com

Ph.D. Students

drs. K. Schoots (Koen)

Undergraduate Students

P.V. Ruijgrok (Paul)

Technicians

L. Crama (Bert)

E. van Tol-Homan (Els)

R. Koehler  (Raymond)

P. Schakel (Peter)

Staff

prof.dr. J.W.M. Frenken (Joost)

dr.ir. T.H. Oosterkamp (Tjerk)

dr. M.J. Rost (Marcel)

Introduction | Stabilizing and Comparing Images | Recognizing Features | Conclusion

Summary

10.7

templateimage

Hough Transform

Introduction | Stabilizing and Comparing Images | Recognizing Features | Future ?

The Future: Superresolution ?

10.7

templateimage

Hough Transform

Introduction | Stabilizing and Comparing Images | Recognizing Features | References

[1] G. Binnig, H. Rohrer, C. Gerber, and E. Weibel, Phys. Rev. Lett. 49, 57 (1982).

[2] M.J. Rost, L. Crama, P. Schakel, E. van Tol et al.; Rev. Sci. Instrum. 76 (2005) 053710

[3] J. Martin and J.L. Crowley. Experimental comparison of correlation techniques. In Proc. International Conf. on Intelligent Autonomous Systems, 1995.

[4] J.P. Lewis. Fast normalized cross-correlation. In Vision Interface, pages 120–123, 1995.

[5] H.Huang, D.Dabiri and M.Gharib. On errors of digital particle image velocimetry.Meas. Sci. Technol. 8 (1997) 1427-1440.

[6] P. Viola and M. Jones. Robust real-time object detection. Second International Workshop on Statistical and Computational Theories of Vision, 2001.

[7] J. Bolinder. On the accuracy of a digital particle image velocimetry system. 1999.

[8] B. Lucas and T. Kanade, An iterative image registration technique with an application to stereo vision, in Proc. Imaging Understanding Workshop, 1981, pp. 121--130.

[9] R. Duda and P. Hart. Use of the Hough transformation to detect lines and curves in pictures. Comm. ACM, Vol. 15, pp. 11–15 (January, 1972).

References

Introduction | Stabilizing and Comparing Images | Recognizing Features | References

Du-Ming Tsai , Chien-Ta Lin, Fast normalized cross correlation for defect detection, Pattern Recognition Letters, v.24 n.15, p.2625-2631, November 2003

Ian T. Young, Jan. J. Gerbrands and Lucas J. van Vliet. Fundamentals of Image Processing. 1998.

W.H. Press, S.A. Teukolsky, W.T. Vetterling, B.P. Flannery. Numerical Recipes in C: The Art of Scientific Computing, 2nd edition. Cambridge University Press. New York, NY, USA.

Ullrich Köthe. STL-Style Generic Programming with Images. C++ Report Magazine 12(1), pp. 24-30, January 2000.

Leiden Probe Microscopy

Interface Physics at Leiden University

This presentation from author’s web-site

Other Information

Introduction | Stabilizing and Comparing Images | Recognizing Features | Software

Stan Birchfield. Dept. of Electrical and Computer Engineering. Clemson University.KLT: An Implementation of the Kanade-Lucas-Tomasi Feature Tracker.

Quantitative Imaging Group at the Faculty of Applied Sciences, Delft University of Technology. The Delft Image Processing library. 1999-2004.

Quantitative Imaging Group at the Faculty of Applied Sciences, Delft University of Technology. DIPimage, A Scientific Image Processing Toolbox for MATLAB. 1999-2004.

Insight Software Consortium. National Library of Medicine Insight Segmentation and Registration Toolkit (ITK). 1999-2003.

Cognitive Systems Group, University of Hamburg, Germany. The VIGRA Computer Vision Library. 1999-2005.

Chair of Technical Computer Science, RWTH Aachen University. LTI-Lib library for image processing and computer vision. 1999-2003.

Software