introduction to image processing (signal processing) neu 259 gina sosinsky may 13, 2012

Introduction to Image Introduction to Image Processing Processing

(Signal Processing)(Signal Processing)

NEU 259NEU 259

Gina SosinskyGina Sosinsky

May 13, 2012May 13, 2012

Quantization of images Quantization of images is key….is key….

The Electron Microscope(Example of a Physical

System)

H0

H1

f

Hn

g

Simian SV40

• Physical systems can be modeled as input signals that are transformed by the system, or cause the system to respond in some way, resulting in other signals, e.g., all imaging devices.

H0

f0

g

g0

H1

f1

g1

Hn

fngn

H0

f g

H1

H0 H1Hn

f0 gnf1 fn

g0 g1

Hf g

Systems and Signals

What is image processing?

• The analysis, manipulation, storage, and display of graphical images from sources such as photographs, drawings, and video.

• Any technique either computational or photographic which alters the information in an image.

The analysis, manipulation, storage, and The analysis, manipulation, storage, and display of signals in a multi-dimensional space.display of signals in a multi-dimensional space.

• Image Enhancement (Real and Reciprocal Spaces).

• Image Segmentation.• Feature Recognition and Classification.

• Three-dimensional Imaging.• Image Visualization (2D and 3D).

Common Topics in Image Common Topics in Image ProcessingProcessing

• Acquisition of Images.• Representation and Storage.

• Boolean Operations.• Morphological Operations.• Image Measuring.

• Correction of Imaging Defects (Image Restoration).

What is image processing?• Any technique either computational or

photographic which alters the information in the image.

– Examples• Reversing the contrast of an image (black becomes white and vice versa).

•Maximizing the values for color tables (histogram stretching)

•Pattern recognition & analysis (correlations).

– As Misell points out, image processing will not turn a poor image into a good one, but extracts the maximum amount of structural information from the original.

Types of operations

an input image a[m,n] ==> an output image b[m,n] (or an input image a[m,n] ==> an output image b[m,n] (or another representation) another representation)

* Point- the output value at a specific coordinate is * Point- the output value at a specific coordinate is dependent only on the input value at that same coordinate.dependent only on the input value at that same coordinate.

* Local - the output value at a specific coordinate is * Local - the output value at a specific coordinate is dependent on the input values in the neighborhood of that dependent on the input values in the neighborhood of that same coordinate.same coordinate.

* Global - the output value at a specific coordinate is * Global - the output value at a specific coordinate is dependent on all the values in the input image.dependent on all the values in the input image.

Image Enhancement

Objective: process an image to obtain the most information from it for a particular application.

Example: Common operations to enhance images depend on the convolution of masks and the Fourier transform.

How do I know that my new/corrected/enhanced image is

correct?

• Does it resemble the original image? • Are any unusual features being introduced (e.g. aliasing)?

• Is it consistent with other results outside of this image (e.g. biochemistry, NMR, MRI etc.)?

Two simple operations• Reversing the contrast

new_pix = max - old_pix + min

Histogram stretching(contrast stretching)

Can use histogram to replace out-lying points.

A bit of history• For electron micrographs, first applications: Markham et al. in 1963, Klug and Berger in 1964. – Involved the signal from a periodic specimen was separated from the non-periodic noise (electron crystallography).

– Seminal publications:•Crowther, DeRosier, Klug, Nature 1968•Unwin and Henderson, JMB 1976

Specific topics

• Shannon sampling and Nyquist limits• Fourier analysis• Projection Theorem• Convolution theorem• Resolution & Filtering• Correlation analysis

Shannon Sampling & Nyquist limits

Nyquist Limit is defined as 2 r/M.

M = magnification; r = step size of scanner or camera

Basically, the Shannon sampling theorem tells you that you need at least 2 data points to sample a function. But…in practice, in order to get a given resolution, you need to use

r/(3 M) or r/(4 M)

(e.g. if you want 12 Å resolution, you need to use a pixel size of 3-4 Å)

This is referred to as oversampling your data. Undersampling can result in an image processing artifact known as aliasing.

Effect of sampling interval on recovery of information. In this example, a sampling interval of 32 appears to be just fine enough to recover the shape of the 1D function without loss of information. At coarser sampling intervals (4-16), the subtler features in the data are lost. In practice, one aims to digitize the data at a fine enough interval to be certain that no information is lost. Thus, using the three-times pixel resolution criteria, in this example one ought to sample the data 96 (= 3 x 32) or greater to be certain to recover all the information contained in the data.

Shannon Sampling & Nyquist Limits

Raster size versus Nyquist limits

Jean Baptiste Joseph Fourier

(1768-1830)

Fourier Theory using trigonometrical series expansion done in ~1807http://www-groups.dcs.st-and.ac.uk/~history/Mathematicians/Fourier.html

“The profound study of nature is the most fertile source of

mathematical discoveries.”

Fear not the Fourier Transform, it is your

friend!Continuous FT

Discrete FT(what we calculate)

Inverse FT

= T-1 (T())Inverse Theorem

Terms for Fourier analysis

• Real space: Our coordinate system (x,y,z)

• Reciprocal, Fourier, inverse, tranform space: Coordinate system after Fourier transformation

• Amplitudes and phases or real and imaginary parts due to complex number analysis

(xyz) Fourier transform (F (hkl)exp i(hkl)) – Where F(hkl) is the amplitude and is the phase and

(xyz) is the density function

Need both phase and amplitude data

Original

correct amplitudesrandom phases

random amplitudescorrect phases

• Dimensions in the object (REAL SPACE) are inversely related to dimensions in the transform (RECIPROCAL SPACE).

• Small spacings or features in real space are represented by features spaced far apart in reciprocal space. Resolution is inversely proportional to spacings.

• Outer regions of the transform arise from fine (high resolution) details in the object. Coarse object features contribute near the central (low resolution) region of the transform.

Reciprocal Space: The Final Frontier

Advantages of using Fourier analysis

• The recorded diffraction pattern of an object is the square of the Fourier transform of that object.

• FT are linear process (like multiplication and division). Can go backward and forward easily if functions are known. Advantages for micrographs where the FT is calculated and we want to do noise reduction, filtering or averaging.

• Projection Theorem (next slides)

• Convolution Theorem: Deconvolution is more easily computed in Fourier space rather than in real space (slides after Projection Theorem).

The Fourier Duck

Behold the duck.It does not cluck.A cluck it lacks.It quacks.It is specially fond of a puddle or pond.When it dines or sups, it bottoms ups.

• The Fourier Duck originated in a book of optical transforms (Taylor, C. A. & Lipson, H., Optical Transforms 1964). An optical transform is a Fourier transform performed using a simple optical apparatus.

http://www.ysbl.york.ac.uk/~cowtan/fourier/fourier.html

The Fourier Cat and its Transform

FTFT

FTFT-1-1

Resolution in real versus reciprocal space

The effect of taking only low angle diffraction to form the image of a duck object. A drawing of a duck is shown, together with its diffraction pattern. Also shown are the images formed (as the diffraction pattern of the diffraction pattern) when stops are used to progressively more of the high angle diffraction pattern. (From Holmes and Blow)

FTFT

FTFT

FTFT-1-1

FTFT-1-1

This is a test – If we take the amplitudes of the Fourier Duck and the phases of the Fourier Cat and back transform, what will we get?

Yes, we get the Fourier Cat!

Duck amplitudes, Cat Phases

Projection TheoremThis is the most fundamental principle for 3D reconstruction from electron micrographs.

Every micrograph we obtain in TEM is a projection (sum) of everything in the specimen.

When examining 3D objects, 2D images

may not provide the complete picture

The Projection TheoremSimply stated it says:

The Fourier Transform of the projected structure of a 3D object is equivalent to a 2D central section of the 3D Fourier transform of the object.

The central section intersects the origin of the 3d transform and is perpendicular to the direction of the projection. The 3d structure is reconstructed from several independent 2d views by the inverse Fourier transform of the complete 3d Fourier transform.

Radon and X-ray transforms

Illustration of Projection Theorem

Baumeister et al. (1999) Trends Cell Biol., 9, 81-85.

Projections Central Sections

The Convolution Theorem

The convolution theorem is one of the most important relationships in Fourier theory

It forms the basis of X-ray, EM and neutron crystallography.

• Holmes and Blow (1965) give a general statement of the operation of convolution of two functions:

"Set down the origin of the first function in every possible position of the second, multiply the value of the first function in each position by the value of the second at that point and take the sum of all such possible operations."

c(u) = f(x) * g(x)

Convolution symbolAlso use

Properties of Convolution

•Convolution is commutative.

c = a * b = b * a

•Convolution is associative.

c = a * (b * d) = (a * b) * d = a

* b * d

•Convolution is distributive.

c = a * (b + d) = (a * b) + (a * d)

where a, b, c, and d are all images, either continuous or discrete.

A simple example of convolution. One function is a drawing of a duck, the other is a 2D lattice. The convolution of these functions is accomplished by putting the duck on every lattice point. (From Holmes and Blow, p.123)

FTFT

FTFT

FTFT-1-1

FTFT-1-1

Molecule Molecule convoluted convoluted with lattice with lattice pointspoints

Convolution & Fourier Transforms• Fourier transform of the convolution of two

functions is the product of their Fourier transforms.

T(ƒ *g) = F x G

• The converse of the above also holds, namely that the Fourier transform of the product of two functions is equal to the convolution of the transforms of the individual functions.

T(ƒ x g) = F * G

Computationally, multiplication and fast-Fourier transform algorithms are speedier processes than deconvolution.

Words to image process by

Convolution is easy, Deconvolution is hard. (Thursday’s lecture)

Need to know:

T(ƒ x g) = F * G

Simple Filtering Operations

• Low pass filter• High pass filter• Band pass filter• Crystalline masks, Layer line masks

(All the above can have hard or soft edges)

• Median filter• Sobel filter• Rotational harmonics filtering (Fourier-Bessel analysis)

The Fourier Duck

The Fourier Duck: Low pass filtering

• If we only have the low resolution terms of the diffraction pattern, we only get a low resolution duck:

High pass filtering• If we only have the high If we only have the high resolution terms of the resolution terms of the diffraction pattern, we see diffraction pattern, we see only the edges of the duck only the edges of the duck but see internal features but see internal features (missing box function): (missing box function):

Inverse band pass (missing shell of data)

• The edges are sharp, but there is smearing around them from the missing intermediate resolution terms. The core of the duck is at the correct level, but the edges are weak.

Hard versus soft edged filters

• Gaussian falloffs at the edges prevent aliasing artifacts.

• Butterworth filters also have soft edges

GaussianButterfield

Crystalline masking

Median Filter (Real space filter based on

image statistics)• Ranks the pixels in a neighborhood (kernel) according to their brightness value (intensity). The median value in the ordered list is used as a brightness value for the central pixel.

• Excellent rejector of “shot noise” and for smoothing operations. Outlying pixels are replaced by a reasonable value -- the median value in the neighborhood.

Sobel Filter (Real Space Filters)

• Uses the derivitives of the values and filters based on the square root of the sum of the squares for the values.

• Good for edge detection

• (computationally intensive!)

Edge Detection Filters

From Russ’ Book on Image Processing

Filters within Photoshop

Correlation Analysis & Pattern Recognition

Correlation Analysis & Pattern Recognition

Pearson coefficient as a measure of correlation of labeling:a measure of the correlation (linear dependence) between two variables X and Y, giving a value between +1 and −1 inclusive.

Rotational Filtering (Fourier-Bessel)

Friedrich Wilhelm BesselFriedrich Wilhelm Bessel1784 - 18461784 - 1846

Good General References• D.L. Misell, "Image Analysis, Enhancement and Interpretation" (1978) (Practical Methods in Electron Microscopy series vol. 7, Audrey Glauert editor) North-Holland publishers

• J. Frank, "Three-Dimensional Electron Microscopy of Macromolecular Assemblies" (2004, 2nd edition) Academic Press publishers

• J. Russ, "The Image Processing Handbook" (1995, 2nd edition) CRC Press

• Image Processing Fundamentals Web Site http://www.ph.tn.tudelft.nl/Courses/FIP/noframes/fip.html

• Baker lecture notes