High-Level Computer VisionSummer Semester 2013

Prof. Bernt Schiele, Dr. Mario Fritz<{schiele,mfritz}@mpi-inf.mpg.de>

Dr. Fabio Galasso, Dr. Zeynep Akata<{galasso,akata}@mpi-inf-mpg.de>

Exercise 3: Interest Point Detection and Panorama Stitching(due May 25th 2014, 23:59)

Question 1: Hessian Detector (5 points)

In this question, we will implement a Hessian interest point detector. This detector operates on the second-derivative matrix H (called the “Hessian” matrix)

H(x, y) =

[Dxx(x, y;σ) Dxy(x, y;σ)Dxy(x, y;σ) Dyy(x, y;σ)

]. (1)

It defines interest points as those points for which the Hessian determinant is greater than a certain threshold t.We also include an additional scale normalization factor σ4 so that we can use the same threshold t independentlyof the value of σ:

det(H) = σ4(DxxDyy −D2

xy

) !> t (2)

a) Implement a function gaussderiv2.m which computes second Gaussian derivatives of the image, necessaryfor the computation of the Hessian determinant. (2pt)

b) Implement Hessian interest point detector in the file hessian.m. Start with computing the Hessian deter-minant for each pixel in the image, and then use the provided function nonmaxsup2d.m to find its localmaxima, i.e. points for which the value of Hessian determinant given by Eq. 2 is larger then for each of the8 neighbors. Return all local maxima with value of Hessian determinant larger then a given threshold. Testyour implementation using code provided in the file ex3.m. (3pt)

Question 2: Harris Detector (7 points)

Next, we will implement a Harris detector. This detector identifies corner-like structures by searching for pointsp = (x, y) where the autocorrelation matrix C around p has two large eigenvalues. The matrix C can becomputed from the first derivatives in a window around p, weighted by a Gaussian G(σ̃):

C(σ, σ̃) = σ2G(σ̃) ?

[D2x(σ) DxDy(σ)

DxDy(σ) D2y(σ)

]=

[σ2G(σ̃) ? D2

x(σ) σ2G(σ̃) ? DxDy(σ)σ2G(σ̃) ? DxDy(σ) σ2G(σ̃) ? D2

y(σ)

], (3)

where “?” denotes the convolution operator, and Dx(σ), Dy(σ) denote Gaussian partial derivatives of the image.Instead of explicitly computing the eigenvalues λ1 and λ2 of C, the following equivalences are used

det(C) = λ1λ2 (4)

trace(C) = λ1 + λ2 (5)

to check if their ratio r = λ1

λ2is below a certain threshold. With

trace2(C)

det(C)=

(λ1 + λ2)2

λ1λ2=

(rλ2 + λ2)2

rλ22=

(r + 1)2

r(6)

we can express this by the following condition

det(C)− αtrace2(C) > t. (7)

In practice, the parameters are usually set to the following values: σ̃ = 1.6σ, α = 0.06.

1

hessian, sigma: 2, threshold: 530, #points: 160

50 100 150 200 250 300 350 400

50

100

150

200

250

(a)

harris, sigma: 2, treshold: 1, #points: 162

50 100 150 200 250 300 350 400

50

100

150

200

250

(b)

Figure 1: Hessian interest points (a), and Harris interest points (b) computed on the image gantrycrane.png.

Figure 2: Example of matching Harris interest points between two images using RG histograms.

a) Write a function harris.m which computes the matrix C for each pixel of a given image, calculates its traceand determinant, and combines them according to Eq. (7). After applying non-maximum suppression onthe result image, it should return the coordinates of all points that pass the threshold. The function shouldalso return the values of the cornerness measure (7) computed for each image pixel. (7pt)

Question 3: Region Descriptors (10 points)

In order to find correspondences between interest points, we need to design region1he window size is given as aparameter in the code descriptors. In this question, we will implement a function necessary for computation ofimage descriptors, and a new image descriptor based on the Laplacian operator.

a) Implement a function compute descriptors.m which should take the coordinates of interest points as inputand compute image descriptors for a region around each interest point. Test this function with histogram-based image descriptors rg hist.m and dxdy hist.m implemented in Exercise 2. (2pt)

b) Implement the function laplace.m which should compute the Laplacian of the image given by:

∇2I =∂2I

∂x2+∂2I

∂y2. (8)

In order to make Laplacian of the image more robust to noise we replace the image derivatives in this expres-sion with Gaussian derivatives, which can be computed using previously implemented function gaussderiv2.m.This is equivalent to convolving the image with Laplacian of Gaussian linear filter. (2pt)

c) Implement a new maglap histogram-based image descriptor in the file maglap hist.m, which computesa combined histogram of the magnitude of image gradient and Laplacian. Use the provided gradmag.m

function and laplace.m function from Question 3b to compute the necessary quantities. Experiment withexample images to select the histogram range appropriately. (3pt)

d) The supplementary code provided with the exercise contains the function show point matches.m, whichallows to visialize the correspondences between the interest points in two images. An example of such

1t

2

(a) (b) (c)

Figure 3: An example for panorama stitching. The two images (a) and (b) are automatically registered usingplanar homographies and composed into a larger panorama image (c).

visualization is shown on Fig. 2. Use this function to test your implementation of interest point detectorsand region descriptors. You can look up the reasonable values for the parameters of detectors and descriporsin the provided script ex3.m, but you are also free to experiment with them on your own.

Start with the images graff5 img1.ppm and graff5 img2.ppm. Test both Harris and Hessian interest pointdetectors in combination with rg and dxdy region descriptors.

Try the images NewYork im1.pgm and NewYork im5.pgm. Compute point matches using Hessian interstpoints detector and dxdy histogram descriptors. Then try the maglap histogram descriptors. Whichdescriptor performs better and why? Submit the summary of your observations as part of the solution tothis exercise. (3pt)

Question 4: Panorama Stitching (13 points)

a) Homography estimation with RANSAC. (3pt)Using the provided code get ransac hom.m, explain with your own words:

• What does homography estimation do?

• What are the steps of RANSAC algoithm and why it is useful?

b) Image Resampling. (6pt)To create a single panorama image, write a function pan sample.m which takes the two images and thehomography as arguments and which performs the following steps:

• append a sufficient number of black columns to the left image

• loop over all newly appended pixels plus some overlap and do the following:

– transform the current pixel coordinates x to a point xH in the right image

– look up gray-values of the four pixels nearest to xH

– bilinearly interpolate the gray-value at xH and assign it to the source pixel x in the left image. Thebilinear interpolation is implemented in the provided function interpolate 2d.m.

c) Intensity adjustment. (4pt)To mask the seam between the left and right image content, modify your function such that it also resamplesa part of the overlapping image content and linearly blends from the left to the right image as you moveacross the overlapping part. The amount of each image’s contribution should depend on the distance to theborders of the overlapping region.

Please turn in your solution by sending an email to Zeynep Akata <akata@ mpi-inf. mpg. de> including allrelevant m-files before Sunday, May 25th, 23:59.

3