The Rendering Algorithm:• Multiresolution strategy over both depth and spatial resolutions.

• At each scale:

Goal: To speed up the image-based rendering algorithm in [1]:

• Energy minimization framework:

• Regularizes output images using input image patch based prior:

The problem with image-based priors:• 75d (5x5 RGB patches) nearest-neighbour search per output image

patch (no. patches � no. pixels).

• Large (~107) input patch database, .

• Iterative algorithm, using ICM.

� Several hours rendering time.

Our Approach:• Observation: similarity of patches at one resolution is highly

correlated with their similarity at a coarser resolution.

� Embed a patch hierarchy in a multiresolution framework

Creating the Patch Hierarchy:• Cluster input patches at each resolution, to reduce size of .

• Group clusters into overlapping bins, to form a spill tree.

• Designate all the patches in the same bins as patch the children of patch , its coarser resolution equivalent.

Oliver Woodford and Andrew Fitzgibbon

Results:

Fast Image-based Rendering using Hierarchical Image-based Priors

References:[1] A. Fitzgibbon, Y. Wexler, and A. Zisserman. Image-based rendering using image-based priors. In Proceedings of the International Conference on Computer Vision, October 2003.

New ViewSynthesis

(NVS)

Input images, ,and projection matrices,

.

��� ��� ��

��� ��� ��

Output image, , and implicit depthmap, .

��

����������� ����������� ������ � �� ���

�

�

�

�

��

Conclusion:We produce considerable speed improvements (over 100 times faster), with equivalent or higher quality, compared with the previous patch-based NVS algorithm [1].

Output images at each scale of the algorithm. Each coarser scale is sub-sampled by a factor of two.

Depth is refined in the coarse-to-fine manner shown.

� (Variance)

Depth �

Input imagesamples �

Mean

����

An output pixel is projected into input images at a number of depths, and the input images sampled at these points. This figure shows those samples for one output pixel.

Leave-one-out test – Plant sequence������ � ������ ��� � �����������

Depth �

y �

x �Colour cube, , showing depth

slice, , (light blue).�

�

Output image�����

Ground truth output patch

Closest input patch

Coarse scale patches: images are low-pass filtered and sub-sampled.

Finer scale patches: centred on the same point as coarse patches, in a higher res. image.

(b’) Multires .(b) only.(a) Ground truth.

(c’) Multires .(c) .(d) Difference (a)-(c’).

(C’) Zoom of (c’).(C) Zoom of (c).(A) Zoom of (a).

���� � ��������

���� ����

���� � ��������

• After the final energy minimization, replace each pixel in the output image with the centre pixel of the closest input patch found � Improves high frequency detail and colours.

At all but the coarsest scale the patch search (yellow crosses) is constrained by the closest patch (blue cross) at the previous scale.

Coarse ����� Scale 3 Scale 2 Scale 1

��������• Constrain the patch search in at each scale to the children of the closest patch found at the previous scale.

Acceleration

1. Cache colour (mean) and (variance) for each pixel at each depth into colour cube, .

2. Estimate an initial depth map, , from . alone.

3. Minimize over depth for each pixel simultaneously, with neighbours’ depths fixed and .

4. Iterate until energy minimum reached, giving final depth map .

����

�

������

��� ���

� � ����

��

(c) Synthesized steadicamframe, rendered using

proposed algorithm in 93s.

(b) Steadicam frame from [1], with a quoted rendering time

of 4.4 hours.

(a) Input image from the monkey sequence [1], with a similar viewpoint to (b) & (c).

93 seconds4.4 hoursRendering time: