Improved Seam Carving for Video Retargeting

Authors: Michael Rubinstein, Ariel Shamir and Shai Avidan Source: ACM Transactions on Graphics (TOG), Volume 27 Issue 3, August 2008Presenter: Hon-Hang ChangDate: 2011/ 09/ 30

1

Outline• Introduction• Preliminaries• Seam Carving using Graph Cut• Forward Energy• Results• Conclusion

2

To treat video as 3D cube

Introduction 1 D path on 2D image (Seam)

2D manifolds in a 3D volume

• In this paper the author define a new formulation of seam carving using graph cuts.

• Because of human perception, the temporal may even be more disturbing than spatial in video, as the human eye is highly sensitive to movement.

• 2D manifolds

3

IntroductionA seam is a monotonic and connected path of pixels going from the top of the image to the bottom

Or from left to right

• The image size is reduced by one either in the horizontal or the vertical dimension.

• video should support retargeting capabilities as it is displayed on TVs, computers, cellular phones.

• A whole seam

4

Preliminaries

Among this, {It} t=1~N

Extend the spatial L1-norm to a spatiotemporal L1-norm:

• Spatiotemporal L1-norm

Spatial term

Temporal term

Where,α ∈ [0, 1]

• Motion artifacts are more noticeable• Taking α= 0.3

5

|}),(||),({|max),(1

jiIy

jiIx

jiE tt

N

tspatial

|}),({|max),(1

jiIt

jiE t

N

ttemporal

temporalspatialglobal EEjiE )1(),(

Seam Carving using Graph Cuts

S (Source)

T (Sink)

Source Sink

arc1

arc2

arc3

arc4

• Graph cut

6

Seam Carving using Graph Cuts

An optimal seam must be:1.1.Valid

- A simple cut cannot define a seam carving2. Monotonic - Including only one pixel in each row (or column)3. Connected -The seam must be connected

• The proposed formulation extend seam carving to video and define a monotonic and connected 2D manifold seam inside the video cube.

• Optimal seam in graph cut

7

Seam Carving using Graph Cuts• Graph cut for image (Non-monotonic)

• Every internal node P is connected to its four neighbors Nbr(Pi,j) = {Pi−1,j , Pi+1,j , Pi,j−1, Pi,j+1}.

Pi,j

Pi-1,j

Pi,j-1 Pi,j+1

Pi+1,j

Neighbors pixels

∂x(i, j) = |I(i, j + 1) − I(i, j)|

∂y(i, j) = |I(i+1, j ) − I(i, j)|

Horizontal direction

Vertical direction

8

Seam Carving using Graph Cuts• Graph cut for image

(Unconnected)

• Prove_1:

9

Seam Carving using Graph Cuts

E1(i, j) =∂x(i, j) + ∂y(i, j)

• Graph cut for image (Original)

• Prove_2:

10

Seam Carving using Graph Cuts• Graph cut for image

• We can use any energy function defined on the pixels as the weight of the forward horizontal arcs .

• Achieve the same results as the original dynamic programming based seam carving

11

Seam Carving using Graph Cuts• Graph cut for video

• The extension to video is straightforward. 12

Seam Carving using Graph Cuts• Graph cut for video

• Computation time is quadratic in the number of voxels. O(mn2)- n #of nodes ; m#of arcs

• Performance issues are encountered already for high resolution images

• The graph cut approach to seam carving allows us to extend the benefits of content-aware resizing to video. Still, the method is not perfect and no single energy function was shown to perform properly in all cases. Therefore, they introduce a new energy function that better protects media content, and improves video results.

13

Forward Energy

• Remove the seam with the least amount of energy from the image

• The inserted energy is due to new edges created by previously non adjacent pixels that become neighbors once the seam is removed

• To measure the real change in energy after a removal of a seam

They measure the difference in the energy of the image after the removal (It=i+1) and the energy of only those parts that were not removed in the previous image

14

Forward Energy

15

+LR = | I(i, j+1)− I(i, j−1)| (arc weight between pi,j and pi,j+1)difference between the Left and Right neighbors

+LU = | I(i-1, j)− I(i, j−1)| (arc weight between pi,j and pi-1,j)difference between the Left and Up neighbors

-LU = | I(i+1, j)− I(i, j−1)| (arc weight between pi,j and pi+1,j)difference between the Left and Up neighbors with respect to the end point of the arrow

Forward Energy• Forward Energy in Graph Cut

16

Forward Energy• Forward Energy in Graph Cut

+LR = | I(i, j+1)− I(i, j−1)| (arc weight between pi,j and pi,j+1)difference between the Left and Right neighbors

+LU = | I(i-1, j)− I(i, j−1)| (arc weight between pi,j and pi-1,j)difference between the Left and Up neighbors

-LU = | I(i+1, j)− I(i, j−1)| (arc weight between pi,j and pi+1,j)difference between the Left and Up neighbors with respect to the end point of the arrow 17

Results

18

Results

19

Conclusion

20

Video retargeting is achieved using graph cuts and we have shown a construction that is consistent with the dynamic programming approach.