Seam Carving for Content-Aware Image Resizing

Shai Aidan (Mitsubishi Electric Research Labs)

Ariel Shamir (The Interdisciplinary Center & MERL)

ACM SIGGRAPH 2007

Resize

Seam carving & insertion

AbstractSeams are optimal 8-connected paths of

pixels cross the imageCarving out or inserting seams to achieve

content-aware resizing

OutlineIntroductionBackgroundSeam-carving operatorDiscrete image resizingMulti-size imagesLimitationsConclusions and future work

INTRODUCTION

MotivationHTML can support dynamic

changes of page layout and text. Why can not an image deform to fit different layout automatically ?◦ iGoogle

How about aspect ratio of an image , such as fitting photo into PDA or phone cells ?

Solution ?◦ Resize – content independent◦ Crop – remove pixels from the image

periphery only

Basic Idea of Seam-CarvingUse energy function to define the

importance of pixelsDefine seam-carving image operator

◦ Image reduction Carving out seams - the connected low energy

pixels crossing the image

Preserving the image structure

◦ Image enlarging Insert seams on low energy area The order of seam insertion ensures a balance

between the original image content and the artificially inserted pixels

ApplicationDiscrete image resizing

◦ Aspect Ration Change, Image Retarget, Image Enlarging, Content Amplification, Seam Carving in gradient domain, Object Removal

Multi-size images◦ An image can continuously change their

size in a content-aware manner◦ Storing the order of seam removal and

insertion

BACKGROUND

Image RetargetSeek to change the size of the image

while maintaining the important features ◦ Face detector

An automatic thumbnail creation [Suh 03] ◦ ROI

Fisheye-View warp [Liu and Gleicher 05, 06]◦ Visual saliency []

[Suh 03]

origin

[Selur 04, decompose image to foreground obj and background

Saliency map[Itti IEEE99]

◦Simulate neuroscience of human visual system

◦Pyramid tech. to compute 3 feature maps, color, intensity and orientation

[Suh 03], an automatic thumbnail creation, based on either a saliency map or the output of a face detector

[Chen 03], adapting most important region of images to mobile devices.

[Liu 03], suggesting to trade time for space. Given a collection of regions of interest, they construct an optimal path through these regions and display them serially.

[Santella et al. 06] use eye tracking, in addition to composition rules to crop images intelligently.

ROI (Region-Of-Interest) Such a method was proposed by [Liu

and Gleicher 05, 06] for image and video retargeting. For image retargeting they find ROI and construct a novel Fisheye-View warp that essentially applies a piecewise linear scaling function in each dimension to the image. This way the ROI is maintained while the rest of the image is warped. The retargeting can be done in interactive rates, once the ROI is found, so the user can control the desired size of the image by moving a slider. In their video retargeting work they use a combination of image and saliency maps to find the ROI. Then they use a combination of cropping, virtual pan and shot cuts to retarget the video frames.

Feature-aware warping The first solution to the

general problem of warping an image into an arbitrary shape while preserving user-specified features was recently proposed by [Gal et al. 06].

The feature-aware warping is achieved by a particular formulation of the Laplacian editing technique, suited to accommodate similarity constraints on parts of the domain.

Since local constraints are propagated by the global optimization process, not all the constraints can always be satisfied at once

Seam Perfect seams to combine parts of a set of photo into a single

composite picture [Agarwala et al. 04]

Drag-and-Drop Pasting that extends the Poisson Image Editing to computer an optimal boundary (seam) between the source picture and target images [Jia et al. 06]

AutoCollage, a program that automatically creates a collage image from a collection of images. [Rother et al. 06]

Simultaneously solve matting and compositing. They allow the user to scale the size of the foreground object and paste it back on the original background. [Wang , Cohen 06]

evaluated several cost functions for seamless image stitching and concluded that minimizing an L1 error norm between the gradients of the stitched image and the gradients of the input images performed well in general [Zomet et al. 05]

Sear Optimal SeamDijkstra’s shortest path algorithm

[98]Dynamic programming [Efros 01]Graph cuts [Kwatra 03]

SEAM-CARVING OPERATOR

Stra

teg

ies o

f Imag

e R

ed

uctio

n

Optimal global remove the lowest energy pixels

Pixelremove the least energy in each row

Original

e1 energy

Columnremoving columns with minimal energy

Cropfind a sub-win with the highest energy

Original

e1 energy

Stra

teg

ies o

f Imag

e R

ed

uctio

n

Vertical Seam

Horizontal Seam

Optimal Seam Search

Optimal Seam Search

Dynamic Programming

G

S

e1 energy

Image Energy PreservationThe average energy of all pixels during

resizing

Energy FunctionsL1 and L2-norm of the

gradient, saliency measure [Itti 99]

Histogram of Gradient (HoG)Histogram of Gradient (HoG)

[Dalal and Triggs 95]1. Dividing the image window into cells2. For each cell accumulating a local 1-D

histogram of gradient directions3. Normalize cells by the measure of local

histogram energy over larger blocks

The average gradient image

R-HOG descriptor

Weighted R-HOG descriptor

Energy FunctionsHistogram of Gradient

(HoG) [Dalal and Triggs 95]◦ max(HoG(I(x,y)) makes sure

the seams run parallel to the

edge of objects and not

cross them

Energy FunctionsEntropy

◦ Compute the entropy over a 9 x 9 window and add it to e1

eEntropy(x,y) =

+ e1 (x,y)

Energy FunctionsSegmentation and L1

1. Image segmentation [Christoudias 02]

2. Apply e1 on the results

No single e function performs well across all images

Similar range for resizing

e1 or eHoG works well

DISCRETE IMAGE RESIZING

Aspect Ratio Change, Retargeting with Optimal Seams-Order, Image Enlarging, Content Amplification,

Aspect Ratio ChangeCarving-out /insert seams

Original

Original

Original

1D aspect ratio changing

Optimal Seams-Order Search

Dynamic Programming

+

+

=

min

n x m

n‘ x m’

2D aspect ratio changing

Retargeting with Optimal Seams-Order

optimal

h-first

Original

alternate

Transport map

v-first

Image Enlarging

enlarged image

I(t): smaller image after t seam-carving

I(-k): enlarged image after k seam insertion

I(t)I(-1)

t

I(-k)

I(-k)

1. Find first k seams for removal

2. Duplicate them in order to arrive at

I(-k)

insert seams in order of removal

origin

Image Enlarging (＞ 50%)1. Break into several

steps2. Each step does not

enlarge the size of image more than a fraction

origin

Content Amplification

Original

Scale

Seam

Carving

Original Amplified

Seam Carving in the Gradient DomainSeam + Poisson

Reconstruction [Perez 03]1. Compute e

function

2. Work on the gradient domain

3. Remove seams from the x and y derivatives of the original image

4. Use Poisson Reconstruction

original retarget

retarget in Gradient Domain

Object Removal

1. Mark the removing target

2. Remove seams until all the marked pixels are gone

3. * Employ seam insertion to maintain the original size

Object RemovalOrigin

Multi-size imagesStore the pre-computed

representation that encodes, for each pixel in V/H map◦ The index of the seam that removed it◦ The negative index of the seam that

inserted it

origin V(i,j)=t : pixel (i,j) removed by t-th vertical seam

H(i,j)=t : pixel (i,j) removed by t-th horizontal seam

Blue (first seam) Red (last seam)

LimitationsSeam-Carving does not work well on all

imagesEx: face

Origin Crop Scale

Bottom up feature detection

Constraint the face

Face the flower

LimitationsThe amount of

content◦ Too density, no

“less” important area

The layout of the image content

origin

origin

ConclusionsPresent a content-aware resizing

using the seam-carving image operator

Seams are the optimal paths on a single image◦ Carve-out seams◦ Insert seams

Application of seam-carving operator◦ Aspect ratio change, image retargeting,

content amplification, object removalMulti-size images that support

continuous resizing in real-time

Future WorkVideo resizingCombination of scaling and seam-

carving◦ Define more robust multi-size image

Better solution to combine horizontal and vertical seams in multi-size image

END