scale and object aware image retargeting for thumbnail browsing
DESCRIPTION
《Scale and Object Aware Image Retargeting for Thumbnail》的PPT perillaroc@BITTRANSCRIPT
Scale and Object Aware Image Retargeting for Thumbnail Browsing
Jin Sun, Haibin Ling
Temple University, Philadelphia, PA, 19122
By perillaroc
ICCV 2011
Introduction
• Image browsing tasks
• Tiny thumbnails: a fixed small size
• Scaling
– bring difficulties in searching and recognition
• Automatic image retargeting methods
– target size is comparable to that of the original image
• Several important issues for thumbnail browsing
2012/6/19 2
Thumbnail Browsing
• Thumbnail scales
– Studies have shown that scales can have
significant effects on human visual perception
• Object completeness
– Low-level gradient-based information
– NOW! The object-level completeness
• Structure smoothness
– The contamination caused by pixel removal
methods, e.g. seam carving
2012/6/19 3
Scale and Object Aware Retargeting
• Thumbnail scales
– An image perceived by human vision system
– A new scale dependent saliency
• Object completeness
– integrating the objectness measurement recently
proposed by Alexe et al.
• Structure smoothness
– Use the thin-plate-spline(TPS) as warping function
– Use Cyclic seam Carving to guide the warping.
2012/6/19 4
Scale and Object Aware Retargeting
• A continuous retargeting algorithm but uses discrete retargeting to guide the warping transformation.
2012/6/19 5
Overview : Task Formulation
• Retargeting problem:
• Seam carving(SC)
• First get retargeted images to a target size that is comparable to the original image size.
• Then shrink the retargeted images to get the final thumbnails.
( , ( ))J F I S I
,
( , ), ( )
( , 1), ( )
sc
i j
I i j if j seam iF
I i j if j seam i
2012/6/19 6
Overview : Framework Overview
• Scale and object aware image retargeting(SOAP):
• Warping function: thin-plate-spline (TPS)
• Landmark points : cyclic seam carving (CSC)
• Scale dependent saliency
• Object awareness
• Scale and object aware saliency
• SOAP
soF
( )scaleS I
( )O I
( ) ( ) ( )so scaleS I S I O I
( , ( ))so soJ F I S I
2012/6/19 7
Flow chart of the proposed method
2012/6/19 8
Scale-dependent Saliency
• Original image size
• Display size
• Projection size
• DPI(Dots Per Inch)
,d rpo
d p
s Dss s
DPI D
2012/6/19 9
Scale-dependent Saliency
• How well such a system preserves the image information?
• We want to make the foreground object/ content/theme of the image as “clear” as possible
• According to the study, not all patterns in an image are recognizable by human
2012/6/19 10
Scale-dependent Saliency • the perceived image, denoted as , can be derived as
• is the mean value of N-connected neighbors of pixel I(i,j)
• N is determined by display device specifications.
• κ and ρ are the lower and upper bounds(in cycles per degree) respectively of human visual acuity.
• κ and ρ define the limits at which the visual stimuli frequency becomes too low or too high to be recognized by human.
• We use κ = 0.0175 and ρ = 0.83.
pI
( , ), / [ , ]( ( , ))
( , ),
d
p
N
I i j if s DI I i j
I i j otherwise
( , )NI i j
2012/6/19 11
Scale-dependent Saliency
• A pixel may become indistinguishable from its neighbors to a human observer.
• An image patch that was salient in the original image may not appear salient to a human observer.
• Scale dependent saliency
2012/6/19 12
Scale-dependent Saliency
• First, the original image is scaled in homogeneous into the final thumbnail size, i.e. the display size, 60x60 pixels in our experiment.
• Then, the minimum recognizable pattern, denoted by
is determined by Eqn.4.
• Finally, the scale dependent saliency is defined as
where S(.) is the standard saliency (Itti 1998)
( ) ( )scale
pS I S I
( )scaleS I
ds
2012/6/19 13
Scale-dependent Saliency • the minimum recognizable pattern
• In our monitor : 1680X1050@65Hz , 120 DPI
• is in average 0.009 inches.
• The size is approximately the distance between two pixel lines on the screen, i.e. N = 4 in Eqn.5.
• As a result, in the final thumbnail four adjacent neighbors of one image pixel patch with value differences in color space within certain threshold will be assigned their mean value, which means those pixels are unable to be distinguished by human.
• Differences : 50 in our experiment
ds
ds
2012/6/19 14
Scale-dependent Saliency
2012/6/19 15
Objectness Measure
• Preserve object completeness as much as possible
• Alexe et al. What is an object? (CVPR,2010).
– A novel objectness measure, which is trained to distinguish object windows from background ones.
– For a rectangular window w = (i,j,w,h) with a top-left corner at (i,j), width w and height h, its objectness is defined as the probability p(w) that w contains an object.
2012/6/19 16
Objectness Measure
• First sample windows for an input image
• Then calculate the objectness map O as the average objectness reponse at each pixel
Where
• = 10000 in our experiments
wn1{ }w
i iW w
( , )
1( , ) ( )obj
w W i j w
O i j p w
,max ( , )i j O i j
wn
2012/6/19 17
Scale and Object Aware Saliency
( , ) ( , )so scaleS S i j O i j
2012/6/19 18
Cyclic Seam Carving • In many cases a seam has no choice but to cross objects due
to the original definition of seam.
• Discontinuous seams
• Discontinuous seam-carving for video retargeting. CVPR, 2010
• Problem: object-structure damage, e.g. pixel shift problem
2012/6/19 19
Cyclic Seam Carving
• Cyclic Seams: warp the image
2012/6/19 20
Cyclic Seam Carving
• Energy function
where ρ is the weight and set to 0.3 .
• The improved energy is then combined with the CSC algorithm to provide landmark point pairs needed for estimating TPS warping.
(1 ) so
scale scE E S
2012/6/19 21
Cyclic Seam Carving
2012/6/19 22
Image Warping Function
• Many discrete retargeting methods generate excellent results in general but they sometimes create serious artifacts when the target has a size much smaller than the input.
• We combine a continuous warping model with a discrete retargeting guidance.
• Thin-plate-spline(TPS) as our warping function
2012/6/19 23
Thin-plate-spline(TPS)
• Landmark points and
, where p is mapped to q.
• the TPS transformation T is defined as the transformation from P to Q that minimizes the regularized bending energy E(f) defined as
• The TPS warping is defined as
2 , 1,2, ,i lP p i n
2 , 1,2, ,i lQ q i n
2
2 2 22 2 2
2 2
( ) ( )
( ) 2( ) ( )
i i
i
E f q f q
f f fdxdy
x x y y
argmin ( )fT E f
2012/6/19 24
TPS Warping Function
• Landmark point pairs P and Q is derived from the CSC retarget algorithm
• A two-way solution
• First, We sample randomly a landmark set b P (Fig. 6(a)) from original image and then trace their shifting in the CSC process.
2012/6/19 25
TPS Warping Function
• Then, point set Q is re-sampled uniformly on the target image, which is generated by CSC.
• Finally, a sample set P is generated by mapping Q to the original image using warping estimated by Q and P.
2012/6/19 26
Quantitative Experiments
• To find the thumbnail that matches the description
• From 10 x 10 image thumbnails
• Different methods
– Scaling(SL)
– Seam carving (SC)
– Improved seam carving (ISC)
– The proposed SOAR algorithm (SOAR)
2012/6/19 27
Quantitative Experiments
2012/6/19 28
Quantitative Experiments
2012/6/19 29
Qualitative Experiments
• In general our method performs the best regarding the (foreground) object size in the thumbnails.
2012/6/19 30
RetargetMe
2012/6/19 31
Limitations
• Saliency distribution is scattered
• Object is to big
2012/6/19 32
Conclusion
• A scale and object aware image retargeting method for thumbnail browsing.
• Several new techniques
– Scale dependent saliency
– Objectness
– Cyclic seam carving
– A TPS-based continuous warping model
2012/6/19 33