YOUNGBIN KIM
2013/01/16
Korea University
Computer Graphics Lab.
Pengfei Xu Hongbo Fu Hong Kong Univ. City Univ. of Hong Kong Oscar Kin-Chung Au Chiew-Lan Tai City Univ. of Hong Kong Hong Kong Univ. SIGGRAPH Asia 2012
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 2 KUCG |
Abstract
• Lazy Selection
Scribble-based tool • Quick selection of one or more desired shape elements
• By roughly stroking through the elements ‗ Automatically refines the selection
‗ Reveals the user’s intention
Introduction
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 4 KUCG |
Introduction
• Manually selecting shape elements or objects
One of the most frequently performed tasks
Click-based selection • One of the most commonly used tools
‗ Mainly due to its simplicity
‗ However, since it requires every click to be placed exactly
• Even a slight improvement in its efficiency would save significant user time as a whole
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 5 KUCG |
Introduction
• New element selection tool based on a scribble-based user interface
Scribble-based UI to provide the advantages 1. More effectively recognize the user’s intent
‗ Through shape matching between the scribble and its covered elements
2. Support the selection of multiple elements in a single action ‗ Allow the user to quickly select a desired group of elements
3. More tolerant to errors due to either imprecise input means or casual selection ‗ Selection decision is made not based on only the input location
information alone
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 6 KUCG |
Introduction
• Automatically picking the desired element(s)
From the set of elements (partially/fully) covered by a user-specified scribble • Not an easy task
‗ No obvious decision strategy for revealing the user’s intent from the scribble
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 7 KUCG |
Introduction
• Present a two-phase approach
1. Establish a set of selection candidates, one of which the user might intend to select
2. Rank each selection candidate based on its location and shape with respect to the scribble
• Paper presents the main application of our tool
Smart selection of shape elements in images • Represented as 2D closed polygons
Related Work
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 9 KUCG |
Related Work
• Tools for Ungrouped Elements Selection
• Perceptual Grouping of Elements
• Scribble-based User Interfaces
• Sketch-based Retrieval
• Shape Writing on Stylus Keyboard
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 10 KUCG |
Related Work
• Tools for Ungrouped Elements Selection
Click-based selection + lasso selection • Inconvenient to select the elements of interest that are
interlaced with other elements
Various research prototypes have been proposed • Sloppy Selection [Lank and Saund 2005]
• DynaSpot [Chapuis et al. 2009]
• Harpoon Selection [Leitner and Haller 2011] ‗ These techniques select a target solely based on whether the target
is touched or enclosed by a selection gesture with tolerance
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 11 KUCG |
Related Work
• Perceptual Grouping of Elements
Magic Wand in Adobe Illustrator • Selecting all similar objects in the entire scene
‗ Lack local or direct selection control
Several techniques have been proposed • Select a group from possibly overlapping groups
‗ Including repeated clicking cycles, pose matching and path tracing
Our work offers a more general and powerful selection tool • Rendering pose matching and path tracing as special cases
of ours
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 12 KUCG |
Related Work
• Scribble-based User Interfaces
Extensively used for interactively cutting out regions of interest • From bitmap images, videos or 3D models
Adopted for various interactive applications • Appearance editing, repeated element extraction, and
colorization of images or cartoons ‗ Require the scribbles to be precisely within the regions of interest
• AppProp [An and Pellacini 2008], LazyBrush [Sykora et al. 2009]
‗ Support imprecise scribble placement
∙ Different nature of the problems
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 13 KUCG |
Related Work
• Sketch-based Retrieval
Our work is also related to content-based retrieval of images or 3D shapes • Using a hand-drawn sketch as a query
‗ Our scribbles are typically drawn across shape elements instead of along their boundaries, thus posing a different matching problem
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 14 KUCG |
Related Work
• Shape Writing on Stylus Keyboard
Our solution bears some resemblance to several methods • Literature of shorthand writing on stylus keyboard
‗ ZHAI, S., AND KRISTENSSON, P. –O. 2003. Shorthand writing on stylus keyboard. In CHI
‗ KRISTENSSON, P. –O., AND ZHAI, S. 2004. Shark2: a large vocabulary shorthand writing system for pen-based computers. In UIST
• Our problem imposes its own unique set of challenges ‗ Arbitrary shape of elements
‗ No given set of selection candidates
Algorithm
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 16 KUCG |
Algorithm
• Develop a two-phase approach
1. Establishes a set of selection candidates by examining the underlying patterns (Section 3.1)
2. Picks the best selection candidate(s) by ranking individual candidates with respect to the scribble (Section 3.2)
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 17 KUCG |
Selection Candidates
• Attributes for Grouping
Various types of attributes can be used to group elements and thus reveal the underlying patterns • Magic Wand tool in Adobe Illustrator
‗ Selects objects with the same or similar attributes based on stroke weight, stroke/fill color, opacity, or blending mode
• Nan et al. [2011] ‗ Explored the integration of five principal Gestalt rules
∙ Similarity, proximity, continuity, closure, and regularity
Law of Proximity Law of Similarity Law of Closure
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 18 KUCG |
Selection Candidates
• Attributes for Grouping
Focus on modeling the interactions between the three principles • Kubovy and van den Berg [2008], Luna and Montoro [2011]
• Proximity
• Similarity in shape
• Common region in visual perception ‗ Common region principle applies to elements within another
element
∙ i.e., enclosed by the boundary of another element
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 19 KUCG |
Selection Candidates
• Attributes for Grouping
Our experimentation on additive models
‗ First term : PCA-based similarity measure
∙ 𝜆 𝑖𝑘
: k-th eigenvalue obtained by applying PCA to element i
∙ 𝑐𝑖 : Shape compactness of element i, defined as the ratio of its squared perimeter to its area
‗ Second term : Strengths of proximity
∙ 𝑔𝑖𝑗 : Spatial distance between the two centers of elements
∙ 𝛾: Weight to control the contribution of the proximity principle(=0.3)
‗ Third term : Strengths of common region
∙ 𝑏𝑖𝑗 : 0.7 if elements are inside a common region, 1 otherwise
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 20 KUCG |
Selection Candidates
• Grouping
By a single scribble, a user may target • A single element
• A group of elements that have one or several characteristics in common
• Multiple groups of elements
• All the elements covered by the scribble ‗ This naturally demands a hierarchical representation of the scribble-
covered elements
To this end, we take a greedy bottom-up grouping approach
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 21 KUCG |
Selection Candidates
• Grouping
Find the minimum grouping error 𝑓𝑖𝑗
• Iteratively group the pairs of elements ‗ Using a greedy bottom-up approach
‗ Until all the input elements form a single group
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 22 KUCG |
Ranking
• Based on the following key observations or rules
Rule 1 • For elements of small to moderate sizes, the user prefers
to draw a scribble which is close to the elements and whose shape is similar to the overall layout of the elements
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 23 KUCG |
Ranking
Rule 2 • For elements of rather large size, the user may specify the
scribble more freely as long as the scribble is within or strokes through the elements
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 24 KUCG |
Ranking
Rule 3 • Since the user’s effort should be respected as much as
possible, the more the scribble covers a candidate, the more likely the candidate is desired
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 25 KUCG |
Ranking
Rule 4 • When the user draws a scribble very fast, the location of
the scribble might be less accurate. In contrast, the shape of the scribble is largely insensitive to the scribbling speed
Rule 5 • As the user usually pays more attention to the beginning
and ending points of the scribble than the rest of the scribble, the elements close to the two endpoints of the scribble are more likely to be picked
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 26 KUCG |
Ranking
• Ranking algorithm
Starts with uniform point sampling • Regions of the elements in a candidate 𝑐 ∈ 𝐶
‗ 𝐶 denote a set of selection candidates
‗ 𝑃𝑐 denote the resulting sets of points for the candidate
Equidistant point sampling of the scribble curve ‗ 𝑃𝑠 denote the resulting sets of points for the scribble
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 27 KUCG |
Ranking
• Ranking algorithm
Next, compute the closest distance from every point in 𝑃𝑠 to 𝑃𝑐
• Denoted as 𝑇𝑠→𝑐 = {𝑖𝑛𝑓𝑝∈𝑃𝑐𝑑(𝑝, 𝑞)|𝑞 ∈ 𝑃𝑠}
‗ 𝑑(𝑝, 𝑞) is the Euclidean distance between a pair of points
Also measure the closest distance from every point in 𝑃𝑐 to 𝑃𝑠
• Denotes as 𝑇𝑐→𝑠 = {𝑖𝑛𝑓𝑝∈𝑃𝑠𝑑(𝑝, 𝑞)|𝑞 ∈ 𝑃𝑐}
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 28 KUCG |
Ranking
• Ranking algorithm
Let 𝜇∗ and 𝜎∗ be the mean and the standard deviation of 𝑇∗, respectively • 𝜇𝑐→𝑠 and 𝜇𝑠→𝑐 measure the degree of matching between
the scribble and a candidate in terms of location
• 𝜎𝑐→𝑠 and 𝜎𝑠→𝑐 measure the degree of shape matching
Matching error function 𝑟(𝑐) • 𝑟 𝑐 = 𝛽 𝜇 𝑠→𝑐 + 𝜇 𝑐→𝑠 + (1 − 𝛽)(𝜎 𝑠→𝑐+𝜎 𝑠→𝑐)
‗ 𝛽 controls the relative contribution of the location and shape information
‗ 𝜇 ∗ and 𝜎 ∗ are the normalized versions of 𝜇∗ and 𝜎∗
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 29 KUCG |
Ranking
• Enforcing Rule 4
Dynamic weighting scheme
• 𝛽 = min{1,𝑒1−𝑣
2}
‗ 𝑣 is the estimated scribbling speed, normalized by the pre-computed average scribbling speed
• Enforcing Rule 5
Simply decrease the matching errors • By a scaling factor less than 1
‗ For the candidates that cover the beginning and ending points of the scribble
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 30 KUCG |
Ranking
• Click Selection
Matching function is reduced • 𝑟 𝑐 = 𝜇 𝑠→𝑐 + 𝜇 𝑐→𝑠
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 31 KUCG |
Feedback and Output Interface
• Adopt a linear menu
For the organization of the top-ranked candidates
Extension to 3D Elements Selection
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 33 KUCG |
Extension to 3D Elements Selection
• Easy selection of 3D elements Preprocessing stage
• Manually pre-segment 3D models into segments as shape elements
Runtime • Covered by a user-drawn 2D scribble are grouped on the
fly again ‗ Via a bottom-up clustering process
‗ Uses a PCA-based similarity measure alone as the grouping strength measure
• To match the scribble and a candidate 1. Pre-sample 3D points on the surface of each candidate element
2. Project them to 2D for matching with the sampled points from the scribble curve
Results and Discussion
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 35 KUCG |
Results and Discussion
• Tested algorithm on a wide range of images
With various selection targets using the mouse or touch input • Click selection, scribble selection, and paint selection
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 36 KUCG |
Results and Discussion
• Impact of the brush size on the selection results
Our tool permits scribbling with different brush sizes • Size of the brush can be increased/decreased for
rougher/finer selections
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 37 KUCG |
Results and Discussion
• Largely insensitive to the exact scribble locations
• Geometrically similar but semantically different elements nearby may still confuse our algorithm
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 38 KUCG |
Results and Discussion
• Compare our tool to the LazyBrush tool and a straightforward method
LazyBrush tool • Sykora et al., Eurographics 2009
• State-of-the-art technique ‗ pixel-level cartoon coloring or segmentation
∙ Preference of short boundaries
∙ Unawareness of underlying patterns
Rule-of-majority method • Assigns individual elements to the scribbles that cover
them the most
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 39 KUCG |
Results and Discussion
• Limitations
In some cases, might not produce satisfactory results • Desired candidate might not be included in the list
‗ At most 10 candidates in our current implementation
∙ Pick the most satisfactory one from the list and refine it with more selection
∙ Deselection scribbles or simply undo and reissue a new scribble
‗ Desired elements might fail to form a candidate in the grouping process
∙ Interesting to explore alternative grouping strategies like [Nan et al. 2011]
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 40 KUCG |
Results and Discussion
• Limitations
In some cases, might not produce satisfactory results • Less effective for rough hand-drawn images
‗ Instead, such problems are the focus of pixel-level segmentation methods like LazyBrush
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 41 KUCG |
User Study
• Comparison with the standard click and lasso selection tools
Apparatus • IBM Thinkpad X220t with 12.5-inch Multitouch HD
‗ Size of the painting brush : roughly 1.5cm
Participants • 17 university students
‗ Had extensive experience in shape element selection
‗ Had various experience with multitouch interaction
Task • 17 (participants) x 2 (techniques) x 2 (window sizes) x 39 (target
selections) = 2652 trials
Performance Measures • Completion time of individual trials
• Number of operations used for selection, deselection, undo, and image translation
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 42 KUCG |
User Study
Results
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 43 KUCG |
User Study
Results • Superimposed visualization of the scribbles by all the
participants
Conclusion and Future Work
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 45 KUCG |
Conclusion and Future Work
• Presented for the first time a scribble tool
For smart selection of shape elements
Our technique has the following advantages • Applicable to the selection of single or multiple elements
of various sizes
• Supporting click, scribble and painting selection unified in a single tool
• Tolerant to imprecise input means like touch input
• Working consistently well for the selection of 2D and 3D elements
• Selection with our tool is significantly faster than with the standard click and lasso selection tools
Korea University Computer Graphics Lab.
YOUNGBIN KIM | 2013/01/16 | # 46 KUCG |
Conclusion and Future Work
• Devised a bottom-up grouping approach
With the simple encodings of three grouping attributes • Similarity, proximity and common background
This approach can be improved by incorporating the scribble information • Coverage ratio of elements by the scribble, relative
location of elements to the scribble
Different grouping algorithms can be devised • Supports a larger set of grouping attributes will be one of
our future works