Download - SKETCH-BASED USER INTERFACE STUDY
SKETCH-BASED USER SKETCH-BASED USER INTERFACE STUDYINTERFACE STUDY
Presented ByPresented By
Jin XiangyuJin Xiangyu
Department of Computer Science and TechnologyNanjing University
June 2002
PART I: PART I: INTRODUCTIONINTRODUCTION
The rise of the research issue of Human-Computer Interaction (HCI)
The idea of this revolution is to bend computers to people’s way of interactiThe idea of this revolution is to bend computers to people’s way of interacting, not the other way around (Landay 2001) ng, not the other way around (Landay 2001)
Human and computer, which one should be the center of computer assisted tasks?
Computer-OrientedComputer-Oriented
Human-OrientedHuman-Oriented
1.1. A Revolution is Undertaking1.1. A Revolution is Undertaking
Computers are becoming more and more powerful and easily available today
1.2. Why Sketch-based User Interface?1.2. Why Sketch-based User Interface?To write down designer’s improvisatory ideas by diagrams is very
important for creative tasks.
(3) Unsuitable for handled devices
No area to accommodate so many stencils and buttons.
Traditional menu/toolbar button-based user interface Demo
(1) Inefficient
A three-step process.
(2) Unnatural
Leaving a sketch uninterrupted, or at least in its rough state, is key to preserving this fluidity (Hearst 1998).
The Solution is “By Sketch”.
Sketching with a pen is a mode of informal, perceptual interaction that has been shown to be especially valuable for creative design tasks (Gross 1996) .
1.3. Research focus: What Kind of Sketch-1.3. Research focus: What Kind of Sketch-based User Interface We are Interested in?based User Interface We are Interested in?
On-line VS Off-line
Mode Input Process Segmentation Vectorization Stroke
On-line f(t): R+ R×R Interactive Easy Not required Cursive
Off-line 2D-Boolean Matrix One-step Difficult Required Formal
Three application level
Sketchy symbol recognitionStroke-number and stroke order free
Not very strictly defined
2
Sketch-based image retrieval
Handwriting recognition
Example
Totally free
Generally agreed among users
Drawing ApproachSymbol SetLevel
Undefined
Strictly defined
3
1
Our research focus
1.4. Three Designing Principles: 1.4. Three Designing Principles: Humanistic, Intelligent, IndividualizedHumanistic, Intelligent, Individualized
How to make the UI to be humanistic?
An graphics inputting user scenario is proposed, which employs an interactive sketching-recognition-rectification process in one-fluent-step.
How to make the UI to be intelligent?
On-line graphics recognition is employed to predict user’s original intention.
How to make the UI to be individualized?
SVM-based incremental learning is employed to adapt different user in shape classification.
These three characteristics are harmoniously combined in our prototype system
—Smart Sketchpad.
PART II: PART II: HIGHLIGHTSHIGHLIGHTS
2.1. Critical Technique 1:2.1. Critical Technique 1:User Scenario for Graphics Inputting User Scenario for Graphics Inputting
Employing one interactive, fluent sketching-recognition-rectification process instead of three split ones.
Step 1: Sketching Recognize and Regularize
The suggested candidate objects
Step 3: Clicking on the intended object and replace the strokes with the very object with proper parameters
Step 2: Sketching Recognize and Regularize
2.2. Critical Technique 2:2.2. Critical Technique 2:On-line Graphics Recognition On-line Graphics Recognition
Strokes Primitive Shape Composite Graphic Object
Primitive Shape Classification and
Regularization
Primitive Shape Classification and
Regularization
Composite Shape Recognition
Composite Shape Recognition
Sketching (user) : Decompose
Composite Graphic Object
Recognition (computer) : Assemble
Primitive ShapeStrokes
2.2.1. Primitive Shape Classification 2.2.1. Primitive Shape Classification and Regularizationand Regularization
Strokes
Primitive Shape Classification & Regularization
Primitive Shapes
The input stroke
Preprocessing Shape Classification
Quadrangle
Shape Fitting
The fitted shape
Shape Regularization
The regularized shape
The User Intended Shape
The User Intended Shape
The User Sketchy Shape
The User Sketchy Shape
By Vertex Combination
By Vertex Combination
Experimental Experimental ResultsResults
0.2 0.4 0.6 0.8 1.0 1.2 1.482
84
86
88
90
92
94
0.2 0.4 0.6 0.8 1.0 1.2 1.482
84
86
88
90
92
94
User 2R
eco
gn
itio
n P
reci
sio
n (
%)
Threshold of Force
User 1
Shape Classification Precision for 1367 samples
User1
0.75
User2
0.35
The optimal thresholds are different
Experimental Experimental ResultsResults Shape Regularization Results
Inner-Shape Regularization
The results of primitive shape classification and regularization
2.2.2. Composite Graphic Object Reco2.2.2. Composite Graphic Object Recognitiongnition
A “Partial” “Structural” Similarity Assessment Strategy is Proposed
B
O
A C
In order to suggest the user in an early stage, the system should recognize graphic object in an incomplete form.
2.2.2. Composite Graphic Object Reco2.2.2. Composite Graphic Object Recognitiongnition
A “Partial” “Structural” Similarity Assessment Strategy is Proposed
A1 A2 B1 B2
The similarity assessment strategy should not only invariant to shifting, rotation, mirroring, but also should invariant to inner distortion.
The Proposed The Proposed Approach Approach
The Source Object The Candidate Object
Spatial Relation Graph (SRGs)
The computational complicity is Pnm.
Conditioned Partial Permutation Algorithm
L1
L2 L3
L4
P1
P2 P3
P4
P5
Graphic Primitive Extraction
Line-segments, arc-segments,
and ellipses/circles
Performance Performance Evaluation Evaluation
Query Generating by adding noises and eliminating some parts.
Experimental results show that our approach can achieve good performance with noises for incomplete objects, and our approach is also invariant to shifting, rotation, mirroring, and inner distortions.
When the user draws 80% of his/her intended object (for users may miss some parts of the object inadvertently) with 10% distortion (this is similar to noises in real user drawing situation), R6 is nearly 90% (averagely of 304 graphic objects).
The Original Object Adding Noises Eliminating some partsThe Generated Query
2.3. Critical Technique 3: 2.3. Critical Technique 3:
User Adaptation User Adaptation
User adaptation is a classical problem in user interface study. Many pattern recognition problems are user specific, for users’ handwritings, drawing styles, and accents are different.
Rule-based feedback may yield “conflict” results due to its intrinsic deficiency, which may lose its general performance when it adapts to a specific user further.
Question: A triangle or a quadrangle?
An ambiguous case
SVM-incremental learning are introduced into the user adaptation problem of shape classification.
2.3.1. 2.3.1. QuestionsQuestionsFour questions need to be solved:
(1) Whether SVM-based Incremental learning can overcome “conflicts”?
(2) What is the advantage of Incremental leaning compared with repetitive learning?
(3) Which one is better, Syed’s or Xiao’s?
(4) Which structure is better, one-against-one and one-against-all?
2.3.2. Experiments2.3.2. ExperimentsExperimental Environments:
Feature Extraction (20-dimensional vector) by turning function
Virtual Sample Generation (with 40 samples each)
40 incremental training sets and two test sets are created
Training time, open-test precision, closed-test precision are tested for different algorithms and structures.
2.3.3. Answers2.3.3. Answers
(1) SVM-based incremental leaning can overcome “conflict”
(2) Incremental learning is much faster than repetitive learning without loss of precision
(3) Syed’s algorithm is better than Xiao’s
(4) One-against-one structure is much faster than one-against-all in our environments
Theoretical analysis and experimental results both show
PART III: THE PART III: THE SMART SMART SKETCHPAD SKETCHPAD SYSTEMSYSTEM
3.1. System architecture of Smart 3.1. System architecture of Smart SketchpadSketchpad
Stroke Recognizer
Closed-Shapes Open-Shapes
Shape Recognizer
Composite Graphic Object Recognizer
Graphic-DB
A New Stroke
Closed Shape Detector
Graphic Primitives Primitive Shapes
Visual Tokens
Interactive Editor
Feedback
Documents
Gesture
Semantic Level Application
Graphic-DB Manager
3.2. The Sketch-based User Interface of 3.2. The Sketch-based User Interface of Smart SketchpadSmart Sketchpad
The Sketching AreaThe Sketching Area
The just inputted shape are recognized and regularized
The just inputted shape are recognized and regularized
The candidate shape are shown
The candidate shape are shown
Part of the intended object are sketched
Part of the intended object are sketched
Candidate object name and their similarities are shown
Candidate object name and their similarities are shown
Candidate object list Candidate object list
Inputting two graphic objects and then delete them
Demo
3.3. UI Evaluation3.3. UI Evaluation
10 different subjects are required to draw the following two diagrams with traditional UI and the sketch-based UI. There are 304 objects listed in 26 stencils (with 12 each) for traditional UI. There are 6 objects can be shown in the Smart Toolbox for sketch-based UI.
Diagram 1 Diagram 2
A demo of inputting Diagram 2 by sketch
Drawing Time for Sketch 1 (s) Drawing Time for Sketch 2 (s) #User ID
Sketch-based Traditional Sketch-based Traditional 1 104 125 157 190 2 93 99 151 288 3 69 98 156 294 4 59 81 122 156 5 64 178 135 191 6 63 100 85 231 7 61 120 92 203 8 72 91 119 195 9 70 70 156 252
10 78 110 125 201
Drawing Time for Different Sketches Under Different UIs
Averagely, the sketch-based UI is 22.4% and 42.9% more efficient than the traditional toolbar button-based UI for sketch1 and sketch2, respectively.
The ultimate comments of all users unanimous agree that they’d like to choose the sketch-based UI instead of the traditional one.
PART IV: PART IV: SUMMAYSUMMAY
Algorithm Algorithm LevelLevel
Solution LevelSolution Level
System LevelSystem Level
(1) Agglomerate Point Filtering, Vertex Combination, Shape Fitting, Shape Regularization
(2) Conditioned Partial Permutation
(3) Comparison of SVM-based Incremental Learning Algorithms and Structures
(1) Interactive graphics inputting user scenario
(2) On-line graphics recognition: primitive shape classification and regularization; composite shape recognition.
(3) SVM-incremental learning for user adaptation in shape classification problem.
(1) A prototype system for conceptual/schematic designing tasks is implemented.
(2) User evaluation between the traditional and sketch-based UIs is performed.
Future WorksFuture Works
(1) How to perform stroke segmentation?
(2) How to cut down the computational cost and improve the recognition precision?
(3) How to make the system learn aggressively?
THANKSTHANKS