bayesian networks for sketch understanding christine alvarado mit student oxygen workshop 12...
TRANSCRIPT
![Page 1: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/1.jpg)
Bayesian Networks for Sketch Understanding
Christine AlvaradoMIT Student Oxygen Workshop12 September 2003
![Page 2: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/2.jpg)
Sketching in Design
Mechanical Engineering
Software
![Page 3: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/3.jpg)
A Challenge In Sketch Understanding
Noisy Input
There is no one threshold for shapes or constraintsInterpretation depends on context
![Page 4: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/4.jpg)
Naïve Approach
Why not just try all possibilities?
![Page 5: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/5.jpg)
Naïve Approach
Why not just try all possibilities?
Arrow?
![Page 6: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/6.jpg)
Naïve Approach
Why not just try all possibilities?
Arrow?
![Page 7: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/7.jpg)
Naïve Approach
Why not just try all possibilities?
Arrow?
![Page 8: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/8.jpg)
Naïve Approach
Why not just try all possibilities?
Arrow?
![Page 9: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/9.jpg)
Naïve Approach
Why not just try all possibilities?
Arrow?
![Page 10: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/10.jpg)
Naïve Approach
Why not just try all possibilities?
Si ik
nMust consider interpretations
n = number of strokes/segments
S = set of shapes
ki = subcomponents in
shape Si
![Page 11: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/11.jpg)
Naïve Approach
Why not just try all possibilities?
Si ik
nMust consider interpretations
n = number of strokes/segments
S = set of shapes
ki = subcomponents in
shape Si
And this only considers shapes independently
![Page 12: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/12.jpg)
Previous Approaches Use Rigid Segmentation
Single stroke shapes Palm Pilot Graffiti Long et. al. [1999]
Explicit Segmentation Quickset: Cohen et. al. [2001]
Pause between strokes
![Page 13: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/13.jpg)
Recognition Using Partial Interpretations
Recognition: Build partial interpretations (PIs) as the
user draws based on easily recognizable low-level shapes
Prune unlikely PIs and use likely PIs to find misrecognized low-level shapes
Evaluating PIs Graphical Models: Missing data = unobserved nodes Interpretation influenced by top-down
and bottom-up information
![Page 14: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/14.jpg)
BN fragments [similar to PRMs, Getoor et. al. 1999]
(Define Arrow (Components (Line shaft) (Line head1) (Line head2)) (Constraints (connects shaft.p1 head1.p1) (connects shaft.p1 head2.p1) (= head1.length head2.length) (< head1.length shaft.length) (< (angle head1 shaft) 90) (< (angle shaft head2) 90) (> (angle head1 shaft) 0) (> (angle shaft head2) 0)))
L1:L2:L3:
C1:C2:C3:C4:C5:C6:C7:C8:
Arrow
L1 L2 L3 C1 C2 C3 C8…
Instantiated and linked together as recognition proceeds[Hammond and Davis, 2003]
![Page 15: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/15.jpg)
Primitive shapes/Constraints
Observation node added when primitive linked to stroke
P(Obs|Prim) determined through data collection
L1
Obs
![Page 16: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/16.jpg)
Example
Sq. error(Stroke a)
Sq. error(Stroke b)
Line(l1)Connects
l1 l2
Arrow
Line(l2) Line(l3)
Quad
LineLine
RemainingArrow
Constraints
Force(F)
Force-pushes-body
Body(B)Touches F B
0.99
0.95
0.95
0.970.99
0.5
0.59
Observation
![Page 17: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/17.jpg)
Example
Sq. error(Stroke a)
Sq. error(Stroke b)
Sq. error(Stroke c)
Line(l1)Connects
l1 l2
Arrow
Line(l2) Line(l3)
Quad
LineLine
RemainingArrow
Constraints
Force(F)
Force-pushes-body
Body(B)Touches F B
0.99
0.95
0.95
0.970.99
0.5
0.59
Observation
![Page 18: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/18.jpg)
Example
Sq. error(Stroke a)
Sq. error(Stroke b)
Sq. error(Stroke c)
Line(l1)Connects
l1 l2
Arrow
Line(l2)Line(l3)
Quad
LineLine
RemainingArrow
Constraints
Force(F)
Force-pushes-body
Body(B)Touches F B
1
1
1
0.971
0.47
0.61
0.95
Observation
![Page 19: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/19.jpg)
Example
Sq. error(Stroke a)
Sq. error(Stroke b)
Sq. error(Stroke c)
Line(l1)Connects
l1 l2
Arrow
Line(l2)Line(l3)
Quad
LineLine
RemainingArrow
Constraints
Force(F)
Force-pushes-body
Body(B)Touches F B
Observation
1
1
1
11
0.47
0.95
1
Sq. error(Stroke d)
Ellipse
0.99
0.970.99
Observation
![Page 20: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/20.jpg)
Current/Future Work
Expand domain/include other domains
Gather sketches from users
![Page 21: Bayesian Networks for Sketch Understanding Christine Alvarado MIT Student Oxygen Workshop 12 September 2003](https://reader035.vdocuments.us/reader035/viewer/2022062217/5697bff91a28abf838cbfb5f/html5/thumbnails/21.jpg)
Conclusion
Graphical models evaluation Partial Interpretations Context-guided search
More drawing freedom + More robust recognition =
More natural interfaces (i.e. The goal of OXYGEN)