my group’s current research on image understanding
DESCRIPTION
My Group’s Current Research on Image Understanding. An image-understanding task. Low -level vision. Color, Shape, Texture. Low-level vision. Simple Segmentation. Color, Shape, Texture. Low-level vision. Simple Segmentation. Color, Shape, Texture. Object recognition. Low-level vision. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/1.jpg)
My Group’s Current Research on Image Understanding
![Page 2: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/2.jpg)
An image-understanding task
![Page 3: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/3.jpg)
Low-level vision
![Page 4: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/4.jpg)
Color, Shape, Texture
Low-level vision
![Page 5: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/5.jpg)
Color, Shape, Texture
Simple SegmentationLow-level vision
![Page 6: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/6.jpg)
Color, Shape, Texture
Simple SegmentationLow-level vision
Object recognition
![Page 7: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/7.jpg)
Color, Shape, Texture
Simple SegmentationLow-level vision
Object recognition
High-level perception
![Page 8: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/8.jpg)
Color, Shape, Texture
Simple SegmentationLow-level vision
Object recognition
High-level perception
Pattern recognition
![Page 9: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/9.jpg)
Color, Shape, Texture
Simple SegmentationLow-level vision
Object recognition
High-level perception
Pattern recognition
Analogy-making
![Page 10: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/10.jpg)
Color, Shape, Texture
Simple SegmentationLow-level vision
Object recognition
High-level perception
Pattern recognition
“Meaning”
Analogy-making
![Page 11: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/11.jpg)
Color, Shape, Texture
Simple SegmentationLow-level vision
Object recognition
High-level perception
??? Pattern recognition
“Meaning”
Analogy-making
![Page 12: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/12.jpg)
Color, Shape, Texture
Simple SegmentationLow-level vision
Object recognition
High-level perception
Pattern recognition
“Meaning”
Analogy-making
The “SEMANTIC GAP’
![Page 13: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/13.jpg)
Color, Shape, Texture
Simple SegmentationLow-level vision
Object recognition
High-level perception
Pattern recognition
“Meaning”
Analogy-making
HMAX model of visual cortexRiesenhuber, Poggio, et al.
The “SEMANTIC GAP’
![Page 14: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/14.jpg)
Color, Shape, Texture
Simple SegmentationLow-level vision
Object recognition
High-level perception
Pattern recognition
“Meaning”
Analogy-making
Active Symbol Architecturefor high-level perceptionHofstadter et al.
HMAX model of visual cortexRiesenhuber, Poggio, et al.
The “SEMANTIC GAP’
![Page 15: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/15.jpg)
Color, Shape, Texture
Simple SegmentationLow-level vision
Object recognition
High-level perception
Pattern recognition
“Meaning”
Analogy-making
Active Symbol Architecturefor high-level perceptionHofstadter et al.
HMAX model of visual cortexRiesenhuber, Poggio, et al.
The “SEMANTIC GAP’
![Page 16: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/16.jpg)
The HMAX model for object recognition(Riesenhuber, Poggio, Serre, et al.)
![Page 17: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/17.jpg)
1. Densely tile the image withwindows of different sizes.
2. HMAX features are computed in each window.
3. The features in eachwindow are given as inputto the trained support vector machine.
4. If the SVM returns a score above a learned threshold, then the object is said to be “detected” .
…
…
Recognition Phase
Streetscenes “scene understanding” system(Bileschi, 2006)
![Page 18: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/18.jpg)
Object detection (here, “car”) with HMAX model (Bileschi, 2006)
![Page 19: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/19.jpg)
Some limitations of the Streetscenes approach to scene understanding
![Page 20: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/20.jpg)
Some limitations of the Streetscenes approach to scene understanding
• Requires exhaustive search for object identification and localization
![Page 21: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/21.jpg)
Some limitations of the Streetscenes approach to scene understanding
• Requires exhaustive search for object identification and localization
Exhaustive search over:
![Page 22: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/22.jpg)
Some limitations of the Streetscenes approach to scene understanding
• Requires exhaustive search for object identification and localization
Exhaustive search over:
• Window size and location in the image
![Page 23: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/23.jpg)
Some limitations of the Streetscenes approach to scene understanding
• Requires exhaustive search for object identification and localization
Exhaustive search over:
• Window size and location in the image
• Object categories (e.g., car, pedestrian, tree, etc.)
![Page 24: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/24.jpg)
Some limitations of the Streetscenes approach to scene understanding
• Requires exhaustive search for object identification and localization
Exhaustive search over:
• Window size and location in the image
• Object categories (e.g., car, pedestrian, tree, etc.)
Exhaustive use of HMAX features in each window
![Page 25: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/25.jpg)
• Does not recognize spatial and abstract relationships among objects for whole scene understanding
![Page 26: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/26.jpg)
• Does not recognize spatial and abstract relationships among objects for whole scene understanding
• Has no prior knowledge about object categories and their place in “conceptual space”
![Page 27: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/27.jpg)
• Does not recognize spatial and abstract relationships among objects for whole scene understanding
• Has no prior knowledge about object categories and their place in “conceptual space”
• HMAX model is completely feed-forward; no feedback to allow context to aid in scene understanding.
![Page 28: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/28.jpg)
Goal of our project
• Perform whole-scene interpretation without exhaustive search.
– Incorporate conceptual knowledge
– Allow feedforward and feedback modes to interact
![Page 29: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/29.jpg)
Person Dog
leash attached to
walking
actionaction
holds
A Simple Semantic Network (or “Ontology”)
“Dog walking”
![Page 30: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/30.jpg)
But...
http://www.dogasaur.com/blog/wp-content/uploads/2011/04/dogwalker.jpg
![Page 31: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/31.jpg)
But...
http://www.vet.k-state.edu/depts/development/lifelines/images/dog_jog_1435.jpg
![Page 32: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/32.jpg)
Person Dog
leash attached to
walking
actionaction
holds Dog Group
running
“Dog walking”
![Page 33: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/33.jpg)
Person Dog
leash attached to
walking
actionaction
holds
running
Allowing “conceptual slippage”
“Dog walking”
Dog Group
![Page 34: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/34.jpg)
But...
http://3.bp.blogspot.com/_1YuoCTv4oKQ/S71jUDm7kOI/AAAAAAAAAak/jz4Pg7zzzQ8/s1600/23743577.JPG
![Page 35: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/35.jpg)
http://lh3.ggpht.com/-ZZrYWeBFTjo/SFQH_0ijwaI/AAAAAAAABjA/8nwryW2BmEw/IMG_0356.JPG
![Page 36: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/36.jpg)
Person
leash attached to
walking
actionaction
holds
“Dog walking”
running
Cat
Iguana
Dog
Dog Group
Tail
![Page 37: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/37.jpg)
But...
http://www.mileanhour.com/post/Dog-walking-bike.aspx
![Page 38: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/38.jpg)
http://cl.jroo.me/z3/Z/e/C/d/a.aaa-Thus-walking-dog.png
![Page 39: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/39.jpg)
ttp://thedaemon.com/images/DARPA_Segue_Dog.jpg
![Page 40: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/40.jpg)
http://www.bikeforest.com/product45422.jpg
![Page 41: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/41.jpg)
http://www.k9ring.com/blog/image.axd?picture=2010%2F3%2Fwalking_dog_from_car.jpg
![Page 42: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/42.jpg)
http://www.guy-sports.com/fun_pictures/dog_walking_helicopter.jpg
![Page 43: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/43.jpg)
http://static.themetapicture.com/media/funny-dog-walking-horse-leash.jpg
![Page 44: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/44.jpg)
http://macwetblog.files.wordpress.com/2012/05/dog-walking.jpg
![Page 45: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/45.jpg)
Person Dog
leash attached to
walking
actionaction
holds
running
Cat
Iguana
Biking
Car
Helicopter
“Dog walking”
Dog Group
Driving
Segue-ingTreadmill-ing Horse
Tail
![Page 46: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/46.jpg)
Active Symbol Architecture(Hofstadter et al., 1995)
![Page 47: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/47.jpg)
Active Symbol Architecture(Hofstadter et al., 1995)
• Basis for – Copycat (analogy-making), Hofstadter & Mitchell
– Tabletop (anlaogy-making), Hofstadter & French
– Metacat (analogy-making and self-awareness), Hofstadter & Marshall
and many others…
![Page 48: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/48.jpg)
![Page 49: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/49.jpg)
Semantic network
Temperature
Workspace
Active Symbol Architecture(Hofstadter et al., 1995)
Perceptual agents (codelets)are “active symbols”
![Page 50: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/50.jpg)
Petacat:
(Descendant of Copycat, part of the PetaVision project)
Integration of Active Symbol Architecture and HMAX
Initial task: Decide if image is an instance of “taking a dog for a walk”, and if so, how good an instance it is.
![Page 51: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/51.jpg)
Workspace
![Page 52: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/52.jpg)
Semantic network
Workspace
![Page 53: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/53.jpg)
taking a dog for a walk
outdoors
has location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
horse
swims
ropebelt
leashsidewalk
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
is on Spatial Relation
Semantic Network
cat
![Page 54: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/54.jpg)
Property links
Slip links
taking a dog for a walk
outdoors
has location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
horse
swims
ropebelt
leashsidewalk
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
is on Spatial Relation
Semantic Network
cat
![Page 55: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/55.jpg)
Semantic Network
taking a dog for a walk
outdoors
has location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
horse
swims
ropebelt
leashsidewalk
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
is on Spatial Relation
cat
![Page 56: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/56.jpg)
taking a dog for a walk
outdoors
has location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
horse
swims
ropebelt
leashsidewalk
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
is onSpatial
Relation
cat
![Page 57: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/57.jpg)
taking a dog for a walk
outdoors
has location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
horse
swims
ropebelt
leashsidewalk
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
is onSpatial
Relation
cat
![Page 58: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/58.jpg)
taking a dog for a walk
outdoors
has location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
horse
swims
ropebelt
leashsidewalk
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
is onSpatial
Relation
cat
![Page 59: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/59.jpg)
taking a dog for a walkhas location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
cathorse
swims
ropebelt
leash
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
sidewalk
outdoors
is on
Spatial Relation
![Page 60: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/60.jpg)
taking a dog for a walkhas location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
horse
swims
ropebelt
leash
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
is on
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
sidewalk
outdoors
Spatial Relation
cat
![Page 61: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/61.jpg)
taking a dog for a walkhas location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
cathorse
swims
ropebelt
leash
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
is on
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
sidewalk
outdoors
Spatial Relation
![Page 62: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/62.jpg)
taking a dog for a walkhas location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
cathorse
swims
ropebelt
leash
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
is on
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
sidewalk
outdoors
Spatial Relation
![Page 63: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/63.jpg)
• Measures how well organized the program’s “understanding” is as processing proceeds
– Little organization high temperature
– Lots of organization low temperature
• Temperature feeds back to affect perceptual agents:
– High temperature low confidence in decisions decisions are made more randomly
– Low temperature high confidence in decisions decisions are made more deterministically
Temperature
![Page 64: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/64.jpg)
Input image
![Page 65: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/65.jpg)
Input image Weak segmentation
![Page 66: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/66.jpg)
Input image Weak segmentation
Location “heat map”(probability distribution over pixel locations)_
++++
+
![Page 67: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/67.jpg)
Input image Weak segmentation
Location “heat map”(probability distribution over pixel locations)_
++++
+
Scale “heat map”(probability distribution over scales at each pixel location)
![Page 68: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/68.jpg)
Dog?
Scout codelets: Send C1 features in window to corresponding SVM.If positive result, post builder codelet with urgency equal to SVM’sconfidence.
![Page 69: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/69.jpg)
Dog? Dog?
Person?
Scout codelets: Send C1 features in window to corresponding SVM.If positive result, post builder codelet with urgency equal to SVM’sconfidence.
![Page 70: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/70.jpg)
Dog? Dog?
Sidewalk?
Person?
Scout codelets: Send C1 features in window to corresponding SVM.If positive result, post builder codelet with urgency equal to SVM’sconfidence.
![Page 71: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/71.jpg)
Dog? Dog?
Sidewalk?
Person?
Dog?
Outdoors?
Scout codelets: Send C1 features in window to corresponding SVM.If positive result, post builder codelet with urgency equal to SVM’sconfidence.
![Page 72: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/72.jpg)
Dog?negative Dog?
negative
Sidewalk?positive: 0.4
Person?negative
Outdoors?positive: 0.7
Scout codelets: Send C1 features in window to corresponding SVM.If positive result, post builder codelet with urgency equal to SVM’sconfidence.
Dog?positive: 0.8
![Page 73: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/73.jpg)
Builder codelets: Ask HMAX to compute C2 features using prototype shapesspecific to the object class, and send them to corresponding SVM. If positive, decide to build structure with probability equal to SVM confidence. Break competing structures if necessary.
Dog?negative Dog?
negative
Sidewalk?positive: 0.4
Person?negative
Outdoors?positive: 0.7
Dog?positive: 0.8
![Page 74: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/74.jpg)
Outdoors
Dog
![Page 75: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/75.jpg)
taking a dog for a walkhas location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
cathorse
swims
ropebelt
leash
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
is on
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
sidewalk
outdoors
Spatial Relation
![Page 76: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/76.jpg)
Object-specific heat maps are updated.
+
Dog
Person heat map
+
![Page 77: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/77.jpg)
Object-specific heat maps are updated.
+
Dog
Person heat map
+
Dog
Person?Person?
![Page 78: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/78.jpg)
Object-specific heat maps are updated.
As codelets build structure, heat maps are continually updated to reflect prior (learned) expectations about location and scale as a function of location and scale of “built” objects.
+
Dog
+
Person heat map
Person?Person?
![Page 79: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/79.jpg)
Dog? Dog
Leash?
OutdoorsLeash?
Sidewalk?
Person?
Person?
![Page 80: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/80.jpg)
Dog
PersonStrength: 0.75
Outdoors
Sidewalk
PersonStrength: 0.6
![Page 81: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/81.jpg)
Dog
PersonOutdoors
Sidewalk
![Page 82: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/82.jpg)
taking a dog for a walkhas location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
cathorse
swims
ropebelt
leash
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
is on
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
sidewalk
outdoors
Spatial Relation
![Page 83: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/83.jpg)
Dog
PersonOutdoors
Sidewalk
Leash?
Leash?
Dog?
Sidewalk?
Dog? Rope?
![Page 84: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/84.jpg)
Dog
PersonOutdoors
Sidewalk
Leash
Dog(weak)
![Page 85: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/85.jpg)
Dog
PersonOutdoors
Sidewalk
Leash
Dog(weak)
Dog(strong)
![Page 86: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/86.jpg)
Dog
PersonOutdoors
Sidewalk
Leash
Dog
![Page 87: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/87.jpg)
taking a dog for a walkhas location
persondog
has action
is onis touching
has component
aroad
abeach
trail
drives
runsflies
cathorse
swims
ropebelt
leash
string
walkswalks
is in front of
has location
has action
has component
has componenthas component
stands
is on
sits
is in front of
is touching
is behind
is next to
is on
agrass
is touching
Object
Action
indoors
sidewalk
outdoors
Spatial Relation
![Page 88: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/88.jpg)
Dog
PersonOutdoors
Sidewalk
Leash
Dog
Once objects begin to be built, relation and grouping codelets can run on them.
is next to
is next to
Dog group
![Page 89: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/89.jpg)
Once objects begin to be built, relation and grouping codelets can run on them.
Dog
PersonOutdoors
Sidewalk
Dog
is next to
is next to
Dog group
Leash
![Page 90: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/90.jpg)
Dog
PersonOutdoors
Sidewalk
Dog
is next to
is next to
Dog group
is next to
Leash
Once objects begin to be built, relation and grouping codelets can run on them.
![Page 91: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/91.jpg)
How Petacat makes a final decision
Temperature
taking a dog for a walk
Dog
PersonOutdoorsLeash
Dog
is next to
is next to
Dog group Sidewalk
is next to
![Page 92: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/92.jpg)
How Petacat makes a final decision
Temperature
taking a dog for a walk
Dog
PersonOutdoorsLeash
Dog
is next to
is next to
Dog group Sidewalk
“Situation” codelet is more likely to run when temperature is low.
is next to
![Page 93: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/93.jpg)
Dog
PersonOutdoors
Leash
Dog
is next to
is next to
Dog group
is next to
Sidewalk
Situation codelet tries to match prototypical situation with existing workspace structures, possibly allowing slippages.
![Page 94: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/94.jpg)
Dog
PersonOutdoors
Leash
Dog
is next to
is next to
Dog group Sidewalk
person
taking a dog for a walk
leash
dog
outdoors
is next to
has componenthas component
has component
has location
is in front of
Situation codelet tries to match prototypical situation with existing workspace structures, possibly allowing slippages.
![Page 95: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/95.jpg)
Dog
PersonOutdoors
Leash
Dog
is next to
is next to
Dog group
person
taking a dog for a walk
leash
dog
outdoors
is next to
has componenthas component
has component
has location
is in front of
is next toDog group
Sidewalk
![Page 96: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/96.jpg)
Dog
PersonOutdoors
Leash
Dog
is next to
is next to
Dog group
person
taking a dog for a walk
leash
dog
outdoors
is next to
has componenthas component
has component
has location
is in front of
is next toDog group
If resulting temperature is low enough, classify scene as positive
Sidewalk
![Page 97: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/97.jpg)
Dog
PersonOutdoors
Leash
Dog
is next to
is next to
Dog group Sidewalk
person
taking a dog for a walk
leash
dog
outdoors
is next to
has componenthas component
has component
has location
is in front of
is next toDog group
If situation codelet fails enough times or does not run for a long time,program has increasing chance of ending with negative classification.
If resulting temperature is low enough, classify scene as positive
![Page 98: My Group’s Current Research on Image Understanding](https://reader036.vdocuments.us/reader036/viewer/2022081604/56816060550346895dcf887c/html5/thumbnails/98.jpg)
Temperature at the end of the run gives a measure of how good an instance the picture is (e.g., of the “dog walking” situation).
Temperature