cs 378: autonomous intelligent roboticsjsinapov/teaching/cs378/slides/...project deliverables •...
TRANSCRIPT
![Page 1: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/1.jpg)
CS 378: Autonomous Intelligent Robotics
Instructor: Jivko Sinapovhttp://www.cs.utexas.edu/~jsinapov/teaching/cs378/
![Page 2: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/2.jpg)
Multimodal Perception
![Page 3: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/3.jpg)
Announcements
Final Projects Presentation Date:
Thursday, May 12, 9:00-12:00 noon
![Page 4: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/4.jpg)
Project Deliverables
• Final Report (6+ pages in PDF)
• Code and Documentation (posted on github)
• Presentation including video and/or demo
![Page 5: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/5.jpg)
Multi-modal Perception
![Page 6: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/6.jpg)
The “5” Senses
![Page 7: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/7.jpg)
The “5” Senses
[http://edublog.cmich.edu/meado1bl/files/2013/03/Five-Senses2.jpg]
![Page 8: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/8.jpg)
The “5” Senses
[http://edublog.cmich.edu/meado1bl/files/2013/03/Five-Senses2.jpg]
![Page 9: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/9.jpg)
![Page 10: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/10.jpg)
[http://neurolearning.com/sensoryslides.pdf]
![Page 11: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/11.jpg)
![Page 12: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/12.jpg)
How are sensory signals from different modalities integrated?
![Page 13: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/13.jpg)
![Page 14: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/14.jpg)
[Battaglia et. al. 2003]
![Page 15: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/15.jpg)
Locating the Stimulus Using a Single Modality
Standard Trial
Comparison Trial
Is the stimulus in Trial 2 located to the left or to the right of the stimulus in Trial 1?
![Page 16: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/16.jpg)
Locating the Stimulus Using a Single Modality
Standard Trial
Comparison Trial
Is the stimulus in Trial 2 located to the left or to the right of the stimulus in Trial 1?
![Page 17: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/17.jpg)
![Page 18: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/18.jpg)
![Page 19: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/19.jpg)
Multimodal Condition
Standard Trial
Comparison Trial
![Page 20: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/20.jpg)
![Page 21: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/21.jpg)
![Page 22: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/22.jpg)
[Ernst, 2006]
![Page 23: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/23.jpg)
![Page 24: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/24.jpg)
Take-home Message
During integration, sensory modalities are weighted based on their individual
reliability
![Page 25: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/25.jpg)
Further Reading
Ernst, Marc O., and Heinrich H. Bülthoff. "Merging the senses into a robust percept." Trends in cognitive sciences 8.4 (2004): 162-169.
Battaglia, Peter W., Robert A. Jacobs, and Richard N. Aslin. "Bayesian integration of visual and auditory signals for spatial localization." JOSA A 20.7 (2003): 1391-1397.
![Page 26: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/26.jpg)
Sensory Integration During Speech Perception
![Page 27: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/27.jpg)
McGurk Effect
![Page 28: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/28.jpg)
McGurk Effect
https://www.youtube.com/watch?v=G-lN8vWm3m0
https://vimeo.com/64888757
![Page 29: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/29.jpg)
![Page 30: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/30.jpg)
Object Recognition Using Auditory and Proprioceptive Feedback
Sinapov et al. “Interactive Object Recognition using Proprioceptive and Auditory Feedback”International Journal of Robotics Research, Vol. 30, No. 10, September 2011
![Page 31: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/31.jpg)
What is Proprioception?
“It is the sense that indicates whether the body is moving with required effort, as well as where the various parts of the body are located in relation to each other.”
- Wikipedia
![Page 32: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/32.jpg)
Why Proprioception?
![Page 33: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/33.jpg)
Why Proprioception?
FullEmpty
![Page 34: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/34.jpg)
Why Proprioception?
Hard Soft
![Page 35: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/35.jpg)
Exploratory BehaviorsLift:
Drop:
Push:
Shake:
Crush:
![Page 36: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/36.jpg)
Objects
![Page 37: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/37.jpg)
Sensorimotor Contexts
lift
shake
drop
press
push
audio proprioception B
ehav
iors
Sensory Modalities
![Page 38: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/38.jpg)
Feature ExtractionJ 1
J 7.
. .
Time
![Page 39: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/39.jpg)
Feature Extraction
Training a self-organizing map (SOM) using sampled joint torques:
Training an SOM using sampled frequency distributions:
![Page 40: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/40.jpg)
Discretization of joint-torque records using a trained SOM
is the sequence of activated SOM nodes over the duration of the interaction
Discretization of the DFT of a sound using a trained SOM
is the sequence of activated SOM nodes over the duration of the sound
Feature Extraction
![Page 41: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/41.jpg)
Proprioceptive Recognition Model
Auditory Recognition Model
Weighted Combination
Proprioception sequence Audio sequence
![Page 42: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/42.jpg)
![Page 43: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/43.jpg)
Accuracy vs. Number of Objects
![Page 44: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/44.jpg)
Accuracy vs. Number of Behaviors
![Page 45: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/45.jpg)
![Page 46: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/46.jpg)
Results with a Second Dataset• Tactile Surface Recognition:
– 5 scratching behaviors
– 2 modalities: vibrotactile and proprioceptive
Artificial Finger Tip
Sinapov et al. “Vibrotactile Recognition and Categorization of Surfaces by a Humanoid Robot”IEEE Transactions on Robotics, Vol. 27, No. 3, pp. 488-497, June 2011
![Page 47: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/47.jpg)
Surface Recognition Results
Chance accuracy = 1/20 = 5 %
![Page 48: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/48.jpg)
Scaling up: more sensory modalities, objects and behaviors
Microphones in the head
Torque sensors in the joints
ZCam (RGB+D)
Logitech Webcam
3-axis accelerometer
![Page 49: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/49.jpg)
100 objects
![Page 50: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/50.jpg)
Exploratory Behaviors
grasp lift hold shake drop
tap poke push press
![Page 51: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/51.jpg)
Object Exploration Video
![Page 52: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/52.jpg)
Object Exploration Video #2
![Page 53: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/53.jpg)
Coupling Action and Perception
… … …
… … …
Time
Action: poke
Perception: optical flow
![Page 54: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/54.jpg)
look
grasp
lift
hold
shake
audio (DFT)
proprioception (joint torques)
Color
drop
tap
poke
push
press
Optical flow
SURFproprioception (finger pos.)
Sensorimotor Contexts
![Page 55: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/55.jpg)
Sensorimotor Contexts
look
grasp
lift
hold
shake
audio (DFT)
proprioception (joint torques)
Color
drop
tap
poke
push
press
Optical flow
SURFproprioception (finger pos.)
![Page 56: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/56.jpg)
Feature Extraction: Proprioception
Joint-Torque values for all 7 Joints
Joint-Torque Features
![Page 57: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/57.jpg)
Feature Extraction: Audio
audio spectrogram
Spectro-temporal Features
![Page 58: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/58.jpg)
Feature Extraction: Color
Color Histogram (4 x 4 x 4 = 64 bins)
Object Segmentation
![Page 59: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/59.jpg)
Feature Extraction: Optical Flow
… … …
Angular bins
Co
un
t
![Page 60: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/60.jpg)
Feature Extraction: Optical Flow
… … …
![Page 61: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/61.jpg)
Feature Extraction: SURF
![Page 62: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/62.jpg)
Feature Extraction: SURF
Each interest point is described by a 128-dimensional vector
![Page 63: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/63.jpg)
Feature Extraction: SURF
Visual “words”
Co
un
t
![Page 64: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/64.jpg)
Dimensionality of Data
audio (DFT)
proprioception (joint torques)
ColorOptical
flowSURF
proprioception (finger pos.)
100 70 64 10 2006
![Page 65: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/65.jpg)
Data From a Single Exploratory Trial
look
grasp
lift
hold
shake
audio (DFT)
proprioception (joint torques)
Color
drop
tap
poke
push
press
Optical flow
SURFproprioception (finger pos.)
![Page 66: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/66.jpg)
Data From a Single Exploratory Trial
look
grasp
lift
hold
shake
audio (DFT)
proprioception (joint torques)
Color
drop
tap
poke
push
press
Optical flow
SURFproprioception (finger pos.)
x 5 per object
![Page 67: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/67.jpg)
Overview
Category Recognition Model
Sensorimotor Feature Extraction
Interaction with Object Category Estimates
…
![Page 68: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/68.jpg)
Context-specific Category Recognition
Observation from poke-audio context
Mpoke-audio
Recognition model for poke-audio context
Distribution over category labels
![Page 69: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/69.jpg)
• The models were implemented by two machine learning algorithms:K-Nearest Neighbors (k = 3)Support Vector Machine
Context-specific Category Recognition
![Page 70: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/70.jpg)
Support Vector Machine• Support Vector Machine: a discriminative learning algorithm
1. Finds maximum margin hyperplane that separates two classes
2. Uses Kernel function to map data points into a feature space in which such a hyperplane exists
[http://www.imtech.res.in/raghava/rbpred/svm.jpg]
![Page 71: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/71.jpg)
Combining Model Outputs
. . . .
Mlook-color Mtap-audio Mlift-SURF Mpress-prop.. . . .
Weighted Combination
![Page 72: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/72.jpg)
Model Evaluation: 5 fold Cross-Validation
Train Set Test Set
![Page 73: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/73.jpg)
Recognition Rates (%) with SVMAudio Proprioception Color Optical Flow SURF All
look 58.8 58.9 67.7
grasp 45.7 38.7 12.2 57.1 65.2
lift 48.1 63.7 5.0 65.9 79.0
hold 30.2 43.9 5.0 58.1 67.0
shake 49.3 57.7 32.8 75.6 76.8
drop 47.9 34.9 17.2 57.9 71.0
tap 63.3 50.7 26.0 77.3 82.4
push 72.8 69.6 26.4 76.8 88.8
poke 65.9 63.9 17.8 74.7 85.4
press 62.7 69.7 32.4 69.7 77.4
![Page 74: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/74.jpg)
![Page 75: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/75.jpg)
![Page 76: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/76.jpg)
![Page 77: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/77.jpg)
![Page 78: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/78.jpg)
![Page 79: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/79.jpg)
Distribution of rates over categories
![Page 80: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/80.jpg)
Can behaviors be selected actively to minimize exploration time?
![Page 81: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/81.jpg)
Active Behavior Selection• For each behavior , estimate such that
• Let be the vector encoding the robot’s current estimates over the category labels and let be the remaining set of behaviors available to the robot
![Page 82: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/82.jpg)
Example with 3 Categories and 2 Behaviors
A B C
A
B
C
A B C
A
B
C
A B C
Remaining Behaviors and Associated Confusion:
B1
Current Estimate:
B2
![Page 83: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/83.jpg)
Active Behavior Selection: Example
A B C
A
B
C
A B C
A
B
C
A B C
Current Estimate: Remaining Behaviors and Associated Confusion:
B1 B2
![Page 84: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/84.jpg)
Active Behavior Selection
![Page 85: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/85.jpg)
Active vs. Random Behavior Selection
![Page 86: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/86.jpg)
Active vs. Random Behavior Selection
![Page 87: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/87.jpg)
Discussion
What are some of the limitations of the experiment?
What are some ways to address them?
What other possible senses can you think of that would be useful to a robot?
![Page 88: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/88.jpg)
References
Sinapov, J., Bergquist, T., Schenck, C., Ohiri, U., Griffith, S., and Stoytchev, A. (2011) Interactive Object Recognition Using Proprioceptive and Auditory Feedback. International Journal of Robotics Research, Vol. 30, No. 10, pp. 1250-1262
Sinapov, J., Schenck, C., Staley, K., Sukhoy, V., and Stoytchev, A. (2014)Grounding Semantic Categories in Behavioral Interactions: Experiments with 100 Objects. Robotics and Autonomous Systems, Vol. 62, No. 5, pp. 632-645
![Page 89: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/89.jpg)
THE END
![Page 90: CS 378: Autonomous Intelligent Roboticsjsinapov/teaching/cs378/slides/...Project Deliverables • Final Report (6+ pages in PDF) • Code and Documentation (posted on github) • Presentation](https://reader033.vdocuments.us/reader033/viewer/2022060304/5f08fcf87e708231d424b144/html5/thumbnails/90.jpg)