16-785: integrated intelligence in robotics: vision...
TRANSCRIPT
![Page 1: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/1.jpg)
16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning
Spring 2018
Lecture 01. Introduction
![Page 2: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/2.jpg)
Instructor Name: Jean Oh (Preferred to be called “Jean”) Office: NSH 4521 Email: [email protected] Prefix subject line with [16-785]
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 2
![Page 3: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/3.jpg)
Course objectives • Learn how to develop intelligence for robots – Learn good old technologies that are still useful – Catch up with new technologies that have been
published recently
• Learn how to start and complete a project from proposal to execution
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 3
![Page 4: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/4.jpg)
Academic integrity • CMU academic integrity policy – http://www.cmu.edu/policies/student-and-
student-life/academic-integrity.html
• Acknowledge if you receive help from others
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 4
![Page 5: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/5.jpg)
About intellectual property • Software is easy to steal – Copy & paste – Rewriting
• Acknowledge sources of information • Learn to read license terms before using
open-sourced software • Never take credit for someone else’s work
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 5
![Page 6: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/6.jpg)
Reasonable Person Principle (RPP) • Base culture of CMU School of Computer Science • Everyone gives/gets the benefit of doubt for trying
to be reasonable o Everyone will be reasonable.o Everyone expects everyone else to be reasonable.o No one is special.o Do not be offended if someone suggests you are
not being reasonable.
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 6
![Page 7: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/7.jpg)
Extensions and late assignments • Everything is due BEFORE class begins on
due date • 5 no-penalty late days: use them anyway you
want except for final project presentation & report
• 50% deduction within 24 hours from due • 0% after 24 hours past due
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 7
![Page 8: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/8.jpg)
Grading • Homework 30% • Class participation 30% • Class project 40%
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 8
![Page 9: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/9.jpg)
Homework (30%) • 10 (+1) reading assignments (3 pt each) • 1 free pass; if used student must attend the class that
discusses the papers • For each paper write 1-2 sentences for:
– Problem definition – Technical challenges – Summary of approach/main ideas – Evaluation methods and results – Discussion points
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 9
![Page 10: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/10.jpg)
Class participation (30%) • Present paper & lead review classes (20%) – At least two papers per student – Sign up on Canvas (Google doc)
• Active participation during class (10%) – Attending – Raising or answering insightful questions – Sharing additional literature review on related topic
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 10
![Page 11: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/11.jpg)
Questions about class participation?
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 11
![Page 12: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/12.jpg)
Class project (40%) • Proposal (10%) – Page limits: 5 (1 extra page for references)
• Midterm report (10%) – Page limits: 5 (1 extra page for references)
• Final presentation (10%) – 20 minutes
• Final report (10%) – Page limits: 10
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 12
![Page 13: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/13.jpg)
Project topics • Requirement – Must be interdisciplinary – 1-3 members per team
• Examples – Vision + language – Planning + language – Vision + planning – Vision + language + planning
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 13
![Page 14: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/14.jpg)
Class project: proposal (10%) • Maximum 5 pages • Project title (Use self-explanatory titles) • Problem definition • Technical challenges • Proposed approach • Milestone schedule (include time commitment of each member) • Expected outcomes • Team members
– Contact information – Bio sketch for each member describing one's technical background and
intended contributions to the project • 10-minute team presentation in class 1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 14
![Page 15: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/15.jpg)
Class project: midterm report (10%) • In-depth literature review on existing work • Progress report (maximum 5 pages) – Detailed technical approach – Schedule update (what has/hasn’t been done) – Detailed plan for experiments – Preliminary results
• 10-minute team presentation in class
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 15
![Page 16: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/16.jpg)
Project final report (10%) • Maximum 10 pages • Yes, it can be build from your midterm report but additionally
include: – Abstract – Complete description of technical approach – Evaluation methods and/or experimental setup – Final results – Technical contributions – Limitations – Future directions
• Project website (Extra points)
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 16
![Page 17: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/17.jpg)
Class project: final presentation (10%)
• 20 minutes per team • Motivation • Problem definition • Existing work • Proposed approach • Results • Future directions
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 17
![Page 18: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/18.jpg)
Questions about projects?
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 18
![Page 19: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/19.jpg)
Course topics • Planning, scheduling, and learning • Vision: image understanding • Language: sequence representation • Vision + language • Vision + planning • Vision + language + planning
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 19
![Page 20: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/20.jpg)
Schedule
1/16/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 20
Introduc;on Intelligence design for problem solving
Reinforcement learning Human guidance Project proposal IRL Deep learning Language-‐planning
1/22 Planning Project proposal Problem formula;on Iden;fying challenges
1/29 Reinforcement learning
2/5 RL with guidance Project design
2/12 Project proposal presenta3ons
2/19 IRL (Katharina Muelling)
2/26 AlexNet, Incep;on, VGG
3/5 Language grounding
1/17 Introduc;on
1/24 Overview papers
1/31 Applica;ons
2/7 Human guidance
2/14 IRL
2/21 Deep learning basics
2/28 Scheduling (Steve Smith)
3/7 Deep IRL
Image understanding Sequence learning Vision-‐language Image synthesis, plan to language
Architecture Evalua;on Project final presenta;ons
3/19 CNN
3/26 RNN LSTM
4/2 Image cap;oning
4/9 GANs
4/16 Intelligence architecture
4/23 User study design (Minkyung Lee)
4/30 Project final presenta3ons
3/21 U-‐Net, DenseNet, DeepLab
3/28 Project midterm presenta3ons
4/4 word2vec
4/11 Natural language direc;on following
4/18 Metrics
4/25 Levine end to-‐end JMLR’16
5/2 Project final presenta3ons
![Page 21: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/21.jpg)
Planning, Scheduling, & Learning
• AI planning • Motion planning • Reinforcement learning • Inverse reinforcement learning • Scheduling
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 21
“Convert high-level specification of tasks from humans into low-level descriptions of how to [move]” (LaValle’06)
![Page 22: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/22.jpg)
Vision: image understanding • Convolutional neural nets • State of the art networks in 2012 – 2017 – AlexNet’12, Inception’15, VGG’15, U-Net’15,
DeepLab’17, DenseNet’17
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 22
![Page 23: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/23.jpg)
Vision + Language • Image region à keywords / labels – Image classification – Object detection – Scene classification
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 23
CAT DOG
![Page 24: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/24.jpg)
Image captioning
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 24
“boy is doing backflip on wakeboard.” [Kaparthy & Fei-Fei ’15]
”robot is doing backflip in a garage.”[Image: Boston Dynamics Atlas]
![Page 25: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/25.jpg)
Vision + Language • Vision à Language – Image captioning with domain knowledge – Describing images with various sentiments
• Language à Vision – Semantic understanding based on description,
e.g., object detection, semantic map building – Image synthesis given language description
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 25
![Page 26: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/26.jpg)
CAT
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 26
![Page 27: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/27.jpg)
CAT
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 27
There is a cat siang by the window. A kicen is looking out the window from the red brick house. The cat in the green window frame looks s;ll as if it is in a pain;ng. The cat is posi;oned perfectly in the center of the window pane, looking outside acen;vely. A licle cat is siang by the window of an old brick house. The wooden window pane looks painted over many ;mes. A curtain is closed behind the cat, so nothing else can be seen from the outside.
![Page 28: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/28.jpg)
Sketch demo
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 28
[Eitz et al., SIGGRAPH’12]
• Interactive recognition• Sketch-based image
retrieval• Art recognition
![Page 29: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/29.jpg)
Image to image
1/17/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 29
[Isola et al., CVPR’17]
• Scene classification• Map generation• Sketch-based image
retrieval• B&W to color• Day to night
![Page 30: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/30.jpg)
Language to image Create composite sketch from description
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 30 [Google image search results for “composite sketch”]
A tall male in his mid-40s wearing a dark-colored hat…
![Page 31: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/31.jpg)
Language to image Creating art work from description [Songeun Lee, 2011] https://vimeo.com/17456055
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 31
“It’s got quite a small head and quite a big body.”
“It has two pointy ears.”
![Page 32: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/32.jpg)
More vision-language applications
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 32
Understanding and describing environments, people, and ac;vi;es
Fashion
Law enforcement
Military
Social robots
![Page 33: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/33.jpg)
Vision + planning
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 33
Crusher [Silver et al., IJRR 2010]
![Page 34: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/34.jpg)
Vision + planning
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 34
Silver et al., “Learning from Demonstra;on for Autonomous Naviga;on in Complex Unstructured Terrain,” IJRR 2010
![Page 35: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/35.jpg)
“Navigate left of the building to the traffic barrel that is behind the building.”
35
![Page 36: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/36.jpg)
Plans made by humans “Navigate lej of the building to the traffic barrel that is behind the building.”
36
![Page 37: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/37.jpg)
Language + vision + planning
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 37
“Navigate lej of the building to a traffic cone behind the building.” 1. When the command is given, the robot sees only 4 front walls of the church building. 2. From these wall segments, a complete geometry of a building is predicted. 3. Symbols in TBS (building and traffic cone) are grounded. Because traffic cones haven’t been detected, the robot uses the contextual clue from
the command, e.g., “behind the building” to select an unknown object behind the building. 4. Ini;al plan is generated, considering “lej of building” spa;al constraint. 5. The robot hasn’t found any traffic cones behind the building; it con;nues with the ini;al plan. 6. The robot detects a traffic cone and re-‐plans towards the correct goal. [Oh et al., AAAI 2015]
![Page 38: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/38.jpg)
Semantic Navigation (robot view)
“Toward mobile robots reasoning like humans” in AAAI 2015,J. Oh, A. Suppe, F. Duvallet, A. Boularias, L. Navarro-Serment, M. Hebert, A. Stentz, J. Vinokurov, O. Romero, C. Lebiere, R. Dean. 38
![Page 39: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/39.jpg)
Language + planning • Incorporating human inputs into
algorithmic planner • Generating planning heuristics from
manual • Hypothesizing environment from
description
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 39
![Page 40: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/40.jpg)
“Stay off the grass; go to the back of the shed.”
40
![Page 41: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/41.jpg)
Explaining decisions • Describing plans in natural language • Explaining the rationale behind decisions – I think that it is a cat because ...
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 41
hcps://www.pinterest.com/pin/285486063854038268
![Page 42: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/42.jpg)
42
Instruction following in manipulation Joint work with Katharina Muelling, Tushar Chugh, Shiyu Dong, Bikram Hanzra, Tae-Hyung Kim, & William Seto
![Page 43: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/43.jpg)
Conversational mobile robots
1/14/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 43
Joint work with Matthew Wilson & Ralph Hollis
![Page 44: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/44.jpg)
AI + Art • GANGogh
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 44
hcps://github.com/rkjones4/GANGogh
![Page 45: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/45.jpg)
Robotics + Art
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 45
[Niklas Roy, 2011]
![Page 46: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/46.jpg)
Questions about course topics?
1/17/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 46
![Page 47: 16-785: Integrated Intelligence in Robotics: Vision ...jeanoh/16-785/lectures/IIR-01-Introduction.pdf · 16-785: Integrated Intelligence in Robotics: Vision, Language, and Planning](https://reader031.vdocuments.us/reader031/viewer/2022021817/5a7f708d7f8b9aee018b90e7/html5/thumbnails/47.jpg)
Homework & logistics • Sign up for paper presentation on Canvas
(Google doc) • Read overview papers and write summary • No office hour this week (email me for any
questions)
1/15/18 CMU 16-‐785: Integrated Intelligence in Robo;cs: Vision, Language, and Planning 47