vincze cognitive vision fermo2006 - univpmpsfmr.univpm.it/slide/vincze_cognitive_vision.pdf ·...
TRANSCRIPT
![Page 1: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/1.jpg)
Cognitive Vision
Markus VinczeAutomation Control Institute
Vienna University of [email protected]
www.acin.tuwien.ac.at
PSFMR – Fermo, 11.-16.9.2006
![Page 2: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/2.jpg)
Idea of Today• Overview of Cognitive Vision Methodology
• Scratch at cognitive science and cognitive systems
• Open your view to other disciplines
• Point out many open problems that are simply awaiting a good student for resolution
![Page 3: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/3.jpg)
Content• Overview
• Tracking
• Detection
• Cognitive Vision– Vision systems– Integration
– Computer Vision Cognitive Vision ...
– Cognitive Systems
![Page 4: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/4.jpg)
• Ambient Intelligence
• “Natural” computer interfaces (e.g., MIT, MS)
• Japan: developmental (humanoid) robotics
Ideas, Drives, Future(s)
STARTREKOdyssey 2001
MIT icom MS EasyLiving
IPA
![Page 5: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/5.jpg)
Personal Assistance• Support user by being aware of situation
• Distributed mobile and ambient devices
Example situations:
• Information assistance, guidanceto location, assembly help
• Alerting of dangerous situations
![Page 6: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/6.jpg)
Personal Assistant• User guidance to
– operate a machine (e.g., copy machine, video/CD-player)
– assemble objects (e.g., furniture, machine maintenance)
• Exploit Augmented Reality to display information
• On-line interpretation to aid user
![Page 7: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/7.jpg)
Personal Assistance – Ingredients
Capabilities• Detection, tracking,
recognition, spatio-temporal reasoning, ...
• Interpret human intentions before acting
• Personalised behaviour
Austrian CV Project: understandand react to situations
![Page 8: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/8.jpg)
Robot Helper g„James, please bring me my cup“
Capabilities• Navigate, avoid obstacles• Detect & recognise objects• Grasp objects• Interact with user• Cope with new situations• Dependable and safe behaviour
![Page 9: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/9.jpg)
Vision for Interaction
Capabilities• Robust detection, tracking• Object and gesture
recognition• Spatio-temporal object
relationships (3D)• Interpretation, understanding
ActIPret: interpretation of humans who handle objects
![Page 10: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/10.jpg)
Cognitive Vision ComponentsEU project ActIPret
Object recognition(CMP)
Robust object
detection
Stereo hand tracking (FORTH)
Object tracking
![Page 11: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/11.jpg)
Cognitive Vision ComponentsEU project ActIPret
Hand gesture recognition (COGS)
Spatio-temporal object reasoning (in 3D)
'Hand 0 picked up object cd-linux-0'Semantic interpretation
![Page 12: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/12.jpg)
GUI – Graphical User Interface
On-line display of 3D results(trajectories, recognition and
interpretation results, )
'Hand 0 pressed button ejectButton-2''Hand 0 picked up object cd-linux-0'
Stereo obser-vation
Off-line VR replay of activity
EU project ActIPret
![Page 13: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/13.jpg)
![Page 14: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/14.jpg)
MOVEMENT• Movment of
– Persons, objects, information
• Stereo vision for navigation– Segement floor
– Obstacle detection
– Detection of tables and chairs
person
Infor-mationobject
MOVEMENT – EU IST Projekt 2004-2007
![Page 15: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/15.jpg)
Vision Capabilities• Vision can provide many capabilities
• Vision itself has many capabilities (redundancies)– Temporal redundancy
– Stereo, many views
– Many cues per image
– Vast number of features
– Multiple representations
• Integrate with other system functions
![Page 16: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/16.jpg)
Summary• Many, many vision (perception) capabilities
• Capabilities operate in context
A consequence:
• To solve even a simple task ⇒ system
Another consequence:• More than vision – e.g., cognitive vision
• Integration – architecture and tool
![Page 17: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/17.jpg)
Content• Overview
• Tracking
• Detection
• Cognitive Vision– Vision systems
– Integration– Computer Vision Cognitive Vision ...
– Cognitive Systems
![Page 18: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/18.jpg)
Integration: Vision is not Alone• Other sensors
– odometry, distance, touch
– Time-of-flight, ultrasound, infrared, ...
– acoustic, olfactory, ...
• „Envisioned“ embodiment
• Task, situation
• Knowledge representations, common sense
• Semantics, language
![Page 19: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/19.jpg)
System Requirements• Task and data-oriented
• Context-based
• Reactive control
• Enable distribution
• Separated development
• Modular + scaleable
• Reusability (!)
![Page 20: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/20.jpg)
Some Options• Dataflow: pipes & filters [Unix shell]
• Layer architectures [OS, ISO-OSI]
• Object-oriented [Corba, ...]
• Event-driven [HMI]
• Shared data: blackboard [DBs]
• Agent-based [AI]
• Component based [Software engineering]
![Page 21: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/21.jpg)
Integration based on Components• Components encapsulate functions
– Service principle - „Yellow pages“– Dynamic linking
• Reusable, distributed, scalable
• Simple (installation, programming)
• „Fast“
![Page 22: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/22.jpg)
Capabilities of Each Component• Function
• Communication
• Memory
• Self-evaluation (reports confidence, accuracy, resource demands)
• Control (processing, view)
• Context (exploit it, report it)
ComponentComponent
ComponentComponent
ComponentComponent
ComponentComponent
ComponentComponent
![Page 23: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/23.jpg)
Control& Data
Data
Re-active Dynamic Integration
ComponentComponent
Component
Component
Component
Component
Component
•• Avoiding negotiationAvoiding negotiation
![Page 24: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/24.jpg)
Example Architecture for ActIPret
![Page 25: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/25.jpg)
Task-related Space of Interest
![Page 26: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/26.jpg)
Zwork (Zillich‘s network)• RPC (Remote Procedure Calls)• Asynchronous• Automatic marshalling of messages• Simple debugging (gdb/ddd)• Logging• GUI Component
Service ProviderService ProviderService Provider
Service Requester
managingframe
Implemen-tation;
providerinterface
requesterinterfaceComponent structure.
![Page 27: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/27.jpg)
• Every lab has another approach:– Definition and notation of functions, skills (and context)
• Key: practicability + ease-of-use
• „Standard“ interface definitions– Get more specific along project
• Integration learn to build systems– How do parts work together?
– Learn which parts work together.
Conclusion Integration
![Page 28: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/28.jpg)
Content• Overview
• Tracking
• Detection
• Cognitive Vision– Vision systems
– Integration
– Computer Vision Cognitive Vision ...– Cognitive Systems
![Page 29: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/29.jpg)
Computer Vision• Computer Vision is a subfield of AI concerned with
processing of images from the real world.
• Purpose: program a computer to "understand" a scene or features in an image.
• Methods: detection, segmentation, tracking, pose estimation, mapping to 3D model, recognition of objects in images (e.g., human faces, robot navigation)
• Achieved by means of pattern recognition, statistical learning, projective geometry, image processing, graph theory and other fields.
![Page 30: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/30.jpg)
Pattern Recognition• "the act of taking in raw data and taking an
action based on the category of the data" [1]
• Goal: detect and learn known patterns
• Methods: statistics, machine learning, ...
[1] Richard O. Duda, Peter E. Hart, David G. Stork (2001) Pattern classification (2nd edition), Wiley, New York.
![Page 31: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/31.jpg)
Computer Vision – A SummaryMany solutions and many more problems, e.g.:
1. Real world?
2. Brittle, thresholds.
3. „Understand“ scene?
4. „Understand“ features?
5. Segment ⇔ recognise?
6. Replication of experiments?
7. Formal description of capabilities?
![Page 32: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/32.jpg)
Computer Vision – Lessons Learned
Serious work with real world images.
Robust, threshold-free methods.
To „understand“ act in scene.
Huge set of features.
Segment AND recognise.
PETS, DBs.
... .
Problems: start to work on the core problems, e.g.:
1. Real world?
2. Brittle, thresholds.
3. „Understand“ scene?
4. „Understand“ features?
5. Segment ⇔ recognise?
6. Replication of experiments?
7. Formal description of capabilities?
![Page 33: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/33.jpg)
Machine Vision• = application of computer vision to factory
automation.
• A MV system is a computer that makes decisions based on the analysis of digital images.
LightLighting Object Sensor
Image
DataResultsControl
Media Reflection
Processing
Figure: Components of a machine vision system.
![Page 34: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/34.jpg)
Machine Vision• Problem 1: narrow applications
• Problem 2: understanding?
• Lesson learned: more options to control
• Lesson learned: consider complete system
LightLighting Object Sensor
Image
DataResultsControl
Media Reflection
Processing
Figure: Components of a machine vision system.
LightLighting Object Sensor
Image
DataResultsControl
Media Reflection
Processing
![Page 35: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/35.jpg)
Cognitive Vision
Cognitive Vision
• embodiment (AmI, PDA, AR, VR, robotics)
representations •
computer vision •
• system architecture
• machine learning
• user experiments, usability
neuro science •information theory •
• cognitive science
• artificial intelligence
systems engineering •
Cognitive science Computer Vision
Cognitive Vision
![Page 36: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/36.jpg)
Cognitive Science• = scientific study either of mind or of
intelligence, it is inherently interdisciplinary – E.g., psychology, neuroscience, linguistics,
philosophy, computer science, and biology
• Cognition = „coming to know“ = act of acquiring knowledge
• be aware of and judge the result of this act
• "cognitive" - any kind of mental operation or structure that can be studied in precise terms[Lakoff, Johnson, 1999]
![Page 37: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/37.jpg)
Cognitive (Computer) Vision One definition:
CV is the act of seeing to obtain empirical factual knowledge– Act: some form of body, self-awareness,
communication, evolving
– Seeing: all computer and biological vision has to offer
– Empirical: based on observation and experiment
– Factual: objective reality, repeatable
– Knowledge: facts acquired, models, procedures
![Page 38: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/38.jpg)
Cognitive (Computer) VisionAnother definition of Cognitive Vision
• 4 levels of generic computer vision functionality– Detection, localization, recognition, and understanding
– Purposive goal-directed behaviour
– Adapting to unforeseen changes of environment
– Anticipate the occurrence of objects or events.
• Achieves capabilities through – Learning semantic knowledge (i.e. contextualized
understanding of form, function, and behaviour)
– Knowledge about environment, itself, and relationsips
ECVision (2002 - 2005), www.ecvision.org
![Page 39: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/39.jpg)
Why Cognitive Vision?• Old terms did not succeed, new term might
• Interdisciplinarity is something good
• „New“ understanding:– Active (since 1987)
– Learning, evolving
– Embodied (envisioned)
• 1000x more computing power in last 15 years
• Understanding cognitive science
![Page 40: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/40.jpg)
Cognitive Vision - Essentials• Seeing: eyes, head, vision data processing
• See what? Objects, humans, environment– „Come to know“ about them
– Only knowledge relevant to seeing
• Act of seeing upon – what is visible or
– becomes visible before it becomes invisible again
![Page 41: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/41.jpg)
For Example: Hide & Seek
Kind.mov
• Key: combining top-down (cognitive) with bottom-up (vision) processes
![Page 42: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/42.jpg)
Cognitive Vision - Challenges• Object permanence (hiding but existing)
• Spatial and dimensional awareness (close or far range, spatial relationships, stacking objects)
• Temporal awareness (synchronous events, e.g., pointing)
• Hierarchical object concepts
• Detect something new
• Awareness of camera/body (view point reasoning, self-localisation)
![Page 43: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/43.jpg)
Cognitive Vision System (CVS)• Instantiation of the bits and pieces necessary
for cognitive vision– System = interacting group of items forming a
unified whole [Merriam Webster]
• Is it only the „seeing“ part of a system?
• Action? How much action?
• Body? How much body?
![Page 44: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/44.jpg)
CVS as part of a Cognitive System
![Page 45: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/45.jpg)
Cognitive Vision System• Vision under egomotion (e.g., + inertial sensors)
• Interaction with other (vision) systems
• Hand-eye coordination (throw objects, eye-body coordination)
• Interpreting gestures
• Search for sounds with eyes (+ auditory cues)
• Viewing the world as seen from a third person’s perspective (= “perspective taking”)
![Page 46: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/46.jpg)
More Challenges• Representations: objects, relationships,
situations, context, „visual“ semantics
• Understanding: function and use of objects
• System: support multiple tasks & autonomy
• Real world: work with it and use it (context)
![Page 47: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/47.jpg)
Possible Methodological Approach• Cognitive reference: people
• Learn from system evaluation by people
• Build system close to people, i.e., in their environment
• Learn from, not copy, biological vision systems
![Page 48: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/48.jpg)
Content• Overview
• Tracking
• Detection
• Cognitive Vision– Vision systems
– Integration
– Computer Vision Cognitive Vision ...
– Cognitive Systems and examples
![Page 49: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/49.jpg)
• Foresight Cognitive System Project, UK– natural or artificial information processing systems
– perception, learning, reasoning and decision making
– communication and action
Cognitive Systems• FP6, EU
– physically instantiated (embodied) systems
– perceive, understand (semantics) and interact
– evolve in order to achieve human-like performance in activities requiring context specific knowledge.
![Page 50: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/50.jpg)
Interdisciplinarity of CognitiveSystems and Cognitive Science• E.g., psychology, neuroscience, linguistics,
philosophy, computer science, biology, …
• Human as reference for cognitive capabilties
• Several examples– Navigation
– Vision
• Source of inspiration
![Page 51: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/51.jpg)
Path Integration in Insects
[Mallot]
![Page 52: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/52.jpg)
Path Complexity does not Impair Visual Path Integration
• Path segments in VR
• No effect on number of segments
• Directionand distance encoding?
[Wiener]
![Page 53: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/53.jpg)
Geographical Slant as Compass• Ground plane
slanted 4 degrees– Perceived visually
and via force feedback
• Pointing the right way becomes easier
[Restat, Mallot]
![Page 54: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/54.jpg)
Orientation in Children• Children re-orient by the shape of the room
• Sensitive to surface layout: distance, angle, sense
• Do not user landmarks for orientation
• Landmarks are detected and remembered
[Gouteux, Spelke, 2001]
![Page 55: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/55.jpg)
Orientation in Adults• Landmarks are used and described verbally
• Not used with verbal interference
• With interference, adults become like rats.
• Orientation on surfaces: children, rats, fish
[Ratliff, Necombe, 2005]
![Page 56: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/56.jpg)
SLAM?• Do humans SLAM?
• Orientation based on main structure
• Icon-based navigation– Plus obstacle avoidance
• Knowledge about what to expect– On airport, train station, ...
![Page 57: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/57.jpg)
Cognition as Control• Hierarchy to cope with complexity
[Hollnagel]
![Page 58: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/58.jpg)
Neuroscience• Study of the human nervous system, brain, and
biological basis of consciousness, perception, memory, and learning
• Brain has a triad structure – reptilian brain controls basic sensory motor functions
– mammalian brain: emotions, memory, biorhythms
– neocortex or thinking brain that controls cognition, reasoning, language, and higher intelligence
• Continued reconnecting and learning – Learn from real experiences, integrated "whole" ideas
![Page 59: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/59.jpg)
Cortex – Examples• About 30 regions involved in vision, half of
cortex
• MT: detecting areas of motion in images (0.1 s after motion is in image)
• V1: cells respond to oriented edges
• V1: BUT 85% of axions come not from retina
• Hippocampus – place cells, direction cells
• Cognitive map [Tolman 1948]
![Page 60: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/60.jpg)
Illusions• Study human vision
system
• Experience of eye: world is benign– Counter example: Gorilla
• Computer Vision suffers from serial processing– Human: all cues in parallel
– Subsequent fight for what is most plausible
![Page 61: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/61.jpg)
Object Perception• Spatio-temporal constraints to form objects
(4-month olds)
[Spelke: Principles of Object Perception, 1990]
![Page 62: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/62.jpg)
Human Vision Learns• Child: perceived
as one object
• First:motion, surfaces (see before)
• Later: shape, appearance
[Spelke]
![Page 63: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/63.jpg)
Gestalt Laws• Develop in humans
• Occlusion: completion depends on experience
• Criteria– Good continuation
– Similarity
• Animals?
[Spelke]
![Page 64: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/64.jpg)
Chicks• Perception of occluded objects without
experience of occlusion
• Inborn object completion
[Regolin]
![Page 65: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/65.jpg)
Number• Multiple Object Tracking
– A) Following several moving targets
– B) Connections disrupt expectations
– C) Too many to follow
• Set size limit: 3-4– Children, adults, animals
• Perception– Cohesion, contact, continuity
– Auditory set size limit: 3-4[Scholl, Wynn, Mittroff, van Marle]
A)
B)
C)
![Page 66: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/66.jpg)
Sensitivity to Geometry• Response to geometrical relationsships
[Dehaene, Izard, Pica, Spelke, 2006]
![Page 67: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/67.jpg)
Sensitivity to Geometry – Results • Strikingly similar patterns
– Munduruku live in Amazonas region
– Adults in Boston improve over children
• Relationships represent Euclidian geometry
![Page 68: Vincze Cognitive Vision Fermo2006 - UNIVPMpsfmr.univpm.it/slide/Vincze_Cognitive_Vision.pdf · 2006-09-13 · • Spatial and dimensional awareness (close or far range, spatial relationships,](https://reader034.vdocuments.us/reader034/viewer/2022042222/5ec843b78f61130c5b201aba/html5/thumbnails/68.jpg)
Final Conclusion• Human is the only working vision and
cognitive system
• Cognitive Science and related fields throw some light on how it may work inspiration
• Cognitive science tries to put it all together
Experience, recommendation:• Design methods without parameters
• Work with system, not individual components