images, words, and gestures ware chapter 9 university of texas – pan american csci 6361, spring...
TRANSCRIPT
![Page 1: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/1.jpg)
Images, Words, and Gestures
Ware Chapter 9
University of Texas – Pan AmericanCSCI 6361, Spring 2014
![Page 2: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/2.jpg)
Introduction: Pictures and Words
• Addresses relationships between visual information and verbal or textual information
• Dual coding theory
• When should we use a visual display?
• What is a visual language? – And does it make sense to use one to program a computer
• How to integrate images and words
![Page 3: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/3.jpg)
A Memory Demonstration
• Remembering words (and lots of things) is not easy
• Given a is a list of 10 words
• Try to remember them
![Page 4: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/4.jpg)
Method of Loci - a Mnemonic
• Remembering things, especially long sequences, not a new problem– Speech, tasks to do, names, …
• Dating back to Greek times, have been various mnemonic devices
• Method of loci, or places– Uses places and encoding of items– E.g., take a walk through your apartment, or across campus, or your parents’
house – some well known place• A “memory palace”
– In Renaissance, mol widely used, cathedrals served nicely
– Identify several points• E.g., 10 for the demonstration
– When a sequence is given, place each element at a (physical) point on the path• The more “memorable”, or bizarre, the image the better
– To retrieve the elements, walk through the physical place
• Memory demo again, but will do it using method of loci
![Page 5: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/5.jpg)
“How to Build a Memory Palace”http://www.wikihow.com/Build-a-Memory-Palace
• Decide on a blueprint for your palace. – The larger or more detailed the real place,
the more information you can store in the corresponding mental space.
• Define a route. – If you will need to remember things in a
certain order, it is essential that you follow a specific route through your palace, both in the real world and in you mind.
• Identify specific storage locations in your palace or along your route.
– Identify as many locations as you think you will need.
– Walk through your structure or along your route and really observe it.
• Memorize your memory palace.
• Place things to be remembered in your palace.
– Put a manageable amount of information in each place.
• Use symbols. – Generally, all you need to store in each
location is something that will jog your memory, something that will lead you to the actual idea you’re trying to remember.
• Be creative. – Generally, images will be more
memorable if they are absurd (out of the ordinary
• Stock your palace with other mnemonics.
– “Every Good Boy Deserves Fudge," which would in turn allow you to recall the order of notes on the lines in treble clef (EGBDF).
• Explore your palace. – Go through it and look at them.
• Use your palace. – just follow your route in order as you
do so.
• Build new palaces. – If you need only commit things to
memory for a short time. Just replace the existing contents with new ones
– If you need to remember the contents of your palace for a long time, you can keep that palace as it is and create new ones in which to store other information as needed
![Page 6: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/6.jpg)
Memory Demo, Again
• Same song – new technique
![Page 7: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/7.jpg)
Coding Words and Images: Two Systems
• Bertin, Semiology of Graphics (1983)– Two distinct sign systems
• Associated with auditory information processing– Includes mathematical symbols, natural language, music
• Based on visual information processing– Includes graphics, abstract and figurative imagery
• Pavio (1987), dual coding theory– Two different types of information in working memory
• Imagens – mental representation of visual information– Objects, natural groupings of objects, whole parts of objects (e.g., arm), spatial
information about layout/arrangement• Logogens – mental representation of language information
– Basic information pertaining to language, but not sounds of words– Processed by a set of functional subsystems that provide support for reading and
writing, understanding and producing speech, logical thought– Not necessarily tied to speech, e.g., deaf and Braille and sign language
![Page 8: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/8.jpg)
Pavio’s Dual Coding Theory, 1
• Account of memory (with implications for perception)– Refines/extends basic approach of 3-stage model of
human memory– Sharpens ideas about object recognition
• “Makes sense” – split between visual and verbal processing
– Long known are different neural processing centers for: • Verbal information (speech areas of temporal cortex) • Visual information (visual cortex)
• Core of rationale for use of visual representations– Especially to facilitate learning
• E.g., concrete mnemonic devices– Greek orators …– Walk through your house and put …
– Primary rationale for “multimedia”!• Provides account of why it works
– For our purposes, informs use of visual representation• With “words”, in larger context of diagrams, etc.• Sharpening up relation of “visual” with “semantic, …”
• Processing:– Visual-spatial information– Visual text– Acoustic verbal stimuli
![Page 9: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/9.jpg)
Pavio’s Dual Coding Theory, 2
• Processing:– Visual-spatial information
• Enters through visual system– Fed into association
structures in nonverbal imagen system
– Visual text• Processed,
– But then fed into association structures of logogens
– Acoustic verbal stimuli • Processed in auditory system
– Then fed into logogen system
• Logogens and imagens can be strongly interlinked
– E.g., word “cat” and language-based concepts related to cats will be linked to visual information related to the appearance of cats
– Method of loci, multimedia, etc.
•Imagens – mental representation of visual informationObjects, natural groupings of objects, whole parts of objects (e.g., arm), spatial information about layout/arrangement
•Logogens – mental representation of language informationBasic information pertaining to language, but not sounds of wordsProcessed by a set of functional subsystems that provide support for reading and writing, understanding and producing speech, logical thoughtNot necessarily tied to speech, e.g., deaf and Braille and sign language
![Page 10: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/10.jpg)
“Thinking” Visually
• Usually associate processes of “thinking”, e.g., greater than, follows from, “logic”, etc., with “verbal” (logogen based system)
• More recently, such operations as part of the “visual” (imagen based system) are finding evidence
– E.g., Subjects report using imagery when compare sizes of light bulb and tennis ball, color (green) of pea and Christmas tree
– Positive emission tomography (PET) evidence• Visual processing centers active when imagery
invoked• Buy, when mentally change size and position of
imagined object, different visual areas of brain activated
• Imagery alone vs. operations on (processing, thinking with) imagery
– When see a cow and imagine a cow same neural pathways (in part) excited
– In accord with accounts of object recognition in which object is “recognized” through interaction of stimuli (bottom up) and memory (top down)
• See figure at right– Visual memory traces stored as part of processing– Hence, recognition is easier than recall
• Matching with something stored (which in part drives recognition)
• Vs. reconstructing all pieces from input of event alone
![Page 11: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/11.jpg)
The Nature of Language(s)
• Noam Chomsky:– Innate deep structures (meaning)– Surface structure (syntax, form)
• That nature of natural (everyday, spoken) language is quite similar to formal description of languages appropriate to describe all languages (natural and formal) is among larger advances of 20th century
– Universality of human language– And, e.g., Chomsky hierarchy of (deep)
interest to computer scientists
• Critical period for language development
• But being verbal is not essential to language development
• Sign languages for the deaf are the most perfect examples of visual language
![Page 12: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/12.jpg)
What is Language?
• Language provides:– Description – Communication of intention – Ability to communicate procedures
and sequences of operations – including logic
• if, but, causes, do a then b then c
• Sign Languages – Arose spontaneously – Are not related to verbal
languages – Have syntax – Become more abstract– To be fluent in visual (or any)
language we should be trained from early in life
![Page 13: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/13.jpg)
Again, Visual Languages (or Not)
• Recall, some fundamental questions addressed in first of class
– Sensory and arbitrary symbols, etc.– Visual language
• Consider that hieroglyphs gave way to more abstract symbols
– Why turn back the clock?
• Can there be a true visual language?– Yes, but not for most of us!– A critical (developmental) period
• ontogenetic– Consider verbal language– Abstraction, logic
• (if, while, perhaps) – Based on speech– Sign languages are true visual languages
![Page 14: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/14.jpg)
Visual and Verbal Pseudo-code
• Ware argues that visual languages in fact not good for expressing sequential operations common to computer programming,
• Sequence below:– Get a line of text– Change characters to all upper case– Write line to output file– While there is more input
• Better expressed by “verbal psuedo-code”:Repeat
Get a line of textChange characters to all upper caseWrite line to output file
Until there is more input
• Flow charts were wrong– Probably just wrong paradigm
• But, billions of dollars spent– Cause inappropriate/unnatural focus on detail
• Visual programming languages have history of failure
![Page 15: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/15.jpg)
Still, Certainly Uses for Visual Represt.
• Gives rapid recognition and pattern finding (again)
• Has distinct advantages over text (linear, serial) for conveying some kinds of information
• Consider the text below:– Jane is Jim’s boss – Jim is Joe’s boss – Anne works for Jane – Mark works for Jim – Anne is Mary’s boss – Anne is Mike’s boss
• And it’s visual representation
![Page 16: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/16.jpg)
When to Use Static Images vs. Words
• In general (rules of thumb)– Images are better than words for:
• Spatial structures• Location• Detail
– Words better for:• Procedural information• Logical condition• Abstract verbal concepts
• Images best for showing structural relationships– Links between entities and groups of entities
• E.g., bus routes shown as graphical representation led to better performance in trip planning than with tables
• Visual information generally remembered better than verbal, but not for abstract images– Visual information need be meaningful and capable of incorporation into a cognitive framework for this
advantage– Image memory can’t be so relied on if information is new and out of context
• Images best for providing detail and appearance– Amount of detail extracted (and remembered) depends on time to study
• Recall, silhoettes first, so line drawing best for rapid extraction
• Information that specifies conditions under which something should or should not be done is better provided using text or spoken language
![Page 17: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/17.jpg)
Gestures, and Linking Images and Words
• Deixis and the deictic gesture– Gesture that links subject of a spoken sentence with a visual reference – Can be a glance or a nod – Pre-speech – Shown to disambiguate verbal communications
• Why the mouse is so powerful …
• Other kinds of gestures– Beat gestures for emphasis – Verb gestures showing how to do something – McNeil Hand and mind
• Issues in shared environments– Speech + Pointer + Visuals – most important components – Subtle ways of directing attention also important in meeting dynamics.
• Linking images and words– Deixis– Pointing is an elementary speech act. – Pointing links images and words – Put that (points) there (points) – Subject verb predicate
![Page 18: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/18.jpg)
Attaching Words to Images
• Turns out the use of words and images together work just fine – Or even quite well
• “Association” (grouping) can be explained by Gestalt principles
![Page 19: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/19.jpg)
Examples of Integrated Pictures and Words
• Examples
![Page 20: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/20.jpg)
Another Ex. of Integrated Pictures and Words
• More examples
![Page 21: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/21.jpg)
Yet Another Ex. of Integrated Pics. and Words
![Page 22: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/22.jpg)
Examples of visual languages
• As noted, limited success of visual languages, especially for programming
• Still, some examples– Sanscrit , Petri-nets, Khoros
![Page 23: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/23.jpg)
Sanscrit
• To program:– Count from 1 to 3– for i = 1 to 3 do
![Page 24: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/24.jpg)
Petri Net Language
• Petri nets are stochastic– timed attributed (tokens on
nodes, transitions)
![Page 25: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/25.jpg)
Khoros
• Combine operations
![Page 26: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/26.jpg)
Conclusions
• Interesting, and not fully understood interaction between the two processing subsystems– “Visual”, images, imagens– “Verbal”, auditory, logogens
• Though certainly there can be “visual languages”, finding useful applications has been elusive
![Page 27: Images, Words, and Gestures Ware Chapter 9 University of Texas – Pan American CSCI 6361, Spring 2014](https://reader035.vdocuments.us/reader035/viewer/2022062421/56649e165503460f94b00528/html5/thumbnails/27.jpg)
End
• .