Chapter 5: Perceiving Objects and Scenes

Why do we want to understand how we perceive objects?

1. For basic understanding of how we function. For science.

2. For designing procedures that can facilitate object recognition in instances of disability.

Artificial eyes or artificial ears for persons without them

We would try to create the stimuli most salient for object perception.

Think about how an eye could be replaced. Where would we intervene?Receptor implants? Ganglion cell impants? LGN implants? Cortical implants?

There is no point in designing an electronic apparatus to create neural activity if it is not the kind of neural activity that will be useful for object recognition.

For example – we’ll see in the study of hearing that cochlear implants, replacing absence hearing receptors, have benefitted considerably by knowledge of how the hearing receptors actually work.

3. For creating machines that can perform object recognition independently of us.

Optical Character Recognition (OCR): Scanning a printed document and converting the images to editable text.

Speech recognition: Words are objects, too.

Security – identifying only those persons who can access a guarded room. Identifying each person as being unique from every other person using voice, iris, fingerprint, brain waves?.

Visual guidance: Machines that can recognize objects in the terrain and guide themselves. Cars that can drive themselves. Mars Rovers that can cover terrain without human intervention.

Topic 10: Perception of Objects- 1 5/23/2023

Play the Optogenetics video here.C:\Users\Michael\Desktop\Desktop folders\Class Videos\Optogenetics - Controlling the brain with light.mp4

Remind class of Sona opportunities – the Forming Impressions project of Kate Rogers.

Examples of machine object recognition successes.

A. Optical Character Recognition (OCR) – letters of the alphabet are objects.

Recognizing text (remember, words are objects) from images of characters on a page.

The original drawing of the text, from Goldstein’s S&P text, 5th Edition, p. 207

The text as "recognized" by the HP scanning program 10+ years ago.

UIC; 1./ '}. In contrast to the important place that top-down processing plays in the later stages of Treisman's FIT approach and Biederman's RBC approach, David Marr's computational approach relies almost exclu- sively on bottom-up processing. l{is computational approach describes the perceptual process as operat- ing like a computer that is programmed to extract in- formation from the stimulus pattern and does not include the possibility that perception might be af- fected by the knowledge, expectations, or past ex- periences of the person doing the perceiving. However, the computational approach does say that the visual system is constructed to take into account the nature of the perceptual world (Man's natural constraints). According to this idea, the visual system

See also Wordlens on YouTube. – do Wordlens demohttp://www.youtube.com/watch?v=h2OfQdYrHRs

Topic 10: Perception of Objects- 2 5/23/2023

The program did a pretty good job of recognizing the individual letters.

Failure to recognize the end-of-line hyphens is an indication that this program is not very “smart”.

B. Voice Recognition – words are objects Siri

Siri and other voice recognition programs available through smartphones.Play the Alexa Video

C. The DARPA (Defense Advanced Research Project Agency) contest held to develop a car that can traverse 150 miles of desert terrain unaided.

Play G9 VL 5-1 Video of DARPA challenge in 2007 – old old old

Note the vehicle at 3:40 into the recording, trying to enter a busy street.C:\Users\Michael\Desktop\Desktop folders\Class Videos\G9 VL 5p1 Carnegie Mellon Entry in DARPA Challenge.mp4

DARPA Grand ChallengeFrom Wikipedia, the free encyclopediaThe DARPA Grand Challenge is a prize competition for driverless vehicles, funded by the Defense Advanced Research Projects Agency, the most prominent research organization of the United States Department of Defense. Congress has authorized DARPA to award cash prizes to further DARPA's mission to sponsor revolutionary, high-payoff research that bridges the gap between fundamental discoveries and military use. DARPA has technologies needed to create the first fully autonomous ground vehiclescapable of completing a substantial off-road course within a limited time. The third event, The DARPA Urban Challenge, which took place on November 3, 2007 and was broadcast via webcast,[1] further advanced vehicle requirements to include autonomous operation in a mock urban environment.

DARPA has moved on to robotics and other areas.

D. A good introduction to the problem of why we study object recognition . . .

C:\Users\Michael\Desktop\Desktop folders\Class Videos\G9 VL 5-11 Jim DeCarlo on visual Object Recognition.mp4 Why we’re here.

Topic 10: Perception of Objects- 3 5/23/2023

A few of the Difficulties in Creating Machine Vision G9 p 97

1) Ambiguity of the 2-dimension stimulus projected onto the retina.Problems that we humans and other living creatures overcome when we perceive. How can machines do the same?Recall that each eye receives a 2-D view of the world.

Many 2-D views are ones that could be created by multiple different actual objects

Multiple views of the same object, in general, are a terrible problem for machines and for us (although we solve them)

How do we know which is the correct object?

In many instances, the image on the retina is one that corresponds perfectly to many, many different external objects.

2) Objects may be hidden or blurred

3) Objects look different from different viewpoints

The above are all different views of the same object. You see them all as the same because of the magic of your visual system.But the image of each view on the retina is completely different from the image of each other view.

Topic 10: Perception of Objects- 4 5/23/2023

Brief history of the study of object recognition – (G9 p. 100)

The Structuralists

A group of psychologists who proposed that perceptions of complex objects were the result of identification of the elementary sensations of which the external stimulus was composed and then the syntheses of those elementary sensations by higher order brain functions.

Analysis: The identification of basic pieces or components of an entity.Synthesis: The combination of separate elements or substances to form a coherent whole.

Retina Immediate Higher orderVision areas brain processes

Belief: Image Perception

You’re not a pretty face, you’re eyebrows, ear lobes, eyelashes, pupil, iris, lips, small mole on your left cheek, wisp of hair on your forehead, slight dimple on your chin, . . . . That’s what you are to me.

The structuralists, whose movement was founded by Wilhelm Wundt and led by E. B. Titchener, believed that all complex cognitive functions involved the synthesis of more elementary components.

From Titchener, E.B. (1902). Experimental psychology: A manual of laboratory practice. (Vol. 1) New York, NY: MacMillan & Co., Ltd. Image from Wikipedia - - - >

“If we take some such section of consciousness (say, a memory-consciousness, our mind as it is when we are remembering something, - or an imagination- consciousness, our mind as it is when we are imagining something, - or a thought-consciousness, our mind as it is when we are arguing something out) and analyze it, we find that it reduces to a number of quite simple processes, all of the same general kind or class. These are called sensations. The sensation, then, is the structural unit or structural element of these consciousnesses, - just as the cell (so the anatomists and physiologists tell us) is the structural element of our bodily tissues. If we wish to understand the make-up of mind, we must know all about these sensations.”

This was written at a time when there was virtually no information about the workings of the brain. It was not even known by many people that the brain was composed of individual cells.

This fit quite nicely with the successes of chemistry in discovering the elements of which all matter is made and of biology in discovering the elements (cells) of which all creatures are made.

The process of identifying the elements of which a complex thing is made is called analysis. For example, in chemistry the process of discovering the components of which complex compounds are made is called qualitative or quantitative analysis, depending on the types of tests that are used.

Topic 10: Perception of Objects- 5 5/23/2023

SN

S4S3S2S1

analysis synthesis

Structuralism under Titchener had 3 basic tenets.

1. The belief that complex perceptions could be analyzed into elementary sensations – that there were elements to discover and that we would be able to discover those elements.

2. The belief that complex perceptions were the result of the combination, i.e., synthesis, of elementary sensations.

3. The belief that the discovery of elementary sensation would occur through the use of a technique called analytic introspection.

Analytic introspection: Participant was shown a complex object, e.g., an apple, a chair and asked to name the sensations evoked by the object. Participant listed the elementary sensations “making up” the complex perception. A completely conscious, overt behavioral procedure.

Problems encountered by the Structuralists in the early 1900s. Common problems in science.

1. The parsimony problem. The list of elementary sensations became quite large after a few years of use of the method of analytic introspection. One list had 1000s of elementary sensations.

2. The replicability problem. There was disagreement between different laboratories concerning which “elements” comprised perceptions of objects which had been studied in the various labs.

Here’s a current example of replicability from research being conducted at UTC . . .

Study Correlation w GPAOriginal Study -.159Sebren Study -.099Balanced Scale original 50 items

-.062

Balanced Scale other 50 items

-.065

Balanced Scale Thompson MM

-.102

ROFB -.133

For the Structuralists, we now know that the method of analytic introspection was the culprit in the replicability problem. Apparently we do not have conscious access to whatever elementary processes underlie the perception of complex objects.

It’s a shame that the concepts of analysis and synthesis came into disrepute, because there is mounting evidence that the visual system DOES analyze the visual world into elementary features – the neurons we’ve studied in Chapters 2 and 3 and 4 are examples of feature detectors. The face detectors found in the FFA are examples. But those features are not “viewable” in the way that Titchner believed.

Topic 10: Perception of Objects- 6 5/23/2023

The correlations in the table are correlations of inconsistency of responding to personality tests with GPA. They’re negative, meaning thatpeople who are more inconsistent when they fill out a personality questionnaire have slightly lower GPAs than those who are less inconsistent. We discovered the result, published it, then wondered if our result was replicable.

The table at the left shows that the results we published (top line) were replicable across five other studies, although the sizes of the other correlation coefficients were not as large as the correlation in the study we published.

3. The theory-crushing results problem: The Phi Phenomenon. Vision researchers developed demonstrations of perceptions which could NOT have been the result of the combination of elementary sensations, at least not the elementary sensations proposed by the structuralists.

The crucial experiment involved the perception of movement. The Structuralist theory held that the perception of movement was built from the synthesis of a series of elementary sensations of objects in intermediate positions.

So an object moving from Point A to Point B created many intermediate sensations . . .of the object at different places between A and B. The Structuralists believed that these intermediate sensations were synthesized into the perception of movement.

That is, the perception of movement from point A to point B was “built”, i.e., synthesized, from sensations of still images on the eyes between point A and point B

Classic example of a refutation of the belief that movement is sensations combined: The Phi Phenomenon.

Two lights are set up, 12-18” apart. Participant sits 6-10’ in front of the lights. The lights are flashed on sequentially. First the left light is turned on. Then it’s turned off and at the same time the right light is turned on. Then it’s turned off and at the same time, the left light is turned on. And so on, and so on.

The sequence of stimuli: Time 1:

Time 2:

Time 3and so on.

See a demo at http://www.yorku.ca/eye/balls.htm

Virtual Lab Chapter 5. Demonstration 2: Apparent Movement.

Most people report a single light moving back and forth between the two end points.

This, of course, could not be the perception according to the Structuralist theory of complex perceptions being built from elementary sensations, because there could have been no elementary sensations – the ball was not ever in the space between the two endpoints. But participants in the research clearly saw the light moving across the area between them. (So did Titchener, I assume.)

The Structuralist movement faded from the psychological scene.

Topic 10: Perception of Objects- 7 5/23/2023

Rightoff

Leftoff

Righton

Lefton

Lefton

Rightoff

The Gestalt movement. G9 p. 101

The Gestalt movement rose in the early 1900s partly as a revolt against the Structuralists.

The movement did NOT believe that complex perceptions were combined/synthesized from elementary sensations.

Instead, their belief was that perception of an object occurs immediately to the whole object, not to pieces of it identified as a result of analysis.

They believed that we do not sense parts and then combine them into percepts.

Instead we perceive the configuration, the whole – innately and immediately.

This is summarized by the phrase: The whole is greater than sum of its parts.

The emphasis of the Gestalt approach was on an isomorphic, unitary response to the whole object, rather than on the synthesis of a collection of individual responses to parts of the object.

They emphasized the importance of the configuration of the scene in the ultimate perception.

Example: Consider the following two lines, obviously of equal length

Now compare their length in the figure below. Note that the top line now appears longer.

How does adding the two diagonal lines change the perception of the two horizontal lines? This was difficult for the Structuralists to explain because the two angled lines were different from the two horizontal lines.

The Gestalt psychologists argued that the answer has to be that since our perception of the figure is a response to the whole configuration the added parts affect our perception of the original lines.

They said that if our perception of the whole were a simple summation of elements, there is no way elements of the added parts, since they’re not parts of the lines being judged, could affect the elements associated with the original lines.

Gestalt psychologists also believed that a miniature 3D representation of the external object being perceived exists inside the brain. This isomorphism assumption is clearly not true, although its spirit is maintained in the retinotopic maps that have been found.

Topic 10: Perception of Objects- 8 5/23/2023

The Gestalt Legacy – the Gestalt laws of grouping – from the 1920s G9 p 102

Gestalt Psychologists discovered several stimulus characteristics that appear to be important for our grouping of different stimulus elements.

a. Pragnanz. Good form/figure. The visual scene will be experienced in such a way that it appears to be comprised of the most parsimonious collection of pieces.

e.g. Is this two circles or two half-eaten apples with a football between them?

b. Similarity. Pieces of a figure that are similar are perceived as grouped together.

This scene is perceived as three columns, as opposed to three rows. The similar elements of each column are grouped together.

c. Good continuation. Elements that define smooth lines or curves are grouped together.

This is perceived as two strings of circles rather than as an unordered collection of circles.

Topic 10: Perception of Objects- 9 5/23/2023

or= + + +

d. Proximity/Nearness. Elements of a figure that are close together are grouped together..

The squares are perceived as 3 columns as opposed to 3 rows, because the squares within each column are closer to each other than are the squares in each row.

e. Connectedness. Elements that are physically connected are perceived as units.

The left row of circles is perceived as 6 individual elements.

The right row of circles is perceived as three groups because of the connectedness.

The point is that our visual processing appears to use these characteristics of the visual stimulus to determine which elements go together as objects and which do not.

Topic 10: Perception of Objects- 10 5/23/2023

The Gestalt Laws of Perceptual Segregation – 2nd part of their Legacy G9 p 104

What do our brains do when the visual system presents a visual scene?

It appears that one of the first processes is a basic segregation into figure and ground

For our perception of most scenes, we automatically separate the scene into two categories – figure and ground.

Figure . . .

That part of the scene that is more “thinglike”, more memorable, in front of the rest of the scene, seen as having a specific shape, possessing borders separating it from the rest of the scene.

Stimulus properties associated with the figure

Parts of the scene that are lower are seen as figure.

Most people see the red as being figure.

The portion of the scene with convex borders are seen as figure

Most people see the dark areas as figure

The portion of the scene that is smaller is seen as figure

The portion of the scene that is most symmetric is seen as figure

Ground

That part of the scene which is not figure.

Topic 10: Perception of Objects- 11 5/23/2023

Some factors which appear to be important in human perception of scenes G9 p 109

What we experience first: Apparently we first categorize the gist of the scene.

Gist – a general description of the type of scene

Examples – Forest, interior of a room, exterior of buildings, sea scape, sky, group of people, ocean

Probably the first characteristic of a visual stimulus that is extracted.

Characteristics that determine what the gist of a scene will be

Naturalness – Textured zones and undulating countours – a natural sceneStraight lines – a manmade scene

Openness - few objects – more likely a beach / oceanMultiple objects – more likely an interior

Roughness - little roughness – oceanMore roughness – interior, wooded

Expansion - converging lines

Color - blue – oceanGreen – forest

The above global features may be the first perceived.

Topic 10: Perception of Objects- 12 5/23/2023

Role of Inference – the visual system guesses a lot

Helmholtz suggested that the process of perception of a visual scene is like the process of solving a crime – arriving at a conclusion based on a collection of ambiguous pieces of information.

This was called the theory of unconscious inference.

We assume light from above and infer the shapes of the mounds / valleys from that.

So we infer that the images on the left are holes in the sand while those on the right are mounds of sand.

A corollary was the likelihood principle – that the most probable perception is the one that will be experienced. (a) below is inferred to be (b), not (c) because (b) is more likely to occur.

Topic 10: Perception of Objects- 13 5/23/2023

Research on neural correlates of object perception G9 p 114

The Gril-Spector et al (2004) Harrison Ford study of the FFA.

Procedure

Experimenter presented pictures of either 1) Harrison Ford, 2) another person, or 3) Nothing

Observers reported either 1) Harrison Ford, 2) Another face, or 3) Didn’t see a face

fMRI factivity in the FFA from trials in which Harrison Ford’s face was presented

fMRI activity in the FFA was greatest on those trials in which they were correct.

The activity was higher on trials in which they reported seeing Harrison Ford.

Activity was least on trials when they reported seeing nothing.

So fMRI activity was correlated with the reports.

Topic 10: Perception of Objects- 14 5/23/2023

The Sheinberg & Logothetis (1997) Monkey Sunburst/Butterfly binocular rivalry study

Binocular Rivalry – One image is presented to the left eye while a different image is presented to the other.

For most people, perception alternates between what is seen in the left eye to what is seen in the right eye, back and forth.

The Sheinberg & Logothetis Procedure

Trained monkeys to press one lever if they saw a butterly or a different lever if they say a sunburst.

Located an inferotemporal cortex neuron that responded to butterfly images – the butterfly neuron.

Then . . .

A picture of a sunburst was presented to one eye, a butterfly to the other.

Monkeys responded indicating what they saw – sunburst or butterfly.

Results

As would be expected their responses alternated – sometimes indicating butterly, sometimes sunburst.

The butterfly neuron responded positively at those times the monkey was indicating it saw a butterfly.

It did not respond above its base rate when the monkey was indicating “sunburst”.

So response of this neuron correlated with the monkey’s overt behavioral report of what it was seeing.

They were “reading” the monkey’s mind.

Topic 10: Perception of Objects- 15 5/23/2023

The Tong et al (1998) colored glasses binocular rivalry study

Procedure

Used binocular rivalry in humans.

Presented a face image to the one eye and a house image to the other eye of humans using specially constructed glasses.

Participants reponded indicating which image they saw. Again, rivalry, so the images and the reports of participants alternated between face and house.

fMRI was used to record activity in the FFA and in the PPA (Parahippocampal Place Area).

Results

When observers saw the house: Increase in PPA activity, decreate in FFA activityWhen observers saw the face: Decrease in PPA activity, increase in FFA activity

So gross neural activity in the PPA and FFA correlated with observer reports of what they saw.

They were reading the participants’ minds.

The Kamitani and Tong (2005) voxel orientation decoder experiment

Procedure

Presented different orientations of stimulus gratings, recording the fMRI activity within a collection of 400 cubic areas of the brain called voxel. They recorded the activity in the 400 voxels associated with each orientation.

They recorded the 400 voxel values for a one stimulus.They then recorded the 400 voxel values for a different stimulus.

Results

They then presented stimuli and recorded the voxel results of what the participants were experiencing to predict which orientation the participant would report.

The important difference between this experiment and the previous is that the same neurons were being recorded for both stimuli. It was the pattern of activity of those neurons that changed when the stimuli changed, not the location of activity in the brain.

In general, the predictions correlated quite highly with the participants’ reports of what they were seeing.

Play C:\Users\Michael\Desktop\Desktop folders\Class Videos\G9 VL 5p15 Frank Tong on Visual Mind Reading.mp4 (Good overview of the problem.)

Topic 10: Perception of Objects- 16 5/23/2023

Are Faces Special? Are they treated specially by the nervous system?

Arguments for “Yes”

1. Faces are pervasive.

2. Faces have their own neural module, the FFA.

3. Turning faces upside down has a huge effect on recognition.

This suggests that faces are processed wholistically – the whole configuration of features is stored – like a template or a stored image.

If faces were stored as simply a collection of features, many of those features would be retained when the image was turned upside down and recognition would not be much affected.

But it IS affect by turning faces upside down.

4. Perception of faces triggers activity in a wide variety of areas of the brain – perhaps more so than any other class of objects. For example, faces trigger activity in the amygdala, an emotional center. Chairs do not.

Topic 10: Perception of Objects- 17 5/23/2023