and the winner is…...1 and the winner is… developing a computer program to investigate neural...

1

And the Winner is…

Developing a Computer Program to Investigate Neural Competition with Multimodal Stimuli

A Computer Science Project with Behavioral Applications

Presented to:

Missouri Junior Science Engineering and Humanities Symposium

University of Missouri

Columbia, Missouri

By

Caleb Jireh Martonfi

Tuscumbia High School

526 School Rd.

Tuscumbia, MO 65082

Mrs. Constance Wyrick

Science Research Advisor

Tuscumbia, MO 65082

2

Abstract Name: Caleb Martonfi School: Tuscumbia High School Sponsoring Teacher: Mrs. Constance Wyrick Project Title: And the Winner is… Developing a Computer Program to Investigate Neural Competition with Multimodal Stimuli

Neural competition occurs when two stimuli fight for representation. One form of neural competition occurs when two stimuli within a single modality compete for embodiment. A form of bimodal neural competition occurs when visual and auditory stimuli compete. The engineering goal of this project was to create a series of computer programs using an icon-based programing language to test neural competition. A Raspberry Pi was purchased on which to run the programs. Scratch, an icon-based programming language, was used to construct the programs. The engineering goal of this project was accomplished. Three different unimodal programs were created. Each of the programs tested different aspects of perception and attention. Two bimodal programs were constructed. One of the five programs was selected for further development. It could be used for the testing of bimodal neural competition with speed perception.

3

Neural Competition

Today’s society is constantly assaulted with many different stimuli. Sorting out these

sensations is difficult for many people. Often, combining two stimuli can result in unique

illusions. However, in many situations, such as driving a car, distractions from competing

stimuli- such as cell phones, radios, music, and even friends- can be hazardous. In order to

avoid tragedies such as these, neural competition must be better understood. Neural

competition occurs when two stimuli compete for representation.

One form of neural competition occurs when two stimuli within a single modality fight

for embodiment. The biased model of visual attention suggests that the visual attention selects

one stimulus over another. Change blindness and tunnel vision are two examples of single

modality neural competition.

A form of bimodal neural competition occurs when visual and auditory stimuli compete.

This can be demonstrated by examining the mechanism involved in conversation. In speech,

both linguistic and paralinguistic cues are used to decipher intent and emotion. ‘Linguistic’

refers to the actual denotation of the sentence. ‘Paralinguistic’ refers to the facial expressions,

intonation, body posture, and physical gestures involved in communication. Together, these

cues can be used to interpret the emotion behind a sentence. When these cues don’t line up,

the brain has to decide what to believe. For example, in the McGurk effect, visual and auditory

stimuli convey contrasting messages. For example, a speaker mouthing “FA” and a sound

playing “BA” would show the McGurk effect. The result is an integration of the two. However,

perception is biased towards visual stimuli [Pollack, 2005], so an onlooker would most likely

4

perceive something closer to “FA”. This is another indication that the integration of audio and

visual stimuli must occur before decision making. The Fuzzy Logical Model of Perception

suggests that bimodal perception comprises of evaluation of each individual stimulus,

integration of the stimuli, and decision making regarding which alternative is supported.

However, “…most individuals show perceptual and attentional biases towards visual stimuli”

[Pollack, 2005].

Other studies have been done regarding the speed at which processing is done.

Perceptual load theory proposes that the time that it takes perception to occur is not constant.

It depends on the perceptual demands of the task. Perceptual load refers to the attentional

demands required to perceptually recognize one thing from another. When distractors are

present, a task that is particularly strenuous leaves less room for the distractors to interfere, so

perception occurs quickly, while tasks that are not demanding leave more room for distraction,

so perception takes longer. For example, detecting color alone is easier than detecting color

and the orientation of an object.

In our technologically advancing world, there are more and more stimuli demanding our

attention. It is become increasingly evident how important it is to understand our brain’s

reaction to this neural competition. There is still much to be learned about how modalities

interact, and how distractors interfere with neural processing.

5

Programming Languages

Many programming languages are available for creating animation programs for

behavioral research and professional work. These include C++, Java/Flash, JavaScript, Python,

Unreal Engine, XNA Game Studio, Dark Basic, GameMaker, and Scratch. This research used

Scratch as its primary programming language. Scratch is an icon-based programming language

developed by the Massachusetts Institute of Technology (MIT).

6

Question Posed

Is it possible to create a series of computer programs using an icon-based programing

language to test neural competition?

Engineering Goal

The engineering goal of this project was to create a series of computer programs using

an icon-based programing language to test neural competition. The programs were developed

for testing the following types of neural competition: unimodal (within the same modality, e.g.

visual/visual) and bimodal (between two modalities, e.g. visual/auditory).

Hypotheses

The following hypotheses were formed:

1.) It would be possible to create at least one program testing the effect of

unimodal neural competition.

2.) It would be possible to create at least one program testing the effect of bimodal

neural competition.

3.) It would be possible to create at least one fully functional program. This program

should be ready for research use.

7

Engineering Process

Designing software usually involves planning, a feasibility study, product design, coding,

implementation/integration, testing, and installation/maintenance. Because the goal of this project was

to develop a series of programs that could test neural competition, not to create market-ready software,

implementation/integration and installation/maintenance were not part of the engineering process.

Testing and selection, however, were included in the design.

Planning:

The goal of this project was to create a series of computer programs using an icon-based

programing language to test neural competition. The first step to accomplishing this was the planning

phase. Many ideas were formed about what the programs should execute. In the end it was decided

that the programs would need to be able to test unimodal and bimodal neural competition. Such

programs would implement and/or examine the following: auditory stimuli with visual distractors, such

as increasing and decreasing pitches; visual stimuli with multiple visual distractors; speed perception of

auditory or visual stimuli, and memory with visual distractors. To conduct this study, an icon-based

programing platform called Scratch was used.

Feasibility:

For some researchers in the behavioral science, their options are limited to buying someone

else’s program with which to test neural competition. This becomes cost prohibitive, especially for

student researchers. In addition, the purchased program may not allow for flexibility when designing

the research. The ability to create one’s own program would be very beneficial. To examine the

feasibility of this project, various factors were examined. Availability, cost, time, and functioning were all

considered. Foremost, the availability and cost of Scratch were surveyed. Scratch was developed by the

8

Lifelong Kindergarten group at the Massachusetts Institute of Technology (MIT). Because it is available

on PC, Macintosh, and Linux operating systems, obtainability was not a limitation. Scratch is free to

everyone, so cost was not an issue either. Next, the time to develop functioning programs was

contemplated. Because the researcher had already been around Scratch for over four years, functioning

wasn’t a question. Scratch could be used to create complex programs using an icon based-platform.

Because the language was already understood, the time to create such programs was minimal.

Due to the nature of behavioral research, portability was also desired. Purchasing and using a

Raspberry Pi solved the problem. A Raspberry Pi is a credit card sized computer designed to help novice

individuals learn programing. Several programing platforms are preinstalled onto it, such as BlueJ Java,

Python 2 and 3, Mathematica, Node-RED, Sonic Pi, Wolfram, and Scratch. A Raspberry Pi only costs

about twenty-five dollars, and it can control external hardware. Because of the availability, low cost,

efficiency in time management, and reliable functioning, a Raspberry Pi was selected as the computer of

this project. A pre-owned computer monitor was purchased under ten dollars. A leftover school

keyboard and mouse were used. A wireless handheld keyboard was also purchased for its remote

feature.

Product Design:

This step was very similar to the planning phase. However, specific goals were set. The following

programs were envisioned.

1.) To test unimodal neural competition:

a.) A program with numerous visual distractors moving across the screen, to test a subject’s

attention to a specific visual tracking task with distractors;

9

b.) A program with a small, color changing dot amid changing background scenes, to investigate

change blindness;

c.) A program with alternating flashing colored objects with memory words, to investigate a

test subject’s association of color with words, to better understand visual memory.

2.) To test bimodal neural competition:

b.) A program to test the accuracy of subject’s viewing descending and ascending pitches

(auditory stimuli) conflicting with descending and ascending or lines (visual stimuli), to

investigate auditory illusions;

c.) A program to test the accuracy of a subject presented with conflicting speeds of auditory and

visual stimuli, to test visual speed perception with auditory distractors, and auditory speed

perception with visual distractors.

Coding:

In Scratch, programs are created by dragging and dropping icons that fit together based on their

functions. Individual characters are called sprites. Each sprite can be programmed to do specific things.

“Hiding” something refers to the object disappearing from view.

In the first program, a subject’s attention to a specific visual tracking task with distractors was

tested. To fashion a program to do this, five sprites were created. Figure 4. shows each of them. Three

sprites were in the form of beach balls, and were equally spaced across the screen. All but one of them

were hidden. The unhidden beach ball started at the top left of

the screen and moved downward. When it reached the bottom, it

hid itself, and broadcasted for the next beach ball to appear and

start moving down the screen. It then jumped back to the top of

the screen ready to start the process anew. Moving from left to

Figure 1.

10

right across the middle of the screen was a little blue car. Once it reached the edge, it would jump back

to the other side of the screen and immediately start moving again. After it had done that 3 times, it

would broadcast to an airplane sprite at the top of the screen to appear and start moving. The airplane

traveled across the screen once and disappeared. A subject participating in this study might be asked to

count how many times two of the five objects touched the edges of the screen. Results would be

evaluated based on the accuracy of the subjects counting. Variables that could be manipulated include

the type of object crossing the screen, the number of objects crossing the screen, or the visual

orientation of the object/s being observed. This was a unimodal test.

The second program that investigated change blindness

was created with a small color changing dot amid changing

background scenes. First, eleven different backgrounds of

random indoor/outdoor/sports/nature backgrounds were

selected. Figure 2. shows one of the possible backgrounds. They

were programmed to alternate every 0.5 seconds. A small, yellow dot was placed in the center of the

screen. Each time the backgrounds changed scenes, the small dot gradually changed color from yellow

to green to blue to purple to red to orange and back to yellow. The test subject would not be alerted to

the changing dot. They would only be asked to count the number of backgrounds. They would then be

asked to recall the order of the colors of the dot, if indeed they noticed the color change at all. This

program could test one of two things: change blindness (an individual’s inability to observe a change) or

memory with visual distractors. This was a unimodal test.

To create the third program that could investigate a test

subject’s association of color with words, to better understand visual

memory, three circular sprites were created. One circle was red,

Figure 2.

11

another was white, and the third was blue. Figure 3. shows the position of the circles. Each circle was set

to spin. Each circle was created with a list of four words. This program was coded very similar to the one

before it. One circle would appear, disappear, change the word within itself, and then broadcast the

next circle to start the process. There was a total of twelve words. To test this program, the subject

would be asked to try to remember as many words as possible. The program would show all of the

words twice, and then stop. The subject would then record the words they remembered on a sheet with

the three different colors. Results would be analyzed based on the subject’s accuracy and how many

words they recalled. This was a unimodal color-association memory test.

The fourth program that was constructed contained

ascending and descending lines and pitches, and investigated

auditory illusions. Four sprites in the shape of lines were placed

equidistance from each other, but at different elevations.

Figure 4. shows the position of the sprites. All the sprites were “hidden”. When the green flag was

pressed, “One” was broadcast to all other sprites. Sprite #1 received “One” and began executing its

code. First, it presented itself, near to top edge of the screen. It then sounded a high pitched note, and

hid itself again. “Two” was then broadcasted. When sprite #2 received “Two”, it presented itself. It had a

slightly lower y-coordinate than sprite #1. Sprite #2 sounded a pitch slightly lower than the preceding

note. It then disappeared from the screen. Next, “Three” was broadcasted. The same process was

followed for the next two sprites. In addition, if the down arrow key was pressed, it caused the program

to start on a low pitch, and ascend in pitch as the lines descended. Note, even though the ascent or

descent of the auditory stimuli varied, the visual stimuli always appeared from the top of the screen to

the bottom, and always from left to right. To use this program in a study, the subject would be asked to

listen for changes in pitch while visually observing descending lines. The auditory stimulus’ “direction”

Figure 4.

12

would be changed using the arrow keys on a wireless keyboard, to eliminate any cues that the program

had changed. Afterward, the subject would be asked to determine if the pitch was going up or down. If

the lines are going down and the pitch is going up and the subject responds that the pitch is going down,

then an auditory illusion has been created, because the visual modality overrode the auditory modality.

Studies using this program could test an individual’s ability to determine the direction of the pitch versus

another’s ability, such as a musician versus a nonmusical individual, an auditory learner versus a visual

learner, or male versus female. This was a bimodal test.

The goal of the fifth program was to test the effect conflicting speeds of auditory and visual

stimuli on rate perception. To do this, a basketball sprite and a soccer ball sprite were placed on the

screen, with the soccer ball sprite hidden. The basketball sprite speed could be adjusted manually to

affect the visual perception. The basketball would bounce and move in the opposite direction when it

hit an edge. Every time the soccer ball sprite hit a wall, it made a slapping sound. The speed of the

soccer ball sprite could be manually adjusted to affect auditory

perception. The two balls always started in the same place, so

when they were set to travel the same speed, the auditory

stimulus was synchronous with the visual stimuli. Figure 5.

shows the motion of the basketball across the screen. To carry out a study using this program, different

combinations of the speed of the ball and the speed of the clicks would be chosen. The time it takes the

basketball to cross the screen from edge to edge at different would be measured using a Scratch script.

To start, the subject would be shown a reference. The ball in this demonstration would take 1 second to

cross the screen. The subject would then be asked to record how fast they thought the ball took to

travel from one edge of the screen to the other. Results would be analyzed based on how accurately the

subject answered the questions. This was a bimodal test.

Figure 5.

13

Testing:

To test the programs, one by one they were run. Bugs were removed or fixed. Any necessary

adjustments were made.

Selection:

To select one of these programs to use for testing, Dr. Nelson Cowan; a neural-cognitive

researcher at the University of Missouri, was consulted. With his assistance, each program was analyzed

for practicality, controllability, and uniqueness.

The program with the bouncing basketball was selected as the best fit for further development.

It had the potential to be used to test neural competition between visual and auditory modalities. This

program was practical. The speed of the basketball could easily be changed. Experiments completed

using it were controllable, because the only independent variables changed are the speed of the ball and

the speed of the click, and the only dependent variable measured was the test subject’s perception of

the speed. Many of the other programs had too many distractors

and too much motion to be able to accurately pinpoint the effect

of one stimulus. Outside factors could easily affect the results.

This program is unique, because few studies have been done on

the topic of bimodal neural competition and speed perception.

Coding:

After this selection was made, several adjustments were made to aid in the actual testing of

participants. To start the modified program, the letters “G” and “O” were pressed. This displayed a

passage stating: “The first demonstration you will see is a reference. It takes the ball exactly 1 second to

Figure 6.

14

travel from one edge of the screen to the other. Use this to

help you on the questions that follow.” Figure 6. shows

that passage. The letters “N” and “X” (for next) are then

pressed to show the reference. A program was previously

used to determine how fast the ball would have to travel to

cross the screen in one second. After the ball bounces 6 times, another passage appears. “This is the

start of the first section. The ball will travel at different random speeds. Use the left and right arrow keys

to move the scale to how fast you think it took the ball to cross the screen from edge to edge. Press

space to submit your answer.” After “N” and “X” are pressed, the program chooses one out of twelve

conditions: fast ball and fast clicks, fast ball and medium clicks, fast ball and slow clicks, medium ball and

fast clicks, medium ball and medium clicks, medium ball and slow clicks, slow ball and fast clicks, slow

ball and medium clicks, slow ball and slow clicks, fast ball with no clicks, medium ball with no clicks, and

slow ball with no clicks. Fast stands for a speed of 0.5 second between wall/click, medium stands for a

speed of 1 second, and slow stands for a speed of 1.5 seconds. The condition was recorded in a list.

After the ball bounced ten times, it disappeared and a scale appeared on the screen. The user could

control a slider on the scale using the arrow keys and submit using the space button. Using an equation

to convert the x-coordinates of the slider to seconds, the user input was recorded in a list, after the

condition. Figure 7. shows the scale. A new condition was then chosen. After all of the twelve conditions

were completed, a third passage appeared. “You will now hear a reference that clicks every second.”

The clicks proceeded without a ball. A fourth passage was presented. “This is the start of the second

section. Instead of asking about the speed of the ball, you will now be asked about the speed of the

clicks. Use the scale as before.” This section tested the subject’s perception of the speed of the clicks. All

of the conditions remained the same except the conditions with the fast ball with no clicks, the medium

ball with no clicks, and the slow ball with no clicks. These were changed to no ball and fast clicks, no ball

15

and medium clicks, and no ball and slow clicks. After these twelve conditions were carried out, the

program displayed “You are finished. Thank you for participating!”, and ended. To view the list of

conditions and times, a box could be checked to display them. Testing with this program would be

simple. The subject would only have access to a keyboard. The progression of the program (e.g. “NX”)

would be controlled by the researcher using a wireless keyboard. The program would start with the

passages and explain what the subject was expected to do. The subject would use the arrow keys to

control the scale. The results would be recorded after the program finished running.

Testing:

The testing phase was repeated after modifications were made to the program. This program

could be used for the testing of neural competition.

16

Figure 1.

Conclusions

The engineering goal of this project was accomplished. A series of computer programs

to test neural competition were created successfully. Programs for testing unimodal and

bimodal neural competition were effectively developed.

The following conclusions can be drawn:

1.) The hypothesis stating “It will be possible to create at least one program testing the

effect of unimodal neural competition,” was supported. Three different unimodal

programs were created. Each of the programs tested different aspects of perception

and attention. See Figure 1.

2.) The hypothesis stating “It will be possible to create at least one program testing the

effect of bimodal neural competition,” was supported. Two bimodal programs were

constructed. See Figure 2.

17

3.) The hypothesis stating “It will be possible to create at least one fully functional program.

This program should be ready for research use,” was supported. One of the five

programs was selected for further development. This was the program that tested the

accuracy of a subject presented with conflicting speeds of auditory and visual stimuli. It could

be used for the testing of bimodal neural competition. See Figure 3.

Figure 2.

Figure 3.

18

Future Studies

The overall goal of this project was to create one fully functional program that would be ready

for research use. This was achieved with the bouncing ball program. In the future, this program will be

used by the researcher to test the effect of bimodal neural competition on speed perception.

19

Acknowledgements

I would like to thank the following people:

Dr. Nelson Cowan- Curators Distinguished Professor of Psychology, for his expertise in reviewing

the various computers developed to test neural competition.

Mrs. Constance Wyrick- my research advisor,

Steve and Robin Martonfi- my parents.

20

Works Cited

Chun-Yu Tse, G. G. (2015). Read My Lips: Brain Dynamics Associated with Audiovisual

Integration and Deviance Detection. Journal of Cognitive Neuroscience, 1723-1737.

Fogerty, K. G. (2015, September 17). Integration of Partial Information Within and Across

Modalities: Contributions to Spoken and Written Sentence Recognition. (N. Tye-Murray,

Ed.) Journal of Speech, Language, and Hearing Research, 1805-1817.

Nathan A. Parks, M. R. ( 2011). Steady-state Signatures of Visual Perceptual Load, Mulitmodal

Distractor Filtering, and Neural Competition. Jounal of Cognitive Neuroscience, 1113-

1124.

Pedram Daee, M. S. (2014, July 24). Reward Maximazation Justifies the Transition from Sensory

Selection at Childhood to Sensory Integration At Adulthood. (R. J. Beers, Ed.) PLoS ONE,

9(7), 1-13.

Pollak, J. E. (2005). Experiential Influences on Multimodal Perception of Emotion. Child

Development, 1116-1126.

Ternaux, J.-P. (2003). Synesthesia: A Multimodal Combination of Senses. Leonardo, 321-322.

21

Appendix for the Bouncing Ball Program Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 3, CPU)

22

The program starts here. Go to

page 37. Make sure that you

read all of the code before you

move on.

Go to page 32 after returning

from Words Start.

23

After Start, go to page 32

and 33.

After all twelve of the

conditions are completed,

go to page 34.

29

Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 1, Scale)

After Start Scale, go to page 22.

30

Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 2, Scale Slider)

After Start Scale, go to page 22.

33

Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 4, Basketball)

After Demo, also view page 34.

Visit page 16. Then go to page

22.

After Start Basketball, go to

page 28 and 29.

34

Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 5, Soccer Ball for Clicks)

After Start Basketball, go back

to page 28.

After Click Demo 2, go back to

page 34.

35

Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 6, Question)

After Start Clicks, go to page 40.

After returning from Click

Demo, go to page 33.

After returning from Click

Demo 2, go to page 38.

After returning from Words

start clicks, go to page 22. If all

of the click conditions are

completed, go to page 39.

37

Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 7, Passage #2)

38


Go to back to page 21.

39


After Words start clicks, go to

page 34.

40

Complete Program for Neural Competition with Visual and Auditory Stimuli (Bouncing Ball) (Sprite 10, Finale)

This is the end of the program.

41


After Click Demo, go to page 14.

42

Views of Sprites

Sprite 8, Passage #1


43

Sprite 1 & 2, Scale and Scale Slider

Sprite 6, Question


44


Sprite 4, Basketball

Sprite 10, Finale

and the winner is…...1 and the winner is… developing a computer program to investigate neural...

Documents