j 3* rton co* j£e4h^ t^t. - stacksfw020vy9950/fw020vy9950.pdf · because of the conditional...

Lynn, *-" S.■*»"'

haw

"2) J 3*_rton_ Unwurety L***nes

CO* J£E4h^ T^t. .-Box I^M_-- Zxm/2ZS-B±2'Fnl 33 Fol. Title —

SD 70-576

DECISION MAKING: AN INDIVIDUALLYPARAMETERIZED DETERMINISTIC MODEL

By

Richard Shelly Lynn

fife Space DivisionNorthAmericartßockwell

.

SD 70-576

DECISION MAKING: AN INDIVIDUALLYPARAMETERIZED DETERMINISTIC MODEL

By

Richard Shelly Lynn

fife Space DivisionNorth American Rockwell


FOREWORD

This dissertation was presented by R.S. Lynn to thefaculty of the University of Southern CaliforniaGraduate School in partial fulfillment of the require-ments for the degree of Doctor of Philosophy, withmajor in Psychology. The format is that required bythe university.

The training and research leading to the dissertationwere supported, in part, by the Advanced Degree programof North American Rockwell Corporation.

The dissertation describes a novel collection oftechniques for the analysis and computer simulation ofmanifest aspects of decision making. The publication ofthis document indicates the readiness and capability toapply and extend these techniques to decision makingresearch of broader economic and scientific interest.

- 1HSD 70-576

fife Space DivisionNorthAmerican Rockwell

ACKNOWLEDGMENTS

I wish to acknowledge my appreciation for the advice

and criticism offered by Drs. Norman Cliff, Ronald Weitzman,

and William Michael, my dissertation committee. I am

thankful to Dr. Albert Marston for setting me back on the

correct track during a difficult period in the research. I

am indebted to colleagues Frank Dean and Maurice Braun for

their suggestions on evaluation of theory parameters. I

wish to acknowledge the influence of the papers of Nico

Frijda, Earl Hunt, and Lee Gregg and Herbert Simon.

My graduate training in psychological measurement

began and continued because of the kindness and encourage-

ment of Dr . J. P. Guilford. I shall always be grateful to

my wife and children, who kept faith during the long years

of study, research, and writing.

- v -SD 70-576

fife SpaceDivisionNorthAmerican Rockwell

TABLE OF CONTENTS

ACKNOWLEDGMENTS V

LIST OF TABLES

LIST OF ILLUSTRATIONS xi

Chapter

I. PURPOSE OF THE STUDY 1

11. BACKGROUND OF THE STUDY 9

Computer ModelsArtificial IntelligenceExisting Game-Playing and Problem-

Solving Machines

111. THE EXPERIMENT 15

The GameExperimental ProcedureSubjectsApparatusAnalysis of Verbal Behavior

IV. THE MODEL 23

Induction of Process RulesDevelopment of the Computer ProgramSimulation Program Structure and FunctionDerivation of Individual Parameter Values

- vii -SD 70-576


Chapter

V. EVALUATION OF THE MODEL 48

Simulation of Individual SubjectsSensitivity of Model to Parametric

VariationCross-validation of the Model

VI. SUMMARY AND CONCLUSIONS 75

APPENDIXESAPPENDIXES

Appendix A: Game Instructions 84

Appendix B: Excerpts from All Protocols 87

Appendix C: Synopsis of Decisions for AllSubjects 113

Appendix D: The Simulation Program SourceCode 118

Appendix E: Example Simulations 126

REFERENCES 132

- viii -SD 70-576


LIST OF TABLES

Table Page

1. Pre-Game Questionnaire Responses 19

2. Parameter Estimates 47

3. Summary Statistics 49

4. Meaning of Bits 59

5. Average Simulation Error Per Cent Associatedwith Each Model Variation 60

6. Partitioning of Average Error of Simulation . . 63

7. Error Per Cent Summary 70

8. Error Matrices for 16 Analyzed Hands 71

9. Error Matrices for Ik Holdout Hands, UsingParameters Estimated from the 16 AnalyzedHands 72

10. Error Matrices for 1^ Holdout Hands, UsingParameters Estimated from the HoldoutHands 7 3

- ix -SD 70-576


LIST OF ILLUSTRATIONS

Figure

1. Flow Chart of Decision-Making Model 35

SD 70-576


CHAPTER I

PURPOSE OF THE STUDY

The purpose of the research was to study and develop

methodology applicable to computer simulation of manifest

behavior in simple decision making or game playing situa-

tions, and to the evaluation of the adequacy of the simula-

tion. It was not the goal of the research to learn anything

about problem solving, choice making, or any other kind of

cognitive behavior. The techniques to be described might,

however, be employed by researches which do have such goals.

The research was not experimental, in the usual sense of

this word. No hypotheses regarding natural laws were enter-

tained or tested. No assumptions were made about the popu-

lation of Ss sampled, and no generalizations beyond the

particular sample were made. The sample of Ss was too small

and too homogeneous, and the sample of stimuli too limited,

to permit such generalization. From its inception, the

research was not experimental, but methodological.

1 -SD 70-576


A simple experimental task requiring a sequence of

binary choices was presented in the form of a one-person

game . The particular game used was not important to the

purpose; the basic research approach would have been the

same for a wide variety of tasks. It is of course simpler

if the choices are binary, but this is not necessary. The

stimuli should be easily quantifiable to facilitate analy-

sis, simulation, and evaluation. The memory load imposed by

the task should be assumed to be negligible; i.e., all

stimuli should remain in view after presentation. Each S_

must receive the same stimuli in the same sequence. The

methodology is especially applicable in situations in which

the effects of each decision are propagated throughout the

remainder of the sequence. The methodology, in whole or in

part, has potential applications to the study of a variety

of natural decision making problems. Possible examples are:

bid or no bid on a request for proposal, acceptance or re-

jection of a proposal, securities portfolio selection, and

personnel decisions such as hiring, promoting, layoff, and

firing. However, no analysis of any natural decision making

was performed to support the suggestion of applicability.

Some of the techniques employed in this research

were suggested by the background literature. Other

2SD 70-576


techniques were invented, only to be discovered later in

more extensive reading. The methodology represented by the

particular combination of techniques is offered as being

unique .The model is based upon formally stated processes

which are strictly deterministic. A formal process model

may be heavily dependent upon stochastic variables for sev-

eral reasons . A theorist may take the approach that random

variables may be used to simulate the effects of a number of

variables he chooses to ignore. Another theorist may admit

that he uses random variables to bridge the gaps in his

knowledge of the natural processes . Without explicit ref-

erence to the need to span areas of ignorance, a theory may

posit that the behavior being simulated, at some points in

an otherwise deterministic process, is fundamentally random

by nature. Stochastic elements were completely excluded

from the model in this research partly for esthetic reasons

and partly because the quantitative evaluation employed

demanded it.

The computer program mechanizing the model makes

decisions which are in effect predictions of what an in-

dividual S_ will do. When these predictions are in error,

the program is set back on the correct track. If an

- 3 -SD 70-576


incorrect move were allowed to stand, the program and S_

being simulated would diverge, and the situations faced by

the machine and the human would no longer be the same. The

bases from which the machine made its subsequent moves would

be different from those used by the S_. The setting-back-on-

the-track technique is fundamental and essential to the

methodology of the research. This technique is necessary

because of the conditional prediction effect, described by

Feldman (196 3) . The use of setting-back -on-the-track in

this research is due to Feldman.

The choice of a programming language in which to

express the model is definitely a methodological matter,

with significant implications for psychological theory.

Some of these implications have been described by Newell and

Simon (1963b) :

To what extent do we make implicit assumptions ofpsychological theory when we decide to write a simu-lation program in an information-processing language?

The mode of expression almost surely influences

that a list-processing language like IPL-V is a (weak)psychological theory. It is an assertion that theelementary information processes that will be dis-covered to underlie human behavior are easily con-structable from the instructions of the list-processing system, that its memory is organized interms of lists and list structures of associatedsymbols, and that it is capable of executing sequences

- k -SD 70-576

:he thought ... It is probable that psychological.ostulates enter the simulation by way of the struc:ure of the programming language. ... We conclude


of behaviors organized as hierarchical list struc-tures, (pp. 422-425)

It must not be understood from the above quotation that pro-

gramming languages other than IPL (Information Processing

Language) could not fairly easily express the same struc-

tures or that IPL must necessarily mechanize only structures

of a certain kind. What is true is that the use of IPL

tends to facilitate characteristic approaches to the model-

ing of human behavior. This is one of the reasons for not

using IPL in the present research; it would have tended to

give direction to the analysis and simulation process when

what was wanted was a more naive approach. The programming

language chosen was a dialect of Fortran. Fortran is per-

haps more neutral with respect to psychological theory im-

plications than other higher level languages, and has the

considerable advantage of being available as a standard (if

not the only) language at nearly every computing installa-

tion.

As a matter of methodology, the analysis which led

to the model was intended to be free of hypotheses, no mat-

ter how plausible, about the processes the Ss brought to

bear upon the decision making task. No particular notions

of memory processes or cognitive concepts were assumed prior

- 5 -SD 70-576


to detailed protocol examination. It was intended that the

protocols speak for themselves, at least in early analysis.

The model predicts the behavior of individual Ss,

not general or ideal behavior. The idiographic approach was

selected because of an interest in predicting individual

variations in behavior. Several Ss were used, not just one

or two. The model was to be structurally homogeneous for

all Ss . To serve as a model for several different individ-

uals, parameterization was needed. Just as Weizenbaum* s

(1966) Eliza program works from different scripts to produce

different verbal behavior, the model in this research works

from profiles of different parameter values to behave as

different subjects.

Frijda (1967) asks,

When a program is constructed on the basis of a given

set of data, how can we make sure that it applies toother sets? With abstract tasks there is not too muchdifficulty. One varies the inputs for both subjects

and program. (p. 66)

This implies cross-validation. The model is individually

parameterized, using parameter estimates which are optimal

for the sample of behavior analyzed. The process of finding

optimal values for these parameters capitalizes upon chance.

Any model which is supposed to simulate human behavior, but

which is not cross -validated, is of unknown generality or

specificity. (However, a model which attempts to predict

- 6 -SD 70-576


the behavior of several Ss is necessarily of some general-

ity, in its structure at least.) The research employs

cross-validation as a means of evaluating the generality of

the estimated values of individual parameters.

Fundamental to the original purpose of the research

was the development of a quantified measure of goodness of

simulation, preferably a measure which could be directly

related to a classical distribution function. In contrast

with this approach would be the usual reliance upon subjec-

tive evaluation of model output, used with nearly all proc-

ess models. Stochastic theories, typified by statistical

learning theories, are also evaluated by essentially sub-

jective techniques, but even when highly quantified tech-

niques are used, they "must be interpreted as validations of

the laws of probability rather than of the psychological

assumptions of the theories, and . . . the classical teststhe classical tests

of statistical significance cannot properly be applied to

testing theories of this kind" (Gregg and Simon, 1967, p.

275). Because the model was not to include stochastic ele-

ments, there was the hope that classical tests of statisti-

cal significance could properly be applied.

It was the purpose of this research to study a

- 7 -SD 70-576


methodology composed of techniques and approaches which were

determined prior to the gathering of any experimental data.

In summary, the methodology included: (a) a strictly de-

terministic process model, (b) with predictions kept on the

correct track, (c) mechanized by a computer program written

in Fortran, (d) based upon decision making processes em-

pirically induced from verbatim protocols, (c) which pre-

dicted the behavior of several Ss , based upon individual

parameterization, (f) with cross-validation input data, and

(g) evaluated by quantitative measures of goodness of fit.

- 8 -SD 70-576


CHAPTER II

BACKGROUND OF THE STUDY

Computer Models

A theory to account for individual playing behavior

in a simple card game was defined pursuant to detailed

The theory is similar to ananalysis of verbatim protocols

information processing model. This class of models has been

described by Feigenbaum (1963) as follows:

a. These are models of mental processes, not

bra in 'hardware. They are psychological models ofmental function. No physiological or neurologicalassumptions are made, nor is any attempt made toexplain information processes in terms of more ele-mentary neural processes.

b. These models conceive of the brain as an

information processor with sense organs as inputchannels, effector organs as output devices, and

with internal programs for testing, comparing, ana-

lyzing, rearranging, and storing information.c. The central processing mechanism is assumed

to be "serial; i.e., capable of doing only one (or a

very few) things at a time .d. These models use as a basic unit the infor-

mation symbol; i.e., a pattern of bits which is

assumed to be the brain's internal representationof environmental data.

- 9 -SD 70-576


c. These models are essentially deterministic,not probabilistic. Random variables play no funda-mental role in them. (pp. 297-298)

Similarly, Simon and Newell (1966) remark:

Information-processing theories of human thinkingemploy unobserved entities —symbols —and unobservedprocesses —elementary information processes. Thetheories provide explanations of behavior that aremechanistic without being physiological. That theyare mechanistic—that they postulate only processescapable of being effected by mechanism—is guaranteedby simulating the behavior predicted on ordinary digi-tal computers. Simulation provides a basis for test-ing the predictions of the theories but does not implythat the protoplasm in the brain resembles the elec-tronic components of the computer. (p. 337)

Basically, the theory posits that the subject inputs

relevant features of the environment as information, proc-

esses this information through stored procedures, changes

the environment, and modifies the stored information. Dif-

ferential employment of specific processes would ideally be

sufficient to account for all observed individual differ-

ences in behavior. The theory is expressed as a process

model (Gregg and Simon, 1967) in the form of a computer

program. Computer simulation of psychological processes is

one facet of the research. Frijda (1967) describes func-

tions in which this simulation can be of value:

Computer programs can serve as unambiguous for-mulations of a theory. The program language is

- 10 -SD 70-576

\

II


*■,

precise; the meaning of a given process is fullydefined by what it does. . . . Computer simulation>is a means to demonstrate and test the consistencyand sufficiency of a theory. If the behavioral datawhich the theory wants to explain are in fact repro-duced by running the program, the theory has beenproved capable of explaining these facts .... Ex-tensive experimentation is possible by running dif-

:> ferent versions of the program; decreasing or in-creasing fit with behavioral data can indicate therole of various components and parameters. (p. 59)i>

i\

Artificial Intelligence

The research was centered on quantitative method-«.

ology in psychological theory construction and testing.i4

Because of its computer mechanized aspects, it has a secon-

1 dary relationship with artificial intelligence. The goal

. of artificial intelligence is the construction of machines

that exhibit behavior one would call intelligent if it wereV

observed in animals..

Slagle (1967) writes:.

The main purposes of Artificial Intelligence andtherefore heuristic programming are to understand(human) intelligence and to use machine intelligenceto acquire knowledge and solve intellectually diffi-cult problems . A researcher having the first purposeis a psychologist. He observes subjects thinkingaloud while trying to solve intellectually difficultproblems. He constructs a model of such problem-solving in the form of a computer program. He noteshow the performance of his program deviates from theperformance of his subjects. He observes the subjectssome more and constructs an improved model. The cycleis repeated over and over . There are two importantadvantages in embodying a model in a computer program.

.

.

.

.

.

.

1

- 11 -iSD 70-576

*

*r


The model is completely specified and consequencesof the model may be obtained by simply running theprogram on a computer. A researcher having thesecond purpose is interested in getting intelligentbehavior and does not care whether the computer usesmethods that people use. (pp. 3-4)

Slagle thus identifies both psychological and ma-

chine performance interests within the area of artificial

intelligence. In contrast are Feigenbaum and Feldman

(1963) . They draw a distinction between artificial intelli-

gence research and simulation of cognitive processes re-

search :

An artificial intelligence researcher interestedin programming a computer to play chess would behappy only if his program played good chess, prefer-ably better chess than the best human player. How-ever, the researcher interested in simulating thechess -playing behavior of a given individual wouldbe unhappy if his program played better (or worse)than that individual, for this researcher wants hisprogram to make the same moves as the human player,regardless of whether these moves are good, bad, orindifferent. (p. 269)

Existing Game-Playing and Problem-Solving Machines

Three categories of machines may be distinguished.

In the first and largest category are machines designed to

behave intelligently without simulating human techniques.

In the second are machines that are supposed to simulate

generalized or idealized human techniques and processes.

- 12 -SD 70-576


In the third and smallest category are machines that attempt

to simulate specific individuals. Category 1 machines may

employ search techniques that are virtually impossible for

humans to use, being based upon extreme depth of search and

superhuman speed and precision. Few of the game-playing

machines reported in the literature examined appear clearly

to be in categories 2 or 3, though the authors of the NSS

chess player indicate that their machine uses some human

techniques (Newell, Shaw, and Simon, 1958; see De Groot,

1965, p. 376). The General Problem Solver (GPS) seems to be

a category 2 problem-solving system (Newell and Simon,

1963a; Ernst and Newell, 1967). Category 3 programs are

Feldman's (1963) binary choice simulator, Clarkson's (1963)

trust investment selection program, and Johnson's (1964)

concept-formation model. The General Game Playing Program

(GGPP) of Thomas Williams (1965) can play many games, by

following rules and making legal moves. Thiele, Lemke, and

Fu (196 3) programmed a machine to play a modified game of

hearts. Balzer (1966) has written a machine program that

decides which cards to pass in the game of hearts. Berle-

kamp (196 3) has described a program for solving problems in

no-trump double-dummy bridge problems. Except for GPS and

the Johnson model, the aforementioned systems seem to be

- 13 -SD 70-576

fife SpaceDivisionNorth American Rockwell

moribund. Still very much alive is the checker player of

Samuel (1959) . No published references have been located on

the chess player of McCarthy at Stanford. Greenblatt (1967)

at MIT has described a successful chess player. Both chess

machines are in current research and development.

Feigenbaum's (1963) EPAM system demonstrates certain

phenomena usually associated with human behavior in paired

associates and serial anticipation learning of nonsense

syllables. Laughery and Gregg (1962) developed a program

that would follow a scheme to simulate the behavior of in-

dividual Ss in serial learning. Simon and Kotovsky (1963)

say they have modeled human techniques in serial pattern

learning. Findler (1966) presents an attempt to model be-

havior in a complex man-machine interactive problem.

Hunt has done a significant amount of research in

this area and presents a good review of computer simulation

studies (1968). He warns, "Teachers of general psychology

should be aware that, as of this date, no program has been

shown to simulate human problem solving, although there have

been several programs which solve problems" (p. 160). In an

overview of artificial intelligence studies, Solomonoff

(1966) reviews no work of much significance to psychological

theory.

ll+ -SD 70-576


CHAPTER 111

THE EXPERIMENT

The Game

Instructing the subject to verbalize about his be-

havior is a good way to get information about aspects of

behavior that are difficult to observe. In a competitive

game, concurrent or post hoc verbalization would disclose

strategy, etc., to the opponents and disrupt the competitive

aspect of the play. Therefore, a one-person game was con-

sidered necessary. A simple card game of unknown origin was

selected. It is coincidental that one £ vaguely recalled a

similar two-handed game. In the game, a 24-card deck of

ordinary playing cards is used, ace through queen of one red

and one black suit. The top 10 cards, one at a time, are

exposed. The player must accept five of the 10 cards. The

goal of the game is to maximize the sum of the values of the

five selected cards, with ace worth one point and queen

worth 12 points. The reds and blacks are summed separately,

- 15 -SD 70-576


and the larger sum is the recorded score. (See the game

instructions in Appendix A.)

The game was not analyzed prior to the experimental

sessions . It is possible that an optimum strategy does

exist, involving the expected value of the unexposed reds

and blacks. Computation of these expected values requires

fractions with denominators which quickly become too large

for unaided calculation in any reasonable time. Regardless

of the rules of thumb, heuristics, or calculations a S_ might

use, the game consists of a sequence of binary decisions,

each of which is necessarily influenced by the effects of

previous decisions in the same hand. When the first of the

10 possible cards to be offered is exposed, S_ must decide

to accept or reject it. S_ may reject the first five cards,

in which case the acceptance of the last five is forced. If

the offered card is accepted, it is held by S_; otherwise it

is left on the table face up. Within the first few cards,

S_ will accept one. Now color (suit) necessarily becomes of

some importance. Only the suit with the highest sum counts

for the score on a hand. A priori, a bias for accepting

cards of the same color as the first would seem reasonable.

If S_ has accepted what he considers a high card of one

color, he might thereafter accept only cards of that color.

- 16 -SD 70-576


If he has selected only one or two cards with a modest par-

tial sum, he may switch to the other color if a card of

sufficiently high magnitude is offered. Whether his handhas one or both suits, he should select cards which willtend to give him the highest final score. He might wish to

hold out for only high cards, but he knows or quickly learnsthat the unseen remaining cards might all be lower than the

ones he has passed up. As more cards are displayed, the

player has increasingly more information about the unexposed

cards, but he has fewer opportunities left to make selec-

tions. The minimum number of choices is five, which obtainsif S either accepts or rejects all of the first five offered

cards. The maximum number of choices is nine, consisting of

four rejections and five acceptances, in any order.

Experimental Procedure

The 12 Ss were recruited and tested by an independ-

ent experimenter as part of a separate research project.

Each S_ was administered the same sequence of decks, 30 in

each of two one-hour sessions spaced one week apart. During

the play, E displayed the top 10 cards, one at a time, call-

ing out the color and value. S was instructed to "think

aloud" as he decided whether to accept or reject each card.

17 -SD 70-576


After all hands had been played, E conducted an inquiry,

asking S what processes he employed. The verbalization wastape-recorded, and the verbatim transcription of the record-

ings constituted the experimental data.

Subjects

The 12 Ss, six male and six female, were enrolled

in introductory psychology classes at the University of

Southern California. They volunteered to participate in the

experiment in partial fulfillment of an obligation imposed

upon all such students. No monetary reward or other incen-

tive to participate was offered. No incentive was employed

to motivate high scores in the experimental task.

Subject responses to a pre-game questionnaire are

summarized in Table 1.

The grade point average is based upon A = 4 points .The last four columns of the table are based upon responses

to the following question and instructions. "Which of the

following games have you played during the past 12 months?

Please answer each of the following with a check mark." The

games specified were blackjack, bridge, canasta, hearts,

pinochle, poker, and rummy. The checks could be made under

"often" (average of once per week), "sometimes" (average of

18 -SD 70-576


19 -SD 70-576

tnQJErdO

S-lQJ> in ro r^ (N ro ro kO

OJ

<tf

r^

tnG

"H>.id

r-ia,

EO

rHQJ

CO

<-i <N ro o CM

CM

ro ro CM r-i in ro

4-1o

inOJE

"H-PQJEo

c/_

OJ

O rH o ro CM CN

CN CM

O o O>ivcI)

en0)s-ife

cQJ

4-14-1O

o o o o O O CN CN o O o Ow0)o.ao0.

cc;

CDSh

"H

CDC

ao"H-P

CQ

Q)

a?

0)or"

oS3inQJ O

2oS3

tna)

inQJ

inOJ1*

in ina;J*

tnat QJ

OJQJ tnrd vv 0)O >

inr-~(N

in

OJ

"roro

OJ

oro

O#

ro(N

ror^

t

CN

m

CM

I

rooro

inOJro

in

t

rooro

0)E

CO

Oi

0)*H

PL,

QJCOJ

rd tn+J >ig rfiaja

rdcrdMP

U

0.

EOJ

rHW

Otno<-i"H

C■HcTSrd

tn3

CQ

r-irdo

"HT_)

QJE0)>-i

rd rdO M

-H Q)tn -C>. -Pfia.

-H Q)rd OU C

"H QJ-P -H"H OrH 03o

«Hidv

"H

gQJVi0.

>itno

rHo"H

CQ

tnS-itncQJ

v1)2

rH cvrd vo c

-H d)-P -H"H UrH 03OOn

S-iO

■i—ird2

QJtn<

CTiiH

00I—l

O

CN

CT.i—l rH

CDrH

CTirH

OJfN rH

00rH

o(N

r^

tntnrdrHv

rC01QJS-lfe

"g,o

c/_

■g,o

COoCO

tnQJi-ife

rCtnOJvfe

■g,oU_

S-lO

"HcQJ

CO

tnOJ

fe

fitnQJS-lfe

S-lo"Hc»-3

fitnOJ

fe

X0)

COfe fe fe 2 2 2 fe 2 2 fe 2 fe

rOl3 S3CO

rH

CN

ro <tf m vo r^ CO (Ti oiH f-i

OliH


once per month), "seldom" (average of four times a year),

and "never."

Apparatus

Ordinary playing cards were used as stimulus mate-

rials. Because the decks consisted of only two suits red

or black and aces through queens—each standard 52-card deckyielded two 24 -card decks. A computer program was used to

shuffle the cards in a pseudorandom arrangement. Sixty in-

dependent shufflings were used to produce the 60 decks used

in the experiment. For each trial, the integers 1 through

24 were randomly permuted. With the cards arranged black

ace through black queen and red ace through red queen,

numerals representing the permuted integers were marked on

the backs with a felt-tip pen. Arranging the decks with the

numerals in monotonically increasing order produced the

desired random shuffle. By marking the trial number also

on the back of each card, it was simple to present each S

with the same sequence of stimuli by reordering the cards

between sessions. A tape recorder was used to record S_'s

response to each stimulus and his concurrent verbalization.

A room ordinarily used for research with human Ss was em-

ployed.

- 20 -SD 70-576


Analysis of Verbal Behavior

S_s were instructed to "think aloud" while responding

to the offered cards. They were not, however, encouraged to

be especially productive verbally. To have reminded Ss

frequently to explain why they accepted or rejected each

card would have been disruptive. Furthermore, it may have

been threatening in instances where choices might have

seemed unwise upon second thought. Requiring a reason for

each choice would have tended to produce behavior that was

more than ordinarily analytic for most S_s . Whatever the

disadvantages of oversoliciting verbal behavior concurrent

with choice making, it appears in retrospect that the in-

structions should have strongly encouraged each S_ to give a

reason for each choice, even if in a highly abbreviated

manner. E could have reminded Ss to give reasons when ver-

balization became too sparse. Only a few Ss gave reasons

for most of their choices. This more or less meager verbal

component to the game -playing behavior dictated that the

analysis focus upon a single bit of information at each

choice point rather than upon stated reasons for making a

move. This facilitated automatic analysis of the behavior,

but it was not included in the preliminary research scheme.

After all 60 hands were played, each S was asked if

- 21 -SD 70-576

ii

t!


he had developed any strategy or style of play. The reasonsgiven were not sufficient in themselves to permit develop-ment of a theory to account for the playing behavior. Ss

did not necessarily play the way they said they had. How-

ever, results of this inquiry, plus the verbalization during

play, were useful in suggesting a model and eliminating some

a priori expectations. For example, Ss were not quantita-

tive in approach. One S stated that he played as if both

colors were equally likely to be offered, although he had

the opportunity to notice from the exposed cards that the

cards not yet offered would be relatively richer in one

color or the other. No complex strategies were verbalized,

hence the model had to be correspondingly simple. Any

theory developed had to be consistent with the verbaliza-

tions during play or inquiry, taking into consideration,

however, the inconsistency in playing behavior itself.

- 22 -SD 70-576

fife Space DivisionNorth AmericanRockwell

CHAPTER IV

THE MODEL

Induction of Process Rules

The first session provided an opportunity for the

Ss to over learn the game and to develop stable playing be-

havior. The second session of 30 hands provided the experi-

mental data. The transcriptions were arranged with two

hands per page. To permit cross-validation, alternate pages

were set aside before detailed analysis was begun. Sixteen

hands thus constituted the corpus for derivation of a fea-

sible model and parameter values. Basic to the research was

the notion of a single model nomothetic in structure but

idiosyncratic in function, with the idiosyncracy due to

individual parameter values. No

structure dictated the analysis,

mation about the subjects except

tocols, the analyst did not know

a priori ideas about the

however. Having no infor-

that contained in the pro-

that of the 12 students

only one had an academic major interest related to

- 23 -SD 70-576

l!


quantitative methods. Point-counting strategies for the

game of blackjack have been well publicized, so some simi-

larly quantitative approach was the object of a shallow

search through the protocols, especially the postplay sum-

mary statement by the S_s . No point -counting strategy was

apparent. One protocol was then selected for detailed

analysis, move by move. The protocol selected was that of

S_ll, a relatively verbal S_, who also had the highest average

score over the entire 60 hands .The recorded verbal content alone was not enough to

permit induction of a theory to account for the decisions

made during play. (See the protocol of S_ll in Appendix B.)

Serious protocol analysis was begun with an attempt to en-

code detailed behavior. It was hoped that some basis for

understanding might emerge once the essential elements were

exposed by stripping away the not -very-helpful verbal mate-

rial. Hand 31 was thus encoded; eight decisions. It became

apparent, even with this very small sample, that context was

involved and that no simple coding of each decision would be

adequate. What seemed better was encoding of the entire

hand. For S.ll on H3l, the very first hand analyzed, no

understanding emerged from the sequence of the successive

decisions, which were related, of course, to the offered

- 2k -SD 70-576

l!


cards. Examining the eight decisions as a whole suggested

that a 7 was high enough to accept as the first card, a 12

was high enough to accept in the opposite color, and that a

4 was too low to accept any time. Two more hands were ana-

lyzed for these three parameters, after which it seemed

that three were insufficient. The insufficiency was indi-

cated in H35. After the first accepted card, no others wererejected. The notion of dominance was defined to facilitate

a description of taking cards of opposite colors. A color

was said to be dominant if the sum of the values of the

cards already accepted of that color was greater than the

corresponding sum for the other color. Before any cards

were taken, neither color was dominant, and tied sums are

not especially rare. The first four hands were scored for

five parameters: highest reject before acceptance, first

acceptance, lowest acceptance of nondominant card, highest

reject of dominant card, and highest reject of nondominant

card. Then, because more context information seemed im-

portant, the dominant sum at the time of the decision making

was linked to four of the previously mentioned parameters,

and lowest acceptance of a dominant card was added. That

made a total of 10 parameters. In addition, the number of

changes in dominance (which might be called color switching)

- 25 -SD 70-576


was recorded, as was the number of forced acceptances. Note

that forced acceptances were not counted as choices, and no

notice was taken of the color or magnitude of these cards inany analysis. The first seven hands were scored or rescored

for these 12 kinds of information. Because color switching

may occur four times in a hand, the number of entries for a

hand under this scoring scheme may exceed 12.

In the first seven hands analyzed, Sll both accepted

and rejected 6 in considering the first card to count toward

his score. In trying to put this into context, it was noted

that a 6 was accepted as the fifth or third offer, but not

as the first. This called for addition of another statistic

—the number of cards rejected before first acceptance. To

facilitate subsequent computation of goodness of simulationanother statistic was added to the list —number of choices .All 16 hands were scored for these 14 statistics . The ana-

lyst then attempted to organize all this information to de-

velop the first model, one that would describe the play of

Sll.

Three pieces of information were recorded about the

first acceptance: highest reject before acceptance, the

number of cards offered before acceptance, and the magnitude

of the first accepted card. On H5l, Sll accepted a 5 after

- 26 -SD 70-576


having already rejected a 5 and a 6. This, plus his accept

ing a 6 as the fifth- or third- but not the first-offeredcard led to this induction: the threshold for first accept-

ance changed as a function of the number of cards offered.

For Sll, it was true that he always accepted a 7 or higher

and accepted as low as 5 on the third or subsequent offer.

This kind of behavior —changing the threshold as play pro-

gressed—became a candidate for inclusion into the model.

The structure of the model related to the first acceptance

seemed satisfactory.

Attention next turned to acceptance of nondominant

color cards —color switching. For the 16 trials analyzed,

the highest reject of nondominant cards ranged from 2 to 12.

There were six rejects of 12. Yet S_ll four times accepted

nondominant cards such as 6, 8, and 9. Clearly, something

besides magnitude would be needed to describe his behavior

in accepting the nondominant color. That something was a

context factor —the dominant sum at the time of the deci-

sion. Thirteen times out of 16 when S_ accepted a nondomi-

nant color he shifted the dominance margin—the amount by

which the dominant sum exceeded the nondominant sum. Put

another way, his partial score at that point would be higher

if he took the nondominant color card. Twice he took as the

- 27 -SD 70-576


second card a card of the same magnitude but opposite colorBut twice he rejected the same kind of offer. He verbalizedabout "opening up options" by taking a card of opposite

color. The analyst formed the hypothesis that Sll's play

with respect to nondominant cards could be described by this

rule: if you have already accepted a card, accept the samevalue in the opposite color to open up options. After ac-

cepting two or more cards, you accept a card of nondominantcolor only if that color will become dominant if the card is

accepted. On H6O, Sll took an 11 red when his partial sum

in black was 19. This was his only instance of taking a

nondominant card, after already taking two cards, when it

did not shift the color dominance. It was assumed to be

anomalous behavior since it did not "open up options" and

led to a below-average score.

A consistent description of behavior with respect

to decisions about dominant-color cards was difficult. He

both accepted and rejected 2, 3, and 4 of dominant color.

No simple threshold for acceptance would be adequate. An

attempt was made to take context into account. Already re-

corded was the dominant total at the time of the decision.Perhaps the desirability of a card was relative to what he

already had in his hand. A detailed analysis of the partial

- 28 -SD 70-576


sums of the cards already accepted indicated that no con-

sistent rule could be formed by including partial sums.

Similarly, taking into account the number of choices

remaining did not yield consistency for S_ll. Magnitude

alone was tentatively selected as the only parameter of the

decision to take a card of dominant color.

The first model, based upon the protocol of Sll, was

expressed as five process rules .Considering first acceptance:

1. Accept first or second offer if greater than or

equal to 7.

2. Accept third or subsequent offers if greater

than or equal to 5.

Considering subsequent acceptances:

3. If only one card has been accepted, "open up

options" by accepting a card of equal value in other color

4. If more than one card has been accepted, "stick

to" dominant color by accepting only cards greater than or

equal to 6 in the dominant color.

5. However, accept card of nondominant color if it

gives a higher partial score.

These five process rules refer only to card-

magnitude values. A fundamental notion to be tested was

- 29 -SD 70-576


that other Ss would behave similarly, but with some varia-

tion in the magnitudes, which constituted thresholds. Im-

plicit in the five rules is another kind of threshold: the

point at which the magnitude threshold is changed. The

offer number on which the threshold for first acceptance

changes can be viewed as a threshold. It was predicted that

some Ss would require more than just a higher partial score

in accepting a nondominant card. They might require the

partial score to be changed by a minimum amount. Similarly,

it was predicted that other Ss might not accept opposite but

equal cards as the second acceptance. They might not want

to "open up options." The latter two predictions were a_

priori , as no other protocols had been analyzed. They were

hypotheses to be tested.

The five process rules were rewritten to permit

seven parameters. With slight wording change, the rules as

hypothesized after analysis of S_ll were used throughout the

remainder of the research.

1. Considering first acceptance, reject values less

than P , until offer P .2. Then first acceptance must be as large as P .3. Considering subsequent acceptances, if the num-

ber of cards already accepted is less than P , accept a card

30 -SD 70-576

1

1


of the nondominant color if it will result in a shift in the

dominance margin (gain) by as much as P .4. If the number of cards already accepted is equal

to or greater than P , then the gain must be as much as P .4 65. Accept a card of the dominant color, or of

either color if neither is dominant, if it is as much as P .7Closely related to these five process rules is a

description of the seven parameters.

P —lnitial threshold for acceptance of first card,

in terms of card magnitude.

P —Offer number on which initial threshold is

changed.

P —Secondary threshold for acceptance of first

card, coming into effect if S_ does not accept a

card before offer P^ .P —Number of cards accepted before changing gain

threshold (for acceptance of nondominant suit

cards). P. is inversely related to the number

of choices remaining.

p —Initial gain threshold (for accepting nondomi-

nant suit cards ) .p —Secondary gain threshold, coming into effect if6

S_ has accepted P cards .

- 31 -SD 70-576

I

1

I

1

1


P ?—Threshold for acceptance of cards of dominant

suit, in terms of card magnitude.

The parameter values for S_ll, based upon the origi-

nal five process rules, would be 7, 3, 5, 2, 0, 1, and 6.

Once the structure of the first model was erected,

the next step was to see how well it fit other Ss . The

process of analyzing the protocol of S_ll was tedious, taking

many hours of labor. To maximize the return from this kind

of labor, it was decided to select as the next subject to be

analyzed one whose behavior differed from Sll's.

Before the next protocol was selected for analysis,

the similarity of responses between Ss was computed by find-

ing the correlation between the Ss based upon the cards ac-

cepted. The maximum number of choices per hand was nine,

because of the forced acceptance rule, but each of the 160

cards that could enter into the scoring was included in the

correlation computation. The correlations with Sll were

ranked, and S_6 was selected. He had a low correlation with

Sll and was reasonably verbal.

S_6 was not difficult to analyze with respect to the

first three parameters, which refer to the first acceptance.

The lowest acceptance was 7, except for a 6 taken after four

consecutive rejects. The first three parameters were

- 32 -SD 70-576

I

l


therefore 7, 5, and 6. S6 was less consistent in decisions

about nondominant cards . Seven times he took nondominant

cards that did not increase his partial score. Defining

gain to be the difference in partial score to be realized if

a nondominant card were taken, it was noted that only six of

the 13 acceptances of nondominant cards resulted in positive

gains, three yielded zero gains, and four resulted in nega-

tive gains. Perhaps the very magnitude of the nondominant

card accepted was the determining aspect of the decision.

However, the magnitudes were 9, 4, 12, and 11 for the in-

stances of negative gain; only two were "face" cards. For

the three instances of zero gain, all were second accept-

ances, equal in magnitude, but opposite in color, with mag-

nitudes of 7, 6, and 9. No simple hypothesis about accept-

ing nondominant on the basis of magnitude could be supported

by the recorded play.

Values for the three parameters related to accept-

ance of nondominant cards were estimated to be 2, 0, and 2,

meaning take nondominant cards for zero or greater gain as

the second acceptance, but raise this threshold to 2 for any

subsequent acceptances.

With respect to decisions about taking dominant

cards, S6 was much more consistent than Sll. While the

- 33 -SD 70-576

Il

f


optimum threshold for acceptance of dominant cards yielded

eight instances of inconsistency for Sll, the corresponding

number for S6 was only two. S_6 both accepted 5 and rejected

6, but he accepted 5 only twice and rejected only one 6. It

was decided to set the parameter at 5, because, in his sum-

mary statement, S6 mentioned "4 or possibly 5 as a minimum"

in reference to taking what has been defined as dominant.

For S_6, the seven parameter values were estimated

to be 7, 4, 6, 2, 0, 2, and 5. Over-all, S6 seemed to fit

the model induced from the behavior of S_ll rather well. In

fact, there were fewer inconsistencies for S_6 than for Sll

(7 versus 10), given that the model was an acceptable de-

scription of the way Ss made decisions. Therefore, the

first model appeared to have some general validity, based

upon this tiny sample.

Development of the Computer Program

The process rules were transformed into a computer

program flow chart (Figure 1) . By ordinary programming

techniques, several procedures, each performing only a small

part in the administration and playing simulation, were

written in Fortran and checked out. It should be noted that

what was simulated were the decisions to accept or reject

- 3^ -SD 70-576

35SD 70-576

£j^ Space DivisionNorth American Rockwell

5I*

zUJs><

zoUJ_J

III

v—1ca__

i>

AlA/ \

/

\

\/

r , A ' \ SI / \ /_ \

i_vK^ii^ <2is§,>3*a I \<z

/

\-°o v/ sI f I

\

/ \

O

/ I

ir I oA r i r i oI \ | I Sal

/ * ! ° I 1 9

Q

5 I eh/3 .. cA v I < <__ § o

\ / s i y 5S »y U lt J g

i

I

fI

It

I.

t

.Ir— '

1 Al " S

O O

■t h iti A s."

UJ>

5-Zuj

_| co

s-

__vUJ__

- Xzi—

o

r—— 1l II °" °"I

i^i- i

ii nj o liI ?x-I "III

oz

f—l!_,..-_I

O

II |I ix-K-iIs| ;

ii \

■or\/

V

': is 1 - A\ -I I 3 I 2 __/_.._ 2 \" r. i z *\

_gj^ / (U. i y ' „ \ So / v

| I- i g V* / So! ! J O. U-l V £

111! c iI UJ

[T J?

'' \ \~ iv* / i !L—i

c

"_

L\->2

C

i

O

z

/

Q

< "a z\ x

___»

a_

5zv.u-o

" < °££_vl

UO

£

i!

IIi

i

■

i

'

t

i

>-a<UJ

Q-

U3is*

oz

___.

UUJ<___

XzI—

o

t_.

<vBz___:UJu-u.o

/UJ >__-<si So\< J V 1? .?_5 2

UJJ-

I:!H

II


the offered cards . The cognitive behavior that accompanied

these decisions was not observable and, therefore, could not

be simulated. The programming language itself has conven-

tions that tend to obscure the psychological functions the

program procedures perform. The administrator functions are

mixed in with the subject functions in most of the proced-

ures. Still, the coding reflects the seven process rules

accurately, and it should be understandable to almost every-

one acquainted with Fortran. Most existing computer pro-

grams related to artificial intelligence or game playing

were written in IPL or LISP. The languages were designed

for these very applications, but are not readily available

everywhere .

Simulation Program Structureand Function

In its latest version, the computer program that

mechanizes the model, based upon analysis of Sll's protocol

and partly supported by analysis of S_6 's , consists of a main

procedure and nine entry points (subroutines). The follow-

ing description should be read in conjunction with the

source code in Appendix D and the flow chart (Figure 1).

MAIN calls SHUFFL, which reads in the already ar-

ranged stimulus cards and the responses of each S_ to each

- 36 -SD 70-576

111

I.ii


card. MAIN then goes into the outermost loop, iterated once

per S_, by calling NXSUB, which reads in subject identifica-

tion and parameter values and prints them out. MAIN then

calls NXDECK, which gets the identifications of the next

deck, the cards for the trial, and S's responses to them.

NXDECK also heads a new page with information about

the hand, initializes several program variables, and estab-

lishes the initial thresholds for first acceptance (P_. ) and

for acceptance of nondominant cards (P. 5) . MAIN then calls

NXCARD, an administrator function that offers each card in

turn. If, by the rules of the game, S_ is forced to accept

the card, MAIN has S_ do so immediately after NXCARD offers

it. Beginning with offer P 2 , the threshold for first ac-

ceptance is changed from P^ to P_ . If the magnitude of the

card is less than the threshold, MAIN calls REJECT, which

compares the decision of the model with that recorded for

the human S_. REJECT helps keep count of the number of er-

rors and choices made. If the model is correct in rejecting

the offer, control is passed to statement 10, and another

card is offered. If the model has incorrectly rejected, an

error message is printed and control passes to MAIN'S call

on ACCEPT. The model is not allowed to diverge from S_'s

track. Along with REJECT, ACCEPT keeps count of errors and

- 37 -SD 70-576

II

,


choices. ACCEPT compares the model's acceptance with S_'s

recorded decision. If they match, the card is added to S_'s

hand. If the hand is not complete, control is returned to

MAIN. Otherwise, ACCEPT calls COUNT, which first determines

if there has been any change in S_'s hand since it was last

called. This is to correspond with the assumption that

human Ss probably do not count up their hands again unless

there has been a change. They remember. The partial red

and black scores are counted, and it is determined which

color is dominant. The partial score for the hand (the

final score if five cards have been accepted) is the red sum

if red is dominant, the black sum if black is dominant, and

the color of the last card accepted if neither is dominant.

COUNT returns to ACCEPT, which then calls QUIT.

QUIT keeps count of the total numbers of errors and choices

over all hands, computes error percentage, and prints cer-

tain information at the end of each hand and at the end of

each subject's session. QUIT returns to ACCEPT, which re-

turns to MAIN at statement 1. However, if ACCEPT is called

after a model decision that does not match the corresponding

human S's decision, the card is not added to the hand, and

control is passed to statement 10 in MAIN, after recording

the error and printing a message. The model is kept on S_'s

- 38 -SD 70-576

1

1

n

i


track. The first card acceptance loop in MAIN continues

until the human S_ accepts a card, either forced or volun-

tary. Forced acceptances are represented by the coding be-

ginning at statement 41 in MAIN. Program control remains in

a short loop until five cards are accepted, in the case of

forced acceptance. (Once S has rejected so many cards that

an acceptance is forced, all subsequent acceptances are also

forced, obviously.) If acceptance is not forced, then con-

trol in MAIN is passed to statement 5, which calls COUNT.

The next statement tests whether the card under considera-

tion is of the dominant or equal color. If it is, then if

its magnitude is greater than or equal to P? , the model de-

cides to accept it.

The error checking described for first acceptance is

performed similarly for acceptance/rejection of dominant and

nondominant cards. Whether the decision was to accept or

reject a dominant card, control is passed to statement 4,

where another card is offered. (Any time ACCEPT takes the

fifth and last card, control goes through COUNT and QUIT, as

described.) If the card under consideration is of the non-

dominant color, control in MAIN is passed to statement 7.

If as many as P4 cards have already been accepted, then the

threshold for acceptance of nondominant is changed from P_5

- 39 -SD 70-576

I;;

II;IIII

!li

:S

I


to P^ . The threshold is for the minimum gain that would be

realized by accepting the nondominant card. MAIN calls

GAIN, which calculates this value. If the gain is as great

as the threshold, the card is accepted; otherwise it is re-

jected. Incorrect decisions regarding nondominant are cor-

rected and recorded, and a message is printed. Whether the

decision was to accept or reject, control is passed to

statement 4 in MAIN to look at the next offer. Note that

the program is never permitted to make a move that differs

from the recorded move of the S it is simulating.

Because of the many IF statements, a verbal descrip-

tion of the flow of control is awkward, although the program

itself is actually as simple as the process rules that con-

stitute the model.

Derivation of IndividualParameter Values

The analysis necessary to derive the seven parameter

value estimates for Ss 11 and 6 was lengthy and tedious.

Partly in anticipation of a more complicated model with more

parameters, many context statistics were determined for each

choice point. As many as 30 or more statistics were re-

corded for a single hand. The analyst attempted to discover

process rules that might account for divergences from the

- kO -SD 70-576

I

I


seven-parameter model. The tedious analysis was carried out

on the protocols of S_l and S_2 . It was thereafter decided

not to complicate the model further with more parameters,

even though this promised to improve upon the proportion of

identical choices between the human and simulated subjects.

The number of statistics for each hand was cut to 10, which

were needed to estimate the existing seven parameters. Any

one hand might have more or less than 10 because of inap-

plicability or multiple changes in dominant color. The

statistics were highest reject before first acceptance and

the associated offer number, the first acceptance and asso-

ciated offer number, the gain (plus or minus) taken in ac-

cepting nondominant and the number of cards already ac-

cepted before making the decision, the gain (zero or plus)

not taken in rejecting nondominant and the number of cards

already accepted, the highest reject of dominant, and the

lowest acceptance of dominant. (Dominant, in general, in-

cludes both colors if partial sums are equal.)

The simulation program was modified to play each

hand exactly as S_ had played it (the model made no deci-

sions) and to output information that would help define the

10 statistics. From this point on, the original protocols,

with their fairly meager verbalization, were not referenced.

- I+l -SD 70-576

fi:if:

|.

!8iii


The verbalization had helped in inducing the basic model,

and in estimating parameter values for Sll and S6 , but S_ll

and S6 were among the most verbal Ss . Only the decision

itself—accept or reject —was used in subsequent analysis.

This modified program played the 16 hands of the 12 Ss, and

the results were compared with the already completed analy-

ses of Ss 11, 6, 1, and 2. Analysis of S3, using the pro-

gram output, took only one hour, instead of the tens of

hours required for the earlier analyses. This was due in

part to the decision not to look for more than seven model

parameters, but was possible only because the output high-

lighted the information needed.

Because only the binary information at each choice

point was to be included in the analysis, the statistics for

any hand would be the same for all Ss playing it the same

way. The only rarely stated reasons for making a particular

decision might differ among various S_s , but this information

was not considered. To save analysis time, it was noted

which hands had been played the same way by two or more S s ,

and thereafter each particular play of a hand was analyzed

for the 10 statistic values only once. The last eight

analyses took only about five hours total, as compared with

tens of hours each for the first few. This great economy

- U2 -SD 70-576

i'l

ji;

If

[!;

i

ii[

1 1'


was due to the restricted set of statistics recorded, the

assistance of a computer program that abstracted the sig-

nificant information, and the one-time analysis of hands

played in a given way by any number of S_s .Only the gathering of statistics was computer-aided.

Estimation of parameter values was a matter of judgment by

the analyst. For some of the parameters, this judgment was

easy to make. P 7 is the threshold for acceptance of domi-

nant. For all S_s, the highest reject was greater than the

lowest acceptance, over the 16 hands. Thus, no matter what

estimate of P 7 was chosen, there would be some errors of

simulation in accepting dominant. The estimate chosen was

the one that would yield the fewest errors. Py may be con

sidered as a one -dimensional parameter, but the other six

parameters belong to two sets that may be considered to be

three-dimensional: accept card value X until V cards have

been offered or accepted, then accept card value Z. This

made it more difficult to find the point of minimum error.

The search for reasonable parameter estimates was

systematic, involving the construction of several sets of

two-dimensional tables and careful search for points of

overlap or inconsistency between rejected and accepted

values . The elements in the tables represented mainly the

- I+3 -SD 70-576

i

j.

X

I.

itt

hi

i;


extrema, e.g., highest reject and lowest acceptance.

After the parameter values were estimated, the simu-

lation program played the 16 hands by inputting the esti-

mates and behaving at the choice points accordingly. The

simulation results were generally good. Only one unusual S_

was simulated with less than 90 per cent accuracy. Because

the parameter estimates were made through a systematic, but

not completely deterministic technique, there remained the

possibility that the estimates were not optimal. Examining

only the extrema might be overlooking some useful informa-

tion. Early spot checks indicated that the critical region,

around the threshold, was similar whether all information

for each hand or just the extrema were included. Still,

there were differences that might affect the over-all accu-

racy of simulation. Consideration of all information was

out of the question for a manual effort, feasible only if it

could be computer-aided. Several of the modules of the

simulation program were used intact and several others were

drastically modified, and a program was developed that com-

puted the error of simulation associated with every possible

parameter value. The MAIN and NXSUB procedures were com-

pletely rewritten to record the errors associated with each

possible P7 estimate in a one -dimensional array, and the

- I+l+ -SD 70-576

li!iii

ifi|:si1

iii

is!';■■■[


errors associated with the possible p i~p 2~P^ anc^ P 4 -P[. -P6

triples in three-dimensional tables. For each S_, 1,120

error -table entries were computed. To estimate optimum

parameter values, it was sufficient to select those associ-

ated with the fewest errors. In many cases, the minimum

error appeared at more than one point in a table, and sev-

eral rules were considered for resolving the ambiguity. The

rule finally chosen was the simplest: to use the lowest

parameter estimate if two or more yielded the same number of

errors. For example, if three errors would result whether

the P7 estimate were 4, 5, or 6 , the one selected would be

4.

By means of this exhaustive-enumeration-like analy-

sis, the 84 (7x12) parameter estimates were made in a matter

of minutes, once the computer output was available. To de-

sign, code, and check out the program to perform the ex-

haustive analysis took longer by an order of magnitude than

the manual but computer -output -aided analysis. Were several

dozens of Ss to be analyzed, the time advantage would be

reversed. Of the 84 parameter estimates, 29 were different

from those made earlier. However, the goodness of simula-

tion was improved for only three Ss . This is attributable

to the minima (least errors) appearing several times in some

- I+s -SD 70-576


tables. Several sets of estimates were optimal. Some dif-

ferences were due to errors in the manual analysis. The

averages presented in Table 2 omit the aberrant S_4 .In a sense, there are only three parameters in the

model, not seven. The parameters within the triples (P -P -P and P.-P -P^) are not independent, but in fact closely234 5 6 J

related. Excluding S4, P > P for nine out of 11 Ss . If

P = 1, then P must equal P , and the converse is also

true. P < Pr in all cases, although this is not necessary.5 — 6

If p, =1, then P. must equal Pr , and the converse is true4 5 6

also. P is unrelated to other parameters. There was most

variation in the first triple, less in the second, and least

in P , corresponding to nine, seven, and four different

sets, excluding S_4 .

- I+6 -SD 70-576


11

II

I

»;

I

f1

3

I

II

i!

Table 2. Parameter Estimates

- I+7 -SD 70-576

SubjectNumber

Derived viaManual AnalysisParameter Number

Derived via Exhaustive(Computer) Analysis

Parameter Number

1 2 3 4 5 6 7 1 2 3 4 5 6 7

1 6 1 6 1 0 0 7 6 4 7 1 0 0 6

2 8 2 7 4 0 5 6 8 1 8 1 0 0 7

3 7 3 6 4 0 6 4 7 3 6 4 0 6 4

4 4 4 3 3 10 0 2 4 2 12 1 10 10 2

5 7 2 6 2 0 6 5 7 2 6 2 0 6 5

6 7 5 6 2 0 2 5 7 4 2 2 0 1 5

7 7 2 6 3 2 3 5 7 2 6 1 1 1 5

8 7 4 6 2 0 3 4 7 4 2 1 0 0 4

9 5 2 6 2 2 2 4 5 2 6 1 8 8 4

10 6 1 6 3 1 3 5 6 1 6 1 1 1 5

11 7 3 5 2 0 3 6 7 3 5 1 0 0 6

12 7 2 7 2 6 6 4 7 1 7 1 6 6 4

Mean(rounded) 7 2 6 2 1 4 5 7 2 6 1 1 3 5

Mode 7 2 6 2 0 3 4, 7 1, 6 1 0 0 4.5, 2, 5

4.

rf

(l

[


CHAPTER V

EVALUATION OF THE MODEL

Simulation of Individual Subjects

The model was nomothetic in structure and idiosyn-

cratic in function: nomothetic because a single model was

used to describe the behavior of the 11 synoptic Ss and

idiosyncratic because the model predicted the peculiar be-

havior of these S_s if initialized with their peculiar

parameter estimates .A model can be evaluated by the amount of agreement

between predictions generated by the model and observations

of natural behavior. Because the computer program mechaniz-

ing the game-playing model was set back on the correct track

after each incorrect prediction, the measurement of agree-

ment was simple. The program played through the 16 analyzed

hands, computing certain statistics for each hand and sum-

mary statistics across the hands (Table 3) .The model shows the least agreement for S_4, which

- I+B -SD 70-576

¥

I+9SD 70-576 .

.-J.H

<NH

O

rH

en

oH

CD

HOJH

ro

r—

t

00

CN

H

ro

00-H

<N

CN

in

CNCTi

CO

roen

ooin

m

CN

41m Space DivisionNorthAmerican Rockwell

I

I 'i

i;'i

fn.'

■J..8&,■»:.Inill'1.1.ii1 '

Ife',

en oro

r—

iIN

r—

l

ro

Cn

en

O

i—lHroH

CNrH

Cn CO

<* vI3 enCN

oo

C^

CN

ro

<_>"H4-30.

"H-Pro4L>

r-CNH

ro (Tiin

!h

CO

aBCQ

<D

CT>

CN"H

00CN

roin

CTi

r^

"*

enH

O

(N

r—

lro

CN

CO

ooinH

t

1,

ion IfiH£)roEH

O

rH

CD

CN (N

CN

CN0>

CO

Is'

CNrH

CM(N

roro

CN

o

c^

oo

CT>

Iii

CD

inCN

oro

ro 143

Ij

(ii

!j.i1JIi

i■i

liij*I:t,F

,1

f

vd)

S-J.

ovvm4-1ov<ii

■i3

CO0)U

■Hov

o1-101

I"y

T3<D

O

Vj.O4-<

>-l

y

U<Dvvoo

catv0)a.

S-l

Ol-l140)

+ 1c01vvQi

0,

I

U

i

i


was because S4 rejected virtually all offers of black cards,

regardless of magnitude. If the model had a parameter for

color bias, better agreement could have been achieved, but

at the expense of introducing an ad hoc modification for a

single S_. This would have been antithetical in a model that

should be descriptive of general behavior. No test for

color bias was made for any S_, but no such bias was manifest

except in S_4 .The model was based upon the behavior of S_ll. How-

ever, this did not lead to predictions for Sll that were

more nearly accurate than those for other Ss . The fit of

the model was better for seven Ss and worse for three (syn-

optic) Ss . The S_ whose behavior was best simulated by the

model was S7 . His optimal parameter estimates were the most

nearly typical. The typicalness of S_7 is further gauged by

the fact that he alone failed to play a single hand in a

unique way. That is, his pattern of choices was identical

to that of at least one other S_ on each hand, while S_4

played nine hands uniquely.

Among the synoptic Ss , the percentage of disagree-

ments between the model and observed behavior ranged from

3.17 to 9.84, with a mean of 6.57. The possible range of

error per cent was from 0 to 100. The obtained range, for

- 50 -SD 70-576

«"«

Pi

,!

I:


the 11 S_s on the 16 hands, was from 0 to 71 per cent.

Averaging across the Ss puts the range for the hands from 1

to 16 per cent. Some hands, apparently, were much easier to

simulate than others. When the decisions are easy for Ss to

make, because of extremes of card magnitudes, then of course

the decisions can be predicted with less error.

The only goodness-of-fit measure presented is error

per cent. Another measure considered was phi, the fourfold

point correlation coefficient. Phi is quite sensitive to

the direction of the error —whether model-accepted but S_-

rejected or vice versa—while error per cent is completely

insensitive to it. Phi is undefined for the several cases

of zero marginal frequencies. When phi is unity, proportion

correct is also unity, but the converse is not true because

of the undefined phi cases. Error per cent was selected as

it is well behaved and easily interpreted. Tests for the

statistical significance of proportions or of differences

between proportions are inappropriate because the decisions,

upon which the error per cent was based, were highly corre-

lated because of the conditional prediction effect. Unfor-

tunately, then, evaluation of the model as a description of

the behavior of this sample of Ss remains subjective. A

sometimes suggested test of computer program output

- 51 -SD 70-576

Ii

li


agreement with recorded natural behavior is a kind of

Turing's test (Turing, 1963, p. 11). Turing called it the

"imitation game." Stylized protocols of human subjects and

computer program output, where the program did not use the

conditional prediction technique, could be presented to

naxve judges. The judges would then try to segregate the

machine and human protocols. However, this would not test

the model, but only whether machines can be programmed to

play the game involved, an obviously unnecessary demonstra-

tion .In discussing the model evaluation problem, Newell

(1966) notes that

in assessing the validity of the program to describeor explain the subject's behavior, two things aremissing to which psychologists have been accustomed.First, there is no acceptable way to quantify thedegree of correspondence between the trace of theprogram and the protocol. This is not a problem ofmaking the inference definite or public. Trace andprotocol can be laid side by side. . . . However,comparison still must be made between an elaborateoutput statement and a free linguistic utterance.Although a human can assess each instance qualita-tively, there are no available techniques for quan-tifying the comparison, or summarizing the resultsof a large set of comparisons .

Second, the program has been created partly withthe subject's protocol in view. Thus, somethinganalogous to the calculation of degrees of freedomused in fitting curves with free parameters to datais appropriate. But programs are not parameterizedin any simple way and no analytic framework yet ex-ists for allowing for degrees of freedom. (pp. 3-4)

- 52 -SD 70-576


The present research cannot claim to have solved the

problem of assessment, and indeed it has not, but error per

cent based upon conditional prediction is a reasonable quan-

tification of comparisons, and the model was cross -validated

with protocols that were not in view during program crea-

tion.

Sensitivity of Model toParametric Variation

The behavior of the computer program was of course

completely determined by the parameter values. The pre-

dictive ability of the computer program was determined by

the proportion of variance in natural behavior accounted for

in terms of the parameters . Information on the relative

importance of the parameters can be obtained by comparing

models with different parameterization. If a parameter were

added to the original model, differences between the pre-

dictive ability of the original and augmented models could

be attributed to the added parameter. By generating model

variations of different strengths, attributable to different

parameterizations , the predictive validity of the individu-

alizing parameters can be estimated.

i ir'i

i .

i

During the manual analysis of the protocols, some

parameters were considered in addition to the seven. For

ii- 53 -SD 70-576


example, after the simulations were performed, and the

optimal parameter estimates computed, it seemed that the

triplet for accepting nondominant was insufficient to pre-

dict the associated behavior. Although during the original

analysis of Sll, the notion of gain seemed important, in

several hands for each of the Ss the magnitude of the non-

dominant offerings apparently led to acceptance of a card

that could not hope to lead to an over-all higher score.

There seemed to be a "face card" effect, and there was some

mention of this in the summary statements. It was decided,

however, not to add a parameter for face card effect in

accepting nondominant offerings. This and any other pos-

sible parameters would have complicated the model. The goal

of the research was not to obtain the most accurate simula-

tion possible, but to explore and understand methodology.

It can safely be contended that additional parameters would

have resulted in a better average fit. It will be shown

that each of the seven parameters contributed some covari-

ance in the prediction process. It would be no surprise,

and no contribution to theory, to discover that adding

parameters (predictors, in effect) resulted in greater cor-

respondence between model and S_ behavior. As long as there

were non-zero correlation between the additional parameter

ii

Ij i

t

- 5^ -SD 70-576


and human play and non-unity correlation with other param-

eters, an increase in correspondence is a mathematical in-

exorability. It was more interesting to make the model less

predictive by removing the effects of parameters in certain

ways. By using exactly the same MAIN program, it is pos-

sible to examine the power of weaker models.I.

Consider a model with no change in threshold for

first acceptance. Using the existing simulation program,

we get exactly this simpler model by setting P 9 = 1, and

P 3 = P 1 ; only P can have any variation in the first trip-

let. In effect, the model then has only five parameters.

Another simplification would permit no change in threshold

for gain in accepting nondominant. This is achieved by set-

ting P = 1, and P = P , leaving only one variable in the

second triplet. Were both of these simplifications made,

the model would have only three parameters . if p were set

to some fixed value, instead of being allowed to take on the

value optimal for each S_, then it would no longer function

as a parameter, but as a constant. The elements of the

triplets must be treated together, as they are closely con-

nected. The triplets and P could all be set constant, and

the result could no longer be termed a parametric model.

I

i

i

ii ■

mli

; ;

ft

If the sample had been very large, then a model

- 55 -SD 70-576

i

fi

I

f

t

1!

II

t

s


that had only average values as constants would still con-

tain information about human behavior. Such a model could

predict only average behavior. The errors of simulation

would vary for each S_, but there would be an average error.

Nullifying the effect of the various parameters by holding

them constant would lead to increases in average error, if

the parameters had any predictive power. For example, al-

lowing only P ? to vary, holding the triplets to modal values

would yield a virtually one-parameter model. The difference

in average error between such a model and one in which all

values were constant would measure the effect of P_ .There are 32 possible model variations of different

strengths mechanized by the same MAIN program. The first

acceptance and nondominant acceptance parameters may have

one or three elements each, and the parameter estimates used

in the simulation may be the idiosyncratic optimal values or

held constant to the average (modal) value. The 32 varia-

tions belong to four submodel groups, according to the num-

ber of elements involved in making the decisions for first

acceptance and for acceptance of nondominant suit cards .Because the several variations are frequently referred to

in the following discussions, it is convenient to label

these four submodel groups with more or less arbitrary

- 56 -SD 70-576

li<

1

.


numeric identifiers .Submodel o—no0 —no change in threshold for accepting

first card as more cards are offered and rejected. Only one

parameter involved in accepting first card. No change in

gain threshold (for accepting nondominant suit cards) as

more cards are added to hand. Only one parameter involved

in accepting nondominant suit cards .Submodel I—only1 —only one parameter involved in accepting

first card. Three parameters involved in accepting non-

dominant suit cards, due to change in gain threshold as more

cards are added to hand.

Submodel 2 —three parameters involved in accepting

first card, because of a change in threshold for acceptance.

Only one parameter involved in accepting nondominant suit.

Submodel 3 —three parameters involved in accepting

first card and three parameters involved in accepting non-

dominant suit cards .There are three distinct kinds of decisions: (1)

accepting cards of dominant suit, (2) accepting cards of

nondominant suit, and (3) accepting first card. The param-

eters involved in making these three kinds of decisions may

be optimal or modal. There are eight ways that the three

3kinds of decisions can be made (2 = 8) . Because parameters

- 57 -SD 70-576

I

I

il

I!

$


can be optimal or modal (two states), binary notation is

convenient. More or less arbitrarily, the low order three

bits of a five bit binary number can be used to convey in-

formation about these three kinds of decisions. Table 4

shows the meanings of all five bits. The bits are numbered

from low order to high order, from Ito 5 . The two high

order bits are used to identify which submodel group is

being considered. The five bits identify the four submodels

and the eight variations within each submodel. Octal nota-

tion is more compact than binary notation, and at some

points in the following discussion, variations are identi-

fied by an octal number. Variations 00-07 are in submodel

0, 10-17 are in submodel 1, etc., according to the first

octal character. The second octal character identifies the

variation within the submodel and represents the low order

three bits in the binary identifier.

As described earlier, there are three kinds of de-

cisions, and predictions of these decisions can be based

upon optimal or modal parameter estimates. In Table 5, the

variations within each submodel group are listed in order of

the number kinds of decisions based upon optimal parameter

estimates. There is one none-optimal variation, three one-

optimal variations, three two-optimal variations, and one

- 58 -SD 70-576

i


Table k. Meaning of Bits

" '

' " -

■

"* """'" ' '—

5 Three first take param-eters

P 2 =1, P 3 = P 1; one firsttake parameter

4 Three nondominant takeparameters

p4 = 1, P, = P ; one non

dominant take parameter

3 First take parameter (s)optimal

First take parameter (s )modal

2 Nondominant take param-eter (s ) optimal

Nondominant take param-eter (s ) modal

1 Dominant take parameteroptimal

Dominant take parametermodal

True (=1) alse (=0)


lP

- Go -SD 70-576

ooi-HQJ

TSOE

rQ

3c/_

CW -H

..-I

J*v v

*X>iH

"OrH

Oro

en

r^oo00

CN

CO

rHoCO

00ro

in

QJ QJ■-H rC"H 4-1fiS 4H

O

rHrH

T. <CGrd

CNi-HQJ

TSOE

■8CQ

c-H «

21 tnS o

c Jv v

in rHIA

xsy>"H

rH

OrH

oroen

en

OJ

en

CN

r^

oo

ro

00

00in00

r^

tnen

tn ts4-J C"H rd

rQ

Sh ojO

44 -,ro

inCQ

romm

IT_J0

-Pro■Hooenco<

tnTS tnC -P ""id -h c4-1,0 0tn -h

tn infi Q) ino > QJ"H -H Sh

d ftrH a.

Sh QJQJ TS QJfi 0 fiE E 4J3 ,0G 3 >i

tn ,0rHrd C Sh4-) -H 0O fi 440 4-)

■H TSC S QJrd 4-i

C Ctn 0 3

■H -H O4-1 U

Sh rd UQJ -H rd

"H Sh44 n_ flj"H > Sh4-> rdG U .QJ O QJ

T3 44 rH"H fi

Sh rdrH Q) 4-1

QJ -HTS 4H inO -H -Hc -v

rQ

C 4-)3 QJtn T> C

"H -H

CQ

ojCQ

CN

"I

mCQiH

i

CQen"l

OJ

mr^co

"I

rHCQ>X>

CO

I

r-in

"orH

4L>aQJo

rH

rHQJ

TSOE

■§CQ

? -H■H "5fe U

c .^4-) 21

§18 v

r^in

"or-i

rHr^

en

en

CN

en

00r^

■

en

Ol

CO

CN

enoo

oo

00

roiX)

a

Sh OO

-H

ft

4Js-i roW -h

ft

c roo >"Hy rtro qj

3 oc s"HCQ X

O

oj ro

CD

>

O

rHaj

TSOE

CQ

cH «fe

c '-

I s° 8

in

O-H

rH

en

o

en

cor^

"enCO

00

OJ

en00

rHenCO

inO

"00

Lf\

QJ rHQJ

TSOE

rQ

3c/_

S-iCD

rH 'H10 -H

° cv OJT.H

o rH

OJ

<* ro in >X> r-

C-HrC-P"H

oooiHOO

orHo

OOrH

rHO

rHOr—l

OrHrH

iHrH<-i

cO

"H4->rd

"HS-ird

S-lQJ

>i-HM 4-1rd -HC 4J

■H CCQ OJ

"OIH

QJ

H S4 QJrd -HC Sh"H 4->

rQ

CQJas

,c oj

4-i ro

;l

■I

I

1

fife Space DivisionNorthAmericanRockwell

three-optimal variations.

Variation 00 uses three distinct values (P , P , and

P), all of which are modal, while P, p p and P are2 3 4 6nullified in effect. Variation 37 uses seven values (the

seven parameters), all of which are optimal. The variations

have from zero to seven model elements that can take on

idiosyncratic optimal values. It can be shown that the num-

ber of idiosyncratic elements in a variation is given by the

expression B ! +B2 +B3+ 2B

2 B4 + 2B3

B5' where B -i represents

bit 1, etc. The products B^ and B B represent interac-

tions (joint effects). The predictive power of a variationis related to the number of idiosyncratic elements. The

expression should be compared with the expression for error

of simulation, to be described later.

For the sample of S_s examined, there is no change

in threshold for nondominant, on the average. That is, P

is modally 1, and the threshold for gain in this case is

always P_ . Therefore, when B is false, B. gives no addi--5 TL 4

tional information. Variations represented by B false have

the nondominant take parameters modal. The modal value of

P. is 1, which means that there is no change in threshold

and thus only one nondominant parameter; B false implies

B false, for this sample. Thus, some of the variations are

redundant. The pairs of identical variations are 00-10,

- 61 -SD 70-576

I

II;

Hi

ii

i!

i


01-11, 04-14, 05-15, 20-30, 21-31, 24-34, and 25-35. There

are only 24, not 32, distinct variations.

The simulation program played the 16 hands based

upon the 24 variations for each of the 12 Ss . The results

are shown in Table 5. Based upon the average errors of

simulation associated with each variation, it is possible to

isolate the contribution of each bit and of each interac-

tion .The interactions are between B and B , and between

B_ and B_ . If there are three parameter elements for ac--3 b

ceptance of nondominant, then the effect of allowing idio-

syncratic variation is greater than for the case of only one

parameter element, the basis for the B -B interaction.

Because, in this sample, the modal values are such that B

false implies B false, B can have no contribution of its

own. By examining the differences among the average errors

in Table 5, it is possible to write an expression that gives

these errors as a function of the bit values. The expres-

sion is exact. Thus the sensitivity of the model to its

constituent parameters can be estimated. Each error of

simulation yields about 0.8 per cent error, considering the

approximately 127 choices predicted. For the experimental

sample, the error of simulation for the several variations

- 62 -SD 70-576

I

p.

it

x

t!ti

f !'lf'.'

.

i

-.

Iliif

if


is given by 10.57 - .868 - -878 - .798 - .OB - .418

.428 B4 - .658 B . To determine the error reduction asso-

ciated with each bit, the average error per cent for the

variation represented by the binary number with only that

bit true is subtracted from the average error per cent for

variation 00 (all bits false) . The difference in error is

the reduction associated with setting that bit true. The

results of the several subtractions are given in Table 6.

Table 6. Partitioning of Average Error of Simulation

- 63 -SD 70-576

BinaryIdentifier

Octal ErrorBit (s) Identifier Reduction

B, 00001 01 .86

B_ 00010 02 .87

B 00100 04 .79

B„ 01000 10 .004

B 10000 20 .41

2&4 01010 12 .42

3&5 10100 24 .65

t

it

f

I!

:i


The difference between 00 and 37 (all bits true) is 4.00,

but the sum of the first order effects associated with each

bit is only 2.93; there remains 1.07 to be accounted for by

interaction effects. If there were no obvious interaction

effects to look for, it would be necessary to make many sub-

tractions, according to a regular search scheme. In the

present case, the obvious interactions are between the num-

ber of parameter elements in a decision and whether these

elements are optimal or modal. The difference between 00

and 24 is 1.85, but this can be partitioned into a B_ ef-3

feet, a B n effect, and a B_B r

reduction is given by 1.85 -the difference between 00 and

interaction effect. The B B3 5

79 - .41 = .65. Similarly,

12 is 1.29, which is the sum

of the B and BB4 effects. (The B 4 effect is zero.) The

According to Table 4, setting any one bit true must

yield a non-negative contribution to the strength of the

model. This is obvious for bits 1, 2, and 3 from the fact

that they stand for optimal parameter values. Bits 4 and 5

each stand for the addition of two parameters to the model,

which implies greater predictive strength. The research did

not examine the theoretical foundations for the assumption

that the effects represented by the bits are necessarily

- Gh -SD 70-576

B^B^ interaction is 1.29 - .87 = .42.2 4

!

I

%

it;

i!


additive in the manner shown. The calculation of inter-

action effects is dependent upon this assumption. It should

be mentioned parenthetically, however, that the partitioning

described is valid for two other error tables not included

in this report.

The coefficient for B4 is zero, as implied previ-

ously. It can be seen from the expression that B , B , and

B3 each contribute to error reduction by about one error,

while the B_B. interaction and B„ each account for about__ 4 _>

half an error, and the B B interaction accounts for about

three-quarters of an error.

The relative contributions of each of the model ele-

ments can be expressed in a slightly different way, using

the same error information:

Dominant acceptance optimal instead of modal . . . 0.86

Nondominant acceptance optimal instead of modal . 0.87

First acceptance optimal instead of modal .... 0.79

Three nondominant acceptance parameters insteadof one

If parameters optimal 0.42

If parameters modal 0.00

Three first-acceptance parameters instead of one

If parameters optimal 1.06

If parameters modal 0.41

- 65 -SD 70-576

li

i

jii!ti

I

1


Table 5 indicates that the model is not very sensi-

tive to parametric variation. There are several reasons for

this. The conditional prediction effect tends to keep the

error per cent low, reducing the range of variation. The

subjects tended to play in much the same way, although the

individual variation was sufficient to be interesting. (See

Appendix C for a synopsis.) The dispersion of idiosyncratic

optimal parameter estimates about the mode is the source of

increased error of simulation when the model element is set

to the modal value. This dispersion was small for the ex-

perimental sample. Also, for many of the parameters, sev-

eral estimates were optimal for the particular Ss . When

more than one estimate was associated with optimal predic-

tion, the convention used was to select the numerically

smallest of the optimal estimates. The modal value was one

of the optimal values in many cases.

In spite of the fact that no one parameter accounted

for more than about one error on each hand, on the average,

the combined effect of the parameters was very significant.

The difference between the performances of models using

all idiosyncratically optimal parameter estimates and

models using constant parameter values (the modal values)

is 10.57 - 6.57 =4.00. Based upon correlated samples, and

- 66 -SD 70-576

f

1


using an appropriate normalizing transform, t_ = 6.62, df =

10, p < .001. The model predicts significantly better if it

incorporates individually optimal parameter estimates, even

though the effect of any single parameter is not very great.

Cross-Validation of the Model

The model was derived from analysis of behavior on

16 trials out of 60. The first 30 were considered as prac-

tice trials . Fourteen trials were set aside from the second

30 as cross-validation data. No information in these trials

was used, or even known, before the model had been derived

and mechanized. If the model could predict behavior only

on the analyzed hands, it would hardly be of interest. If,

on the other hand, the model could predict almost as well

the behavior on the cross-validation sample, it could pro-

vide the basis for further research and improvement, if the

decision-making behavior in this game were intrinsically

interesting .Both the model structure and the individual param-

eter estimates were empirically derived from the analyzed

protocols . The parameter estimates were especially subject

to capitalization upon chance. By playing the 14 holdout

hands using the parameter estimates that were optimal for

- 67 -SD 70-576

.


the analyzed hands, this capitalization is completely

eliminated. Table 7 records the results of this cross-

validation. The cross-validation shrinkage for the strong-

est variation (37) is from 6.57 per cent to 7.86 per cent,

a shrinkage of less than 20 per cent. This difference is

statistically significant, based upon a t-test of the dif-

ference between the correlated samples. Using the appro-

priate normalizing transform, Arcsine square root (p/100),

t_ = 1.91 with df = 10, P < .05. The choices of each subject

were determined jointly by the situation (the offered cards

as stimuli), individual propensity, and personal-situation

interaction. The parameter estimates were determined

strictly from the recorded choices, and therefore from these

three factors (plus "chance"). The cross-validation shrink-

age of 20 per cent reflects the elimination of the personal-

situation interaction factor. But this shrinkage is con-

founded with whatever differences in model structure might

have arisen from the situation (the particular 16 hands

analyzed) . The truth is that the structure of the model was

induced from the behavior of just one S_ and originally sup-

ported by analysis of just one other S_. Still, had the 14

hold-out hands been the behavior originally analyzed, the

model might have been structurally different. The

- 68 -SD 70-576

j

|i.


independence of the model structure from the particular

stimuli in the analyzed 16 hands cannot be tested directly.

If the model used parameter estimates which were optimal for

the 14 cross-validation hands, and if the predictions for

these hands were significantly inferior to the corresponding

predictions for the 16 analyzed hands, also using optimal

parameter estimates, then it could be supposed that the

model was determined largely from the characteristics of the

16 hands. The results summarized in Table 7 show that the

model predicts better for the cross-validation hands than

for the analyzed hands, with an average error of only 5.65

per cent as compared with 6.57 per cent, for the strongest

variation. This supports the conclusion that the model

structure is valid for behavior which could not have in-

fluenced its derivation. That the difference is in favor

of the cross-validation hands can be attributed partly to

easier predictions, but no analysis for this ease of predic-

tion factor was carried out.

Table 7, Error Per Cent Summary, gives the error of

prediction for six combinations of variation and sample.

The information for each subject is averaged over the 16

hands of the analyzed sample and the 14 hands of the hold-

out sample. Tables 8, 9, and 10 give this information for

- 69 -SD 70-576


!

t

lij

I

!

| i{

- 70 -SD 70-576

01

C

IBX

oi —-v oV op0) _, c

V. PIB ii IB0. * -H

InrH IB

rt £

vO

enrt

rort

enort

rororo

rt

O

rt

rt

t*- vo(N

en inrt

r~rtrt

CN

rHrt

vortrt

in

ort 01

T3

C

IB

YDrt

arHOjcmw

01 —M r--0) ro« m cegoIB .5 H>t z p

IB Q. IB

rH IBrH >rt —

vo

CO

encor-

CO

inrt

1~~ VO

in

IN

ro oo voCN

VO

in 00in vo

co■HPIB

-O■HH

>I0101OIH

O

oN>i

C

<

0] —-n odi o4-1D.CI*Bro m H

to ii na. ** Hi-i

■H >

H

>.!HroBE3

01JJ

IN or~

o00

CD

CD

CN

CO

CN

VO CDO

rH

rt

en

CD

enrt

vO

omrH

enrooo 44oc

cQJ

01"acIB.C

ft

QJ

01■ac19

X

01 —I-i r-Ul roa> « cegoIB .5 HM Z 44

rH >< —

in

coft

rt

in

roin

roCN

IC

(N

1^

in

in

ro

VO

H

VO

o invoin

-a0)N>.rHIB

C

IB

5-1o

ft

H

0)r-Ha,Emw

vO

H

CN rH

ID

o

CD CDCD

CN

coi^-

o

CD VO

<NH

enort

orH

ort

rH

en

0)■cPco

"HPIB■a

■HrHIB>I0101oft

01

-~Sh O

4) oP01 * cc rH oIB IB -HIn T3 PIB 0 IBa_ 2 -h

MrH ,fl-H >

oJJ44o&0]0)

.cp"H

01 -~In r~0) roP *fl) rH

C

c fl ora c -hkH -H 4Jo. a-h

O

XrH (0

2 £

(N

O OCO

CN

rtCN

in

roo CN

ID

O

enCN

00

CO

envoen

O

rt

vo

CO

f-

rHIBTJ

O

cvo

rHIBE"rlPao«p

odl

■ro433

CD

rH r>j fi TT in

VO

r^

CD

0> Ort rH

(NH

tn01

C

ffi-HIB P CTkl -P ~ifl. H "I< §.

fife Space DivisionNorthAmerican Rockwell Ii

\'l

t!

m

- 71 - !SD 70-576

-j<a: 0~-»»pJ"»eo,fiA.£ecoj_rr-ai %d—«t7>r^C'~*m—

«

m p»- —» in> """"""""""""O o^ft©fi©r,-«CCJ*m —* — — OJ o

_J__J<tf C'J"-*!^^ -Cr-COdir<_Do>uj <t co ct _d -fi

m^-^"0

,s oji*'t*->*""■""""""""CJ3O4 r"ir>'*'irr*'<\oD*»co* m *

IT.

o -rmcf-ff us a* m or © r- co

Q£

..""""■"""

C en

o

U_lo'Os^"ar^*^*"^CO'CT"C_^Ci', UT>

(J^<ifl^.f\iJ*tfNJ©OOsÔJ^>*<_r*(_oNC4'irio'>O'<olf» r* "^< —»

acIA

c it'nj—c © oj © o — oj oj m-0

Mvj'-OONOO — oj oj Oj*lTi OJ

,-»

Cg IS. i-«

m

t

m*-4

o c~- — moojoc

«^

rg "-* if* ogOJOJ

ff 1

ff- c -« "-* CT -h O —

«

— .-* — —-«i/\

,■""""""""""

0-<-«oj — O—— — — ~< *-

,-*

_

,*

-, .^—

_*

— — — ©

p4

o0*

0-*©-s'.-«0.-00'-«-*

o

oj00 o

Ifl

oo

©IT* """"""<s.r\jr\..*o—«OCOj.-*oj©

ITOJ

CO

o

0_

aroWT_J0)N>_

rH

CD

a<vorH

Uoeh

CDQJo"H

Sh

4-3

£

O

mEh

CO<D

rHrQ

CD

E_H

O

c

sD

<

QO

X

£Ca

__J<o3xX

2CZ

-J<____or

i/iacv.

roozz

UJzo

*-*/lec_!_.

z

i/lCLLUCOX3Z___:

ujO

(—oUJ-9as3..A

t/NiT'CCCCi^CCCCOiîTi """"""""""""ov O COC'N'OOOOO'VJ

** r\i O* — —*fN_mOmc.mOoOir»oo*riA

■

"""""""""*"

—< u>omfi

*i/>oif\Of»'Mf>^'

(Mm

,■«

rg — O — — i\j .© oj en oj

CD CO©.-«njC©— ~

COO

coc — njoc —

~*o©e

r- iroiroirooiMAOmor\j©r\_OojC©ojoj©ojO_,

,-«

iA — — — — «*

4- ©i\_OOCm©©ooo©"4- " " -? -Io(\ioiA©ojeoooco

<m

r- -*

r.rg r\jf\.OjH(\NfVJfV.Nf\J

O

l\i©.-«fliAr''l©©fl

f>'© pfc

injC —

firM-a-COfi

l^©*(V m — — fl " *■'

tr ooc-cooooooc.-*fi ,*"""""»""""

OCOmCOCO©©©-*in —'

* .oonj.fj.CO'JOlAU*'©'^©fi """"""""""""ccocgceeoCccojogmfio

r\j og i\i oj eg ,-. — oj **!

in

o©rvJ~fsJ>'

>pnp,0 *N'',n,sjOm """"""""""""eg im pg —fi oj pj f nj

rsjOfl©©o©oo©OOr~IT.

#""""""""""*

©■* OOOOOOOO^'0—> IM **— lACiAOjiAiAiAfirtAf^mnj(*.""""""""""""

f\j(NjrgI\JINjOJOJOjr~fIOJOJ— 0J — — <*

_»f\jflcriA*ofto «DC7'O«-t 'NJ

4"<C

r~-*c*a

r-o"

ng

fi

<*

*Ap*

fi

*oCO

ODfl

■&m__*>

OJmCD

f-

og

"oom

UJo<ACUJ>

(Acr-

©"a

00m

4-c-t

m

©mat

o

*<>ooo

C

r-

©

. (*-

CDOJ

.*■O

OJ

UJ I/)o4 toec 3uj Z> ~< X

m

l—

Q-

CXoa

t—

aoXszcz

t-ao

t—

inecIL

Xacz3Z

UJUJacx>-

acv.UJUJecXI—

acUJccX3Z__C(JUJo

oUJ->cc3

iTii>cCf'iCirCf , woc ©\T\ """""""■"■" "ojcc^-oo^.

o^-^rcoo

r\jO*>©oi©©c-*o©—*o IT

—i t-i rg —< -n

~*

come »ôj —t

y\ """"""■"""""OjirtC

cC CCoÔ—

tog-^rg —. (M —

CCOCC-i— CC— — CO O«* OOC—

-HCC-**-t©o

c

r«-if*Cir>foir« ©ciriur\©if\ ©"""""""# """ "ojooj(*.ouo©o»;OJ©OJ©

~* «-"-

-*■*

Oo«-40_rt©o_iooooo o— oj —

«

— oc OOOv'—<rg©©ogfg ©—i rg — ~~ —

»t

—o—-« — — omo—ooo o4- """"#

-4- O — O O © ©

O'OOOO.COOOOOO —fl """"""""""""oocooooooooo -«pg —<

■^""""""""""« "iroco^-— o©o*-«o— o

oo© — — o o© — o — o

oj©f>©oo©o©f>oo oo-r

oooooo^roo

o— kTifll_n— 000-Jf OMT —m«. """"""""""rs. .*-— OOO— O^soj —f-ivmO^— f- — — —«

— r\jf^-*-in*_Sr-ac O»o— oj

Owf>

r-

mr-

r-CD

c

"f\flog

mo"4-

o

CO

OJCO

"If!

CO

"og

O*

o

m

iro

UJo«acUJ>

"in

"o-ui

"CO

flc"fl

o

oj

"o

cr

4-fl

"ino

fifi

"flo"fl

o

"og

OO

"

4-UJ _/.o< Mac 3UJ z> —< X


I

'I

I i

■

iii

■li

itii

!72

SD 70-576

-J<acuj pg — o ao gr cc O .__) 0

s

p~ 0* O>«""" ♦"oo*

(M — -^ — —CO

o

4

©"

.J<ecujpgOOpgmOpgoojtoÔ>"""""""■""""Op-p-co — m r» j) ff 1 ooff1 o**- to

CO

b3 «j

>r-maoco4, P-p- on cd< rg — -* — — CO

IAo

oo Opgi*icoin^mo*coo, fiO'*or************>r*j)p-r> fig3ineop

fc

O'Ox —< — — mco

o

P-

tn

ft

0)4->(UBro

ft

roPh

eoooo-ooo©

00000!/>"""""""*****ooomooo 00000

tn 4"

r^tA.— m.omiAn_ in in m in c"""""rvj — rg en Pg oj rg f- pj oj oj O

_,_.*rg — — pgrn — — — in

..j-o—

r-oO—

Op-p-000

O-J. 00-« O *D O O O

r-

OJOsO

gr"of\

"*■

CO

m

000000 00000000tA " " " * "*""""■"

000-J- 00000000

mp-oc oo — ooomoIA """■

»"""""""

im— mo oo — mmoogO— — oj — pg Pg —g_r 00P-J3O —

OP*P*OOO

iA " * " . """"""""OO£OD o — o^gsooo« rg —

©

o

p»4"

"P*

JOm

g_»in

"ina"r->en to

C- row aro ojlU to

-P rHd roo ct. <rHo voW rH

-=|-

CIJ

rH fl-P

O

Seh

Oft

ra fx.QJ

"H QJv y>y> roro ES -h

4^

ft

mo (_qfn

ft

O

QJr4flroEh

o

o

«c

—I<ooIIoo

__I8zcz_J

<ooX.-i/iocv.

X3Ozoz

UJzo

t-l/>acv.

uuzo

I/)

acUJCOX3z

*oUJa

UUJcc3

trt

m — — — ,-— —O — — —iA ""000 — — 000 — — o — —oo—r-o©c — — op—m— — p-

—m— — — m

crooooooommooo""*"""" """""OOCOOOOOJogoO©

4" — —

,aOfi-

minmtn m m fi in mrgro — f-"t— f>— — —« m — —mooor-00000000

,£-"""*"""

"""""o o o -o o o o 00000

IM — — — — om— — — lA —» O— — oj— ir»—

,»

— — — — — — — — OJ— 0000000 00000,)■ """"""" """""0000000 00000

pg

<m

(DOOOÔONIMrIIOiÔ1

m«**««*» """""OOOfIOOOJOjfIOfIOJfi oj oj fi ia fi g}*

p» No*"fiMiHir''.OHiro

im o — ■* — — oj gJ- o — oj O

g>*_nO_no_noo omooo»*."♦""""" """""oj o im o oj o o OOJOOO

mmop-p-omm rg © ia o o!*>""""""""""""

M — — — — (M —"Mogpngrm-oo- «.

<r

o — rg

p-OJ

pg

firg

f.

oj

»n

opn

oIM

oin

"mp-m

mm

CDP--0*

ojoo

I-

oCOCO

UJo<ecUJ>

mOJ

«rpg

0*

o-Pg

Pg

<*■i-ac

oorg—rgOD

"inco

m"o0*

m*n"gf

CDO

"CO

g>UJ i/1o< Wloc 3uj Z> —< X

P-m

i-___.O

?.O

t—

Ql

oX3©2cz

t—

a.o

acv.

XoazazUJUJCLXh-

t--4/1ecXL.uuUJacX

t—

*/>acUJ

___>

X3_2

__coUJO

oUJ~>tc3

m — — © oj — — — ojm " " " " """"""""— — O Pg — — — OO— — IM_-h rg — — — **_4(M

OOO— O" OOO— OO— 0*m " " " " """"""""oo— pj coo— oo— oj—— — «000- O 0000000-*

"* """" """"""""OO— OOOOOOOO —*Oi*ipgOinflÔOfllAOJ" """"""""""""OWNiTiMfONO OWWO

ojftrgoj—fl— f.(^

mooofioooooooo

o o o f. 00000000

OjojojOO OtA — (M o o OJ ing* """" """"""*"pgpgOO opg — oj©inrgoj

ojPj #-^<rg ojim-

— oooooooomooo-jf """"

""""""«"

0000

00©0<MoOO

— IM

eoOoOPnOO — oj**liAfl©fl

■

" " """""*""""0001*100 — IMfIP-flOfl *-i im fl P"! f!

PÎMO- <0— —

lAflO

— tAONO>«(D->-iNÔmNOrg ,-, rg — — — — — —

j-moiAfiinooo'nooopg O oj 4- pjOOOOjOOO

<*l0— OO OiA»A>»^^«lA>^—O — OO OOJIM— — pg — —— OJ f! -J/ lA

«<_

P- CO O O — OJ

O

O

IMo*

(Ato—<oOp-

cop^

"rg

OinOJ

og

gj-OJ

"oj

rSIA

UJo<acUJ>

ocf

flc

IMC

"OJ

"tf.■G

c"o

fl

mo-»oj

om"OJ

m

"mlAgt

«g)-

CD

gfUJ (/)u>« lâc 3UJ z> —« X

-J-


It.

\

i

-iii

!- 73 -SD 70-576

-Jac

LtiNCOco-tr'fMro-eo-O

Qh-t-eoeo-»"cog_)Off, o*g_»in

oo

a*mCD

-J<acuj mm — -r miAfMfMmfi *"* O

O fifi^f-m-o

CDflg9

m

m

cqU0)

y>0)srov

COPL,

rsrgf<.OOogojo*incomeO(*.tf ,,. ...♦»»"""> r- _ocoeog)"eoinOcoe>in«©< pg — —

acooo*oOOOO©©coOOOiAOOOOOOOO

IA ■*

r__o — Oôo — ooconlr> , """"""""""♦

o — ceeoo—inooofi— IM ""* og fl

XCO

"0*

>oo<

o"

I*lOJCO

-tg_>

fl

IMfl

"

oOm^ff-o >j-o.<f> »Hô_Dog

>< "-"coOOOPgOOOOOOOOi/\ """" """"""""

000*MDOOOOOOO

rg IM

h-OGm<coo- inmcoom """"""""""""OOOJOD O O

■—

«M OJ O

OO

—fM — — —

co

<*>OJmm

fi4-

co4-

"mOJCO

OJgj-

W)a"H

CO

t>

CQ

CQ

Gt3 roc wroW 4^>

4^ O3 T)

O

rHrd

O

rH MoW cv

fly

-* yiH

Sv oo st

CQ Id0) <D

O

4->"H

CD

fi B+3 -H

CO

4-5S

CQ

W?H

O

*HW

orH

QJrHfl

CO

&4

o

o

■*

-I<acxxoo__<

_.

iXszcz—I

©oX

1/1ocv.

sozoz

UJzo

t-ac__L

UJzo

tAecli.CDX3z__coUJc

vUJ-1cc3l/l

,*

O — r-oo — op-p*c oom

,#"""""""*""

qh<cOO"o

*---:-- -- J* J z000 — — coo—— ©— —

OO'-'^-000-

- --'©''^^— lA -* —' """' "1

erooooooommooo.*"♦""""""""""ooooooo«n"moooJ. — ——

tninfiiAiAinfimtn

*OflrtNNrtNlMl>"« N N(Ml*.—4 f+\ —" »*1 — — — fl — —lAOOOP-00000000

,*.».""*""♦"""*

000000000000"o

py———-I©O — — — O — O— — — — ©ia — — — ia — m— — — — rg — — — im — oj— 000000000000J.

♦»""""*""""*

0000000 00000pg «*

CD— — — I*l— — — — IMIAPJJOfl «__»_!-_."

p*.^©— m — — infio — mo(<.»"""""""""""

pyO- -*■—— OJJJ-O-ÔJOrg — — — — —— — —-JtiAOinOIAOOOIAOO©

fl * "N3MOIMOOONOOO

rn'iA O p* f* otnun OJOIAOOm 2*22pgO^ÔOJOJOjOOJOO

— *m f* ** *n * **■ eecro—oj

OJo*0

p»OJOJ

f.OJfl

Og*rin

oI*l

oJM

»nin

fl

flfl

"flp-gj-

"CO—OJo

t

o

o---4-oCDCO

UJo«acUJ>

p-m<o

p--I*l

"OJ—g/IMtr

p-OJOJ

J*p~

"CD

o»o

COfl

OJCD

OJ

r--I*l

iningr

CDOCO

4-UJ wio41 l/>ac 3uj z> —« X

p"fl

gj-

t-a.o__Cl

_-a.orszc

a.at~tAecv.

X3O__zUlUlacXp-

p-fc/>

v.UJUJacX

i/.AcUJ__QX3

*UJo

vUJ-1CD3</»

O — P-'O

OÔf^r-000

m """"""""""""© — <CCOO— OJ) <0000— —(M — — —fiO-Ôm— —» © o

«-^

©m """"""""""""o — om— — — oo— — o—m—— — — —omo —fiooo — oo— 0>

m """"""""""""pg © — <? oo©-— OO— oj— — —

_*

— 4-

e>oo — 000000000oo—ooooooooo

o o —rgom- moo — mm4-

«""""*"""*""

O — og O rg— rgoo — ojrgIM— OJ ,-_i-^— — — —

mooo — 00000000000 — 00000000

pg — — O — OO— ojOm — O>*■ """"""""""""— — o — oo — rg©oj — m— — — — rg — — pg— ooooooooomoo>*■ ""*" ""■"""""

ooo©

OoOoOrgOOfM —

cd ©oo — oo— — ojoojofl

"""""""""»"*

© O O -^ O O — — Pg iA pg O— — — rg rg im

r-rgo — fi— < —

mÔ

— mom """*"""""■""rg O — g? — —ojô— (MOfM ——— — —— — —£moinomoG omooom """"""""""""pgopgoojoooojooo

p*imooooom—— 000m**»« """"""""IMOOOOOPJ— — 000

«4Pufi<j>m4>p»cDO< 0- oj

o

CD

flflCD

CDmCD

fl

o

«gj*

o

fl

o—p~

OJ

m"to

ojo"o

p-—gj-

"fi

UJo4acUJ>

r-m"

o

"oo"CD—o

—oo

o---4-"0*

4-

cnpn

"cof.

mm

IMg}-

gj-UJ l/tcg| i/>ot 3uj Z> —4 X

T.

1

i

I

i

i

j

I

'i

:l

81d


each hand, and the same information averaged over Ss . Table

8 gives this information for the 16 analyzed hands, using

the parameter estimates which were optimal or modal for the

16 analyzed hands, according to the rules for forming the

variations of different strengths. Table 9 gives the errors

of prediction for the 14 holdout hands using the parameter

estimates which were optimal or modal for the 16 analyzed

hands. Table 10 shows the results of using the parameter

estimates which were optimal or modal for the 14 holdout

hands in the prediction of behavior on the 14 holdout hands .

- 7^ -SD 70-576

1:

I

I

It

I

M


CHAPTER VI

SUMMARY AND CONCLUSIONS

The purpose of the research, as outlined previously,

was essentially methodological. The basic approach and most

of the techniques applied were established before any data

were collected, and in fact before the game was selected.

The outcomes of the research will be discussed with respect

to the stated purpose.

Five process rules constitute the model. Decisions

to accept offered cards are based solely upon the rules of

the game, the color and magnitude of the offered card, and

the previous decisions. The model is completely determinis-

tic .The computer program is successfully kept on the

correct track by very simple features in the source code

associated with the entry points ACCEPT and REJECT. After

a prediction which differs from the recorded human choice,

an error of simulation is scored and the program is forced

- 75 -SD 70-576

M


to make the recorded choice. Thus, there are no problems

associated with the conditional prediction effect.

Evaluating the use of Fortran as the programming

language has two aspects. Certainly the model was simple

enough to express in Fortran, and only basic skills were

needed to develop and test the program. The main purpose,

however, for using Fortran was to make public all of the

model mechanization. The understandability of the source

code was not tested or measured in any way, but most sci-

entists do have some familiarity with Fortran or access to

others who do. The source code presented in Appendix D

does include some dialect features associated with a par-

ticular manufacturer's implementation of Fortran, but the

code could quite easily be rewritten in any other dialect.

The successful use of Fortran in this research is therefore

a reasonable but unproven conclusion.

The only evidence for the strictly empirical deri-

vation of the model is in the details of the analysis proc-

ess described in Chapter IV. The analysis was successfully

free of any conscious bias . No assumptions regarding cog-

nitive processes or the structure of the model were held in

advance of protocol analysis . The process rules induced by

the analyst (the author) are not the only ones possible, and

- 76 -SD 70-576

SD 70-576 11

M


other rules might emerge from the same protocol data for

other analysts. Had the first protocols analyzed been those

of Ss other than SjLl and S6 , somewhat different rules could

be expected. Had the sample Ss been different, the model

might have been very different. Nonexperimental anecdotal

support for this is found in the fact that a haphazard

sample of two scientific computer programmers, upon reading

the rules of the experimental game, at once proceeded to try

to discover optimal playing strategy based upon very complex

probability calculations. It would appear in retrospect

that to empirically derive a more nearly general model it

would be necessary to ensure a more diverse subject sample.

The model was reasonably successful in predicting

the behavior of 11 individual S_s , each of whom played the

game at least somewhat differently. A twelfth S_, unlike any

of the others, had a very strong color bias which was not

accounted for by the model. The model has seven numeric

parameters which were fitted to individual S_ behavior. Each

S had a unique set of parameter estimates. One of the pur-

poses of the research was to develop a model which was gen-

eral in structure but individual in function. The general-

ity of structure is indicated by the fact that a single

short main computer program predicted behavior for all

- 77 -

f

.


subjects, with 93.4 per cent accuracy for 11 synoptic sub-

jects. The individuality of function was provided by the

incorporation of individually fitted parameter estimates.

The importance of these individualizing parameters is seen

from the fact that when their values are held constant in-

stead of being allowed to vary according to individual dif-

ferences, model performance is very significantly degraded.

The results may be interpreted to support the conclusion

that individuals can be characterized in terms of their

parameter estimates, and that the parameterized model can

account for individual differences in behavior.

The research purposed to develop a measure of the

goodness of simulation. The only measure presented was

error per cent. For the case of sequential decisions, the

conditional prediction effect obtains, so that error per

cent reflects the highly interdependent decisions made by

the model. It would be invalid to use error per cent in the

usual formulae for the significance of a proportion, the

significance of differences between proportions, etc. The

research did not develop a method to test for whether a

model predicted significantly better than chance. For the

cross-validation 14 hands, the average error per cent was

5.65 per cent. The question which should be asked and

- 78 -SD 70-576

r

/

t.


answered is this : does this model account for all but an

insignificant proportion of the variance in the observed

natural behavior? For the author, this remains a challeng-

ing open question.

On the other hand, it may be that error per cent is

quite satisfactory for many purposes. In classical statis-

tical hypothesis testing, the researcher chooses, presumably

in advance, a type I error per cent he is willing to toler-

ate. For experimental decision-making situations like the

one used in this research, the model can incorporate in-

creasingly more parameters, and use up degrees of freedom,

until every single decision point is represented by a

parameter. As the number of parameters approached the num-

ber of choice points, the error per cent would approach

zero, inexorably. The researcher could establish in advance

an error per cent tolerable for his purposes, and add param-

eters to his model until this value was reached. Comparing

models in terms of average error per cent is valid. Error

per cent is directly based upon numbers of errors, which are

arrived at by counting. Ratio scale status for the error

per cent measure is not required for the arithmetic opera-

tions used in the t-tests, however. The theoretical distri-

bution of this error per cent measure is not known (to the

- 79 -SD 70-576

f:


author, at any rate). For each S_, the denominator used in

calculating the error per cent might be different, as it was

for 10 out of the 12 Ss in this study. The conditional pre-

diction effect upon the distribution is probably not amen-

able to any closed-form analysis. It is, however, reason-

able to assume that this measure may be manipulated as be-

longing to an interval scale, with tolerable inaccuracy.

The research developed a technique for studying the

effects of the parameters on the predictive power of the

model. The technique involves the generation of models of

varying strengths by selectively nullifying the effects of

the various parameters, and calculation of the error reduc-

tion associated with particular parameter combinations. The

technique permits the derivation of an exact expression for

the contribution of the parameters to the predictive

strength of the model. Arbitrary though it might seem, the

bit notation used facilitates the calculations of parameter

contributions, while the octal identifiers for the varia-

tions are much more convenient than mnemonic identifiers

composed of English words. The technique involving model

variation and error partition emerged from the search for a

means to test simulation adequacy in terms of classical

hypothesis testing. That search is not over, but the

- 80 -SD 70-576


methodology described in this report is offered as part of

a feasible approach to that goal.

i.

81 -SD 70-576

jm


APPENDIXES

- 82 -SD 70-576


APPENDIX A

GAME INSTRUCTIONS

- 83 -SD 70-576

JO*


APPENDIX A

GAME INSTRUCTIONS

The card game you are about to play was developed as

part of a research program designed to define a methodology

for understanding observed behavior in a problem-solving

(game-playing) situation.

1. The card deck for this game has 24 cards (12 of

one red suit, ace through queen, and 12 of one black suit,

ace through queen) . Each deck has been randomly shuffled

using a list of random numbers, and the sequence of decks is

itself a random arrangement (there is no pattern in the pres-

entation of successive decks). Each person will play the

same hands in the same sequence.

2. I will show you the top 10 cards, one at a time.

As each card is exposed, I will call out the color and value

of the exposed card and you decide whether to accept or

reject it. "Think aloud" as you make your choice. Of the

top 10 cards, you must accept five.

- 8^ -SD 70-576

Jr


3. The numbered cards have a value equal to the

number on the card; the ace is worth one point, the jack,

11 points, and the queen, 12 points. The goal of the game

is to get the maximum total score for the five cards you

select. The values of the red cards and the black cards

will be added separately, and the larger sum will be your

total score for that particular hand. Thirty hands of the

same game will be played during each one-hour session. You

should be able to improve your performance with practice.

4. You are asked to do all of your thinking aloud,

and a tape recording will be made during the course of play.

You may use pencil and paper if you wish, provided you con-

currently verbalize what you are doing.

- 85 -SD 70-576


APPENDIX B

EXCERPTS FROM ALL PROTOCOLS

- 86 -SD 70-576

jf

Jy


APPENDIX B

EXCERPTS FROM ALL PROTOCOLS

SI (after H3O)

I think it works best to try to get all cards of the

same color instead of mixing them.

(Summary comment, on inquiry)

I found that the best system that I worked out was

trying to get all of one color, instead of mixing the two

colors and having half red and half black, and get all reds

or blacks, which generally gave a higher score.

S2 (at start of H4 )

810 I accept it; I'm going to work on blacks and

high numbers

Rll I think I'll accept that too so I can work on

both.

R3 I reject it.

812 That settles it; I'm going to be working on

the blacks now.

- 87 -SD 70-576

j£


(At start of H26)

R9 I accept it and hope I don't see a high black.

810 Oh darn! Since the 10 is higher I'll take

that.

B6 I accept it and work on blacks .(After accepting three cards on H4O)

Rll Darn! Since the jack and 10 are higher than

the 7 and 9, I'll switch back to reds again and

accept it.

(After accepting three cards on H5O)

Rl2 It's too late to switch colors.

(Summary statement)

It's best to wait for a high card when you're first

starting and stick with the high cards in that one color.

If two high cards come up in a row, one in each color, then

you take them both and wait for the higher cards in which-

ever color shows up. But half the time, when the last cards

are coming around, you have to gamble, and that's what I do.

S3 (on H5, after accepting four cards)

B9 I'm holding 12 black in my hand, 26 red in my

hand. There are nine cards showing. I'll

leave it and gamble that the last card is red.

- 88 -SD 70-576

M


(On H2l, after taking two cards)

B5 I'll take it, because I'm building blacks and

the other cards have been small, so there might

not be many high cards in this hand.

(On H2 3, after taking B8 and Bll)

R9 Six cards showing. I'll take the red 9, in

case the rest of the cards are red.

Rll I'll take it; high red which gives me a higher

score in the red than in the black.

(Summary statement, on inquiry)

My strategy would be that I wait for a 7 or better

in the first three cards before I accept a card and the

color of that card determines which color I will start col-

lecting, trying always to get five of one color. If a high

card shows up in each color in the first three cards, I

usually take them both and let the color of the next high

card decide which color I will build on.

S4 (after Hi)

Since I started collecting blacks, it doesn't seem

worthwhile to take high reds since in all probability that

section won't be counted anyway, and I don't know how the

blacks will be stacked because of the random distribution.

- 89 -SD 70-576

Jt


(After H4)

It seems as though in the previous hands, when I had

started accepting one color, high cards in the other color

would come up and I didn't choose those and it seems as

though the large numbers have been coming in series, so I

decided to collect high numbers in both colors.

(During H7 )

After you've passed up one or two high cards in a

color (red) because you've started with the other color

(black) you may as well continue with the blacks and reject

all reds .(During H2l)

the theory now being that now that I've

started with the black series I'll take any blacks over

five, and reject the rest.

(On H3l, the beginning of the second session)

R7 I accept, because it's red and on the way up

the stairs today I decided to go with the reds

on all hands today. On some hands I might have

less of a chance, but on the over-all 30 hands,

I thought it would be nice to have some kind of

a system.

90SD 70-576


(On H32, considering first card offered)

B9 Reject, black. it looks like a nice card and

I'm tempted to take it, but I'm going to go

ahead with the reds so that I can have the

security of knowing that I'm doing something.

(After H39)

Once again, I think it is just as well to go on with

the reds, because although it might not always be best on an

individual hand, since there is no way of knowing what is

coming up on the different hands or what their distribution

is, sticking to the reds seems to work well over-all. The

fact that so many blacks came up this time is just a special

case .(During H44 , after 812, 8 7 , B 6 )

I have the urge, when I see a high black card on the

first card, to accept, but I decide to be consistent with

the reds because it doesn't mean anything, it's just like a

good omen .(At start of H54 )

B9 Accept. I just want to see what happens if I

go with the blacks this time .(After H54)

My new philosophy is this: since it doesn't seem to

- 91 -SD 70-576

k.

fife SpaceDivisionNorth AmericanRockwell

make much difference whether I stick to the reds or change

to the blacks, I'm going to play each hand as I please for

the remainder of the session.

(During H56 )

As it turns out, I would have been better off to

change to blacks when the black 9 came up because the black

9 and 12 would have been higher than the red 11 and 8, but

it's too late now and I'll have to take the last three.

(At start of H57)

Accept, red, and I'm going back to my original sys-

tem for this session and accept only reds, because when you

have to play 30 hands, it's better to stick with one system.

(Summary, after H6O)

Since you're not gaining or losing anything, it

seems at first that it would be better to play each hand by

ear and kind of try a system for each hand, but then, be-

cause of my own temperament, I decided to go with the reds

for the second session. It would seem that it is better to

have some system rather than none at all, and over the long

haul it might be more beneficial to go with the reds, even

though I knew definitely that I'd miss out on a few hands.

- 92 -SD 70-576

__.


S_s (H_4 complete)

810 I'll take the black 10; high black.

Rll I'll take the red jack, too, so I can accept

high cards in either color.

R3 I'll let the red 3 ride; too low.

812 I'll take the black queen, high black, and now

I'm looking for blacks.

B6 I'll take the black 6. I'd rather take a

fairly high black early than be stuck with the

low cards later in the hand.

B8 I'll take the black 8, high black, and there

may not be a higher black left.

(At start of H8)

r6 I'll let the red 6 go; I'm looking for some

thing higher to start with

B7 I'll take the black 7. Sevens are high enough

to start with, and I have a feeling there will

be more high blacks.

(Hl7, after rejecting R5 and accepting R9)

r6 I'll take the red 6 because it looks like there

might be a string of reds .(Summary, after H6 0)

What I did, I examined the first few cards coming

- 93 -SD 70-576


out, and saw whether they were high cards or low cards, and

after I had accepted two relatively high cards in one color,

I stuck with that color and tried to get as many in that one

color as I could, because that's how the high point hands

came. That basically was my strategy. That is, to try to

get as many cards as possible in one suit and try to get

them high.

S6 (after Hi)

For the first time around, I wasn't exactly sure how

they would come up but I suspected that I wouldn't take any-

thing lower than a7or 8 . Other than that I had no real

strategy for the first time around.

(Start of H2O)

r3 I'll probably never take a 3, especially so

early in the hand. I'll pass it up.

B2 Same with the 2.

r8 I'll take that; I'd take that most any time.

(After taking R7 , RIO, 89, on H43)

812 I didn't want this, but I better take it be-

cause the 12 and 9 are higher than the 7 and 10.

(Summary statement, after H6O)

First of all a hunch that I had in the beginning

- 9h -SD 70-576

____.


proved out: that it was never really worth taking lower

than a 4 and possible a 5. I think I learned a little later

in the game than I should have that I shouldn't have split

in many cases where I did, early in the game. That is, I'd

have two or three good cards in one color and a high card

would come up in the other color, and I was so impressed

that I grabbed it real quick, without realizing that my

chances were so slim of bettering my hand in this second

color, even though it might be a jack or queen. Lastly,

whenever I had three [cards] in one color, it was never

worth taking the opposite color; you'd never build up any-

thing on two cards .

S7 (summary statement, after H6O)

I'd wait for the first high card to accept, and if

it were red, for example, I'd pretty much play the reds.

Sometimes I'd have to wait through as many as five cards

before a card, high enough to take, would come up. I tried

to get them all of one color, even if I had to take a rather

low number like a 4 or a 3 , rather than to take a high card

of the other color because getting all five cards of one

color seemed to give more points than taking three of one

color and two of another. This, generally, seemed to work

- 95 -SD 70-576

__.


best and is what I tried to do.

S8 (at the start of H3l)

R7 I'll reject it; too low.

B_7 Reject it; too low.

812 Accept, high.

B4 Reject; too low.

Rl2 Accept; high and possibility of going either

way .(Summary statement, after H60)

If a card comes up which is a 7 or an 8 in either

the red or black suit, you take that as the first card; be-

low that you wait until you get one of those. Then you

build on the color of the first high card accepted, always

taking a 7 or above in that color. Then, toward the end,

when there are only three or four cards left and you still

need two cards to complete the hand, you take any card in

the suit you're building on no matter how low it is. Also,

if in the beginning, you have rejected three or four cards

for being too low, and two high cards come up, one in each

color, take both of them and see what color the next high

card is, and accept it, build on that color, and take any

cards of that color that come up after that.

- 96 -SD 70-576

_w


S9 (H24, complete)

B2 Pass; low card.

B3 Pass; still too low.

R9 Take it.

810 Take it.

Rl Pass .B_6 I will take it; semi-high.

R7 Will not take; stick to blacks.

R8 I will take it.

RIO I'll take that one.

(Summary statement)

Basically, I think how I did try to play the game

was if a card was a 5 or higher, I would try to take that

card- if it was lower, I would just pass. And whatever

color it would be, I would try to stick to that color in-

stead of trying to split colors. I would split points or

whatever and not come up so high. Usually, when we got past

four cards, I would take [pause] . You see, if the cards

were coming out quite low, usually I would try to take the

color that came out the worst high, which doesn't make

sense. But say if it was a split color and they were coming

out black and reds, and they were coming out quite low, I

would try to take that color that came up next. But,

- 97 -SD 70-576

____


basically, what I did try to do, within the first three or

four cards , I would try to take the card that would be 5 or

higher and basically stick to that color, unless I saw that

the opposite color was coming up and quite high cards, I

would try to maybe take one of those cards and build up the

other color .SlO (HlB, complete)

R4 Reject it; too low.

B7 Accept it ; high .B_2 Reject; too low.

R2 Reject it; how.

B9 Accept it; high.

81l Accept it; it's

B6 Accept it; it's

high.

black and it's high

high.810 Accept it; it's

(Summary statement)

I guess it would be take the first high card and,

depending on what color it is, take mostly the cards that

are of that color. Or else, if you have a high black and

right away have a high red, if the next card is a high

black, take all the rest blacks, or if the next card is a

high red, take all the rest red.

- 98 -SD 70-576


810 Yes, I'll take that, too.

B6 No; too low.

R3 No.

R5 Not too low any more; I'll take the 5.

B5 No; I'm going for reds.

R6 I'll take the 6.

Rl2 Take it.

Sll, session 2 (H3l)

R7 Accept, high.

B7 Reject, black.

- 99 -SD 70-576


812 Accept, high.

B4 Reject, low.

Rl2 Accept, high.

R6 Accept, red.

R4 Reject, too low.

RIO Accept, high red.

B8

R2

Score 35R

(H32)

B9 Accept, high.

r4 Reject, low.

B3 Reject, low.

Rl Reject, low.

R2 Reject, low.

Rll Accept, high.

Bl Reject, lov.

B8 Accept.

Rl2 Accept.

B6 Accept .Score 23R, 238

(H35)

R3 No, low.

100 -SD 70-576

____.

_X


R5 No, too low.

B_4 No, low.

Rl No , low .R6 Take it, high enough.

B6 Take it, high enough.

Rll Take it, high red.

R4 Take it, red.

810 Yes, I'll take it, high and I can work both

ways .B5 No , too low .Bl No, too low.

B2 I'll take it.

R9 Yes, high red.

B9 Yes, I'll take it.

812

Score 218

101 -SD 70-576


(H39)

81l Take it, high.

B9 Yes, black.

B2 No, low.

B7 Yes, high.

R5 No, red.

R2 No, red.

Rl2 Nope, red.

RIO No.

B8 Accept .B4 Accept .Score 398

(H4O)

R6 Too low.

B_7 Yes, take it, high enough.

B2 No, too low.

RlO Yes, I'll take it, high.

R4 No , low .R2 No, low.

B9 Yes, high.

Rll Yes, high.

810 Yes, high black.

B4

Score 268

102 -SD 70-576


(H43)

R7 Yes, that's high enough

RIO Yes, high red.

B2 No, low

B_4 No, low black

B9 No, black.

B5 No, black

R4 Yes, red.

812 Nope .R9 Yes, high red [had to accept].

R5 Yes [had to accept] .Score 35R

(H44)

812 Yes, high

B_7 Yes, high enough.

B6 Take it .B8 Yes, high black.

R6 No, red.

R3 No, red.

R2 No, red.

B2 No, low.

Rl2 No, red.

R9 Accept .Score 338

103 -SD 70-576

I


(H47)

B4 Too low.

Bl Too low.

81l Well, I hope enough black come up, I'll take

the jack.

R3 No, red.

812 Take it, high black.

B2 Yes, I'll take it, I don't think too many more

blacks will come, so I'll take the 2.

R6 No, red.

Rl No, red.

B3 Accept.

B8 Accept.

Score 368

(H4B)

R8 Yes, it's high enough.

83 . No, too low.

B6 Too low, black.

B2 No, low black.

R7 Take it, red and high enough.

Rl No, I think better reds will come up.

Rl2 Yes, high red.

81l Nope, black.

10U -SD 70-576

!


Rll Accept.

812 Accept.

Score 38R

(H5l)

R5 No, low.

B6 No, too low.

B5 It's kind of hard to make a choice, they're all

mediocre; I'll take the 5.

Bl No, too low.

812 That, I'll take; high black.

B4 Yes, black.

RIO No, I don't want the 10, it's red.

B3 Yes, I'll take the 3, it's black.

810 I'll take it, high black.

R9

Score 348

(H52)

B5 No, low.

Rl Too low.

R3 Too low.

B9 That's good, I'll take it.

R9 I'll take the red 9 too, open up my options.

B8 I'll take the 8, high black.

105SD 70-576

I

i

(H55)

(H56)


R2 No, too low.

Rll Eight cards, I'll take the jack, and take the

higher of the next two cards coming up.

Rl2 Can't be any higher than that, take it.

B2

Score 32R

B7 That's pretty good, I'll take it.

B4 That's too low.

Rl No, low red.

Bl Too low.

R4 No, it's red.

81l I'll take it, high black.

Rl2 It's the seventh, no it's red, I'll take the

last three.

R5 Accept.

R2 Accept .R9 Accept.

Score 188

Rll That's good, high red.

Rl Too low.

R3 I'll take it, red.

- 106 -SD 70-576


R8 That too, high red.

B4 No, black.

B9 No, it's black.

812 No, it's black.

810 Black [reject] .B5 Accept .

(H59)

B8 That's high enough [accept]

81l Yes, high black.

)

Rl2 No, I don't want that, red.

R2 No, that's red.

B4 Yes, I'll take the 4, black.

812 Yes, high black.

B2 It's too low [reject]

Rll No, it's red.

Bl Accept.

Score 22R


B3 No, too low.

Rll Take it, open up my options.

81l It'll be black [his option—accept].

B5 Too low.

B2 It's the ninth card; no, I'll take my chances

on the last card.

R8 Accept.

Score 308

(Summary statement)

Well, the strategy varies with the first three or

four cards laid down. Usually it depended on whether the

first card thrown down, as far as color was concerned, was

a 7 or higher, and if any other card came by of the same

color, 6 or higher, I would usually stay with that color.

But if a card of a very liigh magnitude, let's say a jack or

queen, came by of the opposite color, I would take it and

open up my options, in most cases. Then I would take the

next higher card that came along in either color, and then

play it out in that color. That worked the best, in most

cases. The hands I had the most trouble with most often

were the hands in which the mediocre cards were thrown down,

108 -SD 70-576

R7 It's red.

R4 Nope .


let's say a4, 5, or a

6,

in sequence. In one hand, espe-

cially, the first three or four cards were like that and I

didn't know what to do. I tried to assume that about as

many black cards as red cards would be in the deck. That's

what it came to when I got to about seven or eight cards in

a deck and I held four of one color, I would assume that of

the remaining cards, half would come up black and half red,

and I didn't pay too much attention to the cards that had

already been thrown down, as far as the number of cards of

each color. That, in general, is the strategy that devel-

810 I'll take that; high number.

Rll No, it's red.

R3 No, red and not high enough.

812 Yes.

B_6 Take that .B8 Take that.

Rl No.

R6 No.

B9 Take that.

109 -SD 70-576

oped.

Sl2 (H4, complete)


(Summary statement)

I think the best approach was to pick the color of

the first high card that came up and then continue to choose

that color of card. I took a lot of chances because I found

out that usually I got better results when I took chances

than when I played it safe and just picked the first five.

Number of Times (Hands) S Played Unique!

110 -SD 70-576

1 2

2 4

3 4

4 9

5 3

6 5

7 0

8 1

9 1

.0 3

.1 2

.2 8


For the 16 Hands, the Sets of SsResponding Identically

Number of WaysHand Played

9

4

6

7

3

7

3

4

5

3

8

5

5

4

5

5

The average number of ways each hand was played by

the 12 Ss was 5.2; excluding S_4, the average was 4.6.

11l -SD 70-576


APPENDIX C

SYNOPSIS OF DECISIONS FOR ALL SUBJECTS

112SD 70-576

L

OECK31

DECK32

DECK35

DECK36


SUBJFCT

I23<»56789

101112

SUBJECT

123

A

if56789

101112

SUBJECT

1234 A56789

101112

SUBJECT

<r A56789

101112

- 113 -SD 70-576

-7 7 12 <. -12 -6 -<. -10 8 . -2

AAA A AA A AAA

AAA A AA A A A AAAA A AAAA A AA A A A A

A A AAAA A A A AA A A A AA AAA. AA A A A A

9-4 3-1 -2 -11 1 8-12 6

A AAAAAA

A AAAA A A A AA AAAA AAAA AAAA AAAA AAAA AAAA A A A

A A A A

AAA

AAAAAAAA

-3 -5 <► -1-6 6 -11 -A. -10 -7

A A A A AA A A A A

A A A A AA AAAAAA A AAAA A AA A A A AA A A A AA A A A AA A A A AAA. A A A

A A A A A

-4 -2 -6 10 5 1 2 -9 9 12

AAAAAA

A A

123

A AA A

AAAAA A

A AA A

A AA A

AAAA A

A A

AAAA..... A .AAAAAAA

A A AA A A

A AAAA

AAA

_1_


HECK SUBJECT39 11 9 2 7 -5 -2 -12 -10 8 A

1 A A A A A2 AAA A A3 AAA A A4 A A A A A

.__

5 . A_ A. _A._ A A6 AAA A A7 AAA A A8 AAA A A9 AAA A A

10 AAA A All._. ... A. . ...A . A ...A A12 A A A A A

DECK SUBJECT.40 -6 7 2 . -10. . -A -2 9 -11. 10 4

I A A A A A23

AA

A AAAA AAA

4 A A A A A5 A

_.

A __.A A A6 A A AAA7 A

AA AAA

8 A AAA9 A A A A A

10 A A A A A11 -A A A ..... A A12 A A A A A

DECK SUBJECT_ -7 10 2. _ A

__.

5 _ -443 12__ -9 -5

1 A AAAA

._

A_

AA

A A2 AAA3 A A

AA A

4 A A A__5 -A

6 A

.-

A A. AA A A A

7 A A AAA8 A A A A A9 A A AAA

10 A A AAA11 A A A_ A A12 A A AAA

DECK SUBJECT44 _12 7 .6 a. -6-_-..-___ ... -2 2_ -12 -9

1 AAA

AAA

A AA

A AAAAA

. AA

A23

A A A45 A

AAAAAAA

6789

101112

- Hi. -SD 70-576

AAAA AAAA

A A AA AAAAAAAAAAAAAA A AAAA

DECK47

DECKA8

DECK51

DECK52

.____


SUBJECT

123456789

101112

SUBJECT

123456789

101112

SUBJECT

1234

___.-

-6789

1011-12

SUBJECT

123456789

101112

4 111-3 12 ...2 -6 -1 3 8 _A AAA

AA

AAA

AAAA AA

AAAA AA .. AA . A A

AAAAAA

A AA A

A AA A AA AAA A .

AAAA AA AA. A A

AAAA A

11 -11 121 -123 .6 2 -7-8

A A A AA A A AA A A AAAA AA .... A

._

AAA A A AA A A AAAA AAAA AA A A AA A . . _ A AA A A A

3 _10 -94 -10

A AAAA

A AA AA

AAAAAAA

A AA.AAA

AAAAA AA AA A

A AAAA AA

AA

A_ AAA A

11 -12 . 28 -2

A A AAAAAA A AA A AAA

A A AAAA

-_

A AAAA A AAAA A AAAA A AA A A AAAA A AAAA A AA A A A

115SD 70-576

AAAAAAAAAAAA

-5

A

A

5

A

A

6_ 5

._

1 12

A A

A ___..._

A A

A AA

A A

-1 -3 9 -9

A A


DECK-55..

SUBJECT7 _ 4 _rj J -4 11.-12 -.5 -2 -J

1 A234 A

-.5 A

A A A AA A A A AA A A A AA AAAAAA

._A__ —A A A.67

AA

A AA

A AAAA

8 A

...__.

A A A A9 A A AAA

10 A A A A A11 A A A A A12 A A A A A

DECK SUBJECT-56 -11 rl ___-3 -B <t_ 9 12 IQ A..... 1.

1 A A AAA2 A A AAA3 AAA A A4 A A AAA5 A A A A _A_6 A A AAA7 A A AAA8 _ A

__._

_ . A A A A9 AAA A A

10 A A AAA11 A A__..._.A _A A12 A A AAA

DECK_59 12 ____-!_ 9 3

1 A A A A A2 AAA A A3 A A A A A

A A4 A A AAA5. -A A A A6 A A A A A7 A A A A A8 A A A A A

A A9 A A A A A10 AAA A A

. 11 A A_. A .A AA12 A A A A

DECK SUBJECTS 10 3_. rll .11 5 .-7 -A 260

1 A A A A A2 A A A A A3 A A A A A4 A A A A A5 . --A... _A .._ A ......A. . . . A_A A6 A A A A AA A7 A A A A A8 A A A A A9 A A A A A

A A10 A A A A A11 ... A . A A . A . _ A12 A A A A A

n6 -SD 70-576

SUBJECTa u__=i2 =2 a :


THE SIMULATION PROGRAM SOURCE CODE


118 -I SD 70-576

"o

UJr-<a

i

z<acfr-O-

CD

ii.

o-o

<«.

■vI/.

O

r~a.11lIS)

COrH

-IUJ>LU_J

o»

QLUOza.<x " "o " »z » »" * »

Q » ♦< * "

O

* »H " »" * - «

1_ » . "v " r- '■£ »vi » o. i *Q " < »O "

►

X *z * <o o *►*a a «>- * "v. #

» »

"— * m e_ "-j »p ♦ a. uu i-

of

"o » v

Q-

< »z * "o_ x ♦» » *" o " o "o » a. v. -o o »— ♦ a. "

O

»

»

"

«

»o»»rnl-o» —a_»*o. — m ai » ouj * * oc a. v * -H""»"»»Z ac » J

uj » ♦ rg « 3 » —

o**o.»-»-ll*

t-a: " * a a * orj ♦ ♦ «Ac » " lv

o*»-h<»x»

->_o»»aoror-# uj

" » »

Q.

o_ » DC

O

* * » » Z *J- » » OC IN » » _J»» » 2IN "" P

>-»»ai/>a.O*

<»N «Z ♦ Ac ac «» to a. vLU » U^HO* l/l -HMz » a. » x "* —— » " » " Z -H_JX*»-H»IN» X -H

-< « l_I_I» P OLfIX

HOI« U^ZVIO

P t-3_LOoo»iuacoac» < ax=_MO»OlO_I» U U»-Iy- CC * " ►"-<!►-» I—O. (X ♦ o " — —□ » » * — " ~"Z* .-«» rH (Nt-O

Z— » * X ♦ -OQ.H-""_r < "

>_

U i/J * "U« X ■* lil

aC

» " "" "x » v

ox*

o a _» — -i

" Z ■— I- » LU _J_.i__.u_»__. " . ..a_ —H#<».

_ < »-vi *x * P __ O a_ a_ o

Q.

X**

IIniUtCZZ-ul

<»">. » lIOUKQDWOz » ♦ o_ " nuiauoc'reou

* « oi

#xoxx< «.<.«_

|»»Z 13

»</»

— ZZtJO —* » O UJ *</> " * X t- * P — P P PZ*»IZ " P PP Jo » »

O

— »<U.«»<U-U.U-<L: » # v »o— o. o~ — — vt- » " "

o*

» *_

*" " o(* " " » -1 —OJ » "pu v ■ v"■_

CLXoo

rg <*|

O O OOOOOrH^H.^

o o 00000000o o 00000000

z Z ZZZZZZZZi/l IA IS)lAlS)ISllS)lS)iS).lS)

o

in

o_-__o

Tl

LUp

Xo_ J-Z * <J

I o -.UJ u3

ec U —z oc ►-o o o a.a_ oc v. uj«tv < uUU II vX < *z — a

LUP—U P op «Jl-< v.

O

<

O

v — v. o

O

rH

"O f» 0" O h-h *h »h rg rg0000000000z zzz zIS) IS) IS) IS) IS)

J-

uLU-.

r- lv__

O

PI- po <o v

O

f-a..

-I h- —" p-»"

UJ "CC —

rt

<QJX 0- —o va

►-

#

— LUZ

O

i^ v__; 0_ a_ va < "< < "»v o —p 1 op -ll-< v. a. <

O

v—— v o

mv

in r^ m corg rg rg n N0000000000zzz z z

I IS) IS) <S) IS) IS)

"*vUJ->LUoc

p-0 Pa. <vn

IN —IlA (NCC XX LI►- oc

X

■* — "v.

Q.

I- -►- J -H. JJ " _JLU Z —U I-

" — ar- UJ

X Z lvv—— zooc «t <i *z u>— _J -H- -J

O

p Jl-u- < v. «t

C

—v— v o

f-

u

ff HMr\j r*. r*\ ro r*^0000000000zzz z z

I LH l/. LO '/. UT)

v

I

ozLU

-orooozIS)

o*IIzvLUzp

oII

aoz<X

IIUJX<z

"vUJli.v.UJ

z

zo►-

o

v.UJacXozo

I—

aLUoz

Q.

<tXaza<op

-_

vUJoozt-l/l

pozvova.LU

_"

UJvCC

oIS)

*vUJv.v.LU

z</)zoo.oJ_


~i o

t

—

119SD 70-576

3

II

<t3

XZ<eci—ac

O

x

o.0rr\*vl/lO

ff-0

►-a.UJi/.

CO

aUJoza.<x ♦o »z «" "o »< *a »p »

* "v r- 13 «uj a. Ac *a < "o"x » - »z -O

O

v. z »" a o *: v; »

I- U Q »IS) »

►

UJ I- »—

«\3 >OJ

lo *p a. lv r- z ac p »□ v a. < "Z - CC « X " *

»Nfo»za

at *o a. u_ o ■* a Z » ac-. o.

O

* » ocq»»i-»u » uj

U-OX-uOOLU » I—to a. — in „j oj o * vuj ec cl o *» Z - at * »aj in » ac

O

lo ♦-» »

O

a- * >*■ Z v. Ac *cc a »" * a * *-. » oc v " oe »jj 3

O

h < .vi X ac ♦o t-_o

Q.

O ro

O

— UJ » — Uo. oc »

O

a.o

«"

»z - *x.»■ 0C rg » — UJ »

O

"II Z X "N <N » O * sv Qv- a is) a. " a — ■»-» lvzacacpacOO * o0«*X»« Xlv ♦ <S> OC_uu*-rH>-uuac * ec c_Z Q.O k- «s "

O

U.— »- - ♦ Ac_l .-4.rg—

»O»0C

" t ac ► I in* " * vi—. 0"/)ZCv^O — v o» UJo uKca'jjiczo^ »»i vii on-KKZJ en» —

►_

__,>_<i_i-i«_a_ » * aa. o Ii- v»

uiO

Xo ... >^ » v to. .*_. oc»— uz _umi^ o a: » a a. » ac_

x. u<looc uj*x uj lo z«» uj j_ OO M-«ui3sei!ex i- v o co i o ac a»— z a o o

_» __t_>-LOuj jkt-o<j*Ji3 — X ac n *

r

; ■/) a o

«-

im 30»0-''Nr»\._

;

zs

>_.

>-x*-> oo.roz

__.

ac U * II f ff O ff.- uj acc»n ff

" -_)__■

O

Z UJI-» * I- ►- '—'a o lv o -J

»*acz

z >- z„y ,_ X OJ<»

OU-

-"Z —zfflX Z x

--*>->-ac*-3C

£55 »_ c z<»uuQ.i__3ca.2^u v -a* a. a a.v. *

Q.

U_ Lj <*_ '__ <-_ *a

'

:* :111p <->a.Xo

pj^ in .0 *- co ff OHigfl

OO O

O

OO OOrH-1-.rH

OO O

O

OO 00000000 o o

OO

000000

oLUvac 1

O

'v.Qz<LOacoac□cUJ

LOUJvoXoX00

UOU

<L-<

O

ac> aca: lv< fc-X — vx a.a —ist 3 »

IV- O *z a oXI- l-ac Ua. a.

Q_ Ulv a ac acX to e

O

LO UJ Ul li.—" ac *- az a 1--_ r_ H- "ojinoii —O OC OJ Z I o—

C

U LO ii * X0 ac oc — a

«"" O

v1 oc o t- 0 "- a

OUJU-H-LOIO

r-u p v a. "♦ ♦ ♦ uj 3 ac—> " *~ I ac

_j _ uioaj lv

O

ac ac a z — o t-

XluOlo

" — " OU t- LL * O t—

O O

a lo

aC

O O ""►—►—"— — z 0 -" -000x *: -" in *11 11 11 tv 11 o o 11 11 11UJ II ff ff— 3CUU.Q OC

"-OCO

O ac ac — _c oc h »— a ac acxujO — aujzzxxia01-U. h-f-i-r — UH-U.oca u-uoocncoao*->-)— — a. a. a. &. >- >-

1 v '

.a

sffOHNrt>lin

W~H-H -i^hINIMNNIM000 00000000000 00000000

v

»

«

»

*»

»

»«

**

»

*

#

z »

OC

*a »t- »Ul "ac »

»

I v

<org

O

O

-ILU>LU

ZZ z z Z Z Z Z ZZ z zy.^ uo lo uolo i/i 1/1 v. ifl i/i

z z z zzzzzzzzIS) IS) IS) IS)IS)IS)IS)IS)IS)IS)IS)

zIS)Ml

fife SpaceDivisionNorthAmericanRockwell

ii

120SD 70-576

■^ oo oc" oc

in ujLL" k-

Z II zCC LUa is) vV- UJ OCIU U UJoc — a.a o— x -o *»vr -<v v. "lv o iriQ LLp ac -a lv ii

CD

" Xl-a a vLU Z UJ

" o aco - acac -»■ -" v.

O

z o o* "v

O

—

O

in I—LU L-

r-

L- J. Z

O O

— 3 " UJc 3 a O II v. — z ocQ — 4 IS) UJZ -» — X — ac a

<O—

3 Oi OZ*Org O

J-~aC-

-4 lv —" "—

-»

m ac n.

* Ac " X *c ac » " m — lv in "UZ *— |_) ►- -H I— " M I-I >_v a (J*< m j - n . ll " acq_j " p + " i ii " n a il 4_j o a -a o - x3 — PL-*—* r* OCIIUJ3COX

t-»tz —' oc o x*zacujjjaz cc n ic x ~ i— " 4 — — o *s_ai-o ooBZIuXU —q -h _j o ac 4 o3oXac "a

■

—Ol * COD ZP UJ 3vo 3 3 «op a: a o

z<

o l- a: _c — o_ a >- o z v. o

CIZ

"1034 <>— Ol- O O " O rg— I ioh«jilJIU I X Z - " " OC t-ee -C ii ii v. a: _ o _ an

<n

— —" u_ —>ujz __* ii — -ror zua zx z>— ►— l- *— to ~ t—So

«OO

—OC

POC

I_«<<<<JH<i-Lu — zukqh «-< — aca — acaxxxxa«x2 1_ oOplu zx 3 ►- oi-ococacaczinacaujzllpp»c*3 c3i_.oluu.op a cocou-oz— —

O

a ►- ►- 3 O O — LOOC— LO 0C LL LL li. LL - LL UJ

i-3 -8*r

O

-H rg rg in*h —i —« —. o r*i

O Ort

O ff O ff ffff ff

o-J-II

*-zoLUzp

oII

*"a.az4X

IIUJX<z

*oLUILLLUJ

LLUJocX3Z

'

a

L-

oUJaza.<JX3za<3P_CoLUa3Z

uO

P3ZoaoCOUJ

LUOOC

3LO

*I—oUJLLLLUJ

rgffO-H

II

UJ*g

lO

X4CCo3aca

*m

ilIS)k—zLUXUJ

<LO

LUoacaauO

ap

4ac .LUzujU

uO

O

L-iS)3Zo4a

oz

"Z3>—4P

axaoLL

Mff-Mn^JNOlffdrMfl^^JXffOHNr, .»" ,nrgrgrgrAmi»>i»\r»i(->-orr^j-.1-J--»--»->>'-J,^mmmJ> in m000000003000000000000000 OOooooooooocaooooooooooooo oo

zrzzzzzzzzzzzzzzzzzzzzzz z zis>is: IS)I/>IS))S)IS)IS)IS)IS)ISIIS)IS) IS) IS)

zLOz3t-a.3

"

Z

LOz3

*—a

C

*

LOo)—IS)

)—4H-LO

"

*IS)o

LO

►-4►-

"

O

azLU

»*»


- 121 -SD 70-576

ff-H

"op-

ui»--4

Q

Xz4OC

a:3LL

o-o

>.

LO3

ff"O

L-a.UJIS)

CO

rt

a

t-

oLUaza4

5 * *" » "r. » #2 * «a " «-" * »» *O f- <J * .lv a ac * »o 4 » Ja » « x " Zz z g. o lo " Z"*aa * " #

l- a o " JLOL-»*LU-_ * -— i/ima.aoig * Ippauji--zac-H " X "a o a 4 * « m.z.-ac.r. * ct "

-aCJ'O-za^J

" o "

ozau.^*a>H

» o -~ * a a " * "a o »">-". " — " a. -ouj-ox.looo o " Z-ooa-_m_jiu-<aeuj " _o «.uj ac a o— z o " — I»»»z»jc_ia__c " _o Z_j_org-3c3330 # v Joaca.«-#zu-3i/>LL " _. Zac a aa * a » < J

T0C«aCO»O».

" » Z3_C-h4»luli/1-»_j " < Z

/.Luaor^a—

olu # Zaoc.-Ho » i »o>>>>z-.wae * £ "* vi _n-- jie o * Z

lIOZX^J'>J»rHOU_ » -x "h- — QUIO. »o "ill > " JZOOoCacpacoo " mj «

OX'P4X»44-H

. *LO_J "

IuOOCO*--hl-O>-»-> »XcO

"zk-a3p3 * * < — m »

"-»

» ►» *

O

Z o # ac i- »P -»»»-"» "VJ34 4P " 4 4 *" .nr*jjj # a ao "—<—

OOIOZOIO

— o_o * x ►- «

O

o<lil_CJJ_ " >tk_ * » 33 lo *ii zpax=coci-»~» c »oc ou x "I— 4CO~k-4l— I— Or*-— »v.

#Z

4 *all— O rH w — —

*03r-

a "3 .._l^_-H.»==_

< »" »*rH<XLOw—

»*LOa

a «Z *Ol*4X_JO » Z HOC I- »— *s.

O4LOOOC44r*-»*

— M LULU* *4<sio l-jccu^klwohh " a o oc *__ a cc o ac;xaa43a-H»ii uju_44Z »looi z — ►-*- — a. ►- — — * o xzujuiaa »xi — »» — a x * z >-.xxa »

ll Zl < «* :a ju * Jj i/. JJ>uj3X v *dq n *

i/iaf

* » »X JJ

OLO4»OLUZ--

IN . "4Z"v~- Z

IXl*o*olooo.

»Z "— OC LU_)X»ff4affffO ♦I- LU p 4 « --IX L 0 ff "lazzo 4o»»mn v-z»03 a lv » — »»— * a — c z >- tx »

LOaCX X r- — 'J p " O o 4 4 — Z 3 a. *ZfflX X Z a 34»4LULULLPCt-«>-3*333 3 — Q) ji4i_3ac_.Hi.i_L "— i/> o o lv ac » ac

■—

aacu.*i- » l- »a u3lJl.l_ll.l_lJ » >< #

O

" LU *" z "ac " m

rt

"lv " «Poo o oa.x3O

rgr*. g. in -O r- ao

ffO

h^ ol

OO

O

O O OO O-h

—th —. —- —« -h

OO

O O O OO OO 000000

OO

O O O

OO OO

000000

***»*«*

L- "o "LU *-> »UJ "ac »

"a "p ». 4 "> »z "— "*ac "O »LL "-" *o »z "X Z *3 ac "►- a #

UJ >~ »OC ' LU "X »"" »— »

" ac . »- Z I- "O " " o "* OC -" -H LU »— 4 -» "" O I ♦ LU *— LU — OC ")- o a iv lo c "o — — o ac ac 0»u_ 3 C_. — 3 4 — »Tiaooc J _!♦-U O 3 I OC 4 »ac lo o lv " _» »

II * »" ii ii o ac "I- ff o "

>UJOLULO

zll»acozoaci-ac "i- —

t— C

Z 3 -h »ZCJ-OOC-L *■jJla.lacociuz»v — OLuaacac»

3 »>- »UJ *ac ""i~> v v

t ff O N I*l

■.

IT—i -4 rg og rg rg rg00000000000000

PUJ>JJp

ZZZZZZZZZ zz z r z zIS) IS) IS) l/l LO l/l l/l IS) IS) IS) IS) ISI 'A IS) IS)

zzzz z z zIS) ISI IS) IS) IS) IS) IS)


122 -SD 70-576

*

*»

»

*»

*— *>- »— — »."za "

_

-t

X

O

»3 "* * >- OC "UJ LU «

_v is) ec t- *

O

oc v. »—3P 4 »

O rg

ec 4 «- P *X

■

ac X iv »o uj ac a »

O

3 " 3 "ii y- n Z isi X *3—

O

*UJ

O

I/* OC > *O OC 4 CO #—3"O *

O

~ QC ►- ►"» »i — oc a m a *O OC Z LU LU LU ♦z ac v o o #—oaoz o ♦

* — cC L- -» 4 — 4 #« 4 LU I— **poocp a a Q »—. UJ -~ LU "- IP — »i-oo a— ao -j v -i * ia. cc — acin cc ac

404*

LUQO 4 43 ><>#

OJL

00 O " OU. Z » i

03-Hiik- ocac — »< llfl »

»

JL - " JJIU »i— x I " z >- m*- L-k-ac*3 . — — 3 — -<03rgu.u.a»zi-acocxaa crz 4 4 a. »

J-tOZZ-OOZ " Z *a. zii— ac

aC>-

ac O — M "i -__OZPP3Z — 3 »Z — —Z— PP>-—

b-ZZZ*

LUu.u.a:4LL44ujo:LLUjacacac*_ — ZX — ooaca —

3C333»_->-►-»

UJ LU LU *ac ac ac "rg «*U i^3 ijU U

*

***»

**»

»

-H ~ "ac »♦ z »

a *o oc oc »OC Z 4 #4ao »v oc — *X4* »zoo »

UJ »ii a »

">" OC II z »oc z cc *t- a a a *ZXXLIUJ 4 4 LU »o o ac "»»

*i o o

*«»

*»

**

*»— "— »

CC

*LU "X *I- »3 ♦— "p #4 "r- "a »i- "— »LO "__

"4 »— "I *rg — »— ""g II l/l »— "II LO X »— I- *a i — »

Ul I- p »ac 4 *— lo >- *~"— a *I- I -. — L- »LU L- Q O + *z ii cc lo a *

»

4 " -> OC »— ac * o I-X4 »ZjJO— Pl-O » I— I4 at " — ♦4 r- -I

O

I IS) "

O

3 a_ _j a ie »3 rH oc ro 4 »

CC O 4 — »UJ II o II *3- (J II II Z ♦

OC UJ LO — oc oc *r- I— LO"-

LJ*-3*

Z Z -> I v. I lv ►- »UJ— Ih-r-

ZJJ*

L-

O

OC *"**ioo o o

aczCDaLO

*oUla»

— p

* 3O

O

vi a.a a*

_i

LOo —lv 4 IIa * ox o

-t

—Zui «-a r* ~— ii aJo e ooc zkiiiii- * azo lolv uj aa o

i

s ff rg ro i/> HDF«"3_ffrHrgrgrgroiO'O'OO^iOfO^000000000000000000000000

NIO ». iT.»■*»■-»"-»"00000000

*0 I"- Iff HfJI.■j- -j" in in inO O 00000

O

O 00000

Ifl O !*■m in in in00000000

zzzzzzzzzzzzisiiS)iS)iS)iAiS)iS)iS)i/)iS)isiiS)zz z zIS) IS) IS) IS)

zzzz z z zIS) IS) IS) IS) IS) IS) IS)

zz z zISI IS) IS) IS)


123 -SD 70-576

«««**

**»**»»*a "— »a »a_ "3 "LO """ »

*

rt

»o »IP II *Q' ** *" O "ac vi "Z

O

»* — *O *-v *a — *— "-i *♦ "— LO ** »

* o *v uj "LU

O

»Q Z »—. — *-oc ♦~z " »— a. a *p a -< vi *3uo * "

t

»

C

*

r<

in " o »a

»

o * m o aao o *

O

LU O » O II ♦ii -" aiu -> ii non ii ii ii "ff — a ii — ♦

__. ii*— — >—

ocii-HiNLOLuaz*

— i- * —Z I X nc o vi ac »— zo —

-raoXL^Loa

—

oa*

* — lv z a. — acacacaac>-»oa:3u.c44acllnclCuj»uj__----oxozk-h-ujou.ac»a

■

»»"J- »

m *o o

»

*♦ z" -r "» 4 OC

"

O

UJ

" ro — L-» — 4« >» Z Ul» <*> — a:* — zoo" - 4 o:* Z X 4 ac» 4 >» i- x a— » X <*)

rg * r- —. i/i lv p-H » LO

O

4

* » lo . lv z a

-t

* LO LO P 4

O

II " LU 4 Z UJ_C ♦ P lo ~«" » LU — - X LO— * LO o oc o -lo » lv ac a aa* » ■V34UIP a

O*

rr»_jPL3LULUuj* — < a o I l.a » > lo vi l. aZ * - 4 O L- Ul» » ac»-

"OZZo

—* » LUO IU * 4 4 -h oll* rO al XI IS) Z 4

">* X-> UJ >" — *-4 »» 3uji-oaxu.>

* — » ZQCLOZ43—QO

O*

34P0X4Ul n » I- " X L- 0C I l/l Ul

O

» » ouj

O.PZ

ac— -<»

LUOLULU434P

ll* -> Z U U Z 4

" — * 3- 4 Z O i/» Zlo - * 3r <<Quj- in* — » Loai-ax ao * * _uat-4L-i-acUJ * *

OUJZOP4

O "J*

OOLUU.3O

Z

»

* 4 O 3 li-

O

lo«-»

-V4CJOUJU.

" — *

k-muiaacâ

Z P* l_ wLI/11. CC I*l

* 3* aC-o_aP4p — aca 3 * — -" 1-3 JUJ Ul—J L- _J a *

__

iLflt -»

Jl I —.

_■

coLL LOP 3 0 » 4 JJ U. 3 4 _* l/> XLL POO LO » o J LL 13 Z 433 3a — * ~-h Z li. CX Z L-L- zX -a _i— » -. a. . ~3lu — via — I

LO g_ ". 3 __. * in 4 -H

,^c

I— e_ X Ul OUIO

»

»

C

* » —

O

- -*UJJU._IhUIL3Iffiôaff »co—* — -a — 40 o — x i—

-h

Off

O »— — i— — — l— LU — U. 4

>llOff OffZ*L-l— 41-LOZDCLO—

IS) a.Ac * 4 L-—

OC*44X4Z3LUZ

4 —l-Oi*jOZC33»XXoCXa 13zm _ < - _D4i— „ a ac QiCu l-o

>■

luo iLi_3_j'jjii3.. _.Oo o - »a_— _caLoaca:*u.u. u."»r*-s.»*v "* -â * cacara.'aci.racarv's4 » rr,LU » O rH rg

O

oc * o o o ff

* ff ff ffo o

a_a*O-_ro*j'in O N X ff

O

-him

0000000000000000000000000000

rO

>$"

Jl .0 r~ 00 ff O H rg Xlr*r*h-h-p- r-p»

coco^co

00000 oo 000000000

OO

0000zzzzzzzzzzzzzzIS)iS)*S)iSIiS)tS)IStiSIISIiSIIS)IS)IS)IS)

ZZZZZZZ z z z zIS) I/) IS) IS) IS) IS) IS) l/l l/l l/l l/l


rt

- 12U -SD 70-576

i__.

o -_) LOa <o —X "

OC

o »-vi a —X Xi- a— LO LUUJ 4 h-

»

0. L-«n LL

lO

Ul —■m a -« v— o aac uj 4 uj— a a i-LO P l/l

O

4*4 UJ

OC

> "" 1x a uj "».O -I LO "

OC

* —

"so*-

» — *>X U "- * </> « ""— noLO r~ U. - "♦ —< z — a — gc "4 UJ - — Z *lv z » >- » m *

O

a_ -" r- o

»>■

—

uO

x z uj — a lv l-►- o4 -» " -h a I

i/ioziu>

" " a_a — ac a — a » axlux lv — " x i/>

X

O

-J H- *» — "1z *- a uj a. —"v ""— a vi x " *

4U.L/100

—

i/i

—00 — xo " * — -«.4—mo — n —I_Q_ " 1-UJK

O

:« vi »_» — — o — oc —►- 4 X in uj *o

O

» Z

Z — t-0»»P — 3"UJ4LU- x in x 4 — lo1

ZOO

— 00 — o —L- L- " "

rg

— " — -O.U.— — — —

"»

— <-

UixKl-L-h-k-^l-SC

O 44444440O XXXXXXXUJ4 acacccacocecacoo

" o » ooooaoooz"* - SU.ILU.LLU.ILU.> LU

c« ra c« -j

#

00 oo in g> r- o ff -<ff o oo ff o ff

ff ff ff ff

O

—0 co co oo oo 0 ff ff0000000 o0000000

O

Z Z Z ZZ ZZ zi/i i/i i/i i/i m it i/i is)

o■»II

zvLUzp

*-HoNI-aoz4X

IIUJX4z

oUlLLILUJ

zLOzoL-o.o"

LLlvocXoz»a»

3UJoz

«>

CL

4zQza<o_J

Pi__oUl

O

□zLO

-1azv_M

ao39UJ

UJoacol/l

*I-oUJILLLUJ

zI/)zoL-a.o"

CM

IIUJr>g

IS)

X4ocoaaca»

o

Oi

NtowzUJZUJL--4I—IS)

IPvecoLO

*v.vi/l

H--4t-l/l

*

oUJt—4acLUzUJliIS)

o►-1/1ozo4

o

oz

v¥-IS)

r--4►-IS)

*

*

IL--4p

azov

*ozUJ

**


EXAMPLE SIMULATIONS

SD 70-576

Space DivisionNorth American Rockwell

—ISJ .-4

I

00 -*

-* o

I

M OI

■c

rv -» Of\lIH

m O f^

- 126 -SD 70-576

o

<zv©x

oczvUJO

i-4-4

H

QC

zCD

«r o.

ooIo cc

UJ Z1

O */»

CO

I

fl___■ao

Cm

UJecva

o

l/tocaat3tLU

U.aacLUtoXZ

NIrtUJvoIvv.oacUJCDX

___>

z

HOUJ

acav.atUJCOXZ

ooo—nvUlaca.oaV-ZUJOatUJo.

oo

MatvatatUJ►-XUJvatUJo.

r»jm

itaZoa

iA<03

<X

t-

in

iA iAUJ <-J

UJ<S\ oLU CC© <-J -J

<.

> l*<

U UJoi a.UI

t-

ac=1x

UJO uj

z v< z»- <

Q.

*-uj Cllj uj

< U4

on en "—at i<n— a-u. Ac —LU LLc u-z v. at-UiJa

►-

OJ__

lLO — <

fc-

*- liJi/i z at_r __; ivo Iv

►-

Z -— oc< UJ© fc-— <— UJZ ac— L 5z o

< ac ofx < v

t-

X.J

L/. UJ *tuj Z O-J < UJ

ZWl — l/l— at x —O Oo -j o auj O uj

*" U UJ H» Oa. la.LU

t-

P- UJv z v«""U< Z U <

>a

< Z — <UJ - Xo > I ■- > <JZ O □ U.

Q

3«J < O — < XhIU I lva at Z Ln ac

***>

ijj -J O -J <u<z < <v UJ< t/. UJ Z L/l (X3oI — o

h- 3t

t-

at e--7 4

►-

< «nlv O U. -J O -33a 3 xO v. l/l U.uj O O uj C Zvn a a —co ix < ac <___! lv O -J UJ OiA to —I tO

X < — X UJO3* 3 XZ Zc- Z -— a. h-a uj uj — uj z-v x v x uj

o .-" v v, »- x— < — *"i/i v. v.Z — —Ov

QC

<0O-JO t/>0 <ac xUJ o1 3*- XUJ iA

<iL.O "*acO vi

* -JQC <O >-Ja i/»v

t-

z v.< —zX

t-

o zO<z

LU —X X*- o

Ov.vLn

oa ac4LUv x« —UJ»- ZaUJu_v—v"«

m

ItaczUJo

HatzCO

iA

IN. O

CO O

O —I

* oI

"O —I

OJ —I

«ro

oj—!«- o

I

ao

* IL_) CCUJ

z>

o »/»

ovi

UJLUatiA

f-

OUJa.UJv<

UJaX

o—i"oi

fsj

t

rg

I

II

OO

mI

tlUJacOv

"H

ataatacUJv.oacUJtoXZ

(0

RiAUJ

OXvv.oaUJCDXz

o

oUJvacav.atUJCOXZ

IA

CO

ai—

UJecaoo

zUJoatUJa

rgp^

Mataatatat

zLUoatUJa.

m

Hatz__:aUJo


k-

J

- 127 -SD 70-576

w o—UJ

O.OJvv

"* OI/.

o> — rgO —

c* —IOUJ

►-

Ul-IUJec

—I0* —— O

_jUJoax

OJ oI

_A O "r oi

tr o-4

m

acZ*vUJo

—HatzCO3iA

o —■C —I

rg O1

* oI

ttoo

tt IO COUJ 3O i"

erI

rg

o

<oI

II

aot-

rsj

nUJacov

ni/iatoatacUJ

v.aatUJCOX3z

HiAUJ

axvv.oatUJcoXZ

o

IIoUJoatav.aUJCDX3Z

mCDHt-UUJacacOv*-ZUJoatUJ

"

acvacacUJi-zUJvatUJ«.

en

Naztto

UJo

o

Haczttv

UJv

Hatzmi/l

o

<*

i

rg o

r>

—<c oi

ttoo

tt tv av>uj 3o */i

I

0»

oI

r-

iittca

"org

tlataovI/)

o

Hl/latOacacLU

LLoacUJtox3Z

IIiAUJOOXvLLaatox3Z

O

H□UJwacov.a

COX3z

oooIIh-uatataij

t-zvaUJa.

oo

Nataacocuj

t—

zLUvaUJa.

o**>R

aczttv

UJa

»-

*- ot

oUJ►-a.UJ(J

■# —(/»

o ——I "—I

I

oUJ

oUJ

UJat

o oI

rg O

"0 — _jUJOOX

rg OI

IAfl

N

atz

__t

*->UJa

iatzCO3

"o —I— oI

* o

m oI

t* ot

ttoo

t-

* IU CDUJ 3O LA

oI

g>i

I

Mtt0O

mt

ttU-JaoviA

—Hi/latoecocUJ

v,Oa.UJCOX3X

0>

ttl/>UJvraxwv.vatviCOX3z

o

IIoUJ

aoIXaUJODX3Z

<r

"COVHI-UUJececovt-zUJoatUJa.

HatoatacUJ

zUi(JatUJa.

if*en

NatZttv

UJo

"atzttvUJUl

Hacz_c3l/l

lf\ OI

*»

—rg O

» —

tt

ao

* IU '0uj 3O Lrt

CO

f-

-0»

H

ttvol-

cr

MUJatt_)ol/»

o

HinatoatatLU

v.aaUJenX3z

co

Hl/lLUvUX

<_>

ILCJaLUCDX3Z

rg

HvUJoatov.OLUJCDX3Z

O

"ooIII-vLUa_acOv

zUJvocUJ

O,

O

"O

HatoatacUJ

zLU

acUJ

0*i*i

itatzttoUlO

4|m Space DivisionNorthAmerican Rockwell

rst -*" i

<M O

r-a

t

rg O -*o

rg ot

rsi -"

f*. oI *^ O

I

-o OI I

loi

as imoOD

N N I.O -i O* "o o »

128 -SD 70-576

zttvUJa

—HOfZCD34/1

h- —rg ——

ttOOI-

* IO

cQ

UJ 3a**

*0

r»

rg

H

OO

iUJacOviA

fliACCaacacUJLLOatUJcoX3Z

IIiAUJUaxILaatLUtoX3Z

HOUJvacOv.atUJCOX3Z

oo—Ah«OUJotocOoI-zUJ

atUJ0>

Oo

IIataatatvii-zUJvatUJa

aczttvUJO

zttvUJo

Natzcc3t/l

(*. o

CD —I

ttooI-

* Io COUJ 3o */_

rg

I

I

COI

IIttOo1-

00

I

HUJaovl/»

HiAacaacac

a.oaUJCOX3Z

It</ludooX*_>

v.aaLUCOX3z

f\J

IIOUJvocov.atUJCOX3z

"Oo—R»->oUJntatoot-zUJvatUJa.

oo

Hacoececvit-zUJ

atUJa.

CO

"**eczntvUJo

X ►"

ir. —Ovii—CLUJ!__>a< 00 —

aUJ

t—

OlUJUU<i/>

iI

I*l — rg

rg o—I

OUJI-vUJUJat

—oI

_ooI

3UJI-oUJ-»UJec

m o .jUJooX

rg — -jUJOox

o_ o rg —irI

<0

RacZttvUJo

——ItatXCD3

* o

rg O

O —I

f*» —I

ttooI-

Iv COUJ 3O iA

i

oi

i»*t

H

oa

friI

UJatoviA

*

1l/lSoatatUJ

OatUJ■OX3Z

CO

Hi/lUJvoxvv.aatUJCOX3Z

rg

ROUJUaoLL

aUJmx3Z

ITr»COti

►-

vUJatatovI*zUJvacUJa

unrg

RacoecetUJ.-ZUJU___UJa.

«*igr

RatzttvUJo

Raczttv

UJo

——flatzCO3i/l

m oI

-4 —-4— o

<r

o

ttoa

* I

L,

J COUJ 3O i/>

m

rg

rg

HttOa

Pa,

"Om

RUJavvl/l

RWt0LoacatUJ

ILoetUJCDX3Z

CO

IIiAUJvoX

IXaatUJCDX3z

rg

ROUJ

atov.aUJCDX3Z

"ccitI-vUJatacov

z111vacUJ

*—■acaatacUJi-zUJvacUJa.

*r

Hat

oUJo


aUJ

\

129 -SD 70-576

rg O

UJaa<iA

rg ——I v*> — f*.I— —

—t

I

IN OI

O O—rg O(4

OUJoUJ

UJctt

CO — a o UJaax

cr —l

o* — I"oin

"r o

CO —I

H

atziCoUJa

—NOtZaa3iA

I— oI

m o

'

oott Io COUJ 3O V>

—ICO

erl

c*

ttttooI-

rqmI

ttUJaoi/i

o

iii/iacOeta.

u<

v.OecLUCCX3Z

0"

Ml/lonxvLLOecUPCCX3Z

o

HCJUJacOLL

aUJ03X3Z

oooHI-vUJacecoL->H-Z

UJvaUla

oo

flocvccacUJt"*zUJvatUJ&

rg

NatziCi_>UJO

RaczttvUJa

Meczo3L/l

(*l —I— OI

I

ttaatt iO CDuj 3a i/i

CDI

flI

H

Cofr-

rgCSI

IIUJac

Q

Ul/l

t

—H

iAOLacca.

ILUatUJCOX3Z

*CO

Hi/lUJvaxvLLOaUJccX3Z

rg

NO

atoLL

aLUCOX3z

"r-coH

OLUataa

i-zUJUatUJa.

inrg—MatOacecLU

H-ZUJvacUJa.

<oIT

Haczi£UUJo

L- i-

0* oI

oUJi—exUJvLj

oUla.ujvv< o —t

l/l iA

o —— "* **.rg —I

I*l —o o—I

oUJt-vUJUJac

oUJt-vUJ-1UJec

IP "*I

rg O—iI

* — _jUJoo

-jUJaax

rg —— O* O

eri

if. — O■n

IT —oIMI

HOCzttvUJa

HeczfO3i/l

m —« o

ifi oI

ttooI-

.c Iv xUJ 3O fc"

"r

rg

ir

nttooI-

f«

UJavvI/1

rg

Ni/latOatacUJ

LLCaLUtoa3z

c

NiAUJUOIvv.vaUJa_i

3z

O

MOUJavv.XM.1-.X3Z

CDl«-P*

H

OUJIt.atov

zUJvatUJa.

rg

"rgrg

ROCoatatUJ

zUJLJ«UJ&

amiti

HiXzTataUJ

IIatZtt

" vujo

ReczCD3i/l

— oI

<r

o

r- —ttoo

* IC. cPuJ 3O iA

ITI—

r-

ttL.a

cc

itUJavoi/i

O

Rl/»aoecOlUJv.oa.UJer»3Z

f-

Ri/iUJvoXvv.oOLUJccX3z

c.

ROUJvacov.atUJtox3Z

o"ooR

t-

oUJatacovt-zLUoacUJa.

oo

ROCoatatUJi-z

LUvaUJa.

if*

Hatz

UJa


»»

130 -SD 70-576

!

CO —I

UJ

a.UJvol/l

rgo

I-toI

h-oi

aUJvUJnUJ

OC

in o _JUJaax

-4 —— ODI

oo

ReczttvUJa

roczOD3

— —I

**lo

o —a* —

oant Il_) _oUJ 3

Q

i/i

am

amamI

O—C*

MntOa

om

RUJacOvl/l

"HWlaoocatUJ

v.OatUJoax3Z

er

Mi/lUJooXvILoatUJCDX3Z

oUJvaoILatUJCOX3Z

CTCOCO

I—vUJacacovI-zUJvacUJOl

■4—HOCo

OCatUJ

zUJvacUJ01.

Vac<XX3_/l

o10

ftazttvUJO

o

HutaoacacUJiLOacUJ03X3Z

gfm

liiAUJvoXwILoacUJCDX3Z

n

«aviUacov.AcUJCOX3z

mrgoRI-UUJacaco

zUJvacUJa

in

NacoatatUJ

zUJvaUJa.

N»

I*l o

oUJI-___.UJvviA

o — *—oI

fM O

oUJvUJUJac

rg — -jUJooX

♦— c*

0» rgoI rg

RofZttvUJo

—H

atz_D3l/l

rg OI

«___

amam

CO —a<oo

tt IU COUJ 3

O

wt

CD

Bntcja

gr

RUJac(jvVI

HwtatoececUJiLDacUJcc___3Z

er

Ri/»UJv

Q

Xvv.vatUJCDX3Z

O

oUJva.oLLaUJCDX3Z

er"CO

COflt»vUJacocowI*zUJvatUJOL

■acoacatUJt~zUlvaUJ&

crtn

iiazttvUJo


REFERENCES

t1131 -

SD 70-576


REFERENCES

Balzer, R. M. A mathematical model for performing a complextask in a card game. Behavioral Science, 1966, 11,219-226.

Berlekamp, E. R. Program for double-dummy bridge problems—A new strategy for mechanical game learning. Jour-nal of the Association for Computing Machinery,1963, 10., 357-364.

Clarkson, G. P. E. A model of the trust investment process.In E. A. Feigenbaum and J. Feldman (Eds.), Computers

and Thought. New York: McGraw-Hill, 1963. Pp.329-346.

De Groot, A. D. Thought and Choice in Chess . New York:Basic Books, 1965.

Ernst, G. W. and Newell, A. Some issues of representationin a general problem solver. Proceedings of theSpring Joint Computer Conference, 1967, 583-600.

Feigenbaum, E. A. The simulation of verbal learning behav-ior. In E. A. Feigenbaum and J. Feldman (Eds.),Computers and Thought. New York: McGraw-Hill, 1963Pp. 297-309.

Feigenbaum, E. A. and Feldman, J. Simulation of cognitiveprocesses. In E. A. Feigenbaum and J. Feldman,Computers and Thought. New York: McGraw-Hill,1963. Pp. 269-276.

Feldman, J. Simulation of behavior in the binary choiceexperiment. In E. A. Feigenbaum and J. Feldman(Eds.), Computers and Thought. New York: McGraw-Hill, 1963. Pp. 329-346.

132 -SD 70-576

- 133 -1SD 70-576

A


Findler, N. V. Human decision making under uncertainty andrisk. Kybernetik. 1966, 3., 82-93.

Frijda, N. H. Problems of computer simulation. Behavioral

Greenblatt, R. D. The Greenblatt chess program. Proceed-ings of the Fall Joint Computer Conference, 1967,801-810.

Gregg, L. W. and Simon, H. A. Process models and stochastictheories of simple concept formation. Journal ofMathematical Psychology. 1967, 4, 246-276.

Hunt, E. B. Computer simulation: Artificial intelligencestudies and their relevance to psychology. In P. R.Farnsworth (Ed.), Annual Review of Psychology. PaloAlto, Calif.: Annual Reviews, 1968. Pp. 135-168.

Johnson, E. S. An information processing model of one kindof problem solving. Psychological Monographs, 1964,78 (4, Whole No. 581) .

Laughery, K. R. and Gregg, L. W. Simulation of humanproblem-solving behavior. Psychometrika , 1962, 27 ,265-282.

Newell, A. On the analysis of human problem solving proto-cols. Presented at International Symposium onMathematical and Computational Methods in the SocialSciences, 1966. AD658863.

Newell, A., Shaw, J. c, and Simon, H. A. Chess-playingprograms and the problem of complexity. IBM Journalof Research and Development, 1958, _2, 329-335.

Newell, A., and Simon, H. A. GPS, a program that simulateshuman thought. In E. A. Feigenbaum and J. Feldman(Eds . ) , Computers and Thought. New York : McGraw-Hill, 1963a. Pp. 279-293.

Newell, A. and Simon, H. A. Computers in psychology. InR. D. Luce, R. R. Bush, and E. Galanter (Eds.).,Handbook of Mathematical Psychology. New York :Wiley, 1963b. Pp. 361-428.

Science, 1967, 12_, 59-67.


Samuel, A. L. Some studies in machine learning using thegame of checkers . IBM Journal of Research and De-velopment, 1959, 3, 210-229.

Simon, H. A. and Kotovsky, K. Human acquisition of con-cepts for sequential patterns. Psychological Re-view, 196 3, 70_, 5 34-546.

Simon, H. A. and Newell, A. Information processing computerand man. In W. R. Brode (Ed.), Science in Progress .New Haven: Yale University Press, 1966. Pp. 333--362.

Slagle, J. Unpublished lecture notes for UCLA Extensioncourse, The heuristic programming approach to arti-ficial intelligence, 1967.

Solomonoff, R. J. Some recent work in artificial intelli-gence. Proceedings of the lEEE, 1966, 54_, 1687--1697.

Thiele, T. N., Lemke, R. R., and Fu, K. S. A digital com-puter card-playing program. Behavioral Science,1963, 8, 362-368.

Turing, A. M. Computing machinery and intelligence. InE. A. Feigenbaum and J. Feldman (Eds.), Computersand Thought. New York: McGraw-Hill, 1963. Pp. 11--35.

Weizenbaum, J. Eliza, a computer program for the study ofnatural language communication between man andmachine. Communications of the Association forComputing Machinery, 1966, 9_, 36-45.

Williams, T. G. Some studies in game playing with a digitalcomputer. Unpublished doctoral dissertation,AD634 821, Carnegie Institute of Technology, 1965.

13U -SD 70-576

j 3* rton co* j£e4h^ t^t. - stacksfw020vy9950/fw020vy9950.pdf · because of the conditional...

Documents