computer-assisted and traditional psychological assessment of elementary-school-aged children

CONTEMPORARY EDUCATIONAL PSYCHOLOGY 6, 314-322 (1981)

Computer-Assisted and Traditional Psychological Assessment of Elementary-School-Aged Children

LARRY KATZ University of Calgay

AND

J. THOMAS DALBY Alhcrtu Children’s Hospital. Calgary

Computer-assisted psychological assessment was evaluated as a possible alternative to traditional assessment methods. Forty gifted children and forty behavior problem children were administered the State Anxiety and FIRO-BC personality inventories. Half from each group were tested using a cathode ray tube computer terminal while the others were given standard written forms of the tests. All children were retested at a mean interval of 1 week using the same procedures. Test-retest correlations were significant for all six FIRO-BC personality scales regardless of assessment method. Time required to administer and score the inventories was significantly reduced by the computer procedure. While the children’s initial perceptions of computers were generally favorable, the perceptions of those children who received the computerized tests significantly increased. Mea- sures of mean state anxiety were uniform with all groups showing significant de- creases from first to second testing sessions. These findings suggested that interactive computer assessment may provide a viable alternative to traditional assessment methods with children.

Computer assistance in psychological assessment has been advanced as a method of controlling testing variability or bias and making optimal use of highly trained personnel by releasing them from routine functions. While computers are now widely employed in the scoring and interpreta- tion of standardized psychological and educational tests, there remains relatively infrequent use of computers to administer these tests (Lushene, O’Neil, & Dunn, 1974).

Experimental programs have been developed to administer the Wechsler Adult Intelligence Scale (Elwood, 1969), the Minnesota Mul- tiphasic Personality Inventory (Dunn, Lushene, & O’Neil, 1972), the

This paper is based on a thesis submitted by the first author in partial fulfillment of the requirements of the M.Sc. degree at the University of Calgary. Thanks are due to the Thesis committee: Professor Ann Brebner, Dr. H. J. Hallworth, Dr. Roy Ferguson, and Dr. E. S. Edgington. The help received from the Staff and Children of the Education Assistance Service for Gifted Children and the Alberta Children’s Hospital was also appreciated. Re- quest for reprints should be sent to Larry Katz, Department of Educational Psychology, University of Calgary, Calgary, Alberta T2N lN4. Canada.

314 0361-476X/81/040314-09$02.00/0 Copyright 0 1981 by Academic Press. Inc. All rights of reproductmn in any form reserved.

PSYCHOLOGICAL ASSESSMENT BY COMPUTER 315

Peabody Picture Vocabulary Test (Overton & Scott, 1972), the California Psychological Inventory (S&sons, 1976) and the Eysenck Personality Inventory (Katz & Dalby, 1981). These computerized formats have pro- duced scores highly and positively correlated with those from traditional testing methods. They have also yielded comparable validity and test-retest reliability.

The apparent resistance to large-scale automized administration of psychological tests may stem from the impression that the “coldness” of the computer approach would inhibit the subject’s performance. Indeed, much attention in clinical training is focused on establishing rapport between tester and testee in an effort to promote optimal or typical responses. Subject’s reactions and attitudes toward computerized administration of tests should be included in test format comparisons. Research on this question has generated mixed results. Evan and Miller (1969) reported that students tended to be more open and honest in responding to computerized administration of a psychological inventory, particularly if the items were of a personal nature. Hedl, O’Neil, and Hansen (1973), however, found higher levels of state anxiety in subjects after computerized testing compared to manual test administrations.

The present investigation studied computer-assisted and traditional administration of two personality tests with gifted and behavior problem children. The experiment addressed three specific issues: test-retest reliability, the effects of administration method on personality assessment, and the differences between subject groups and administration methods on measures of state anxiety and attitude toward computers, when prior experience is taken into account.

Subjects

METHOD

Eighty children between the ages of 8 and 12 years served as subjects. Forty children (22 male, 18 female) were enrolled in a special program for gifted children (i.e., superior intellect or creativity) offered by the Calgary Board of Education. The mean age of this group was 10.2 years and the mean grade level was 5.1. The remaining 40 children were from the outpatient referral program of the Psychology Department at the Alberta Children’s Hospi- tal. This group was composed of a heterogeneous sample of children who displayed be- haviors inconsistent with the expectations of their parents. The mean age of this group was 10.1 years and the mean grade level was 4.5.

Instruments PreViOUs experience item. Children responded to a Likert-type item to determine

the extent of their previous direct experience with computers. Nine gifted children and five behavior problem children reported some previous experience.

Semantic Differential Questionnaire. Children’s attitudes toward computers was assessed by a IO-item semantic differential with the word “computer” as the stimulus word. Six of the adjective pairs were chosen from Hallworth and Wane (1966) with the

316 KATZ AND DALBY

balance from Osgood, Suci, and Tannenbaum (1957). To maintain scoring consistency those adjectives considered to be favorable toward computers were assigned a value of 7, and unfavorable pole adjectives were given the score of 1. Order of scale and adjective presenta- tion was randomly assigned.

State Trait Anxiety Inventory for Children (STAZC). The self-report measures for the state anxiety scale (Speilberger, 1973) were used to assess changes in anxiety between testing sessions. The scale is comprised of 20-item statements designed to measure situational anxiety.

Fundamental Interpersonal Relations Orientation (FIRO-BC). The FIRO-BC was used to monitor test-retest reliability. Developed by Schultz (1967), the FIRO-BC measures children’s characteristic behavior toward other children.

Computer A PDP-11170 Digital Equipment Corporation time-shared computer system was utilized.

Access to this computer was through an Ann Arbor terminal connected to the computer by means of an acoustic coupler and phone line. This terminal has a cathode ray tube (CRT) display screen for written instructions, with a capacity of 24 lines of 80 upper case letters. Responses are typed on a portable keyboard.

The computer-assisted testing sequence was written in BASIC-PLUS. The courseware consisted of a set of four computer programs, administrative control, test management for both the state anxiety and FIRO-BC personality questionnaires, and tile management.’

The state anxiety test was presented to each subject initially. The instructions were presented on the CRT screen followed by the questions, displayed one at a time with the associated response choices. Each question had to be answered before the next one was shown. After completing the inventory, the system automatically accessed the FIRO-BC test which was administered using a similar process. The system was thus self-paced with the total amount of time to complete each questionnaire automatically recorded. Errors in typing responses were identified and rejected by the program and the question was repeated.

Procedure After obtaining parental consent children from the gifted program were transported in

small groups to the computer facility. They were administered the previous experience item and semantic differential questionnaire. After this they were randomly assigned to traditional or computer testing formats. Traditional manual administrations of the state anxiety test and FIRO-BC were timed by a concealed stopwatch. Before computer administration, subjects were instructed in the use of the computer terminal. Subjects were tested individu- ally. At a l-week interval children returned to the testing center and were again assessed using the same format that they had initially received. The same procedures were followed for the behavior problem subjects. They were, however, tested at the hospital before or after a regularly scheduled appointment.

Analyses The evaluation of the experiment was divided into three components. First, the reliability

for both the semantic differential and the FIRO-BC personality inventories was assessed, then sample and treatment differences were analyzed and finally the effects of previous experience were considered.

In evaluating the semantic differential, the principal components were calculated from the

’ Complete details regarding computer programs are available from the first author upon request.


item intercorrelations and then a varimax factor rotation was employed. After extracting the evaluative dimension, test-retest reliability was calculated, using the items which loaded on that dimension. Only scores from those children tested in the traditional manner were used to determine the reliability of the semantic differential. Since these children did not come in contact with the computer, their attitude should have remained relatively stable.

For the six scales of the FIRO-BC, test-retest correlations were calculated for each sample and method. It was hypothesized that FIRO-BC test-retest correlations would not be significantly different for computerized and traditional testing procedures. To test this hypothesis reliability coefficients were converted into Fisher’s Z, statistic and then a t test was used to examine method differences.

In analyzing sample and treatment differences, the between subject independent variables were sample membership (gifted or behavior problem children), administration method (computer or traditional), and sex of child (male or female). Testing session (first and second) was the repeated measure. The nine dependent variables were the score derived from the semantic differential, the state anxiety score, the six FIRO-BC personality scale scores, and the length of time required to administer and score the state anxiety and FIRO-BC tests.

A four-way multivariate analysis of variance with one repeated measure was utilized. In evaluating prior experience a one-way multivariate analysis of variance was used. When differences significant at the .05 level were uncovered in the multivariate analyses. univariate analyses followed. An alpha level of .Ol was set for univariate tests.

RESULTS

Semantic Differential Questiorznuirc

The initial scores indicated generally favorable perceptions of computers from the outset. In the first principal component analysis, a single dimension of connotative meaning appeared to exist (Table 1). Moreover, the first principal component accounted for 25% of the total variance. As

TABLE I PRINCIPAL COMPONENT ANALYSIS OF THE SEMANTIC DIFFERENTIAL

Factor

Item 1 2 3 4

Interesting-boring .60610 .08195 -.27270 .38146 Clean-dirty .45597 .22048 - .42774 -.30125 Good- bad .44681 .58317 .20944 .I7795 Fast-slow .47316 -.34017 -.06092 .46419 Useful-useless .56072 .09817 -.58680 -.09800 Safe-dangerous .66287 -. 12724 .27477 .12344 Wise-foolish .52743 .00468 .03123 -.54394 Kind-cruel .52790 .09052 .62447 -.36740 Easy-difficult .41661 -.70241 .20904 .I1373 Moving-still .03594 .60838 .25728 .37855

Eigen value 2.48 1.41 1.22 1.09 % variance 24.80 14.10 12.20 10.90

318 KATZ AND DALBY

the varimax rotation obtained no identifiable structure, the principal component analysis was retained.

Those items loading above .40 on the first principal component were included in a “perception scale.” Consequently, only the “moving-still” item was deleted. The score from this scale was used to monitor changes in children’s perception of computers.

Test-retest reliability for the perception scale was .81 (N = 40,~ < .Ol) indicating high test-retest stability.

FIRO-BC Scales

Test-retest correlations for the FIRO-BC scales are listed in Table 2. All scale correlation coefficients were significant, ranging from .5 to .9, displaying a high degree of consistency over testing sessions.

In the computerized administration, three out of the six scales had test-retest reliability significantly higher than those of the traditional administration method. These scales were: Control-wished [t(38) = 5.75, p < .OOl], Affection-expressed [t(38) = 7.27, p < .OOl], and Affection- wished [t(38) = 4.75, p < .OOl]. The traditional procedure had significantly higher reliability on one scale, Inclusion-wished [t(38) = 5.01, p < .OOl].

Analyses of Variance

Four-way MANOVA. The MANOVA program compared sample membership, testing method, testing session, and sex on measures of state anxiety, perception of computers, six scales of the FIRO-BC, and administration and scoring time. While there was a significant main effect differ- ence between gifted and behavior problem samples, F(9,64) = 2.31, p < .05, univariate analyses revealed that only one of the nine dependent variables, administration and scoring time, differed between the two samples, F( 1,72) = 17.14, p < .OOl. This indicated that the gifted children took significantly less time (M = 16.5 min) than the behavior problem children (M = 19.4 min) to complete the tests.

A significant main effect for method, F(9,64) = 6.85, p < .Ol, also resulted from differences in administration and scoring time, F( 1,72) = 55, p < .OOl. The computerized method took significantly less mean time to administer and score (14.7 min) than the traditional method (17.7 min).

A testing session main effect was also significant, F(9,64) = 14.7, p < .OOl, with univariate analyses revealing significant differences on three dependent variables: perception of computer, F(1,72) = 12.18, p < .Ol; state anxiety, F(1,72) = 14.81, p < .OOl; and administration and scoring time, F(1,72) = 54.13, p < .OOl. Examination of mean changes from first to second testing sessions showed that there was a significant in- crease in mean positive attitudes toward computers (50.9 to 53.05), a

TABL

E 2

FIR

O-B

C

TEST

-RET

EST

CO

RR

ELAT

ION

C

OEF

FIC

IEN

TS

2 2 Sc

ale

0”

Sam

ple

Met

hod

Ie

IW

Ce

cw

Ae

Aw

6 0 G

ifted

ch

ildre

n Tr

aditi

onal

,*y

**

.66*

* ,lJ

-J*‘“

* .6

4**

Jgj””

.6

6**

5 C

ompu

ter

,3’@

**

,72*

**

.72*

**

,Qj**

” Jl

”*”

,69*

“”

I?

Com

bine

d ,,7

***

,69*

*”

,g(j”

**

,,y+*

* ,6

9***

&

j***

P

Beha

vior

pr

oble

m

ii ch

ildre

n Tr

aditi

onal

,7

g+**

,g

j*:%

” ,7

7***

.5

0*

.53*

.6

0**

Com

pute

r .g

4***

,g

**

,g,r*

* .5

0”

,gl**

:” Jf

j*“”

g

Com

bine

d .g

()***

JO

***

,p**

,5

@“”

,,4*“

‘: 39

**”

2

Gift

ed

and

beha

vior

m

pr

oble

m

child

ren

Trad

ition

al

.g,**

* J6

***

,R(j*

“”

,55”

” <

59”

d:

,(jO

*Ss*

C

ompu

ter

Jy/“*

” ,&

,“*”

,83’

1”

,731

:“:’

,,p*+

,7

4***

8

Com

bine

d Jy

”” ,p

**

,gy*

* ,6

-y:‘%

” JO

**”

,(j7”

“”

s z *p

<

.05.

2

**,I

< .O

l. 7J

***p

<

,001

.

320 KATZ AND DALBY

significant decrease in state anxiety (30.15 to 28.3) and that it took significantly less time to administer and score the inventories the second time (17.95 to 14.4 min).

Only one interaction was significant, a method of administration by testing session effect, F(9,64) = 2.08, p < .05. Univariate analyses revealed that the significant interaction was attributable to changes in children’s perception of computers, F(1,72) = 12.48,~ < .Ol. This meant that children who received the computerized tests significantly increased their mean positive perceptions of computers (50.45 to 54.6), while the perceptions of children using the traditional method did not change appreciably (51.6 to 51.5).

Children tested by computer did not obtain results on the personality tests significantly different from those of children tested in a traditional fashion. There were also no significant differences between sexes.

One-way MANOVA. Since only 14 of the 80 children had any previous experience with computers, the 5-point previous experience item was converted to a dichotomy of no experience and some degree of experience. A MANOVA was used to compare these two groups on initial perception of computers and initial state anxiety. No significant differences were observed.

DISCUSSION

The purpose of this study was to evaluate the reactions of elementary school-aged children to computer assisted assessment. Test-retest re- liabilities for the FIRO-BC scales administered by computer were significant, equivalent to those found in the FIR0 manual, and consistent with the results of previous research. In addition, reliability for the computerized administration was significantly higher than for the traditional administration in three out of the six FIRO-BC scales, reinforcing the idea that computerized testing is at least as reliable as traditional procedures. As there were no mean differences between the administration formats on the six personality scales, it would appear that the two methods had equivalent validity.

These findings are similar to previous comparisons of computerized and manual test administrations. This study extends these favorable generali- zations to elementary-school-aged children. The results were consistent across settings and samples. The gifted children were able to complete the inventory faster than the behavior problem children, perhaps due to better reading skills.

Initially, almost all the children had high positive perceptions of computers. This is consistent with a previous study (Lucas, 1977), which suggested that young people tend to have positive attitudes toward computers. Despite these high initial values, the perception of computers by


those children who were tested with the computer significantly improved. These results conflict with those of Hedl et al. (1973) who found that the group using the computer was initially more anxious and showed a decrease in positive attitude toward computers after computerized testing.

Children’s levels of state anxiety showed a significant decrease from first to second testing session, irrespective of administration method or sample membership. This reduction in state anxiety is a common clinical phenomenon attributable to the children’s acclimatization to an unfamiliar setting.

Automated procedures appear to be useful in educational and clinical settings, but it remains necessary to explore the limits of applicability. Rezmovic (1977) found that computer assessment significantly increased score variability. He discussed this problem in terms of individual subject differences and suggested that computerized testing may have detrimental effects on certain individuals, particularly when the interactive effects of attitude and previous experience are considered. Emphasis should be placed on these variables in future investigations.

REFERENCES

DUNN, T. G., LUSHENE, R. E., & O’NEIL, H. F. Complete automation of the MMPI and a study of its response latencies. Journrrl of Consltlting rr& Cliniccrl P.yvc~ho/og:v, 1972, 39, 381-387.

ELWOOD, D. L. Automation of psychological testing. Americrrn P.rwho/ogist, 1969, 24,

287- 289.

EVAN, W. M., & MILLER, J. R. Differential effects on response bias of computer vs. con- ventional administration of a social science questionnaire. Beha\liorrr/ Science. 1969, 14, 216-227.

HALLWORTH, H. J., & WAITE, G. A comparative study of value judgements of adolescents. British Jourml of Educufionnl Psychology, 1966, 36, 202-209.

HEDL, J. J., O-NEIL, H. F., & HANSEN, D. N. Affective reactions toward computer-based intelligence testing. Jolrrnal of‘Consrt/ring and Clinicfrl Psyhology. 1973, 40, 217-222.

KATZ, L.. & DALBY, J. T. Computer and manual administration of the Eysenck Personality Inventory. Journal of Clinicnl Psychology, 1981, in press.

LUCAS, R. W. A study of patients’ attitudes toward computer interrogation. Internotionnl Jortrnal of Man-Machine Stlrdies, 1977. 9, 69-86.

LUSHENE, R. E., O’NEIL, H. F., & DUNN, T. Equivalent validity of a completely computerized MMPI. Journcrl ofPersoncr/ity Assessment. 1974, 38, 353-361.

OSGOOD, C. E., SUCI, G. J., & TANNENBAUM, P. H. T/lr ~~~‘(ISIITC~II~)II c;f‘nzetrnirlg. Ur- bana: Univ. of Illinois Press, 1957.

OVERTON, G. W., & SCOTT, K. G. Automated and manual intelligence testing: Data on parallel forms of the Peabody Picture Vocabulary Test. Amrriccrn Joumcrl of Mental Dejkienqv. 1972, 76, 639-643.

REZMOVIC, V. The effects of computerized experimentation on response variance. Behrrvior

Research Methods crnd Instrrtmentcrtion. 1977, 9, 144- 147.

322 KATZ AND DALBY

SCHULTZ, W. C. The FIR0 scu1e.r manual. Palo Alto, Calif.: Consulting Psychologists Press, 1967.

SCISSONS, E. H. Computer administration of the California Psychological Inventory. Mea- .surement und Evcrlrrntion in Guidunw. 1976, 9, 22-25.

SPEILBERGER, C. D. Manual for the state twit anxiety inwntory for children. Palo Alto, Calif.: Consulting Psychologists Press, 1973.

computer-assisted and traditional psychological assessment of elementary-school-aged children

Documents