, lAoP4-1 3

AlIR FORCE 'QI~OPTER ANj

Go HU Gerard L. Kiely

A IDavid J. Wess 4.

Ikmlsi of II~mmet

A lstp~eltsm

NMANPOWER AND PERSONNEL DIVISION

Brooks Air Force Base Texas 78235-5601 ,.-.

E'+.o. v

August 198"E Ml Tecbml al fr eriod Apri low a low

0U

;.. % '

R Approved for public release; distribution is unlimited.CES LABORATORY

DTICELECTE AIR FORCE SYSTEMS COMMANDAUG 2 5 19%86 BROOKS AIR FORCE BASE, TEXAS 78235-5601

4L ..-- "

E . .

q*,'.*,...,..

NUTCE

WMn government draings, specifications, or other data are used for anypurpose other than In connection with a definitely bovornent-relatedprecurenent, the United State Government incurs no responsbility or anyebligation whatsoever. The fact that the bGovenset my have formulated orIn amy way supplied the said drawings, specificatioes, or ether data, Isnet to he regarded by implication, or otherwise in any manoer construed, aslicensing the holder, or any ether person or corporation; or as coveyingany rights or permission to manufacture, use, or sell any patentedInvention that ray In any way be related thereto.

The Public Affairs Office has reviewed this paper, and It is releasable tothe Natioal Technical Information Service, where it will be available tothe general public, Including foreign nationals,

This paper has been reviewed and is approved for publication,

NILLIAN E, ALLEY, Scientific AdvisorNfinpower and Personnel Division

RONALD L. KEKCNMME, Colonel, USAFChief, Nanpower and Personnel Division

OL ADDRESS MV Siefe and ZPCOW) 10. SOURCE OF FJNOING NUMBERSBrooks Air Fe Bass, Tonas MU354M ~ PROGRAM PROJECT I TASK Iwom UNIT

ELEMENT No. no. NO. EACSI No.

11. TITLE -~W Sewr-1 miteN I O .1 719i

jgqvlmcs of Computer and faps,'-anud l Armed Services vocationul Aptitude Battery Tests

1I. PERSONAL AUTHOR(SKiely, Gerard W. Zara, PAtbos R.; and Wiss, David J.

13a. TYPE OF REPORT 3b. TIM COVERED 14. DATE OP REPORT (Vbee Aftwh Day O) rS. PAGE COUNTFinmal PROM 64 L TO 3an-86 August IM 6

16 SUPPLMENTARY NOTATION

17. COSATIWODES I&. su1Ucr TuRMs lCone an rovoe if necenuy and Adenwi& by block number)ILD GROUP SU8400j, abilities testing06 o aptitude testing

I I hueN Services Vocational Aptitude Battery (ASVAB) (Continsued)1t. ABSTRACT (Couwam anem N IVfneeoy and "No"~ bJY blo0k nMbe

Rut order to investigate the possibility of obtaining equivalent results between computer and pape-and-penci test adminstration, Selected subtests Of the Armed Services Vocational Aptitude Battery (ASYAB) wereprogrammed for cathosratube (CRT) adm iistrati on. Speeded subtest items were presented two ways: one ata tim, and is blocks of several Item, Paragraph comprehnson Items, which were too lon to fit on the CRTscreen, were presented in three different scrolling modes. The graphical portions of Items were displayedfrom edited cede created by a conertcI digitizer. These CRT subtests were then compared with their paper.a aud il countaerts using Air Force remrits In a counterbalanced design. Results indicated thatobtaining equiwulence bet"ee paper-and-pencIl aid computer administration appears feasible. Graphical Itemspresent the least difficulty. Item production for on type of speeded subtest was best approximated bysingle-Ite Cl? presentatien, aid for another type of speeded subtest by CRT presentation of blocks of Item.Additional research Is required for the *too loag paragraph comprehension Items In which mre practice with

the scrolling a my be usefl.

its is (Concludd):

cathei ray U6e (CRT)claslftoatlncmputer adaWtve tstoqcmputer tsttoaelactle

II

__ DTIC

Accession o AY IE T

Dr T C AUG 2 5 W6

By-I

Av2 v; w;DI ti

, - - - , * '. , .' " . .% , •.,," . ." .% "." ,-,," ,. .. ,1 , . "

MIt. Techmical Paper 6-13 August 196

EQIVALENJI OF COlVT!R A PAPER-AN.-PMICZLANED SERVICES VOCATIONAL. AwITUmE MATERY TEST

Gerard L. Mel)Anthony I. ZrDavid J. Weiss

Department of psychologyUiversity of Mnmesota

75 East River leadNm Ieapell Is, Nn teset 4SU -0344

IMPNP AM PERONE DIVISION

Bres Air Force Base, Tens 78236-501

Reviewed by

Nalcolm James ReSenior Scientist, Force Acqulsltion Branch

Sul.itted for publication by

Lotase D. Valentine. Jr.Chief, Force Acqulsition Branch

This publicatlon is prmrlily a working paper. It Is published solely to document work performed,

p . -. . -r r F "" l - -' .i** ;.. ;;..A. ' .. ..ii ¢

SUMMARY

?be armed services have a goal of administering their operationalselection and classification enlisted test, the Armed Service@ VocationalAptitude Battery (ASVA), on the computer. Intentions are to administerthe same kind of subtests as exist in the present operational paper-and-pencil (PAP) battery and to be able to do so in such a mnner as to make ittotally irrelmnt whether an ocaminee receives the computer-adnisteredsubtest or the P&P version.

Several subtests consist of items which by their very nature wouldseem to be potential problems for transfer to the cathode-ray tube (CRT).There are two different types of speeded subtests (numerical operations andcoding speed), whose Items are easy and on which the final score generallydepends on the number of items accomplished in a short time limit. The CRTis much too small to duplicate a full page of items such as those presentedin a P&P mode. Similarly, the sis of paragraph comprehension Itemsprevents their appearinL fully on the CRT screen. Finally, three of theASVAB subtests have Items with Illustrations, and CRT presentation mustswitch from flat ink drawings to vertical light drawings on the screen.

Speeded subtests were programmed for two different CRT presentationodes, paragraph comprehension for three different nodes, and graphics were

displayed from code created by an off-the-shelf commercial digitizer.These CRT subtests were then compared with their P&P counterparts using AirForce recruits in a counterbalanced design. Results indicated that,obtaining equivalence between P&P and computer administration appearsfeasible. Graphical items present the least difficulty.. Item productionfor one type of speeded subtest was best approximated by single-item CRTpresentation, and for another type of speeded test by CRT presentation ofblocks of items. Additional research is required for the "too long"paragraph comprehension items in which more practice with the computerscope may be useful.

IL

PREFACE

Several Armsd Services Vocational Aptitude Battery (ASVAB) subtestsconsist of items whose characteristics ware expected to change whenadministered on a computer screen. This technical paper examines severalways to administer these items in order to find one which is equivalent topaper-and-pencil administration. This effort is ancillary to the Air Forceresponsibility for item pools for a computer adapted ASVAB,

- ii -

~ U **~~*j , . .:

TANLE Of C01NTENTS

Page

Excperimental Deaign.................e....e...ce......... 2Subjects and Data Collection.......................... 2Tate and Conditions................................7 3Treatment Gop. .... .*.******e* ** 4

Data Anlss .... ................. 7Combined Cntin................. 7

* Analysis of Variance (ANOVA). ........... S0gggO0cC g0.g 9Reliability Analyste. goo.oo.oo..oo..... ............. 14

*Analysis of Structure................................. 17

Analysis of Viac..ggege eg.egee ggegc 19Paper-and-Pencil Baseline Analysis....eoeoooooa*.. ee* 19Computer versus Pae-n-ecl.....e..e..27

Reliability Alyi. e.....g.g.e.e.egce33

Test-Retest Corltos.eggge.gege.e~.33Internal Cnitnis ec...ggcecegeg.e34

Analysis of Stutr. .. ~..ge.. g e... .35

Across Sutas ~... ............ 35

Item Anlas......................42

Paragraph Comprehension Stet............ 43Speeded Stet.................... 44Graphical Subtestoos.. ......... eog...... ................. 45Subtest Structure. goooe.oo.. ........ ..... coe..g....g.. 45

V. CONCLUSI0NS.ooo.eo..... cc........ osoosoo.............oo~o 45

APPENDIX As ITEM FACTOR LOADINGS AND COKNfl4ALITIES BySUBTEST AND PORK FOR TWO PRESENTATION MEDIA. 49

APPENDIX B: ITEM STATISTICS (DIFFICULTY, POINT-BISERIAL,AND BISERIAL CORRELATION) BY SUBTEST, PORM,AND VERSION FOR TWO PRESENTATION MEDIAoooooo 59

7. T! -701 -.

LIST OF FIGIRES

FigurePage

1 Data Collection Design for PC Subtest ................. 5

2 Data Collection Design for NO and CS Subtests.......... 6

3 Data Collection Design for AS, NC,0 and 11

Su t s s . . . . . .. . . . . . . . . .. . . . . . . .

4 Eperimesntal Design Plan for Paper-and-Poel.

5 Experimental Design Plan for Analysis ofParagraph Comprehension Subtest .............. 12

*6 Experimental Design Plan for Analysis of Speeded4Subtests *. .................................. 13

7 khperiantal Design Plan for Analysis ofGraphical Subtests .. . ................ .......... 15

8 Overview of Test Conditions by Group and Sessionfor Computer (CRT) and Paper-and-Pencil (P&P)Groups ..................................... 16

LIST OF TABLES

Table Page

1 ASYAB Subtests Used with Corresponding Number ofItms, Operational Time in Minutes, and Speededness ... 2

2 Pillai-Bartlett Trace Values, Degrees of Freedom,Approximate F Ratios,, and Estimated SignificanceLevels (p) for the Baseline MANOVA (N - 333)..........20

3 Results of Univariate ANOVAs for SignificantEffects Identified in the Baseline MANOVA, (N - 333) ... 20

4 Mean and Standard Deviation of Scores,,Kuder-Richardson Formula 20 Reliability Coefficient(KR2O), and Number of Examinees (N), for the PCSubtest by Group and Session ...................... 21

5 Mean and Standard Deviation of Scores, and Numberof Examinees (N),, for the NO Subtest by Group andSession ................................... 22

-iv-

6 Mean and Standard Deviation of Scores, and Numberof Examinees (N), for the CS Subtest by Group andSession .................................. 23

7 Mean and Standard Deviation of Scores,Kuder-Richardson Formula 20 Reliability Coefficient(KR2O), and Number of Examinees (N), for the ASSubtest by Group and Session............ ..... 24

8 Mean and Standard Deviation of Score.,Kuder-Richardeon Formula 20 Reliability Coefficient(ZR2O)# and Number of Examinees (N), for the NCSubtest by Group and Session .................. 25

9 Mean and Standard Deviation of Scores,Kuder-Richardson Formula 20 Reliability Coefficient(KR2O), and Number of Examinees (N), for the ElSubtst by Group and Session.................... 26

10 Results of the F-tests of Significance from theNZOVA for the Computer Versus Paper-and-Poecl

Administration PC Subtest Conditions (N - 664) ........ 28

11 Pillal-Bartlett Trace V Values with 2 and 655Degrees of Freedom, Approximate F Ratios, andEstimated Significance Levels (p) from the NANOVAfor the Computer Versus Paper-and-PencilAdministration Conditions for the NO and CSSubtests (N - 664) .............................. 29

12 Results of Univariate ANOVAs for SignificantEffects Identified In the NANOVA for the Subtestswith 1 and 656 Degrees of Freedom (N - 664)........... 30

13 Pillai-Bartlett Trace V Values, Degrees ofFreedom, Approximate F Ratios, and EstimatedSignificance Levels (p) from the NANOVA for theComputer Versus the Paper-and-PencilAdministration Conditions for the AS, MC, and ElSubtests (N - 664) .................................. 31

14 Results of Univariate ANOVAs for SignificantEffects Identified in the NANOVA for the AS, MC,and El Subtests with 1 and 652 Degrees of Freedom(Ni - 664) ............................... 31

15 Test-Retest Correlations (r), for Subteuts CS,AS, NC, and El and Alternate Forms RetestCorrelations for PC and NO Subtests and Number ofExaminees (N) by Group and Node ..................... 33

W . .. ... . .... .. gym , Y Vy .o '- ,d .

16 Average Kuder-Richardeon Formula 20 Reliability(KR20) for Subtests PC, AS, MC, and E1 byExperimental Group and Session ...................... 34

17 Intercorrelations of Subtest Scores by Group forSession 1 (Upper Triangle) and Session 2 (LowerTriangle) ........................................... 36

18 Elgenvalues and Percentage of Common VarianceAccounted for from Principal Factor Analyses ofSubtest Scores by Group and Session .................. 37

219 Factor Loadings and Communalities (h ) fromPrincipal Factor Analyses of Subtest Scores byGroup and Session ............... ............ 38

20 Summary Table for Testing Models of FactorEquality Within Session, Showing Model,Chi-Square Value (X2 ), Degrees of Freedom (df),"Goodness of Fit Index (GFI), Root-Mean SquareResidual (RNR), and Estimated Probability (p) ........ 39

21 Eigenvalues (E) and Percentage of Common VarianceAccounted for (Z), by the First Five PrincipalFactors from Subtests AS, MC, and El by Formand Administration Medium ............. 41

A-i Factor Loadings and Communalities (b) for theFirst Five Principal Factors from Session 1Subtest AS Form 11 for Group 1 (Computer) andGroup 3 (Paper-and-Pencil) .......................... 50

2A-2 Factor Loadings and Communalities (h ) for theFirst Five Principal Factors from Session 1Subtest AS Form 12 for Group 1 (Computer) andGroup 3 (Paper-and-Pencil) .......................... 51

2A-3 Factor Loadings and Communalities (h ) for theFirst Five Principal Factors from Session 1Subtest AS Form 13 for Group 1 (Computer) andGroup 3 (Paper-and-Pencil) .......................... 52

A-4 Factor Loadings and Communalities (h2) for theFirst Five Principal Factors from Session 1Subtest MC Form 11 for Group 1 (Computer) andGroup 3 (Paper-and-Pencil) ........................... 53

A" 5 Factor Loadings and CommunalitLes (h ) for theFirst Five Principal Factors from Session 1Subtest MC Form 12 for Group I (Computer) andGroup 3 (Paper-and-Pencil) ............................ 54

- i

, ., , . ,. ,, , , ) , , . , , -, . -, -. .. _. ' . . . . . . . . . . . . . . e n . " , , ', - - " -' ". " - " . " . ". " . . . . . . .

A-6 Factor loadings and Coummalities (h ) for theFirst Five Principal Factors from Session 1Subtest MC Form 13 for Group 1 (Computer) andGroup 3 (Paper-sand-Pancil)....................... 55

A-7 Factor Loadings and Couunalities (h ) for theFirst Five Principal Factors from Session 1Subtest ZI Form 11 for Group 1 (Computer) andGroup 3 (Paper-end-Pencil) ......... *. . ** .. ....... 56

A-8 Factor Loadings and Comumnalities (h)for theFirst Five Principal Factors from Session 1Subtest 11 Form 12 f or Group 1 (Computer) andGroup 3 (Paper-and-Pencil) ...................... 57

A-9 Factor Loadings and Comunalities (h 2) for theFirst Five Principal Factors from Session 1Subtest II Form 13 for Group 1 (Computer) andGroup 3 (Paper-and-Pencil) .......................... 58 .**

B-1 Item Difficulties (p),, Point-Biserial Correlations(rpb), and Biserial Correlations (rbis), forVersion A of the PC Subtest by Session andTest Administration Medium ............. 60

B-2 Item Difficulties (p),, Point-Biserial Correlations(rpb), and Biserial Correlations (rbis), forVersion B of the PC Subtest by Session andTest Administration Medium .... * .............. 61

B-3 Item Difficulties (p), Point-Biserial Correlations(rpb), and Biserial Correlations (rbis), Within:.Session 1 for Version A of the AS Subtest by ..

Form and Test Administration Medium .................. 62 --

B-4 Item Difficulties (p), Point-Biserial Correlations ~.(rpb), and Biserial Correlations (rbis), WithinSession 1 for Version B of the AS Subtest byForm and Test Administration Medium................... 63 . .

B-5 Item Difficulties (p), Point-Biserial Correlations(rpb), and Biserial Correlations (rbis), WithinSession 2 for Version A of the AS Subtest byForm and Test Administration Medium .................. 64

B-6 Item Difficulties (p), Point-Biserial Correlations .

(rpb), and Biserial Correlations (rbis), WithinSession 2 for Version B of the AS Subtest byForm and Test Administration Medium .................. 65 0.vv

-vii- '

RKA"

B-7 Item Difficulties (p), Point-Biserial Correlations(rpb), and Biserial Correlations (rbis), WithinSession 1 for Version A of the MC Subtest byForm and Test Administration Mediun .................. 66

3-8 Item Difficulties (p), Point-Biserial Correlations(rpb), and BLiserial Correlations (rbis), WithinSession 1 for Version B of the MC Subtest byForm and Test Administration Medium .................. 67

B-9 Item Difficulties (p), Point-Biserial Correlations(rpb), and liserial Correlations (rbis), WithinSession 2 for Version A of the NC Subtest byForm and Test Administration Medium .................. 68

B-10 Item Difficulties (p), Point-Biserial Correlations(rpb), and Biserial Correlations (rbis), WithinSession 2 for Version B of the MC Subtest byForm and Test Administration Medium .................. 69

3-11 Item Difficulties (p), Point-Biserial Correlations(rpb), and Biserial Correlations (rbis), WithinSession 1 for Version A of the EI Subtest byForm and Test Administration Medium .................. 70

B-12 Item Difficulties (p), Point-Biserial Correlations(rpb), and Biserial Correlations (rbis), WithinSession 1 for Version B of the El Subtest byForm and Test Administration Medium ................. 71

B-13 Item Difficulties (p), Point-BLserial Correlations(rpb), and Biserial Correlations (rbis), WithinSession 2 for Version A of the El Subtest byForm and Test Administration Medium .................. 72

B-14 Item Difficulties (p), Point-Biserial Correlations(rpb), and Biserial Correlations (rbis), WithinSession 2 for Version B of the El Subtest byForm and Test Administration Medium ................. 73

viii -

1. WhNRODUCTION

As the United States Armed Services move toward the iEplemntation ofcomputer adaptive testing (CAT) with the Armed Services Vocational AptitudeBattery (ASVAR), several major issues must be addressed to ensure the continu-ity ad well-being of the military testing program. At the present time, thetransition from traditional paper-end-pencil testing to computerized testingis being envisioned as a gradual process; therefore, the equivalence of thetwo coexisting methods of testing is of conern, because the resulting scoreswill be required to function interchangeably (Green, Dock, kmphreys, Lin, &ieckase, 1982).

The first question of equivalence that m=st be addressed is that of theinfluence of the presentation modium itself on otherwise Identical tests.That Is, If the ASVAB in its present form is administered by computer to agroup of examinees, how would the resulting scores differ from those thatwould be obtained by testing in the paper-and-pencil medium? More specifical-ly, three questions can be raised:

1. Now would the scores differ by subtest?2. What night account for some of these differences?3. For sme types of Items, which of several modes of presentation on the

computer screen night serve to minimize differences?

The present effort was designed to address these questions in two ways.One way that the equivalence of two procedures can be determined is by a con-parison of the mean scores resulting from then to see if significant differ-ences can be detected. Toward this end, tb# experimental design provided sep-arate conditions for the study of alternative computer procedures againsttheir paper-and-pencil counterparts. In addition, controls were built intothe study to account for possible foe and version differences, as wel aspractice effects. Methods of comparing mean differences were then applied.

The second approach was that of employing correlational methods. Thesetechniques, which attempt to assess the degree of similarity rather then theextent of differences, included test-retest and internal consistency reliabil-ity comparisons across conditions, and exploratory and confirmatory factoranalytic studios to compare factor structures.

For the purposes of this effort, 6 of the 10 ASVAB subtosts were chosenfor adminstration. The selection was based on som characteristic or set ofcharacteristics of the subtests that might be expected to interact with themedium of presentation in either a positive or negative way, or that ightrequire some change or modification from the previous paper-and-pencil form ofpresentatione. These subtests can be divided into three groups on the basis ofthe criteria for selection: Paragraph Couprehension (PC), because of theproblems involved in presenting it by computer that result from its uniqueitem formt; Numerical Operations (NO) and Coding Speed (CS), due to thespeeded nature of these subtests; and Auto and Shop Information (AS), Mechani-cal Comprehension (MC), and rlectronics Information (1I)9 because of theiremphasis on graphical Image@, as well as items with no graphical content.

-1-|

11. METHOD

Test Battery

Tha study used subtests of Forus 11, 12, and 13 of the ASVAB (Preetwood,Vale, Hassey, & Welsh, 1985), The subtests used are shown in Table 1. Thesix subtests chosen for study were administered in two versions, A and B, foreach of the three forms for a total of six alternatives for each subtest: Il,11B, 12At 12B, 13A, and 13B. For the CS, AS, MC, and El subtests, the twoversions differ only in the ordering of the items; by contrast, the PC and NOsubtests use different items in each version. All subtests consist of differ-ent items In each form. Scoring was based on number-correct raw scores foreach subtesto

Table 1t ASVAB Subets Used with Corresponding Number

of It~w,, Operational Tim in Minutes, and Speededuess

Number TimSubtest of items allotted Speeded

Paragraph Comprehension (PC) 15 13 NoNumerical Operations (NO) 50 3 YesCoding Speed (CS) 84 7 YesAuto and Shop Information (AS) 25 11 NoMechanical Comprehension (MC) 25 19 NoElectronics Information (El) 20 9 NO

Experimental Design

Sublects and Data Collection

The initial sample consisted of 1,024 Air Force recruits distributed over30 independent groups with repeated measures for each examinee. The 30 groupswere subgroups of three general experimental groups: (a) Group 1, which wasfirst administered the ASVAB tests by computer and then by paper-and-pencil;(b) Group 2, which took the tests first by paper-and-pencil and then by com-puter; and (c) Group 3 which took the ASVAB tests twice by paper-and-pencil.

After removal of 27 examinees with incomplete data, the final data con-sisted of item responses and subtest scores for 997 Air Force recruits on sixsubtests of the ASVAB, each taken twice in alternate forms, for a total of 12subtest scores per examinee. Randomization was assured by preprinting 1,200cards with coded conditions and ordering then in 40 groups, each group con-taiing conditions 1 through 30 in sequence. Examinees were then assigned togroups by having them line up upon their arrival at the test center and hand-ing out the cards in order until all examinees were accounted for. In thisway, every 30th person was assigned to the same condition. All examinees waremale recruits (to avoid possible confounding of results due to gender differ-ences), and all were in their sixth day of basic military training at the timeof testing. Testing of examinees occurred at the Air Force Human ResourcesLaboratory test facility at Lackland AFB, Texas, over a 2-week period duringFebruary 1985.

-2-

Once conditions were aesigned, Group 3 (the group receiving paper-and-pencil tests In both sessions) was removed to a separate building for testingand s not elposed to the computerized test environment. Group I was broughtinto a room containing 30 carrels with Terk microcomputers 28 of which wereused for testing, hmineea were assigned randomly to computers. Due to com-puter memory limitations, each computer was programmed to administer only ver-sion A or version B tests. Within version, however, each m=chins containedall forms and conditions for all six subtests. The test each examinee re-ceived ws determLned by the condition code printed on his condition card,thich was entered into the computer before the start of testing. This randomassignment of aaminees to conditions bad to computers controlled for varia-tIons in screm resolution between computer monitor--a particularly importantconsideration for subtests containing graphical Images.

Before testing commenced, Group I received a standard set of instruc-tions, over earphones from trained test administrators, covering both thetests themselves and the operation of the computer; then, graphical demonstra-tions and user-paced exercises on computer keyboard use and the 10 keys re-quired for testing were administered. All examinees also received standardASY A test Instructions. Administrators were available throughout testing toanswer questions and to help with problems. Upon completion of teting, exam-inees were allowed to leave quietly. During the first session, Group 2 wasadministered equivalent paper-and-pencil tests in another room. Following abreak at the end of Session 1, Groups I and 2 switched places; i.e., Group 2was administered instructions and testing by computer while Group I was admin-istered paper-and-pencll tests.,

Tests and Conditions

PC subtest, This subtest poses two problems when converted from paper-and-p1enc to computer administration. First, sou of the paragraphs used inthe paper-and-pencil test are too long to fit on a cathode-ray-tube (CRT)screen at ome time Second, most of the reading comprehension paragraphs intho ASVAB tests are accompanied by multiple questions. consequently, In thecomputer presentation medium, three different methods of presenting thesekinds of items on a CRT (explained further on p. 4) were evaluated to deter-mine which gave results most similar to those of paper-end-pencil administra-tion.

O and CS subtests, In the case of these highly-speeded, low-difficultysubtests, item are typically presented In groups in the paper-and-pencil se-dium, with instructions to answer as many itms as possible within the timelimit. A similar approach was taken with the computer presentation of thesesubtests by presenting several items on the screen at a time. This conditionwas compared with a second condition that presented items on the screen one ata time, to determine which item presentation condition was more similar to thepaper-end-pencil adiministration,

AS. IC. and 91 subtests. These subtests consist of both standard mlti-ple-choice test items and multiple-choice Items that use graphical images todescribe pysical, mechanical, or electronic concepts or components aboutwhich the examinee is questioned. They were presented on the computer screenin a single computer-presentation mode very similar to the presentation oftheir paper-and-pencil counterparts, to determine if differences resulted from

-3-

WWI@* Rik

the translation of standard multiple-choice Items and graphical images fromthe paper-and-peic1l medium to a CRT.

Treatment Groups

The overall data collection plan, with the total numbers of examinees percondition (before eliminating those with Incomplete data), is shown In Figures1 through 3. Bach figure is divided Into sections representing the treatmentgroups to which the examinees were randomly assigned. A major advantage ofthis design was that it allowed the computer versus paper-and-pencil test ad-m itration variable to be examined both between aud within subjects for allsix subtests, thus allowing greater flexibility of analysis,

PC subtest. The data collection plan for the PC subtest is shown In FiS-ure 1. To overeom the problem of lengthy PC Items, a scrolling procedure wasdevised that enabled examinees to move forward and backward through the Itemtext a line at a time. Scrolling was activated by depressing a designatedforward or backward scrolling key which erased the text from the currentscreen (field) and replaced It with a different screen (field) of text. De-pending on the scrolling key chosen (forward or backward), examinees were ableto view now or previously presented material.

Three computer-admianstratLon Hode conditions (CRT-I to CRT-3) were in-cluded to assess the effectiveness of three possible solutions to this complexitem presentation problem. In the first Hode condition (CRT-i) the paragraphunder consideration could be scrolled on the screen while one question at atime appeared beneath It In a separate, nouscrollable field. Previous ques-tions about the paragraph were erased following an exainee's response andbefore the next question appeared, and examinees were unable to refer to pre-vious paragraphs or previous questions pertaining to the current paragraph.

The second Hode condition (CRT-2) presented the paragraph and all ques-tious related to It In a single scrollable field with no access to previousparagraphs or their questions. This condition allowed the examinee to use theentire screen to view the paragraph or single questions, and to move back andforth between the current paragraph and Its questions. The third Hode condi-tLou (CIT-3) allowed the current paragraph to be scrolled on the screen, andbeneath It questions could be scrolled separately. The displayed paragraphwould change automatically as the Ltems were scrolled to remain current withthe displayed Ltem, so that an examinee could scroll back to any previousparagraph and Its questLons. This condition also provided an answer-sheettype of display at the top of the screen which allowed examinees to monitortheir pro ress through the test, and pemitted them to go back and change an-soes to all previous PC items at any time throughout the test.

Esch presentation condition appeared twice in the first two experimentalgroups, once with Version A and alternatively with Version B, for a total ofsix conditions with 50 to 60 examinees per condition. Due to the varied ex-peirmntal conditions and limitations in sample size, it was necessary to Ua-It the administration of the PC subtest to Torm 11 of the ASVAB in Groups 1and 2. Group 3, the paper-aud-pencLl only group, was not restricted by thethree computer conditions; so the three forms of the PC subtest, Versions Aand St were administered Lnstead, for a total of six conditions with 50 to 60examinees per condition.

-4-

Group 1 (Nm338): Cosputer-P&PSession 1 Session 2

S, CRT1 IIA 1-59 PAP 11B 1-59

CRTI 115 1-56 PAP 11A 1-56

CT2 IIA 1-54 P&P 113 1-54

* CI2 1IS 1-56 P&PI I A -56

S 813 11A 1-57 PO 11B -57

S338 CT3 1IB 1-56 PAP 11A M-56

Group 2 (N-339): P&P-ComputrSession I Session 2

S3 9 PP IIA -56 CTI 113 1-56

UP lII? 111, 57 CRTlII1A N57

PA 0113 N-57 CET2 IIA N-57P P11A N-58 CR12 11B N-58

k 1r6? 113 1-57 CR12 ILA 1-57

0 P hlA 3-56 CRT3 111 5

867, 10 113 3-55 CR13 11A 11-55

Group 3 (N-347): PP-?&PSession 1 Session 2

S678 P&P IIA N-58 P&P 1IB N-58

P0 IB N-58 P&P IIA N-58

S P&P 12A N-58 P&P 12B 1-58

P &P 12 a58 P&P 12A N-58

P PAP 13A -158 P&P 133 N158

$10241 PAP 133 N-57 P&P 13A N-57

Figure 1. Data Collection Design for PC Subtesto

-5-

-~~~ '

NO and CS subtests. ligure 2 describes the data collection plan for theNO and CS subtestsa Here, two computer presentation Mode conditions were In-eluded (CiT-I and CRT-2) which, when combined with Versitons A and B of eachsubteat, provided four conditions per group with 80 to 85 examinees per condi-tion* In the CRT-i condition, the items appeared on the CRT one at a tine,and the exeaines were instructed to answer as meny of the items as possiblewithin the allotted time. In the CRT-2 condition, several items were dis-played at a time on the screen with the same Instructions; for NO, three itemsappeared on the screen at the sam time, and for CS, two blocks of seven itemswere displayed, Response times were recorded at both the single- and mlti-ple-ite level, Again, limitations in sample sise restricted the adinistra-

Group 1 (N-338): Coputer-lWSession I Session 2

S1 CRTI 12A 14-83 P&P 12B 14-83

M CRT112B 10-86 P&P 12A4 N-86

* CRT2 12A 1"-87 P&P 12B N-87

CRT2 12B N-82 P 12A N-82

Group 2 (14-339): P&P-ComputerSession I Session 2

S339 P&P 12A4-85 CRTI 12,B N-65

P P 1231 N-84 CRT1 12A N-64

S P&P 12A N-85 CRT2 121 N,85

;677 1P&P 12B N-85 CRT2 12A N-85

Group 3 (N-300): P&P-P&PSession I Session 2

$67 P&P IA 14,58 P&P 11B V-58

PiP lIB N58 P&P 11A N-58

PiP 12A Na8 PAP 12B N-58

P&P 129 0-58 P& 12A 1-58

S P&P 13A N-58 PP 13B N-58

510241 P&P 13B 1-57 P&P 13A N-57

FiSure 2. Data Collection Design for NO and CS Subtesta.

-6-

., " "*" " ""- . .f .;v:,: . ' r ,;,',' ,'-..._ . *.

'' W sflS ,S, W YI -~'r . u~ - - -qlrv-

tion of the NO and CS subtests to Form 12 in Groups I and 2. Group 3 receivedboth versions of eli three foats for a total of six conditions with 50 to 60examinees per condition.

AS, MC, and II subtests. The data collection plan for the AS$ NC, and Elsubtests Is shown In Figure 3. Unlike the other subtests, these consisted ofonly a single computer presentation condition that presented all items includ-Lng graphical Images. Ts permitted the administration of Forms 11, 12, and13 to all three groups, for a total of six conditions per group with 50 to 60examinees per condition. The graphical images for the computer-aduinisteredASVAB tests were developed by the four-step process of (a) digitizing, (b)edltlg, (c) moving, and (d) concatenation and Indexing. The digitizing pro-css was dome using a Kurta Series Two 12" x 17" digitizer pad.

The digitizer program that was written for this project (called DIGITI-ZER) allowed for digitizing to a half screen while scaling the image to makeit as large as possible. In this way, the limited 120 vertical dots by 320horizontal dots half-screen graphics window was used as completely as possi-ble. DIGITIZER allows points, lines, and connected lines ("connect the dots"or polygon mode) to be plotted, either entered individually or *streamed" to-gether.

After digitizing, all images were cleaned up and completed using thegraphics editor. The graphics editor used was a reworked version of the Terakgraphics editor GREDIT. It allows lines, points, circles, arcs, and arrowheads to be drawn or erased, or text to be entered or edited. It also allowsthe superimposing of one Image upon another. Ultimately, all images weresuperimposed in this fashion, with one image on the top of the screen and theother on the bottom.

Once the image was refined to the desired level, it was centered within ahalf-screen area and located in the proper half screen (top or bottom), usinga progrem called MOVER, which also allowed the duplication of one part of thescreen on another part of the screen. The graphical portions of the screenswere then concatenated and indexed into a random access file that was utilizedby the test administration program to retrieve graphical images associatedwith text segments.

Data Analysis

Combined Conditions

The experimental design plan provided for three test administrationgroups: Group 1, to be tested by computer followed by paper-and-pencil; Group2, tested by paper-and-pencL followed by computer; and Group 3, tested bypaper-and-pencil twice. Within each group the three forms and two versions ofeach subtest were administered, except for PC In the first two groups wherethe three screen Modes (CRT-1 to CRT-3) were substituted for three forms andall examinees received Form 11. This resulted in 18 experimental conditions(3 groups x 3 forms x 2 versions). In addition, for the computer administra-tion in Groups I and 2, two separate screen conditions were administered forthe NO and CS subtests, thus adding an additional 12 conditions, yielding atotal of 30 experimental conditions.

-7-

Group 1 (1N-338): CoMurter-P&PSession 1 Session 2

S1 CRT IIA 1-59 P&P 11B N-59

S CRT 1iB N-56 pp IIA N-560i

CRT 12A N-54 P&P 12B 1-54

* CIT 125 1-56 P&P 12A 1-56

* CIT 13A N-57 P&P 13D 1-57

S338 CRT 1315 1-56 P&P 13A 10516

Group 2 (1-339): P&P-ComputerSession I Session 2

S339 P&P IIA N-56 CRT 11B 1-56

* P&P IIB N-57 CRT ILA N-57

* P&P 12A N-58 CRT 12B V-58

* P&P 12B N-57 CRT 12A N-57

. P&P 13A N-56 CRT 13B N-56

$677 P&P 13B N-55 CRT 13A N-55

Group 3 (N-347): P&P-P&PSession 1 Session 2

S678 P&P 11A N-58 P&P 11B N-58

. P&P I1B N-58 P&P IIA N-58

* P&P 12A N-58 P&P 125 N-58

* P&P 12B N-58 P&P 12A N-58

* P&P 13A N-58 P&P 13B N-58

;1024 P&P 13 N-57 P&P 13A -57

Figure 3. Data Collection Design for AS, MC, and EI Subtests.

-8-

Analysis of Variance (AMOVA)

The wodel chosen for the ANCA uas a repeated measures multivarLate (MA-NOVA) design using total subtest scores as dependent variables, with uti-variate follow-up ANOTAs for all significant effects. A multivariate analysiswas selected because it offers a substantial advantage with respect to power,while controlling the Type I error rate over all subtests simultaneously atthe nominal alpha level (orrisou, 1976).

A fundamental assumption underlying any NANOVA analysis is that of bomo-genaity of the covarlance matrices within each cell. fowever, the HAiOVA nod-el is widely recognized to be robust to violation of this assumption, espe-cially for largo sample ams. Furthermore, .. ltvarlate tests of homogeneityare very powerful, given large samples, oftan leading to a rejection of thehypothesis of equal covariance matrices on the basis of minor differences(Cooley & Lohnes, 1971). Univariate ANOVA models in a similar way assume ho-mogneity of variances, and in a like mannr, are largely robust to violationsof this assumption. This situation holds at least for between-subjects facto-rial models, but for repeated measures designs violations of the equal vari-ance assumption have been shown to lead to a greatly increased probability ofType I error (O'Brien & Kaiser, 1985). Unfortunately, there Is no known testat present for the homogeneity of repeated measures variances. Therefore, thebest defense against an increased incidence of Type I error is to Interpretsignificant outcomes accordingly, when the variances over sessions appear tobe heterogeneous.

MANOVA Is based upon a comparison of the latent structures of the be-tween-groups sum of squares and cross-products (SSCP) matrix, V, and thewithin-groups SSQ' matrix of subtest scores, ;, for a given effect. This tsaccomplished through the decomposition of the product mtrix, i_-I, and theanalysis of the resulting latent roots. Although there is no uniformly mostpowerful statistic for conducting this analysis, all of the widely acceptedstatistics are some function of the latent roots of If 1. and the choice of anappropriate statistic is dependent upon the rank of the matrix, or the numberof significant roots obtained.

For the situation where the rank of the matrix is 1, the most powerful

test of significance has been shown to be Roy's Largest Root test; whereas,when the rank is greater than 1, the Pillai-Bartlett Trace V test is most pow-erful (Olson, 1976). The Pillai-Bartlett offers the additional advantage ofbeing the most robust to the violation of the assumptions of the MMAZOVA model,whereas Roy's test is extremely sensitive to violation of the homogeneity as-sumption.

Tor the present analyses, four maltivariate tests of significance wereapplieds Roy's Largest Root, Hotelling's T, Wilks' Lambda, and the Pillai-Bartlett Trace V test. For every experimental outcome, the results of allfour tests were in complete agreement. Therefore, the Pillai-Bartlett Trace Vstatistic, which has been recommended for general use (Olson, 1976) is report-ed for the MANOVA analyses.

The Pillai-Bartlett Trace V test isS

V- EC /(l+Ci) (1)

-9-

- 4.. .

where Ci is the ith latent root of the metrix BI- ' and S is its rank. Pillaiderived an approximation to the F distribution (orrison, 1976) for V as

F =(dfe - p + S)V (2)b(S - V)

with Sb and S(dfe - p + a) degrees of freedom, where dfe is the degrees offreedom within groups, p Is the number of dependent variables, S is the rankof 15- 1. and b is the larger of p and the degrees of freedom between groups.

The follow-up analysis for those MAMOVA outcomes found to be significantwas based on univariate ABOVA F tests to identify the specific subtests forwhich the effects were significant. Once a MWOVA result is found signift-cant, however, all follow-up statistical tests are unconstrained in tets ofType I error. That Is, the Type I error rate applies at the individual testlevel, and over numerous tests, the rate is compounded. Caution mast be ex-ercised in such a case In interpreting significant outcomes. Furthermore,standard forms of post hoc statistical tests, while varying in power and flex-ibility, are all Increasingly sensitive with Increasing sample sizes, and forthe present study, relatively large samples were required to adequately pro-vide for the correlatLonal analyses, such as factor analysis. Such large sam-ples serve to increase the sensitivity to post hoc statistical tests to thepoint where meaningless differences become prominent and obscure the importantgroup differences under experimental manipulation.

Paper-and-pencil baseline analysis. The purpose of the first analysiswas to emIne the effec's across all six subtests of (a) Form-the three testforms--ASVAD Forms 11, 12, and 13; (b) Version-the two Versions (A and B)within each form; and (b) Seasion-the repeated measurement of paper-and-pencil tests. This analysis was confined to Group 3 (N - 333), which func-tioned as a baseline comparison group, being tested entirely in the paper-and-pencil medium. Figure 4 shows the experimental design plan for Group 3.

The analysis consisted of a repeated measures NANOVA with univariate ANO-VA follow-up tests for all significant effects across session within subjects,and test form and test version between subjects. The dependent variables werethe total scores for each of the six subtests taken by each examinee (PC, NO,CS, AS, MC, and EI).

Parasraph Comprehnsion. The second analysis was designed to determineif differences were observed in man test scores among the three modes of cor-puter presentation of the PC subtest between subjects, and between the comput-er and paper-end-pencil presentation media both between and within subjects.Test form differences were also examined to eliminate confounding with theother effects.

Figure 5 shows the experimental design plan for the PC analysis. Allexaminees wore tested using Form 11 of the PC subtest to allow sufficient sam-ple size for a fully crossed design. Group 1, consisting of 332 examinees,was subdivided into three groups of approximetely equal size. Each subgroupwas administered one test version in one mode of the computer medium in thefirst testing session, followed by paper-and-pencil testing with the alternateversion In the second session; subgroups of between 50 and 60 examinees withineach mode subgroup were admnistered Versions A and B of ASVAB Form 11 PC in

-10-

.. _ • ..-;4

Testversionpresen- Session 1 Session 2tation Exam- Form & Form &order inees Medium version Subtest Medium version Subtest

665 11A 11B PC, NO, CS,to N-56 ASO MC El 1-56 AS, MC, El720

721 12A PC, N0, CS, 12B PC, N0, CS,1 to P&P N-56 AS, MC, E P&P N-56 AS, MC, EI

776

777 13A PC, NO, CS, 13B PC, NO, CS,to N-57 AS, MC, El 1-57 AS, MC, El833

834 1iB PC, NO, CS, 11A PC, NO, CS,to N-56 AS, MC, El -56 AS, MC, El889

890 12B PC, No, CS, 12A PC, NO, CS,2 to P&P 1-52 AS, MC, El P&P N-52 AS, MC, El

941

942 13B PC, NO, CS, 13B PC, NO, CS,to N-56 AS, MC, El N-56 AS, MC, El997

Figure 4. Experimental Design Plan for Paper-end-Pencil Baseline Analysis.

counterbalanced order. Group 2 (N - 332) was tested with the presentationmedia in reverse order, i.e., paper-and-pencil followed by computer presenta-tion.

The method of analysis was a repeated measures MANOVA with the PC subtestscore as the sole dependent variable. The effects that were examined as be-tween-subjects variables for the PC analysis included three modes of computerpresentation (Mode), computer versus paper-and-pencil presentation (Medium),and test version (Version). The effect examined as a within-subjects variablewsa the computer versus paper-and-pencil presentation medium across sessions(Session).

Speeded tests. The purpose of the third analysis was to determine ifdifferences existed between mean scores resulting from the two modes of com-puter presentation of the NO and CS subtests, and between the computer andpaper-and-pencil presentation media both between and within subjects. Testversion differences were examined to determine if they interacted with theseprimary effects.

Figure 6 summarizes the experimental design plan for these analyses. All

S- 11 -

.I .::.,:. .,:

Mediupreser Session I Session 2tation Exam- Form & Form &order ine. Mode version Subtest Mode version Subtest

to IlA PC iB PC58 U-58 N-58

CRT-1 P&P59to 1iB PC IIA PC

114 M-56 V-56

115to IIA PC 11B PC168 1-54 N-54

CRT-2 P&P169to 113 PC 1A PC223 U-55 N-55

224to 11A PC 11B PC278 1-55 N-55

CRT-3 P&P279to 1IB PC ilA PC332 V-54 -54

333to IIA PC 11B PC387 1-55 N-55

------- P&P CRT-I388to 1iB PC 11A PC441 N4-54 N4-54

4"2to 11A PC 11B PC499 N-58 N=58

2 P&P CRT-2500to 11B PC 11A PC553 4-54 N-54

554to IlA PC lib PC609 N-56 N-56

P&P 6.CRT-3610to 11B PC 11A PC664 N-55 N-55

Figure 5. Experimental Design Plan for Analysis of

Paragraph Comprehension Subtest.

-12-

Mediumpresen- Session 1 Session 2tation Form & Form &order N Mode version Subtest Mode version Subtest

* 1

to 12A N0O CS 12B N0, CS83 N-83

CRT-I P&P84to 12B N0, CS 12A No, CS168 N-85 N-85

169to 12A NO, CS 12B NO, CS252 1,-84 P-84---"---CRT-2 - ~ - - P&P" •

253to 12B NO, CS 12A NO, CS332 N-80 N-80

333to 12A NO, CS 12B NO, CS417 N-85 1--65

- P&P ' CIT-I418to 12B NO, CS 12A NO, CS500 N-83 N-83

501to 12A N0, CS 12B N0, CS584 V-84 N-84

P&P - CRT-2585to 12B NO, CS 12A NO, CS664 N-80 N-80

Figure 6. Experimental Design Plan for Analysis of Speeded Subtests.

testing was limited to Form 12 to allow sufficient sample size for a fullycrossed design. Group 1, with 332 examinees, was subdivided randomly into twosubgroups of approximately equal size. Each subgroup was administered onetest in one of the two comuter odes (single-item or multiple-item screen) inthe first testing session, followed by paper-and-pencil testing with the al-ternate version of Form 12 in the second session. Group 2, also with 332 ex-ainees, was tested in the reverse order, with paper-and-pencil testing in thefirst session and computer testing in the second session. Within each of theNode subgroups of Group 1 and Group 2, half of the examinees were assigned toeither Version A or Version B of ASVAB Form 12 in random order to test theVersion effect. This yielded a total of eight experimental groups of 80 to 85examinees each. (Due to sample size limitations, within-subject test versionand medium effects were necessarily combined into a single experimental condi-tion.)

-13-

The method of analysis us a repeated measures NOVA with the NO and CSsubtest scores as the dependent variables, with univariate ANOVA follow-uptests for all significant tests. The between-subjects variables were testversion (Version), computer presentation mode (Node), and computer versus pa-per-end-pencil presentation medium (Medium), and the within-subjects variablewas the computer versus paper-and-pencil medium (Session) effect.

Grpic81 subtests. The fourth analysis was designed to examine the dif-ferences between th acomputer and paper-and-pencil presentation media for theAS, NC, end El subtests. These differences were analyzed for Forms 11, 12,and 13 of each subtest and by version. Madium and Version effects were exam-ined In a completely unconfounded design. For the graphical subtests, a par-ticular question of Interest Is what differences in performance, if any, re-sult from the presentation of graphical Images on a computer screen versus theprinted page. The analysis of the NC subtest addresses this question direct-ly, since of the 25 items comprising the test, 23 for Form 11, 21 for Form 12,and 22 for Form 13 contain graphical Images. In addition, the NC subtests, aswith all others, were randomly distributed over 28 computers to control forindividual screen resolution differences.

Figure 7 shows the experimental design plan for these tests. The 332exminees in Group 1 were randomly assigned to three subgroups. Each subgroupwas administered one form of each subtest, one version in the computer mediumin Session 1 and the alternate version in the paper-and-pencil edium in Ses-sion 2. The second group of 332 examinees was tested in a similar manner butwith the order of presentation medium reversed.

The analysis was a repeated measures NANOVA with total subtest scores asdependent variables. The effects that were examined as between-subjects vari-ables included test fors, computer versus paper-end-pencil presentation medi-um, and test version. The effect examined as a within-subjects variable wasthe computer versus paper-and-pencil presentation medium.

eliability Analyses

Experimental group and session abbreviations are presented in Figure 8.The first character stands for medium of administration: C for computer, P forpaper-and-pencil administration; the second character denotes the experimentalgroup (1, 2, or 3); the third digit indicates testing session 1 or 2. Due tothe nature of the data collected, two types of reliability comparisons weremade.

Test-retest. The first comparison used a test-retest correlation designand wee computed for all the ASVAB subtests. The subtests CS, AS, NC, and Elcontain the se items in both Version A and Version B, providing true test-retest date. However, subtests PC and NO use different items in Versions Aand B, thus changing the comparison slightly to one of an alternate-forms cor-relation, although the versions are nominally parallel. The comparison ofinterest was that between Groups 1 and 2 (CI then P12, and P21 then C22) ver-sus the paper-and-pencil only (P31 then P32) group. n these comparisons theGroup 3 reliabilities served as a baseline against which the other experimen-tal conditions could be judged. The reliabilities were compared by t-testsfor each contrast after performing Fisher's (1921) r to z transformation on

- 14 -

4

,, ;;- .. . . , , . . -.. . q. . ,e . , . , ., .. , .

Mediumepreaon- Session 1 Sessio_2tation Form & Form &order N Medium version Subtest Medium version Subtest

Ito IlA AS, NC, El liB AS, NC, El58 N-58 N-58

59to lB ASO NC,E I IL& AS, MCOE1114 1-56 N-56

115to 12A ASO C,EI 12B AS, MC, El168 V-54 N-54

1 CET -P&P169to 12B AS, MC,EI 12A AS, MC, EI223 N-55 N-55

224to 13A AS, NC, El 13B AS, NC, El278 N-55 N-55

279to 135 AS, NC, ZI 13A AS, MC, El332 N-54 N-54i333to N-55 AS, NC, El N-55 AS, NC, El387

388to lib ASO NCG, 21 11A AS, NC, El

441 N-54 N-54

442to 12A ASIC, EI 12B AS, C, El499 N-58 N-58

2 P p CRT500

to 12B AS, NC, El 12A AS, MC, El553 N-54 N-54

554to 13A AS, NC, El 13B AS, NC, El609 N-56 N-56

610to 13B AS, ICO El 13A AS, MC, ZI664 N-55 N-55

Figure 7. Experimental Design Plan for Analysis of Graphical Subtests.

- 15 -

Test Admnistration

Session 1 Session 2

1 Ci P12

Naewimental 2 P21 C22group

3 P31 P32

plaure S. Overview of Test Conditions by Group and Session

f Computer (CRT) and Paper-and-Pencil (P&P) Groups.

the test-retest correlations.

internal consistency. The final reliability comparisons were made usingen internal consistency measure of reliability, Cronbech's (1951) coefficientalpha, which reduced to the Kuder-Rlchardson (1937) Formula 20 (KR-20) due tothe dichotomous item responses. Because internal consistency reliability es-timsates are not appropriate for speeded tests, only subtests PC, AS, NC, and9I ware considered in these couparisons.

The data collection plan provided the means to perform both matchedgroups (within-group across sessions) and multiple independent groups (across-groups within session) KR-20 comparisons. In order to perform the completematched group comparison across all subtests by Form and Version for each ex-periental group, 72 tests of the type proposed by Feldt (1980) would be nec-essary. A simple independent groups comparison for each subtest by form andversion within each session would require that 48 tests using Haketian andMalen's (1976) method be made. The sheer number of statistical tests neces-sary to adequately compare the KR-20 reliabilities suggested that the findingsobtained would be extreely difficult to Interpret and confounded by Type 1errer

As an alternative, the KR-20 estimates were averaged separately for eachsubtest within each group and session (ClI, P12, P21, C229 P31, nd P32) endexamined for differences. Collapsing the data in this wsy was justified be-cause the ASVAB form and versions are designed to yield similar measurementcharacteristics; thus, any Important reliability differences across presenta-tion medium or session would be consistent across all versions and forms.

Studying the reliabilities In this manner allowed the discovery of possi-ble trends or differences in KR-20s both within groups across sessions, andacross groups within sessions. As in the test-retest analysis, the paper-and-pencil-only groups (Group 3, P31 and P32) were used as a baseline againstwhich the man KR-20s from the other groups were judged.

- 16 -

Analysis of Structure

Detween-subtest structure. In order to exsmine the possible effect ofdifferent Itm presentation methods on a battery of tests, an analysis wasperformed on the similarity of factor and covariance structures between sub-tests at the subtest score level, under both administration medium conditions.The data were correlation and covariance matrices for between-subtest scorescomputed from examinees' total scores for each of the six subtests.

One method used to determine the equivalence of test presentation mediawas the examination of the factor structure of the between-subtest correla-tions across conditions. For this comparison, unrotated principal factoranalysis vas used as the factoring method. Although sets of common factorloading@ are usually not unique and many different sets of loading values candefine a solution, principal factor analysis yields a factor solution thatdefines both a unique common factor space and a unique set of factor loadings(Berman, 1976). Therefore, factor loadings from the unrotated principal fac-tor analysis solution (defined by extraction of maximum variance from the cor-relation matrices with squared multiple correlations on the diagonals) of thesubtests were directly compared for equivalence across media of presentation.

The equality of between-subtest covariance matrices and factor structureswas examined using the maximm likelihood methods of LISREL VI (Joreskog &S~rbom, 1984). There are three main indices of 2 general model fit yielded byLISREL. The first is the overall chi-square (X ). The test made by chi-square judges the fit of the constrained model against the alternative hypoth-esis that2the estimated covariance matrix is unconstrained. Degrees of free-don for X are calculated by

df - J k(k + 1)- t (3)

where k is the number of observed variables, and t is the number of free pa-rameters astimated. Jireskog and Sorbom (1984) suggest that X2 be used as anindex of the degree of odel fit and not strictly as a test statistic.

The second index of overall model fit is the goodness-of-fit index (Gi).G7i is defined by

tr(d-s - I) 2Sa I - tr(S) 2 (4)

where tr Is the trace of the indicated matrices, j is the fitted matrix, isthe observed covariance matrix, and I is an identity matrix. The range of GFIis between sero and 1,0, and it is a measure of the amount of covariance andvariance accounted for by the model.

The last model fit index Is the root mean square residual (R14R) definedas

k 12 ill2ji- l(a - &ij)2/k(k + 1] (5)

- 17 -

where k ts the number of observed variables in the model, s:j is the observedvariance or covariance, and a is the estimated covariance or variance con-ponento The interpretation o RlUR depends on the relative sizes of the cover-lances and variances of the observed variables, For example, a large valuefor 3M as compared to the average observed covariance or variance would be anIndication of a model that did not fit the data very val.

Joreskog (1971) outlined a method by which the factor structures of testbatteries could be tested for equality across different groups of examinees.Models assuming different levels of equalities were tested sequentially untilthe level of structured equality appropriate for the betwee-subtest data wasfound. The first model tested, the most strict test of covariance equality,assumed that, within each Session (1 and 2), the between-subtest scores cover-ancs matrices for the paper-and-pencil adminlstrations were equal for eachgroup (Session 1: E - r__ and Session 2: E M Eid) This test of the modelthat the covariance structures within the paper-ana-pencLl administrationswere equal across groups provided a baseline against which the covariancestructures of computer-adminLstered tests could be judged. Next, the sametest wa made, again by session, with the addition of the covariances for thecomputer administrations -G1 = -XG2 - -G3)' These tests are generalizationsof the Bartlett test for hoboneLty of variances (Morrison, 1976) and are.susceptible to having a high degree of power when sample sizes are large,causing rejection of the null hypothesis when minor differences are present(Cooley & Lohnes, 1971).

The next model tested, holding a much less strong equality, was thatwithin each Session (1 and 2) the between-subtest scores yielded the same fac-tor structure In each group.

Within-subtest structure. The analysis of the similarity of factorstructures within subtests was also performad for subtests AS, NC, and El.These were the only non-speeded subtests containing the same Itens in bothversions (A and B). By combining examinees tested on either version, the sam-ple size requirements of factor analysis were nearly met, with approximately100 persons per subtest available across Forms (11, 12, and 13) within eachcell (Cl, P12, P21, C22, P31, P32) of the experimental design.

The factor structures were compared through uniterated principal factoranalysis, as described previously, of the item intercorrelation matrices com-posed of phi correlations. Five unrotated principal factors were extractedfrom each correlation matrix. These comparisons were made across both sub-jects and madia using the first session baseline paper-and-pencil group forall comparisons (C11 and C22 versus P31), The subtest factor structures fromboth computer-administered conditions were compared against the subtest factorstructures of the Session I paper-and-pencil-only group (Group 3) because It

yielded an adequate representation of the factor structures of the subtests inall the paper-and-pencLl groups.

Item Analysis

Due to limitations in sample size (between 50 and 75 per cell), onlyclassical test theory Item parameters (point-biserial, biserial, and propor-tion correct) were calculated and analyzed for the neer-speeded subteats. The

sample nine demands of item response theory item parameterization using LOGIST

- 8

(Wood, Wingerky, & Lord, 1976)-500 to 1,000 per cell (Wood & Lord, 1976)-greatly exceed the number of examinees in this study.

The subtests under study were chosen because they are the most problema-tic ASVAB subtesta for computer admdinstrat ion. Of particular concern was thepossible difference in measurement properties of items containing graphicalimages (n subtests AS, NC, and ZZ) due to problem in translating the imagefrom paper to a computer CRT screen. This question of masureuent equivalencewas neatly addressed at the subtest level because of the distribution ofgraphical content items emong the three tests. The KC subtest contains be-twoon 21 to 23 graphical items (23 in Form 11, 21 in Form 12, 22 in For 13)out of the 25 total, whereas subtest I has only two or three out of 20 items.Therefore, any computer administration effect on graphical item would impacton subtest NC in its entirety and cause man differences across media, whichwould be found through the NANOVA analysis. Subtest 11, having almost nographical content items, was used to index possible graphical item administra-tion differences since it has similar test objectives, being a non-speeded,technical Information test. Subtest AS was not used in this analysis becauseit contains several graphical items (5 in Form 13, and 6 in Forms 11 and 12),disqualifying it as either a high or low graphical content subtest.

Administration mdium differences for graphical content item were alsocompared at the factor structure level. The differences in the unrotatedprincipal factor solutions for subtest MC across media of presentation werecompared with the factor structure differences for subtest 21 across the sameconditions. Any large discrepancy between MC and EZ factor structure differ-enes would provide evidence that computer administration changes the Lnterre-lationshLps among graphical items, Implying that the translation of imagesfrom paper to CRT screen differentially affects items with graphical content.

MII. RESULTS

Analysis of Variance

Paper-and-Pencil Baseline Analysis

Table 2 shows the outcome of the IAWOVA for the paper-and-pencil baselinegroup, both within and across sessions, for all subtests. Significance (p <.05) is indicated for one within-sessLon factor, test form (Form), the repeiat-ed measures factor (Session), and for the Interaction of test version (Ver-sion) with Form and Session. All other factors and interactions were not IS-nificant; therefore, no further analyses were necessary.

The results of the univariate follow-up analyses by subtest for the sig-nificant affects Identified in the multivariate baseline analysis are given inTable 3. For the Form factor, significance was found for the AS subtest.Significant Session effects are shown for the NO, CS, and EI subtests, and PCshows a significant outcome for the Version by Form by Session interaction.

The next step in the analysis was to examine the difference in the mansfor each level of each effect. Tables 4 through 9 show the means for all sub-tests by condition. A significant difference in the mean scores by Form wasfound for the AS subtest; the largest of these was between Form 11 and Form

- 19 -

* ** P ' ' P P . ' *

Table 2. Pillai-Bartlett Trace Values, Degrees of Freedom,Approxinate F Ratios, and Estimsted Significance Levels (p)

for the Baseline tANOVA (N - 333)

Degrees of freedomSource of variation Trace V Between Within F p<

Between GroupsForm .10 12 646 2.96 .001**Version .01 6 322 .30 .935Fore x Version .04 12 646 1.11 .353

Within GroupsSession .35 6 322 29.21 .001**Fore x Session .05 12 646 1.45 .138Version x Session .02 6 322 .93 .473Version x Form x Session .08 12 646 2.30 .007**

**Statistically significant at p < .01.

Table 3. Results of Univariate ANOVA. for Significant EffectsIdentified in the Baseline MAOVA (N - 333)

Effect, degreesof freedom, Mean squaresand subtest Between Within F p<

Form (2,327)PC 6.82 8.96 o76 .468no 160.97 97o17 lo66 .192CS 317.30 303.33 1o05 .353AS 129.51 39.02 3.32 .037*HC 10.74 3305 o33 o723El 20.22 23.13 .87 .418

Session (1,327)PC .29 3o01 .10 .755No 312,22 14.92 20o93 0001**CS 5506.34 34010 161.47 .001**AS .38 1.18 .33 .568MC 5.23 207 2.53 o113El 12.99 1o85 7.02 .008**

Version x Form x Session (2,327)PC 32.99 3o01 10.94 .001**No 26.34 14.92 lo77 o173CS 14.35 34.10 42 .657AS .37 1.18 .32 .728MC 1.50 207 .73 484E1 1.13 1.85 .61 .542

*Statistically significant at p < .05.**Statistically significant at p < .01.

- 20

g I

1% 40f% %D % I.0 4 LPc a 0 oP

0 Uin 4fA% mw% me%, 0

04 ( *4 94 4 se. . ~ %* %***r.M %P

0 444t4~C4 44~C4; ('C4 (.4 C4 4.0

a 4" -4 14 "C DI 44 t-% DV -Ii is.1 05 f4'" CMM04P s 10%DW% 40 'O. LN I

"4 m 4 i 4 4

0i 0 v I."6 0 ~ o~

a (4

1 9I4 D4 oP %ais .1ILS. E 140do1111 c P

-4l 44P- 4 - t . Wa~bi Ii 0

0;" 0; 0; ; A0: C; r A C4g 4 4 4 444 4 A C4

:i im .4D 00-00-0 D 00 0000. g"4 V4 9-4 3@( 0 O 9-4 P- 4 C4P 41fC 4(10 - 0

me 00

o (21

%N . T -t- -v.; I--3

14 14m. wlo 60 w m 04 4 -040 ff. C

4 0% *eo *e4000ea 0 ak. W- ?4. 0ec oo4 nC% n G

a 0a C14 0

o' a , "4O"' -Z 0-4 on0as 04tA INS a. 4444 l C40 40

46&

140wt v4 -q inP %

cq14 r P% *4 4 0% at -- -4 0 g-aCA 0

0 v 0 : : Jzr

4'4

ft's As%% .0 t4 C 04 @v%40

AV "V

_4 C4

0 0 22

04 woW49 W4'o V4 m 4 1 4 W4

A~ 4. 0 .

UU 'ka in4040 40 cc 40 0 in LN i kbl% %

84%* .04 8q -0ff d 4 oc" w o4 m0%

Ii 0(4 0N -4..Sui4 C40 A 44 4 so4.

go0 A4 (5 .Cn40ae6n 4C,

lie N C~O'.4 c e 'aW'0(4f4 V p W460(4 P4 9.046 4we jj U 4 4woW4"

t4 w 4 .

4.0 .N~f% 4 446VI23

'In

GO 40~ e4 ccd * a or od fbG~* E4 50 0 0 0 00 00 00

mO m. C" ccP.A0.%I I %f% 4-%

0 3O ~ ~ 0 00U C4r%.rC P a0 0 000 V1 N 00400 40 0

C000

ca at% na c k

P4 t4m h D404400G

'44

C4 we"4"4 % 0D @ 4 . .. blU 04C4 %in-*%coi 4f 41% * 6 * .d. i A.

6 1 0 C4c nI 4. d Db DI oC

44a 4glv* . 4 44 M 04 4 A.O Ofl@ am 44 am44 I

.0 0 0- 0o 04 0n 0ne 4C4C

10 o 4- Q CM44I

-24-

*r*l.. r.*I~f rm ru

Si P4

46 u ln SDV 0fM 4 oC "0 -C4.f%% 4fla P% .%IPr'4r0 er~4'

C413 1 00 0 0 0 g * 0

ISO'4 '4

@4; 6 * %Nai o0C C % 4 4 A' 4C%

06 9. s

Ne NOa

*u a

a Icoon vu-ill 0

0. $s4 M N4CN

all P-4 V4 04" -4W-4 4 V"-4" .

9040 ON cn *I4"1 I .ur-...Eo~we

*0 C~gS25

P0 I%%flu in11 % % - - 0

43 r4 U9*OO4ho am En 0 D'r0%r o 0 0 0 %D-.

4b er'- P%~~ca t-4u inft D% ... i

4I -&

0 w ~ ui.. c" C44O"

u to

0 C4%D 40 V4 rft C4 4 In o - CD 0 % V4 N L % "

0

U34

M M 0 % X r- ' a ob% o0 00 co'p% a n

ad 0 cl I-.

'M4 4

-.c h 0% D %D -o -. %a N - - Ol- N - - -D

* - - - -l -e . -oP - 0 %D -1 - %C " -- I-

C4C 4C4 C4 CC4 4i A;A4 4C

-C.44 - - - go --9. 4- 4 . 4C

-26 -

12, with a difference of 1.41, suggesting that no practical or meaningful sig-nificance can be attached to this outcome. Therefore, a conclusion of no im-portant Fore effect was drawn for the paper-and-pencil baseline analysis.

For the Session effect, the NO subtest showed an increase in mean scoreof 1.37 from Session 1 to Session 2, CS had a gain of 5.75, and El differed by.28. Therefore, a very slight NO increase and a somewhat larger CS increasewere Implied by these statistically significant mean differences.

The PC subtest analysis indicated a significant Version by Fore by Ses-sion Interaction. However, for this subtest, the range in mean scores fromthe lowest to the highest over all 12 testing conditions was only 1.30. Be-cause of the low magnLtude of this man difference, this interaction was In-terpreted as a psychometrically nonmeaningful difference.

Computer versus Paper-and-Pencil

PC subtest. The results of the MANOVA for the PC subtest for the comput-er and paper-and-pencil groups are reported in Table 10 (means and standarddeviations by condition are In Table 4). The statistically significant be-tween-subjects effects include the node of computer administration (Node) andcomputer versus paper-end-pencil administration (Medium). No significant be-tween-subjects interactions were revealed. The within-subjects repeated mea-sures factor (Session) was found to be significant, as well as the Medium bySession, Version by Mode by Session, Version by Medium by Session, and Mode byMedium by Session interactions.

The follow-up analysis for the PC subtest began with the Mode effect.For the three computer screen Modes, the mean scores were 10.32 for CRT-I,8.46 for CRT-2, and 9.99 for CRT-3; the corresponding mean differences ware1.86 raw-score points between Nodes CRT-i and CRT-2, .33 between 1 and 3, and1.53 between 2 and 3. This indicates that screen condition 2 was differentfrom 1 and 3, whereas conditions 1 and 3 were not significantly different fromeach other.

For the computer versus paper-and-pencil Medium effect, the mean differ-ence within Session between the paper-and-pencil group (mean - 12.19) and thecomputer group (mean - 9.59) was 2.60 points. For the Session effect, thefirst and second session mean difference was .82 points. Upon examination ofthe Medium by Session interaction, it was found that the group that took thecomputer test in session 2 scored 2.15 points higher (mean - 11.74) than thegroup that took It during the first session (mean - 9.59). For the paper-and-pencil tests, however, the second session mean score (mean - 11.67) was only.52 lower than the first session scores (mean - 12.19), a finding of littlepractical Importance.

The Medium, Session, and Medium by Session effects were reevaluated withthe low-scoring second presentation Mode group removed. This reduced the Me-dium mean difference to 2.03 points, and increased the Session difference forthe computer groups slightly to 2.43 points.

Of the three-way interactions of Version by Mode by Session, Version byMedium by Session, and Mode by Medium by Session, all revealed only minor meandifferences of less than 1 point across conditions, but the Mode by Session

- 27 -

Table 10. Results of the F-tests of Significance from theMANOVA for the Computer Versus Paper-and-Pencil Administration

PC Subtest Conditions (N - 664)

Degrees of MeanSource of variation freedom square F p<

Betwoen SubjectsVersion 1 .01 .00 .979Node 2 218.66 21.31 .001**Medium 1 590.51 57.56 .001**Version x Node 2 .66 .06 .937

Version x Medium 1 .57 .06 .814Mode x Medium 2 15.77 1.54 .216Version x Mode x Medium 2 21.69 2.11 .122Within Cells 652 10.26

Within SubjectsSession 1 221.47 65.88 .001**Version x Session 1 .20 .06 .810Mode x Session 2 2.47 .74 .480Medium x Session 1 531.81 158.21 .001**Version x Mods x Session 2 15.41 4.58 .011*Version x Medium x Session 1 14.03 4.17 .041*Mode x Medium x Session 2 90.48 26.92 .001**Version x Mode x Medium x Session 2 4.88 1.4 .235Within Cells 652 3.36

*Statistically significant at p < .05.**Statistically significant at p < .01.

interaction was a meaningful one in that Mode differences existed only withinone of the two Medium conditions.

Speeded subtests. The results of the MANOVA for the speeded NO and CSsubtests are shown in Table 11 (mean scores by condition are in Table 5 for NOand Table 6 for CS). For the between-subjects effects, Medium, Mode, and theMedium by ode interaction were statistically significant. Within subjects,Session, Version by Session, Medium by Session, and Medium by Node by Sessioneffects were found to be statistically significant.

Table 12 contains the results of the analyses of statistically signifi-cant effects by subtest. Medium differences were indicated for the NO sub-test, while Mode differences were observed for both NO and CS. The between-subjects Medium by Node interaction was found significant for the CS subtest.Within subjects, the Session factor was significant for both NO and CS, andall of the interactions tested were significant for both subtests except forVersion by Session, which was significant for NO but decidedly nonsignificantfor CS.

Examination of the mean scores for the Medium effects shows that the pa-per-and-pencil group (men NO - 37.79) obtained higher scores than the comput-er group (mean NO - 3112) in Session 1 by 6.67 points for the NO 8ubtest, and

8,22 (mean CS scores of 60.96 and 52.74, respectively) for CS; both differ-

-28-

,.J

Table 11, Pillai-Bartlett Trace V Values with 2 and 655Dagrees of Freedom, Approximate F Ratios, and Estimated

Significance Levels (p) from the MANOVA for the ComputerVersus Paper-and-Pencil Administration Conditions for

the 10 and CS Subtests (N - 664)

Source of variation Trace V F p<

Betwmen GroupsVersion 100 1.11 .331Medium .01 3.29 .038*Mods .23 96.09 .001**Version x Medium 100 .82 .439Version x Mods 100 111 .897Medium x Mods .01 3.30 .038*Version x Medium x Mode 100 .04 .965

Vithin GroupsSession .06 21.59 .001**Version x Session 001 3.54 .030*Medium x Session .63 556.01 .001**Mode x Session 100 08 .925Version x Medium x Session .00 .84 .434Version x Mode x Session .00 1.43 o241Medium x Mode x Session 042 237,07 .001**Version x Msdium x Mode x Session .00 .73 .482

*Statistically significant at p < .05,**Statistically significant at p < '01.

ences were clearly consequentials, even though the ANOVA analysis found the CSdifference morginally statistically nonsignificant. In Session 2, the paper-and-pencil group (man - 38.11) outscored the computer group (man - 34,32) by3.79 for NO, and by 11,29 for CS (moan - 53.52 and 64,81). The CS result isclearly substantive. The Session effect overall, with a 1.77 point gain forNO (means - 34.46 and 36.23), and 2.32 points for CS (means - 56,91 and59.23), does not indicate such of a real difference, since practice effectswould be expected on these speeded itens.

Upon comparison of computer screen Node differences, it was found firstthat the CRT-i Mods condition (mean - 34.02) produced higher scores than theCRT-2 condition (man - 28.21) in Session 1 by 5.81 for NO, an important meandifference; and by 21.86 for CS (mans - 71,89 and 50.03), a highly importantdifference. These disparities hardly diminished for the computer group inSession 2, with an NO difference of 5.42 (means - 37.03 and 31.61) and a CSdifference of 18.45 (mns - 74.04 and 55,59).

Further analyses of the computer versus paper-and-poncil mdium differ-once* by computer presentation mode within Session 1 revealed that, for NO,the CRT-I condition (man a 34.02) shoved no practical difference from thepaper-and-pencil condition (mean - 37.79), but that the CRT-2 condition (man- 28.21) resulted in substantially lower scores than did both the paper-and-pencil condition (man - 37.79) and the alternate computer Mode (mean -

34.02). The same results wore obtained in Session 2, with the CRT-1 condition

- 29 -

I I .* 9,tI , 1 " 3, ,SN .3e ,', S. ;.ic. . ¢,,,..N.,', , . ,N".',: ", ,'.,,,.,6.,.,.., ':,,''.-..,, .I

I 7 1 - -

Table 12. Results of Univariate ANOVAs for Significant Effects Identifiedin the MANOVA for the Subtests with 1 and 656 Degrees of Freedom (N - 664)

Effect and Mean squaressubtest Between Within F p<

MediuaNO 685.89 106.29 6.45 .011*CS 782.94 228.41 3.43 .065

ModeNO 3,006.37 106.29 28.23 .001**CS 40,997.32 228.41 179.49 .001**

lMedium x NodeN O 14.35 106.29 .13 .713CS 1,145.51 228.41 5.02 .025*

SessionNo 1,028.29 40.18 25.59 .001**CS 1,784.08 57.15 31.22 .001**

Version x SessionNO 263.96 40.18 6.57 .011*CS .88 57.15 .02 .901

Medium x SessionNO 9,083.82 40.18 226.09 .001**

• CS 31,570.67 57.15 552.45 .001**Medium x Mode x SessionNO 2,256.84 40.18 56.17 .001**CS 27,114.36 57.15 474.47 .001**

*Statistically significant at p < .05.**Statistically significant at p < .01.

(man - 37.03) not being different from the paper-and-pencil condition (mean38.11), but with both of these conditions revealing mean scores mch higherthan those of the CRT-2 condition (mean - 31.60). For CS, an outcome similarto that for the NO subtest was obtained, except that for Session I the CRT-1computer presentation condition (mean - 71.88) had mean scores such higherthan the paper-and-pencil (mean - 52.74) and the CRT-2 computer presentation(man - 50.04) conditions. For Session 2, the CRT-1 condition (mean - 74.04)was again msuch higher then either the paper-and-pencil (mean - 53.52) or theCRT-2 (man - 55.58) conditions.

Although a significant Version by Session interaction was reported by theANOVA analysis for the NO subtest, pairwise comparisons of mean differencesfound no statistical significance. The largest difference discovered was thatbetween Version A in Session 1 (mean - 33.64) and Version A in Session 2 (seen- 36.30), a difference of only 2.66 points. The Version B means ware 35.26for Session 1, increasing to 36.13 in Session 2. For the Medium by Sessioninteraction, the difference between paper-and-pencil and computer admnistra-tion scores decreased in Session 2 for NO from 6.67 to 3.79, and increased forCS from 8.22 to 11.28, neither by a significant amount.

Graphical subtests. Table 13 provides the results of the MANOVA analysisfor the AS, MC, and El graphical subtests (mean scores by condition for thesetests are in Tables 5, 6, 7). This analysis revealed no statistically signif-

-30-

Table 13. Pillal-Bartlett Trace V Values, Degrees of Freedom,Approximate P Ratios, and Estimated Significance Levels (p) from

the ANOVA for the Computer Versus the Paper-and-PencilAdnstration Conditions for the AS, MC, end El Subtests (N - 664)

Degrees of freedomSource of variation Trace V Between Within F p<

Between GroupsVersion .00 3 650 .32 .810Form .01 6 1,302 1.44 .196Redius 001 3 650 2.34 .072Version x Form 000 6 1,302 .36 .902Version x Medium .00 3 650 1.03 .379Form x Medium .01 6 1,302 1.37 .225Version x Form x Medium .00 6 1,302 .48 .827

Within GroupsSession .09 3 650 20.24 .001**Version x Session .00 3 650 .84 .474Form x Session .01 6 1,302 .82 o552Medium x Session .04 3 650 9.10 .001**Version x Form x Session .00 6 1,302 .35 .908Version x Medium x Session o01 3 650 1.88 .131Torn x Medium x Session 01 6 1,302 1.45 .191Version x Fore x Medium x Session 001 6 1,302 1.37 .226

**Statistically significant at p < .01.

Table 14. Results of UnIvariate AIOVAs for SignificantEffects Identified in the MANOVA for the AS, MC, and ElSubtests with 1 and 652 Degrees of Freedom (N - 664)

Effect and Mean squaressubtest Between Within F p<

SessionAS 58.70 2.19 26.75 .001**MC 122.61 2.93 41.88 .001**El 17.45 2.13 8.21 .004**

Medium x SessionAS 3.89 2.19 1.77 .184MC 70.74 2.93 24.17 .001**ZI 21.86 2.13 10.28 .001**

**Statistically significant at p < .01.

-31-

*" " "- " ""*" " "" " "€ ",'* ,' ' , - 1.. - '. , , ',-

Ieant effects for the between-subjects factors, Version, Fore, or HMdium.Significance vas indicated for the repeated measures Session effect# and theHediun by Session interaction. Table 14 shows the results of the univariatefollow-up analyses for the statistically signif icant e facts, revealing siS-nificance for all three graphical subtests on the Session effect, and for MCand 11 on the Medium by Session interaction.

Comparison of the means by condition for the significant effects revealedsmall differesces in means across session of .42 for AS, .61 for MC, and .23for 11, certainly not of any psychometric consequence. For the Medium by Ses-sion effect for MC and 21, differences of a similar magnitude were revealed,with the MC computer scores Increasing by 1.43 across sessions (means - 16.14and 17.57), while the paper-and-pencil scores declined by .21 (means - 17.57and 17.21). For the 31 subtest, computer scores gained .66 across sessions(mans - 13.23 and 13.89) while the paper-and-pencil man score was .20 lower(means - 13.92 and 13.72), all relatively small changes.

Comparison of the means for the MC Session 1 Medium effect, specificallycomputer (mean - 16.14) versus paper-and-pencil (mean - 17.43) presentation ofgraphical Image items, showed a mean difference of 1.29. Further inspectionshowed that the largest part of this difference was attributable to Form 11,where the paper-and-pencll group (mean - 18.07) outscored the computer group(mean - 15.82) by 2.25. For Form 12, the means were 16.17 for the computergroup and 16.88 for the paper-and-pencil group, for a difference of .71; forForm 13, the means were 16.42 and 17.32, respectively, for a difference of.90.

Summr

No Important psychomtrically meaningful significant differences weredemonstrated for the paper-and-pencil baseline analysis for any subtests, ex-cept for a practice effect on CS. For the computer versus paper-and-pencilequivalence analysis, the PC subtest revealed a ajor difference in the odefactor, with the CRT-2 condition resulting in important mean differences fromthe CRT-1 and CRT-3 conditions. Even with the effects of this condition re-moved, however, differences were still shown between the computer and paper-and-pencil presentation media. It was also found that the group that took thecomputer test second scored higher on it than the group that took it first.However, the group that took the paper-and-pencil test second, did not obtainhigher scores than the other group's first session paper-and-pencil test.

Mh speeded test comparisons revealed that for NO, the CRT-2 presentationnode was decidedly inferior in performance, while the CRT-I condition did notdiffer significantly from the paper-and-pencil results. For CS, it was foundthat the CIT-i computer condition resulted in higher scores than did eitherthe QT-2 or the paper-and-pencil conditions, which did not differ from eachother,

The analysis for the graphical subtests revealed no psychometricallymeaningful differences for any effects for any subtests, including the firstsession Nedium effect for the MC test which addresses directly the question ofdifferences due to paper-and-pencil versus computer presentation of graphicalImage test items.

- 32 -

Reliability Analysis

Test-Retest Correlations

Table 15 ohms test-retest correlations (for CS, AS, HC and ZI) or alter-nate form retest correlations (for PC end NO) for each subtest. Tor the

* speeded subtesto (NO and CS), all the correlations for Groups 1 and 2 for bothcomputer administration modes were significantly lower (at the .05 level) thanthose from Group 3. except for the NO subtest in Group 2 computer-administeredftNde 1. These significant differe*c" In retest correlations were fairlylarge, the smallest being .20 for CS (.65 versus .85) in Group 2 Mode 2,Increasing to .36 for 110 (0-33 versus .69) in Group 1 No&e 2.

Table 15. Test-Retest Correlations (r), forNSu Xtests CS, AS,, NC, end 11 and AlternatePores Retest Correlations for PC and NOSubtests and Number of Examinees (N)

by Group and Node

Subtest& Mode/ LLOU I ru ruversion r N r N r N

1 4 42 114 e75* 109 49 1122 453 164 .509 164

1 447* 168 v5 168 .A9 1082 .60* 164 .5* 164

AS

12 .93 109 .88* 112 .94 10813 .89* 109 .95 ill .95 113

HC .81* 332 .87 332 .88 333

II .79* 332 .89 332 .85 333

Note. Group 3 PC correlation Is based on examineestaking Form 11 only, and the NO and CS correlationsare based on exminees taking Pore 12 only,, Incorrespondence with the forms administered to Groups1 and 2.

*Indicates that the Group 1 or Group 2 correlationis significantly different from the Group 3 corre-lotion at the .05 level.

-33-

* ' - 0-- - - - IF0

The graphical subtests (AS, C, and II) showed a similar pattern of test-retest correlations, For each graphical subtest, the Group I correlationswere significantly lower than those from Group 3, although for AS only Form 13yielded significantly lower correlations. The only Group 2 test-retest corre-lations significantly lower than Group 3 were from AS, Forms 11 and 12. Al-though these relLbllties (test-retest) were statistically significantly lov-er for the groups in which one test administration was computerized, the actu-al differences in teat-retest correlations were not very large, ranging from.05 (AS in Group 2, Form 11) to .07 (NC in Group 1).

The pattern of teat-retest correlations was somewhat different for sub-test PC as there were no differences In test-retest correlations betweenGroups 1 and 3 that reached significance at the .05 level. The Group 2 PCtest-retest correlations were significantly higher than Group 3 for Modes 1and 3. These were the only test-retest correlations for any of the subteststhat were significantly higher in either of the computer administration groups(1 and 2) than in the paper-and-pencil-only Group 3 (.75 and .79 versus .49).

internal Consistencies

Mean 5R-20e for each non-speeded subteet by Group and Session are inTable 16 (5R-20. for each group and condition are in Tables 4 through 9).Tables 4 through 9 show that the differences in KR-20e within cells were ofthe sam magnitude as any difference found between cells; thus, the data werestudied as means instead of as individual values from separate test admnLs-tratons.

Table 16. Average Kuder-Richardson Formula 20 Reliability (520)Tor Subtests PC, AS, NC, and II by Experimental Group and Session

Session I Session 2

Group PC AS NC 2I PC AS NC EI

Computer Paper-and-Pencil

Group 1 .558 .817 .758 .728 .674 .824 .764 .720

Paper-and-Pencil Computer

Group 2 .658 .808 .714 .694 .596 .800 .711 .699

Paper-and-Pencil Paper-and-Pencil

Group 3 .727 .806 .748 .697 .694 .794 .740 .692

Note. For the PC subtest in the computer conditions, only Modes 1 and 3were included, due to the extreme Mode 2 differences. In the paper-and-pencil conditions, only Form 11 was included so as to correspond with thecomputer conditions,

The lowest man reliabilitLes for both computer administration (.558 and,658) and paper-and-pencil (.727) were obtained for the PC subteat, as expect-

- 34 -

': ';' ''-2' '.' .'. ,;-';,',-'. • " . . . - "• " ,". . . ." . . . .. .- - -. . - .- - -•. . .

ed, due to its short length. Also, the largest differences in man KR-20sacross media of administration for any subtest were found for the reliabill-ties of PC, with differences of .100 (Cl1 versus P21) and .169 (ClI versusP31). The PC subtest also showed the only large increase in man KR-20 acrosssessions, with the Group I man KR-20 increasing from .558 in Session 1 to.674 in Session 2.

Subtest AS yielded much more consistent man KR-20 values, with a rangefrom .794 (Group 3, Session 2) to .824 (Group 1, Session 2). This .03 rangeis striking in that both values are from Session 2 paper-and-pencil-admini-stared tests, implying that the computer versus paper-end-poncil comparisonswere more equal, with only .009 (ClI versus P21) and .011 (ClI versus P31)differences in Session 1 and .024 (P12 versus C22) and .006 (C22 versus P32)differences in Session 2.

The Session I and 2 mean KR-20 differences were all smaller than .01 forsubtest NC. This corresponds with the pattern found for subtest AS, where thelargest difference was a .012 decrease from Session 1 to Session 2. WithinSession 1, there was a larger man KR-20 difference for NC between the twopaper-and-pencil administrations (P21 versus P31, .03) than between Cl andP31 (.01), though all the differences were very small.

The man K-20 values for subtest 11 conformed to the general patternfound for the other graphical subtests, indicating no effect on internal con-sistency for computer versus paper-and-pencil test administration. As foundfor NC, there was no computer administration effect within examinees for El,with all differences in mean fl-20s across sessions being .008 or less. Inboth sessions, the man IR-20 was higher in Group 1 than in the other groups,with the overall range being .692 (P32) to .728 (ClI).

Analysis of Structure

Across Subtests

For the calculation of the between-subtest correlation and covariancematrices from the computer administrations, only scores obtained from exami-nees taking Nodes I and 3 for the PC subtest and Node 1 of the NO and CS sub-tests were included. The MANOVA analysis showed that the Mode 2 scores forPC, NO, and CS were such different than both Mode 1 and Node 3 in terms ofsubtest mans and variances, thus their exclusion from this analysis. Table17 shows the correlations between subtest scores by Group and Session whichwere analysed by principal factor analysis to yield the eigenvalues shown inTable 18. The ei4gnstructure is fairly similar within each cell (Group andSession combination), with a large first factor capturing between 63.4Z (P21)and 69.8Z (P32) of the cosmon variance. All calls seen to conform to a two-factor solution except C1i, for which three factors are indicated. Cell CI1is the only one where the third eigenvalue is positive, with the third factoraccounting for 9.9% of the comon variance. The third factor is present atthe expense of Factor 2 which accounts for 13.3% less common variance than the

*. Factor 2 in any other cell in Session 1.

Table 19 gives the factor loadings by Group and Session from the prici-pal factor analyses. Factor I loads highest on, and is defined by, the graph-ical subtests (AS, NC, ZT) in every cell except P31, where PC loads higherthan AS (.572 versus .542). There are also substantial loadings for PC on

- 35 -

o :, .,...

Table 17. Intercorrelations of Subtest Scoresby Group for Session 1 (Upper Triangle) and

Session 2 (Lower Triangle)

Subtest PC NO CS AS MC E1

Group 1 (N-171 Session l8; N-332 Session 2)

PC - .36 -.06 .08 .30 .32No .14 - .27 -.05 .19 .11CS .13 .56 - .08 .18 .10AS .25 -.05 .01 - .59 .49MC .32 .00 .08 .52 - .56El .34 -.02 .00 .56 .59 -

Group 2 (W--332 Session 1; N1167 Session 2a )

PC - .20 .22 .16 .25 .28NO .15 - .62 -.04 .12 -. 02CS .27 .41 - .11 .21 .10AS .24 -.17 .01 - .54 .57MC .34 -.01 .11 .56 - .55El .40 -.09 .00 .52 .54 -

Group 3 (1-333, both sessions)

PC - .39 .34 .21 .36 .35NO .26 - .63 -. 03 .16 .07CS .27 .57 - .02 .21 .09AS .22 -.03 .05 - 153 .55MC .41 .18 .28 .52 - .521E .37 .11 .16 .58 .57 -

Note. Only factors with eigenvalues greater than zeroare presented.

aOnly computer Modes 1 and 3 for PC, and Mode 1 for NOand CS, were included since Mode 2 for PC and Mode 2for NO and CS were found to be significantly differentin the MAIOVA analysis.

Factor 1 for cells C22 (.488) and P32 (.521), crossing both Session and MedL-urn.

The second factor is a little cleaner, with its highest loadings found onthe speeded subtests (NO and CS) in every cell except CI1, where PC and AShave stronger loadings than CS (-.432 versus .350). In Group 1, Session 1,both PC and AS have stronger loadings than CS (.357 and -. 347 versus .184).In Group 3 there is also a substantial loading on Factor 2 for 8ubtest AS ineach session (-.443 and -. 385).

Assuming that a third factor is necessary to adequately explain the data

- 36 -

Table 18. Kigenvalues and Percentage of CommonVariance Accounted for from Principal Factor Analyses

of Subtest Scores by Group and Session

Session 1 Session 2Group and Kigen- Z of Zigen- Z offactor value variance value variance

Group, 1: Computer (-171f Paper-and-Pencil (N-332)

1 1,740 67.8 1.719 65.52 .572 22.3 .906 34.53 .255 9.9

Group 2: Paper-and-Pencil (N-332) Comuter (N-167)a

1 1.740 63.4 1.732 69.32 1.006 36.6 .768 30.7

Group 3: Paper-and-Pencil (N-333) Paper-and-Pencil (N-333)

1 1.885 64.4 1.956 69.82 1.043 35.6 .846 30.2

Note. Only factors with eigenvalues greater than zero arepresented.

aonly computer Modes 1 and 3 for PC, and Mode 1 for NO and CS,were included since Node 2 for PC and Node 2 for NO and CSwere found to be significantly different in the MANOVA analysis.

in cell Cl, it is defined by low to moderate loadings on PC (-.289) and CS(.389). The communality estimate for CS in cell Cl (.220) is lower than inany other cell, the next lowest being .337 in the other computer-administeredcall C22.

Table 20 summarizes the covariance structure and confirmatory factoranalysis results. The first model tested was the equality of the across-subtest covariance matrices within sessions. In both sessions this model wasrejected as not fitting the data, with X2 _ 104.57 and X2 - 75.47 with 42 de-gross of freedom. A check on the sensitivity of the procedure was run testing

the equality of the across-subtest covariance matrices for only the paper-and-pencil administrations within each session. This model was not rejected ineither session (x2 - 22.42 and X2 - 29.91, with 21 degrees of freedom each),indicating that the covariance matrices for computer administration sessionswere not equal to those from paper-and-pencil testing sessions.

The next model tested a less-strog equality that the factor pattern wasinvariant for each group within sessions. The principal factors analyses sug-gested that either two or three factors were present; thus, the first testsassumed the presence of three correlated factors for each group. Assumingthat this model was not rejected, subsequent models would test the equality ofthe factor loadings, error variances and covariances, and factor covariances,

- 37 -

I|

m 04 I- en0 0 0 P0 0. P. 04 0 t M 0 0 0 N V4 0

0041% n0C % 0 -I.4M bUO @~430 cIh' r e co-4

V e 00000 le OS 000 00

In, in0d c - clI in-t - 4fI C% P Iin UN0S 4aglfr 4b nc 4C00 q4 ,O t4OC %DPWMn.0wlI%

C4 u 0

0 ~~ 00. . OOA

T XO 0 =O I0s41 W V-id

N -11 t o4 6"'4

0b an -0 CdU C4 0

coo 0

10 CiP.C4

4

V%41 %a.4f10 @u 4 44 MSA -C 4lf M

.0 0 * 0 * 0 0 0 0 0 0 0 0 o 0 4

mu '4a

r4 &1

@6C CM"MOO

A m

@1.4 46

416 0tJ 4 p.~r%4 o~a'@40~91

64 U ~ ~ O~MO ~ M38

Table 20. Sumry Table for Testing Models of Factor EqualityVithin Sessiou, Showing Model, Ch-Square Value (X2), Degreesof Freedom (df), Goodness of Fit Index (GFI), Root-Mean Square

Residual (INR), and Estimated Probability (p)

Hypothesis x df cn RJ p<

El2 w Z21 - ES1 104.57 42 .977 2.206 .001**-21 - E31 22.42 21 .989 2.090 .374

E12 : E22 - E32 75.47 42 .985 3.513 .001**E12 E 132 29.91 21 .984 4.632 .094

A. Ax - A2 A e6 not positive definite for Cl11 x 2 1 31

A. A1 2 -Ax22 - AX32 56.58 21 0967 2.297 .001**B. A I I A x21 - A X31 not positive definite for C22

B. AX12 A1 2 A O , e8 not positive definite for Cll

C. A1 A 2 f w A 1 157.54 24 .940 2.239 .001**x1 "x21. A31

**Statistically significant at p < .01.

Note. A is a model of the factor pattern: '0 0 0

x 0 01V 0 1X

3 Is a model of the factor pattern: 0 0-X

0 x 0

LX 0 xJis a model of the factor pattern: 0 Xc

0A x

1b 39

providing an indication of the degree of equality of the factor structures.

Only for Session 2 did LISREL converge to meaningful parameter estimateswhile testing a three-factor model. This model (Factor 1 - AS, MC, El; Factor2 1 O, CS; Factor 3 - PC; zeroes elsewhere) produced a significant chi-square(X - 56.58, 21 degrees of freedom) indicating non-fit. In Session 1, thematrix theta-delta was not positive definite for Group I (Cll), indicatingthat the model did not fit the data for that cell.

A second three-factor pattern was hypothesized, allowing two more load-ings to vary (Factor 1 - AS, C, I; Factor 2 - NO, CS; Factor 3 - PC, NC, El;zeroes elsewhere), which lessened the pressure to fit many zero loadings. Forthis model, LISREL did not converge to meaningful parameter estimates ineither session. The Session 1 results found both matrices phi and theta-deltato be non-positive definite for Group 1 (Cll). In Session 2, phi was not pos-itive definite for cell C22 (Group 2). Again, the non-meaningful estimates ofthese matrices Imply that the model did not fit the data within those cells.

Next, a two-factor pattern model (Factor 1 - AS, NC, El; Factor 2 - PC,NO, CS; zeroes elsewhere) was tried for Session 1, in case three factors werenot necessary to explain the data adequately. This model fIt the data lesswell than the most restrictive covariance equality model (X - 104.57, df -42), with a X2 value of 157.54 with 24 degrees of freedom. The implication ofthis test is that neither a two-factor model nor a three-factor model fit theSession 1 data. Thus, the reason for the non-fit of the three-factor modelcould not be the specification of too many factors.

*Within Subtest

Table 21 shows sigenvalues and the percentage of common variance account-ad for from the within-subtest principal factors analysis by Form (11, 12 and13) and Medium of administration. Appendix A shows the factor loadings forthe first five principal factors from the same analyses. Only non-speededsubtests AS, MC, and El were factor analyzed, because it was necessary to com-bine the data from examinees taking both Versions A and B in order to obtainenough examinees to meet factor analysis requirements. Subtest PC containsdifferent items in Versions A and B, making such a combination impossible.Due to the finding of significant Session effects in the MANOVA analysis, allanalyses were performed within Session 1, with Group I (computer administra-tion) being compared against Group 3 (paper-and-pencil administration).

There were no major trends across subtests or forms, or large factorstructure differences for the subtests, under different media of administra-tion conditions. The factor structures of each form of each subtest underboth administration media were adequately described by one-factor solutions.The large number of eigenvalues greater than 1.0 found for each subtest islikely an indication of the small examinee-per-item ratio for each factoranalysis (4:1 or 5:1). The first factors accounted for 43.8Z (MC Form 13,computer-administered) to 59.41 (AS Form 13, computer-administered) of thecommon variance in each subtest. These first factors are more than twice aslarge as the second factors in every case, suggesting the interpretation of aone-factor solution. All subteuts, however, showed a trend for the first fac-tor to be larger (i.e., account for more common variance) under computer ad-ministration than for paper-and-pencil administration. These differences

40

'p.%

R7*~m AR .? I Pz** . V.. . . . . . .V?-

& 60 0% W 00 %a 0 .S N0

is 04 .4 ato e-r'C

en of 0 0 0 6 0 0 S S

III tj in4nIc- ini%0 0.%4CO

0 0 0 0 0 0 0 OS

"4in04 )0 4cc.-O@ Cf4 CC44

0-4 4 * .4 . r-* in at * *" -C N %D 4

N goCIC c. c000 n .VC 0 4 CD0

* 0 c'8 4 0 " i 00 s N cn OC4co In u ~-C9 -tIt%

bIV C4 'I N In *0 m %. o- *l Go *

F4W%0- r4 -4 -J0 lflO 914 4 -4 wl

NN Go 0 0 -e Go 0 Sl 00 -a 4 r- 00-

10 In inO I .l0

00 IA .5 0 ge%..

WN~ P-4 N m.0 in -

41,I."

-41-

ranged from 11.3% in MC Form 11 to 1.1% in AS Form 12. There was no differ-ence in 2I Form 11, and there was a 1.7% difference in the other direction forMC Form 13, but these were the only exceptions.

The only large difference in computer- versus paper-and-pencil-admini-stored subtest factor structure occurred for subtest MC, Form 11. For comput-er administration, Factor I accounted for 56.8Z of the common variance, butfor the paper-and-pencil administration, Factor 1 captured only 45.5Z, causingan 11.3Z difference. This difference was not matched in Forms 12 or 13 ofsubtest MC; in fact, for Yorm 13, the paper-and-pencil Factor 1 accounted formore common variance than did the Factor I from the computer administration(45.5Z versus 43.8%).

Item Analysis

Conventional Item statistics (proportion correct, point-biserial, andbiserial Item-test correlations) for each item In each non-speeded subtest(PC, AS, MC and E1) by Version (A or B) and Medium (computer or paper-and-pencil) are shown in Appendix B. There were no statistical analyses performedat the item level to compare particular values of these statistics, due to themostly nonsignificant and inconsequential differences found for the Mediumeffect in the MANOVA analyses.

The important question of possible differences in graphical content itemsacross administration media was addressed partially through the subtest-levelMAMOVAs, The distribution of graphical items allowed the comparisons of sub-test MC across administration media to substitute for an item-level analysis.In fact, every subtest-level analysis performed on subtest MC could be used asa method to compare graphical versus non-graphical content iteas. By compar-ing computer administration versus paper-and-pencil administration differencesfor KC versus AS and El, information on graphical item was obtained. Theresults of these comparisons were that differences in subtest MC across admin-istration media were not significantly greater than the computer versus paper-and-pencil differences exhibited by AS or El. The within-subtest factor anal-yses also showed no greater differences in the factor structure of MC acrossadministration media then were found for subtests AS and El.

IV. DISCUSSION

The purpose of this study was to determine the extent of equivalence ofmeasurement properties for a battery of ASVAB subtests under conditions ofcomputer versus paper-and-pencil administration. The subtests selected forstudy--PC, NO, CS, AS, MC, and El--were those that presented particular prob-lems for computer presentation.

The paper-and-pencil baseline analysis was performed to yield a lowerbound against which computer versus paper-and-pencil differences could beJudged, both within and between examinees. The only statistically significanteffects found were those for Form on the AS subtast and Session for the NO,CS, and AS subtests. However, the statistically significant mean differencesobserved were small and not psychometrically meaningful, thus providing anacceptable basis for judging comparisons between computer and paper-and-pencilconditions.

- 42 -

Paragraph Comprehension Subtest

The PC subtest, chosen due to its paragraph-length Ltems and multipleitems per paragraph, showed differences In measurement properties across themedium of presentation and within the three modes of computer administration,with one node (CRT-2) differing significantly from the other two (CRT-i,CRT-3). The first Mode (CRT-i) displayed the paragraph in one scrolling fieldwhile the itsm appeared sequentially In a separate nonscrolling field beneathit. Each item was erased before the next one appeared, and the examinee, un-able to retrieve the item, could therefore proceed only n a forward directionthrough the test. The second computer administration Node (CRT-2) containedeach paragraph and all relevant questions in a single scrolling field. Inthis node, the. entire screen was available for viewing the paragraph, withquestions appearing after the final line of text as the paragraph was scrolledup the screen, allowing the examinee to move back and forth within each para-graph. Vor Sessions I and 2, this condition resulted in mean scores sLgnifi-cantly lower than both the paper-and-pencil condition and the other two CRTconditions, suggesting examinee confusion or disorientation arising from thisparticular screen forme. Another possibility is that because whole para-graphs and corresponding items never appeared on the screen simultaneously, amemory component was introduced and becme more important in this conditionthan in the others. The final computer condition (CRT-3) contained separatescrolling fields for both the paragraphs and their related items, and providedan answer-sheet type of display at the top of the screen, allowing examineesto monitor their progress through the test. This condition also enabled exam-inees to return to any paragraph and to change their response to any item atany time during the test. In both sessions, this condition resulted in meanscores almost identical to those for the CRT-i condition.

Clear differences were demonstrated within Session 1 between all threeCRT conditions and the paper-and-pencil administration condition. In Session2 however, equivalent scores were obtained for the paper-and-pencil, CRT-I,and CRT-3 conditions, with the mean paper-and-pencil score being fairly con-stant, while the scores for all three CRT groups were significantly largercompared to Session 1. Only the CRT-2 condition still yielded significantlylower scores than paper-and-pencil in Session 2. This finding suggests thatthose who took the paper-and-pencil PC test first, followed by the computer PCtest, may have benefitted from practice effects in the second session, imply-ing that a more extensive practice sequence and perhaps a more detailed in-struction set preceding the administration of the computerized PC test may beappropriate. In light of the absence of comparable differences on the othersubteats, another possibility may be that, due to the combination of lack offamiliarity with the computer medium and the complexity of the PC subtest, thefirst subtest administered in the sequence produced a heightened level of anx-iety which attenuated test scores for those tested by computer in the firstsession. The equivalence of the CRT-i and CRT-3 conditions may be due to atendency for examinees to ignore the more complex features of the CRT-3 condi-tion, such as the ability to return to earlier paragraphs, responding insteadIn a manner similar to that of the CRT-1 forats, i.e., proceeding straightthrough the test item by item without backtracking. The lower performance forCRT-2 may have resulted from the increased memory requirement caused by theparticular screen configuration used.

The reliability analyses found that the PC subtest obtained lower inter-

- 43-

- ~% *%*%

nal consistencies of a meaningful magnitude (> 010) when computer-adminis-tared. However, the test-retest reliabilitLes when one administration was bycomputer were not significantly lower than the paper-and-pencil-only test-re-test. reliabilities. In fact, the Group 2 tet-retest correlations were sig-nificantly higher (for Modes 1 and 3) than those from the paper-and-pencil-only group.

Speeded Subtests

he NO and CS subtests, chosen on the basis of their speeded nature, wereadministered by paper-end-pencil and by two CRT conditions. The first comput-er mode presented one item at a time on the screen, with each item beingerased and replaced by the next item following the exminee's response. Thesecond mode filled the screen with a block of items, with the examinee re-sponding to the first item in the block. This item was then erased and theremaining items in the block shifted upward following the examinee's response.The number of items present on the screen decreased in this manner with eachresponse until, following the response to the last item in the block, a newblock of items appeared on the screen.

Although the initial analysis revealed significant differences betweenthe paper-and-pencil and CRT media for both the NO and CS subtests in Session1, further evaluation demonstrated equivalence between one CRT condition andthe paper-and-pencil sedium in both sessions. For the NO subtest, this equiv-alence was found between the CRT-1 sngle-ftee mode and the paper-and-pencilcondition, with the CRT-2 group scoring significantly lower. For CS, theequivalent conditions were the CRT-2 aultiple-Ltes screen mode and the paper-and-pencIl condition$ with the CRT-I single-tm =group scoring much higherthan either of the othes. It should be noted that for every comparison be-tween the CS CRT-2 and paper-and-pencil conditions, a consistent trend of mar-ginally lower CRT scares was observed, arguing against absolute equivalence.

For both speeded subtests, a consistent relationship was found betweenthe two computer conditions, with the sinLgle-Ltem presentation causing higherscores than the multiple-item presentation in every case, the mnltiple-itenmode perhaps causing distraction or confusion for the examiLnees. For the NOsubtest, this resulted in highly attenuated scores for the CRT-2 condition ascompared with paper-and-pencil results. For the CS subtest, the single-itemcomputer mode provided such a marked increase in performance over paper-and-pencil administration that the attenuating effects of multiple-Lte computerpresentation brought scores more in line with those for the paper-and-pencilmedium. This finding my be deceptive, however. The CS single-item presenta-tion condition may actually be more parallel to the paper-and-pencil perfor-mance, differing only by a scaling factor due to the greater response speedafforded the CRT examnee. The multiple-item response condition may be sub-ject to negative effects arising from rapid response times which are then ne-gated by the distracting influence of the upward-shLftLng ultLple-item screenfoat, as found with the NO subtest.

The test-retest reliability analysis showed for both NO and CS that ad-ministration by computer in one session significantly lowered the correlationof compute-administered scores with those from paper-and-pencil administra-tion, in comparison to the paper-and-pencil baseline analysis. This suggeststhat the administration of speeded tests by computer might result in test

- 44 -

scores with somewhat different score propertios than the sa, speeded testsadministered by paper-and-pencil.

Graphical Subtests

The AS, NC, and I subtests, chosen for their graphical and standard ul-tiple-choice text content, were presented in a single CRT mode in addition topaper-and-pencil presentation. The format of the computer presentation wasIdentical to that of the paper-end-pencil tests with digitized graphical im.-gee, copied directly from the ASVAB test booklets, appearing on the computerscreen with the appropriate text.

The results of the analyses for these subtests indicated a straghtfor-ward equivalence between the computer and paper-and-pencil media for all threesubtets. The clarity of this finding can be attributed, at least in part, tothe construction of finely detailed computer representations of an graphicalimages. A direct test of the equivalence of examinee perception of these Ima-ges was provided by the comparison between CRT and paper-and-pencil conditionswithin Session I for the MC subtests the content of which is almost entirelygraphical. This comparison Identified no differences between the two modes ofpresentation, thus suggesting equivalence.

Subtast Structure

The interrelationships of this battery of subtests as a whole were some-what different under the conditions of different administration methods. Theprincipal factor analyses suggested that the factor structure of the subtestscores for computer-administared Group I Session 1 contained three factors,whereas only two factors were needed to explain the data from the other firstsession paper-and-pencil administrations. The LTSREL confirmatory analysissupported these results by rejecting a model of the equality of the subtestcovariance matrices when a computer administration group was included, but notrejecting a nodel of equal covariance matrices when only covariance matricesfrom the paper-and-pencLl groups were included. All of the less stringenttests of the equality of factor structure models across administration mediafound either a rejection of the model or the presence of non-positive definitecovariance matrices. A finding of non-positive definite covariance matricesis an Indication that the model did not provide a suitable fit to the data.

These negative LISREL findings could be due to the slight factor struc-ture difference found In the principal factors analyses or to the extreamepower of the LISREL procedure when sample sizes and degrees of freedom arelarge. The model of equal factor structure that did converge to meaningfulparmeter estimates produced a significant chi-square, but the degree of non-fit was quite smll. The estimated covariance matrices for Groups 1, 2, and 3produced only ero, one, and two significant normalized residuals, respective-ly. Thus, although the model did obtain a significant chi-square value, itshould not be rejected entirely.

V. CONCLUSIONS

The results of the present study suggest that attaining equivalent testresults between computer and paper-and-pencil administrations is feasible for

- 45-

the power subtests of the ASVAB. For the case of standard mltiple-choicetext items and those with graphical content, all that my be required is care-ful development of testing software, with adequate attention to the clarityand detail of graphical images.

Tor the speeded aubtests, the present findings indicate that the mode ofscram presentation of test item can drastically Influence the level of exan-mae performances For the NO subteat, an indication of equivalence betweenthe single-item CRT and paper-and-pencil presentation conditions was estab-lished. For CS, however, although the multiple-item screen condition producedsimilar scores to those from the paper-end-pencil medium, the trend over il-tiple comparisons between the two showed that the computer socres were margin-ally lower in every came. 7his finding suggests that further research on al-ternative screen configurations and enlarged computer instruction and practicesets may be appropriate to bring computer performance more In line with Itspaper-and-pencil counterpart. Continued investigation my also reveal thatthe higher-scoring single-ite screen condition is actually more consistentwith papet-aud-pencil performance, and that a scaling factor is required tocompensate for the faster response time made possible by the substitution of acomputer keyboard for a paper-and-pencil answer sheet.

The PC subtest offers perhaps the grantest challenge to equivalence inthe ASVAJ battery. For this subtest, the present results demonstrate the sen-sitivity of examinee performance to alternative screen presentation modes forthis rather compleak test. The findings support, however, the benefits ofpre-test practice, simplicity of screen format design, and detailed instruc-tion sets in equalizing computer and paper-and-pencil performance. Recommen-datLons for further research include eperimentation with varying numbers ofpractice items, alternative screen formats, instruction sets emphasizing ape-cific aspects of the computerized PC subtst, and administration of the PCsubtest after administration of other subtests in the ASVAB battery to allowgreater examinee familiarity with the computer before the presentation of thePC items.

-46-

Cooley,p V., & Lohn.., P. (1971). Multivariate data analysis. NewYork: Wiley.

Croubach, L. J. (1951). Cofficient alpha and the internal structure oftests. Psychometrika, 160 297-334.

Vldt, L. S. (1980). A teat of the hypothesis that Cronhachas alpha relia-bility coefficient is the sam for two tests administered to the samample. Psychometrika, 15, 99-106.

Fisher, R. A. (1921). On the "probable error" of a coefficient of correla-tion deduced from a smal ample. Mtron, 1, 3-32.

Green, Be V., Bock# Re Do. Humphreys, U. G., Linn, R. Lo, & Reckase, He D.(1982, May). Evaluation plan for the computerized adaptive vocationala ttude batter (Research Report 82-1), Baltimore, RD: The. Johns Hop-kisUiversty, Department of Psychology.

Haktianq A. R.0 & Whaleng To Z. (1976). A k-sample significance test forindependent alpha coefficients. Psyoeti, 41 21-21

Sarman, 3. 1. (1976). Modern factor analysis (3rd edo). Chicago: Universi-ty of Chicago Press.

Jioreakog, K. Go (1971). Simultaneous factor analysis in several populations.Pchomstrika, 36, 409-426,

Joreskog, K. G., & Sorbo., Do (1984). LISREL VI: Analysis of linear struc-tural relationships by the method of -maximum likelihood, Mooresville,IN: Scientific Software,, Inc*

Kuder, G V,9 & Richardson, M. W. (1937). The theory of the estimtion ofteat reliability. Paychomtrika, 2, 151-160,

Morrison, D. F. (1976). Multivariate statistical methods (2nd. ed.). NewYork: McGraw-Hill Coo

Nster, Jo & Vassernan, W. (1974). Applied linear statistical models. Home-wood, IL: Irwin.

O'Brien, R. Go# & Kaiser, Me K. (1985). MANOVA method for analyzing repeatedmeasures designs: An extensive primr. Psychological Bulletin, 97,316-333.

Olson, C. L. (1976). On choosing a toot statistic in mltivariate analysisof variance. Psychological Bulletin,, 83, 579-586.

Preatwood, J. S., Vale, Co Dog Massey, R. H, & Welsh, Jo R. (1985, September)Armed Services Vocational Aptitude Battery: Development of Forms 11. 12.nd 13 (AIHRL-TR-85-16(I), AD-A160584), Brooks Air Force Base, Ul: AirForce Humn Resources Laboratory, Manpower and Personnel Division*

-47 -

Viner, B. J. (1971). Statistical principles in experimental design (2nd.ed.). Now York: McGraw-Hill.

Vood, 1. L.9 & Lord, 7. 14. (1976). A user's guide to LOGIST (Research Memo-randus 76-4). Princeton, NJ: Educational Testing Service,

Wood# R. L.v Vingerskys , 1.S,p & Lord, P. M4. (1976). LOGIST-A computer pro-ara for estimating examinee ability and item characteristic curve param-eters (Research Memorandum 76-6).* Princeton, WJ: Educational Testing

-48-

SPPmZNDX A: ITEM FACT OR LOADINGS AND COMMUNALTTES

BYSUBTIST AND P0ORK FOR TWO PRESENTTION MEDIA

.4

04Q 0

4 l00 0 0 0 0 0 0 0 0

0eS e0

&.D 4" 0 q 0 4 000 (4 0 0- 0m 0 00 0 M N a0 0 04 q C 04 0 0 0

41 W ei IV!I I I W I"e Iq CI IIqmIrw q q

M5 0 14 * M 04 0 0 0 C4 * 0 -4 0 0 04 0 0 -0 0 0- - 0*

4I04

Ow. @41. Go r. ,

&I ca 0 00 0 O 0 40 0-0 00 Wi a 0f 0 P% 0% 0 0 0 0 m 0% 0 0 0r 0

f"W nC 4meq00- 40d P 4V4c -8 - - -

* 0 0 0 0 0 0 0 0 * 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

Ch 0 , 00 0 0D 0% 0 0 0 %0 0 Q0 %0 0 0_ 0f m 0 0 0n 0 0 _0 0 0

I0 II M 14 MI N MI I44 ni n0 oV4%U-

a 64 0 00 M 0 0 0D 0 0o 0 0 0 0 -O In %a 0. Go 0 0 0 M -W In5,. eq Ni I4 I4 I I

d5

W0 0 0 6 0 0 0 0 0 10 0 10 0 0 10 10

Id 9 0 0 0 000 ** ** 00010 0. . 0 0 0

2.0 n. P% %0 V P. Nii In Go M Co onc IS o

l E U0 0 0 1* * 0 10 *0 0 0 0 *1 10 01 *1 0 01 o1

610 ,4 990006006,00010 00

~o 0C4 OMDC% OO04000

C4 ISGD - 0 I M P% II% IP - 0 In

Ah8i I I Sil N MIW- # A4 0" %MM9 WM94.

0 0 0 0O 0 a 0C .0 0 0 0 0 0 0 0 * 0 *

M cc 0 6 40 n M 0q M 0n 0D 0 0 0 0-a g 00 0 0- 0M

-W- DC . *% oI 0 -4P Nd -0 No N0 0- Go-- -n -4 %aC4IniP" C 0 -W%00000 o40 000atC4 DD006000000 00- 4 %P

0n ina 0 0 0 b-0 M 0 0 0D In 0 0 0 WN 0 e 0W 0 In m 0 0 C

Pm CU m 0 -0 t- 0 at 0 0s 4 0 - 6 0 0a 0l Go e 0 9. 04 M 0 0n

'4I

C4'

a-. ~i .0 -W C M M 4I~,

0 0 0 0 0 0 6 0 0 e 0 0 0 0 0 0 0 10 0 0

(OA

*0 Mmn 0% .0 nO m n 4 W I %aP4

Il 0 04 0 D 0 V- 00 M 0 0 m M r- in 0 0 04 0 %D 00 0 N 0

64 in In i %a 4

0 9-4 Qq m 0% a r% LIN 0 r 0~a

0 04 0- 04 0c 0 N M 0 0 0 0 0t m 0 0 %a %a 0 0 0 0

C q 0a 0 0 04 0 0 60 0 M 0 0 0 600

.6k0 04 0 0% 6 0r- 0o P% 0 0 0 . 0% M 0% 0 a M S 0

ft g 1 1 oII i l I II

Cv4I 0 0 -0 0 ! M 60 . 000 4 - 0 0 0 0 0 M 0 0 "4 0 04 0 0

1.0

P* C.' M - n %40 i 00c. -4.tIn 0rl@%%(.O Nm 0C Or.

1 0 -0 0 0 0 0 0 m 4 0 -0 0 0 4 0 0 C0 C0 0 0

0 ~ ' ~r%4i~oci'~52

~~~~~~~~~~0 %'1 M 0 0~ @~'~~~-QO i 4 ' ' m

F A'I% 400a mG DO%0 lW m l01C4 CWt.C4 a r-oco NCin0 %D0 0. 04e 0 -e 0q a 0 0 0" 0 0" 0 0- 0 0 en 0 0 04 0 0W 00%

1-w 40 0 40 0 0 0 0 0 04 04 0 04 0 0 0 0 0 04 04 0 0 0000

N W004 nOU.4 4 N 4 P% a %a 9.4 u m.0 P%

At~ I0 I I I Iii III I

NJ co* 0 0 0 NO 0 0- 0 0a -a 0 0 m 0% 0 " 0 0 0l 0 .0 N

"4t %f V a0 . O%D"M o 4 c a %Pd 0% m o a 04 C co-4- O %aC C % nC -e C)

1001 0 0 0 0 00 0101010100 0 *,00 0 0 ,0 0 0

C4

Go3 0Go 0% 0- 0 0 0e a 0 0 -C P% N 0 a 0 0 0 %a 0a 0t 0 m %V

0040 04 0 0q 0 0 0 0 0 0- 0D 0 a 0 04 0 W0 a 0 0 04 a 0" 0

4

si 104 100 1*10* 0 0 0 01 *0 0 0 o e * a, 0 6 010 01

4" C4 li 94 4 ma%- 4 n 4QS ~ F a " " *4 a 0 0 " v* m

44 U) -V4 m 4 -40 0N -6v- 0-I 0 4 0 1 0 0 0 0 0 0 0 0 0 0 000 0 0 0 0-II Dc o C% n0 4m & .

-53-

%* % . -*-

08I 0 0 M .0 0 0 0 0 . %a0 0 %a 0 0 00 % 04 n m 0% 0 -0 P

V 0

0n in M0 m4 0 m in %a 0 0 *A 0 0 00 -0 0 0 in %a N 0h 0

'f I. I

0d 1* a * 00 0 1* 0 1* , * 0 01 *o *e I * I * 0 *

-W0 01 . W- In N In -0i

C40 ' 00CM444"c"0 %cw a 0 000000'- Tr-IaJ V%~ 00 0 0 04 -a 0o 0 0 In 0 In

I I Ci I Ci V: *I C r! 0

0$W 0c *4 O C4 % % 4 - 00in4"-Wt-4C4CI 44O%0 N% aeqa* 4* 00 00 N 40 0 0 9. V 5 0 0 0- 0 m 06 P- C P 0 0 0 04 " "

41.

o 4q-400C 0 " -OOO~ O wa~f4 M0 0 0 0 -

0 w0 011*0 0 0 0 0 0 0 0 0 0 0 0 00 0 0 0 S 0

a i1 0 00 *e00 0 1 0 0 0 0 0 0 0 O 0 0 0 0 0a 0 0 0

u m 0 NC4 0 9O 4 9M P4 .4 C4 0 .4 0 C 4 - -

o Id 0 ,00 ,010 10 0101 0 * 0 0 0 0 0 0 0 6 ** 0 0 0 0

Sa0

W!3t 114 ' .@@ Mshln4'M e m WCe m e '

64 P" 00 0CLO0 1- 0 0 a% C 4 m 0 0- 0 0- 0 0 0 0 C4 m 0e inbe "I III II Ill -4 III4C NC

05

M - 47,. 0@%a% 0 N4sE0 M r. 4 %a%a Mr' 0% W

"* 0 0. 010 0 0 04 04 S 0 " 0 " 0 1 0 4 " 0

ha IV 0-a 14 0 C4 1 d 0 No* 0 0 Go N In 6n 0n 4n %D C4 -4 *hRI i 9 1 vi. 19 9-99:1999i90 i

au " -W -4 N a M 0 N "4 0 0 0

0 0 0 0 0 lii 0 I al

: a** 0 V4 0 " V4 N N *4 * 0 0. 0 0 * * * "160 .ool 0. 01 1 010 0 1 3 I 0*101*19 of

0 4M0 n * c hC4-

" 0 0 C4 N 6 0 0 0 0 04 0 0 0 0 0. 64 C6 0 0

418 - C4~ C - 0 *00 *0 0,a 64 t" 0 0 " M0 0 0 0 CD~

0 n0 t CC er W C 0- n%)K T0MCW oe O"0- *- 0 4r 0C1 ) - P1- D% -C

o I

0" en q In NO 0. GoO 0 64 N -V * l %a 0 0 0 0 0 04 0 M -i in

04V4 4. 4NC 4C

04(1 - o "C %% - "G '0%- .0 %0C o0- oC nG

54 0-4 pqC Cdf4 "C4 V " m% V4 e -4c V8-. -C4

0 0 '0 0- 0C 0 04 06 cc 0 0 40 0 0" 0 0 0

410

ad 0

.F4 C" e . G 4 0 000 -1"%) oG C ni 0 040

sk ~ ~~~~~ Cd en CD Cl 0 C" 4 m09 d fq CD C C4 4"g 000 0 0 00 0 0 0 0 0 0 10 0

4a

0 * * 0 0 V4 LM 0 m Q 0 0 *4 V4 -4 CD 0 t" C* 0 Ca1 10 1 0 1 0 1 I o il 1* 0 0 0 0 0 1 1 1 ,

CIe 0

C" m6 0 n Go 0 r 0 -0 00 0 0o 0 V% 0 0a 0o 0 0W 0-0

41

04 0- aO C4 C4CCm 0 inO' 0 C(C'.-eW4vsin

*e C e .0 m o0 40 4- . %

-4 04 00 04 0 0 04 0 0 0 0 0 0 0 0 04 0- 0 0n

0 0 0 C 0 0 - 4 4 0e00 n 0 en (0 0- 0 0 0

II f i I I

o C00 0 0n N 0 0 04 P4 a 0 04 0 0 Sn 0 0

at. fS I IN II C4 %a INi - qs %- 004 i in N a ad ad d q.4O%ao c% cdi--fi

11.041 0 10110 00000 00

0i" "Olei-~~ IAII. , I. -e .- 0CO O4.-fl %C... 4i

%D4 in0 C4 r.. % 0%O4-4 ~@ 4C4c'4S 000.w co"Ime0oo @r%.EiCan"

0 ~ N V4 " 0 4 4" 0 0 0 0 0 0 0000

4000

94 N 4I DC oaf n0G 4; o C4004I 000nu '@ 'FiO fl0r

0000 l00 00000001944rD0000%D000.c

W.% 24 'rC4: v!0 1V 0 as 0 0 0 0 a 0 0

on0 0% 00 0 0o 0 %D 0 0n 0" 0 C 0 40 t 0 0 Go

C0*000100 1000000 000110 000

V4

a cc 0 0 0 00 0 0 0 0 0 04 0 0 0. fn 0 0

C"aC 4 %a V@4'-4 i00 ~ P4 WN a P

IaC3 4 o4%W0 in0 ~ 00 0 00 f4 0 e n M-%0lu ~ ~~~~~~~~~~ 04 0 V4 M 0)0V 40V 4 l94C 4PS-4

* 04 04 P4 0V00 4 0- 0- 0 f 04 0 0

.~. * II I 11157

AAa

cYd TMu.M4Ci"r"U x4% k0 OP.- 0M IC

C- aO0v4CI .O. dV40o- 0 l 00 0 -a w " 0 %0 0 0F 0 on 0 0 0D 0. m

04*a u

%0.0 ONw

V40 6k F00 ~OO %r 0 .f % 0 f- 0

MU ~ 0006"0000000000 00

0 II SI S SI S ~ kn,-, 6

dJ" r~ 0000 vic0"G l D - 0c o- nN%

4-0 M~4 WNM N - M - V 4 V4

@6 0 S SS S 0 00 0 0 0 0 0 00060

1-19 9 .I ill I1 Ii I l ISic41 -aM m 0-4C n '

,-% 0 N%0'aW a Z0

06 N m 0 In 0% 60 0 0 0 I* In 0 N w 0 0

,~58

~000o 0O00~c~i E~cA

APPENDIX B: ITI STALTISTICS (DIFFICULT!, POINT-BISURIAL,

AND BISERIAL CORULIATION) BY SUBTEST, FORM, AND VEUSION

FOR TWO PRES ENTATION MEDIA

-59 -

* LU.R~~*- --- s~.*~ -----

0 0 0 * 0 0 0

A N 440uI ~ ~4 - ~0 4

A er00.M - .R 1 -0 0%0 0 0 0 0 0 00 0 00 0

0 00 00 00 00a 0 0 0 00 0000 0 0 0 00

P48"d 4'9o~ Wt~' d

1w ad 0 0 0 00 00 00 *00 0 0 00 00 0 00 0 0 00 3

M 00 M0 M 00 in0 0 00 0 0 w0 a a0 i0 00 O*u

100~~~~~~~a " un a %a 0a W a %0448W

0~~~~~ ~ ~ ~ ~ 00 04 em -%;. 0. 4Ce o . DI oI

a % in I%,hW.i 1%tM

04 * on- a

000 0 00 0 L ..% " 04 V% 04 at %D wk %D so so % LNF 0 m04 . -vi .0 a0448. 19i.wl . .. " ,

e1g &p f"11

AIN P% . UaCSCOI 0N040 R 00 A 8 m do & i 14 I'

en t %10. (4 *P.V 94 In - 41,V r. S 44 ., - -0 . 1 0!i0 b1 V- 00 0 0" 00 I M 0 0

%* 0 M %3n 0 Id 0.C owl ~ ?A .IL

*0 9-04 *N . '1f0 % (4 .;0~ FI NoiI4 L CI k , . V - a ,0 040'

A C4 4- -go CI 6,-M--

j -60-C4;C

WN.o.%.

fo a -AP~4@ an~ a *.i**m " N

0 @ 0 0 c0jVq4 U

0e.~~~ ".. O e P.. S S SO SO SS

TCt4 o % in 4 O N .0 FV~n 4 b 0

.0 30.r ,'I'a 4 ' 6-

in lOC4dS 0 ' 4 ihA JI n aia,2f

9'- 4 1 -%

4% 40 5.5. 4,0.. onm55 55 4 S: "S 9 5MASS

*X -A C iIl!S! . . 00 00 . 0* e0 00 4

a If no 0

It w n C4. ES. M~ NS a on A ,0 1A -toit

a:Io a ILI *@~in ~ ~ t~ 01 0~.e W%

ini S ; I S .*a:; r Z M-na .0 20 1.-23so w so 9 1

S'as 66.S . SO 500

0 C4 Ok-1% 6 P%10 l%- A 0 W%; t V4f... W%0 C0. % MW % ZG% 0208 C zW In '34Mr~r C-1 -l" 4 V on , .0 In.0

lp 4.0 ,- 3A 0;soe R1 .466

S S~~i s S 5*0 00 e0 . 0 h

&~ ~~ fad ~ ~ a e 4 EmtrE% 2 *1~ 0 S

8S 5 5 * s e SO S O S I"S .4

-61-

V~WU W7W~ - ~-. ~ ~ ----- ~ ~ U' aU W'E7- w WV WW -4W -H WV VJM -" ~ J j 2£ ~ J1 ~.i F

V! *e% N .0 - e 00 0. 0 0 T0000 l -0 0"

"44in

1Cie 0.

0000 IMblC4okMokNo 4- ( 4 h ,%- 0 000 P. 0 0 04-

Son 0. XV me C.4-0("on.gN6w Q n C

Po 0C0 @*0 eec...... 0

No~

at "ft 0em ma "4!

MC'0a e4 C4 ?44 %aP..U%o 0 ;: % r.. 'at 40 in F%.* .0

Mi 4..tW .. ; ,-4C 4Wa c%

%D -4.4 W - f % 4 PcaDC Cee e . ee .4 a . . "4a4 .4 ;:P

re C4 C WN -0 0I- % N lFt M

InI: *

"me .0 go .. gee. c -a00W000 .00 L

%d 9 .. -040 M:S- -% - D% %--'F.a a4 C d0% % o 0Cd~ C .044 4 0.49% o %0 Os

a ~ ~ ~ ~ ~ m 14S~~S.o i0

eeoc m ge OgO %g 00.00 0

V0ln V! i2 vi il

.0 W %D % 400% 04 M - No % 4

.Fz.

-d ffqq' -4MF% w0122

of -s . . 00M 0 60C4 00a 40 *0. 60 ftS

Ch 10 0 0M P% .. 0 0 00 04 0 "0 1550 O 10

I I

-"" in

~ an

adgo

a

-04" i.p.""" .Q .P4mwa *Ana

~* ID ID . ooooS0S . 0 f"0" b, *0 * .

lo*m ad 0 Cvi

-- ~ea 82 t0 @0. 05 F" 5t 000 a5 0P 0 SO

a..Iva q WWlvwA8AUb% %D f% 140 C14 -0 -0 -A %D P-- In 4 in

In "&,No aS o 0 o ~.-4~ o o~ 6 6 4 o

ho 6 00 005 50 50 0 005 CC 5 5 &

W, .0 a 5- " 0 .

amad.4vp~~m~io- .3

C40 * 0 "* .4 50 ago 0s ao k" in

.40 A co a o o6 a 6 6 o0

sov AP14 in P . 0d-n . . . a a

as3

T 47 A -- d r W, L, W

.4~Goi: .4i 0 0iwz@!

We 6 6 6 6 6 6 * 6 6 6 6 6 6

M MOC %D . nV.f 40 %. n4M i .

S. 0a i. 0 eM' 94i % n f0 4 .- S.C0U

V 4 M @4NbV aM@t0.~ M* on S ia" AN$ . o t l 0 S ~0t 'a & Arl 2-a r 0in. 66 .614N % % "4 4 49 .0 .0004604"04660 6 400 N

.. .. . n III . witiv! C*"ck - h o 2I 66 66 666 666 66%6 6 a6 606 640

P." 44 .0@'# h~0 @ S

PP4 * S SI0 N0 NP. f w Siiii% 'n

0 0- 4 0- .qPI MVN N. %D @q M 0 o nt0.C % %oMM t

40 .0 V-0- 01 -N M N0- ' .C S"

-~~ C- -6 N~*~~

ala 6 60 66 66 0 0 0 6 6 6 6 66

6 U 0.

a 0f. D , Me' t %Do -0 W V %

"O S. .666 *0 66 66 60 6 606* 0 6 0 66 6

W!G 000 0 o s0

P4~~~~i~f~ 0

94 CA 66 soo e o 'a..C.4....go gogo IV% 4

go -4 lo 0 40 w 'W , WV: W 06'117 66.60606660660606666 a~l~a lowa

a0% 4 C- In ftin P 2 4 N 4 r. att% A- . P N %

0~ 44%o . Ul rn v

-64-

a. .000 000 -lm 0041 00 0 0 0a 0 00400 0

No %D %a * 0* 0000000 w-0 000 P 4. mt.zo

,-- Ra 0 a ' 4

.48n. I -0 1000 in 0000000 9 000 0 W Lt

.4.44. 4 3 4A14- MgS- 31%.'n 40 I a-4 bL. WL In 'd4P3t0.0ruL

-A W% 404 IN 00 0000 0 000"00 0P-- 01-MWOWi0 00 0 %a

VON ot n00 000 000 000 0 0.Op

0040we-4*14 4 04doi 1.0V 4 4 4~~ CMA itA- ni IS4 n0W PI 0Lt. r Dc .

0000000 0 0 .0 .0 00 . 00 ON8

-0%aDC 0 4 DV f A~ .600 40s N 4 nb,4L. a D0 00 000 0 0 0 0 0 .

40 a M ta4

8.C- E ' 4 'a 04 q4 4 cph W% UP% F. 0*.l 00 -; g..o o . '.. '..3,....-on

to wt4 44 4h v"W "4 *0a4 40%s % I* 00

*4 6& Ile 00.$ A 0 wo44 4do f" ull CC F% n' ' d 01 0%r ~ ~C4oq4.4%

%DM .- a -6t 0% wl % 4Po0 W84 P% Cfgle~ 4Qr -4 1at 400 N L

o o

44 4 .4 F w 0 Vc qa oa 4 0- nV

C40 00 00 0 0 at0t00 000 %Dan 0 0000 m 8.

o o $a.1 P.a.9 4. 0 4P q0 0 v 4V qP qV.a Di 0.

000 v 0 P0 00 000-4t f %0 00 0 0% c 0 C0 I

p, 44P " DMt4v$- bl'0R r 11 %D 0 0W V 1vC44 ,-

L"0 ' V -65-D4A "InW

%4 q440G - .0n-S N 0. 0 0% .9 .

on m %D .0 0 f. V.Nin4 n4bl s0 31~m0~4004eu4

0i 0 * * 0 0 0 * S e e 0S C @0

%D on C *0 .5.0 aese C *n No e .0 50a4 9%. IA bS on wo

-A 0 O@ a S4 C0 ln~ s 00 SO4 a C ) S P40 Cs

4 A.l coatSOCO MGIA4 SOOpe 0r %a 050w %ar .04

41 aj. 0U 2 ICO 4 a* do 0 4% 0 .0a 6 t mfi 44on. 04 0 *0 o aC *@S SO W%* W% 0 qw0 SO SO ono

* -SoC60 on 4% .4a 5F 04l 00t a0 4_ ,

.0 V

U. A.1%41 Q ." a. 1% 1%q'. 1% W W re ' 1E0%M we

on on 0. .0. 0pU .4 n %D eC%.4 n9 0I 4.0 0 in.-U'%0%.l * 4 4

IOOW 0 ge * @ @ * O S 5

V4 ., 8

%D AM 44 N ~ 0 0% 4"% %Mo UQR-1. 90 "% n-0- o n o 0 0 on r.

Ad 00 54 sO SO e @0 0 g 0 5, 550 a * 6 00e 0 C g.0.

3 do 94C 500 oh 00 * S4 F S6%4 4W% 4W q o

0

W. 9 a. N 'a N .0 10 ..d OA 0 %a C N WNai 410 ~ ~ ~ ~ ~ ~ ~ ~ 4 cc do .4 *I 44f0 4t4oW.0A..m % A.s

C4 4" .0 In %D C 0 G 0-ee 4 , - w NoP%40S t g - No.0S b, 6N4 V44 04NC4C

* 45.66

* 6'.4 ~~ % ~ ~d %

N4 0 v* 0 0 No N.4 A c o"A 4 w "04. e c * * *c

%D M0 cO O SD %@a t% 0 M Se % 0 4 0 in6 cc a WNf % o

354

14 -WN# tI 4 0W Aaa00 cOCO 1" 00 . 5 S e @ .

Id 14W IN 0SS S0% M05 5500 * e 0 C4

IA0 0 a. C6s

-@ 0 @* a in 0 at in 0 m cc. *o cotR 1

A~4 A-.N004 iti4 b.E

a 0

W6 14N

I0 a

A 0, " "I A -3r4"4 b,0V40 NO~ c 04 eec IM in. 4S 4do0N @iN 0 M P 0

W! 3c9 - M~ 0% -0 Oe 4 io C - ------- V% ~ fnN *ceco~~ soc cc 00 eg ce .00

t4 uain V * wS t.q-.4 A .0 %0 W- 0101 0 0 0 0 0 0 0 I a *0 40V0 0 u e' of.. ul

00 'aO~ S4 M L Se co * 0

% 0 - d 40 ' V. g ~ C In - u '-o n n -.

* Sa A A b, n0e N N0 zW

014 a. IPN M M d @~IACh0 -10 M 61 N UN i%.4.N 44 MS N 0 -4-4 -0t4 a Cd '~ 0 0 0 0% 0.

ft M 0i

** 00 050 5 P' f 0 0 W S N M NOs N 00 4 04 0 IMi

102 * D M % * , 0w 1 4F at % 0 --4@. % cogo 0. %re P.~40 ~ O - .o'-e 00 i-4 P% C eDA 0 Ce4 I n n nn 4 %D 8 A T

ato .0 %0 D O L 0 teo c een a, ImC0 SW-e sq S4 4

44

.11 # IY4~VE .1 .1iC .1Vi g4.141i 14

o 6 00 0 0 1 2 00 a' oq oc S e a' %I.- P-P D Am1 -. I.

coo". M, -Ac cc a4cc. 0 M- S % P 05 0% C m 5*0 V

-49 4 o- 4-4O -- 4 4 -6 - C4 N N C4

-67-

2' -, -.

7 -T.. . 77.- . . - ': 7 -. -WIV

64 0 0 0 O a0 0 e0 0 0 0 @ 0 0 00 0 0 0 @0

a 0%~ C40?%60 N P%0 I wl 4at@n44 %a In @4 N 0 600 In -1 cm - 04in ,e .~N

0 m N4 -4. -4 -i 0004a N -4 4" on 4'" -4 en 0 4'4 N

0 * @0 0 @00* 0 0 00 0 0 0 0 a0 000a 0 00 0 0

a *"4

64 %D r .0 N4- 4 -4 % 0 ftP 0 (' 0 mD-e-0% -W m . cc i

* 0 0 0000 0 00 0 *000 0 @0 a @00 0 a0@ 0 0

64 0 1 0 00 0 0 00 00 000 @0 000 00 0

4.- . 4 f . km W 04 on ". at FC0 ' 60 00 N" MN%

N4 GA0 N % A %a N% 04t %a N N4 N N 4 N N so N%0- N 444

-0 44

I P P. -4 NN C CIA4 0 4

m 4 r 1 C@4Et N ~ -a1 4 % 4 % - 4r 0

0 0 00 00r - n4,0 @0C @0000 co 0 000 000"4 0~~ -0

%%~-0 a -4N "04 40 g' f - en4 m c c"'C4 c 0 soD ".m -. P

.% a~~~ a ~ W4 4'~ 40 N Go P44.44 N 1n n"-emP o NW P%4 N1 No 9'4 e*4 * * **0 0 00 00 .0 0 0 00 a 0 00 0 00 0 0

0444f-f mi WF-ft t- t0t. ," % %0 t4 4C "d

4400. CP NN ~ I i N N40 -W WN-4N'CNO% m@4 %D% tdoar ori

A -. No0 ** . 00 C"- o * 0% 000W .0 a 0 000 %0 ** -

*1 Q

W- &A

10 '9%. t

b.4~. IO~c qN"4o

.30 0U - U"4 a il 11%N'C@4o100b4N4Pti4@4.'O!Wnmf@ VitWw0

a0 v% 00 N 0 0I NFI ' O N 00 0 00 000 00 0 0 a 00 @0 00

M 0

A 0d a 0 0 0 0 0 0 0 0 0 0 d

0.14~O 04 In'4C 0(~0 r u m l0 ~ s f .0

W14 94 N 0 4 M0 4-t 00 a 0 0 % 00 0 N 0 00 0% 00 so0 @ 00 0 0 .6

O0l-.41. 0 %f %D0I 0 4- -4 10N0N0'C 4 ON -4 N g 0

44 . ,N "40 U0 N4 C" -a WC %0 Go 0@ 0 -.4@ N6M-.0 W&h~~~~~~~~d~C C4 ~ N 0 N 4 4 ~ O (~ N t~4C4

SN 00 S S S 0 000 0 0 0 0608

"4 ~~ ~ ~ P C)%il W .0W% %a 46 O ~ qt -4

*4 N- rIi 0 -0 %N 0 NO %0 00 -- 00 l -4 N A 0 4 4 0 MEliVi .4 Mt NM -~ in N N-O N l-NM C)Wi N MO N 4

0 ~ ~~~~~~~ 0 66 *6 66 6 6 6 * * 6 6MA. . fnWU.0 -0 00W eO Wr WcN4 f 00 -6 0% -M%4 0

%4 0 -4 O 0C'N 4" -fbN W W C" 00 WOf% 0 P. % PA "C, W IMiP C1@40 N' in I -t.4ri .0 4 i

0 6r N WVI N- V PN4 0 P. wO i mI P n44 ft"dim "Cl 446 -0 666 M6 N6 666 666 64 . 00M4 e DMMMN

wo as 00 @40P. ~ iNC- r%0%% 1-00P % U AInaO W % aM ANoa a. 0ga.a n NO -N oW Nfi cc0 W a a M%* D 0 %0 AD W

"4 F 9- @~O.4 M 4 -4%D 4 M V% N N P N 0C4W .EN % 0 Pd'0O

a. O' 0a .0 @'C .60 -"a g.0m %0 W, g g 0 %a W-0 %0I.at

a4 PN - -4O. N on MN0 M M-6 MM 0 in In m -

In P-I-:DV VN N On N ON Mi'O4U a.%*% , A5.D a6 % 4 lwW . 0

0A 4 M b, 6, V% CD.0 W '6 0 .0 %a 0 n-4W . %M 4 ~ 14 ba. a . 1.4 0 4 ul f1 W % O ~N WW 0b . 4W 4 O. 4 fi CvM6j 66 6 6 6 66 6 6 6 6 6 66 P6

In I

soa 44 0 4 4 4VMN1 0M 4 -1 6" in 0V1E4 i 0

0 9, @~04 00MV P0.OOO~COO ii at M * n C "b o000P .% 0%

Id'a ~ . - 06 No %%' 00 NC4l~ %D~i%Ii0 0% N4 @V iP'4 i@0 4 0 0 W .4Va W %ai -N0fi V

ad 66 to

604

04 0 a NN 0 0 0 rl461n.tiPi OO% W li 00% aO0DPn . 6W %D aO0 IS -4 M M C4 M 0b . 0PN4MiNOWiNiglM44ElNV40 86 66 66 *6 66 66 66 66 6 6 604 6a.. 0 h. U4 4 % nN- a0 ,i I t- 00 %

% O a. . 2 4k 0W% V Na - 0N. 0.

* ona MN 0MM% 1OOMNV%. 4MW . NNN

C)M Mdj Q 0 00@%l'O.i4~ 4f N L, d .NO i~ 'O. %aW% .M%O&V%94 M 0 00 N P N( i E i ( 1 U 4 N ViN %N( 1 N0

c~ N~ m N6 .4 66 w

a ,0Q 0 c l0 WDca.o D .G PP. %Dfl 04O Vii %0 rN4%D f- - N % a

N M" .~0 6 0i0 W ar oC 4N M .0 W, %a .Ci4 a N0 1"4U~~~~~~ N6 66 66 66 00 6 6 6 6 6

691

% SN V4' N Ei PV @ 0 .EiiiO % E 1 1IQ L L I p ., , - -c " . . *' % .

%D In b-%-4 % % 0 r % %I%4 Ap LN I~I% Or% No -$I W Mq AcaF% -* 4 4 N.0 "

0-4

In m f.0N%& 0 60 %a 0 at a N 6wM0 chF 05M @0 506 4 w

.! C4~~C-4OdCO'N'

o~Ii. atP 0 .

Wt4 WN . 4L nc .

W%.4..4

mo do .. C4 -A 'D 0, r . M at-4N

M 60 06 % 0a 5 5, m 56 L% Sn4 n- 0600- .0

4

3d go 00 00 % Data- N00 60 66w0 .0 P. 5 *a 0- nN 0W 0. N0. 0 - N -I.a

I0 ft P 0- . ,0 V

0.0 "4Na0%0 1

94 0..1~ 0 ' ~ O O'0 b%000 -0 05In. @0 0 0006A 0 Mi05

aa

0 inin-

%Dm- 04i-% V 4- W

a 4e t"- " 4c

4~ 0N -a SN W% 05 0% G6 5 o

."C" Wfn0 Nam 0 oND

* 4 P4

@4 d . @U%.4O ~ @.~.@4~efN70

so ong inx . mC1 w4 0. a 4 00 a0 0000

.4h O~~404E4* gjg44*

v44rr 04. %a 4C40W o

41o0 0lot0 00 . 0. .

Ch -4 laa

.4.w

-A4 .4" b, ,% n 44%IE 4On~ r 0m 0% in 0. %D4@ %D Wa % 4P 04

.U 0 ftSq. -a L, do0 4h0 50 00OM %-

0. 0.t 0 a a 0 P I.% S n G - a - W A 1

o 04 a G 0 " 0 0 NC4@ n 2004 C C40,fl

0 00 N 5- 4.4 0 NO0000 0N 0 5 4CI 00P0 930

"42 0 o

.4 C D %- at4"S% A4 4 0 0'0 oc -4k %%

0 00 0 6 0 090 0 0680 * . . .0 'I 00 0

8~ 001 6j:40 0

in0 %S 00 0m6 N0n

4w mub be 4,a 00a-c"%cebc

40..-u InE--~ 0% er- r -NoUDW

a O e 40 ~400% 4 '000M so W% M %D ,wo n 0 660 P%06000 00 %D C 00F 0 V-

-4C

31 S0.4 ', O 0~~0r.o

cm~~ %00 0 %a in0 %a ** %De .0 so m

C4t"-W W,%0r d ct - 4 " 0 , 0 N cCh 0 e

uI ~71

on 94 0% 0 4h( C4 A @ 'C 40 0,40 '" Sw 0@ 55 D5 Se. S %a@ @0 in A

V% 04 C

4:4 rq 4:~ Q!natiI 1%nI !0

0006 4 0660 00%5 W% V n% W 4i nr 0 %4.

64 . V! P.0CVi .4V1%i .1 4i~~~ Vi C ol

S3 i 0ft .400 =I .0 00 06 M P0 4 b0 4 %

14. 94 44q-V P% a. in0N m W o% oP h% 4 .

in SO 5 0 0 @0% a 00 "10

It In W%

.0C .W. s 4P..fig..d -6 .c04-

64 -4 -1 -404

A.N4 u. .0 ' 440 044 00 F%0,WA b 6 M 0

;Ri W ~~V f4'". at W%8G 4 0 n 0 @q

40 , co .0 05 s 5 0 m

.4 0 0 -6 C 0 A eqW e ~.ml.Uu a %a t 4

Go N 0 00 0 00 06 P% - 45V. C 0 0 "I

000at4 0%D % O4 'AOS Fl r% 000 W% % 0 0 0 10

"0 4 A ~ l P A a*Uoo -

b, L N @ 00 @00 055 -0 m 0W%5

I ~ ~ 1% *0@C4 V,a@ 000

* ml 'm

34 ~~ ~ ~ a a-SN.garr.4 a.

SOm-bll% SO d 6 SO W M5 0 WIN PSO 0%0

-72-

-" 4v M 0 N' W % VM0%MNV M-

A V a @~N9N InNZ I n:~~FU %.

64 0 qv b%. " 4 4. .A .B .0 .0 . . . . ....

N at M -a6.0"06660 N N0ft66060

a 0 l0

A g-4 N0060.0

041

14 IN V~P b, in o 4 4" 94 = 4 ilnt 91q4

A ~ M4 1 606 on.0 "66 60 "I

f itU 0 0 i 0 4 4 i .@ 1 I

ii0 N 1 ni I o

400

aa146 -40 hN C oW tW8e 2*@ .4 s0NorW n " W on M0W

I 060n 6.6 0 . .6.*; toi 61" oI 14lI 0 0 10 . 0 4. 0 0 5

I6 IC 00 t00 4M M .0CBor4 4g.q N .4 C4

6~ ~ 14 a 66 * 6 0 6o0e * 0 0 .0 01~64 a

*D bqF M%9. .0$I (4W ;g;_N45 .0 44 b " ' Co N ' 1 4

tiNN f S

0404 *4 S 0 N illO 9ilv! o lvil

0 N0 60mI i 0 6 6666o 060 66 6W

MIwi %a t-6.40 0 0 -0641- %D t% *%0 "o0-1 -4- 4- . 4-

- 73