DOCUMENT PESOSE

.ED 186 469 TM 800 174

AUTHOR Engelhardt, David F.TITLE Motivation and Test-Wiseness. Director's Handbook;

Topics in Testing, Measurement, and Evaluation,Volume V, Fall 1979.

INSTITUTION New Jersey State Dept. of Education, Trenton. Div. ofOperationse Research, and Evaluation.

PUB DATE 79NOTE 13p.

EDRS PRICE MF01/PC01 Plus Postage.DESCRIPTORS *Fducatienal Testing: Elementaty Secondary Education;

*Guessing (Tests$: Guidelines: Student Attitudes;*Student Motivation: Teacher Attitudes; TestingProbleWS: *Test,Wiseness

_IDENTIFIERS New Jersesy.4

ABSTRACTBecause motivational factors and test wiseness can

contaminate testing used for needs assessments or evaluationpurposes, techniques for increasing student and teacher motivationare discussed. Guidelines concerning 'guessing are also presented.While guessing' is encouraged on program evaluation cr screeningtests, it is not advocated for diagnostic tests: thus, differentscoring formulas may be appropriate fcr different applications oftesting. Instruction designed to increase student motivation and testwiseness is described, and is said to be potentially fruitful whenseveral situations are considered: the student's will to win,feelings of indisidual poworlessness, withdrawal due to previousfailure, purposeless testing, and teachers' negative attitudes.Suggestions to combat these problers irclude feedback to students and

parents. Advice is provided concerning several,issues in a testwiseness program: when students should guess: strategies fornorm-referenced, criteric,n-referenced, screening, and diagnostictests; and scoring formulas. It is concluded that students must betaught to pace their testing time, that item cOnstruction cues can be

taught, an'd that practice is useful. (GDC)

***********************************************************************Reproductions supplied by FDPS are the best that can be made

from +he original document.***********************************************************************

(g.

Aft

1

Introduction

c.

"PERMISSION TO REPRODUCE THISMATELUAL HAS BEEN GIIOITED SY

a agr 016TO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)."

_ EVALUATIONU.S DEPARTMENT OP HEALTH.

EDUCATION II WELFARENATIONAL INSTITUTE OP

EDUCATION

THIS DOCUMENT HAS BEEN REPRO-DUCED EXACTLY AS RECEIVED FRO"THE PERSON OR ORGANIZATION ORIOleATING IT POINTS OF VIEW OR OPINIOP.STATED DO NOT NECESSARILY REPRt'SENT OFFICIAL NATIONAL INSTITUTE Or-'EDUCATION POSITION OR POLICY

MOTIVATION AND TEST-WISENESS

. David F. Engelhardt

This paper cites instances where two variables, motivation and test-wiseness, can contaminate needs assessment and evaluation studies whenusing the types of tests most often given by districts in the T&E process. Al-though te,sts claim to measure attainment of skills labeled in the item specifica-tions, the scores often reflect variations in motivation or test-wiseness (thesevariables contribute to both invalidity and unreliableass of data).

The author suggests some methods to increase motivation of students bygerwrally increasing the chances of rewarding students and teachers. The)riic i,of test-wisenciss and the practice of test-wise behavior is advocated.

'A caution ainst guessing is g: for diagnostic tests, while guessing is ad-ymated on tests used for program evaluation-or screening (needs assessment).If a test is used for both diagnostic and program evaluation purposes, twoseparate seorMg procedures could be used.

Recommendations ore given with the spirit of reducing misclassificationsof able students as being in need of basic skill remediation, incecasing thevalidity of measurement, and increasing reliability of test scores. If studentsor non-random groups., are being compared, as is done by scores on norm-referenced tests or is done with New Jersey Educational Assessment tests,the elimination of contaminate measures (if test-wiseness is advocated asan important goal. To forget cultural or psvehological differences in motiva-tnui and test-wiseness ma% lead to gross inefficiency in our remedial andpreventie programs as well as incorrect evaluative conclusions.

Why don't test scores respond more easily to instructional effort? Canour kids really lark so many basic skills? Questions such as these can be askedof test results achieved by !:mne public schools as well as of certain programareas (e.g. grammar) in the curricula of smile elite, private 5(110015. Thequestions can apply to results produced through norm-referenced or standard-ized criterion-referenced testing. As part of a managyment team, test coor-dinators ruminate oveF their assessments and evaluations with eyes of anexa miner, .4 atistician and decision-maker. lid%ing.done this myself for sev er alYears. I began looking at the test from the eaminees* standpoint and concluded,in part, that we:

I)FALL 1979 Vohimn V

fl

(1) Have. failed if) motivate or reward many studentsso they desire to perform to the best of their abilities.and

(2) Have often failed to improve Me ability ofstudents to demonstrate their skills on tests.

The validity of our testing operations is affected bythe above characteristics of the examinees.

This paper deals with observations of good andpoor motivation and the need for test-taking in-struction which leads to what some call test-wiseness.Suggestions and references for remedial action aretr m so that a local school distrietwan launch its ownstudent motivation progcam if desired. Several .whooldistricts and at least one stat e have begun to developtest-wiseness in their students, often under the rubricof tea('hing study-skills which have lifetitne hem*.fit.These sch ( )1)1 districts include Philadelphia:Washington.D.C.: Chicago; Daily Ct4unty. Florida: and Mont-ginner% Coluity. Maryland. 1 um aware only of twofilm strip/cassette programs that aOress test-takingstiategies (Guidanre Associates of Pleasantville, N.J.and 1.amport .et al.. 1976). It is reeommended thatthese programs be previewed before purchasing. Books

sy hal n ri.com11101(Ied fOr Use with the Marylandprogram or others are: lionig (1973), llook (1-967).Huff (1961), Jongsma (1975), Millman and Pank0969). and Slakter et. al. (1979). AdditionalkErickson (1972), Ford (1973). and Millman et. al.(1965)outline the framew_ork around which a .frvi4k.:1test wisetiess program (inild hc construcied htealk.

The Will To Win

volirse, such test-wiseness programs ha%little tsortli !unless students %%i.,11 to do %%cll. Sonicscore adjustments up to chance leel of slICtl'SS. ran.be made tit partially vomiter poor motkation. Such:idjusttnents discussed later actuall% simulate lest-

wise 1)trila%ior. I him' scvn test-A% isc senior, in an ex.priute school fail to nerfonin on a Vissonri

Collegi. English Test. eell %%VI the headmaster urgingstudents to do their best. This helm% low was ,AltilthcolNen tInnigh students knew the headmaster was

-desiring to .legitimatck calitalc the school's Inl%\kelt:tines of English Program. Problems idobtainingOle beSi Vilork Of am school's students sVVIII in be-come more frequent as grade leei increases. 1% ith

proper teacher attitude ill III(' prinuir grades. studentsawn eagerk await the test almost as a oatne.FAidence of this eagerness is (Yencrallv lacking inscrundar Alidenk according. to in\ experience. Thereis a chance that Iest-wisencss instructional units maproe to interest some rebellious students tdio areintrigued I:\ Ihe idea of heating a s stem %shich hasheretofore "turnoi-illem.u11. Nonetheles,4. ttioik ;I-lion should and cati be addressed outside of the test-

wiseness unit.

One mightt conclude, students invariably must,see a reward RI the test's output for themselvesbefore mustering all their test-taking energies andwiseness. Let us consider a feiv situations affectingmotivation:

Feelings of Powerlessness

Especially prevalent in children of lower socio-economic elass is the feeling that no.malter what theydo, they have no power over what happens to them.Being st.thjeet to what appeav.as a capricious,n 'Oon-men!. slreh studetitsppose!I to many mid classchildren, do not seem to develop the attitude thateffort leads --to success and eventually to betterthings. Ei4mherg (1967) points out that middleclass children find reward within a test, feeling thatprogress in scores is the path to success. A professorof elementary science education said to me once,"Perhaps thc hest reason to teach elementary scienceis to show sOme children that their is order to ourworld, and that through their mind and actions theycan control part, of their environment."

A related and hopefully niore rare phenomenonran he encountered -when a student feels his/herdestiny is predetermined in a favorable sense, often'by social bias' (not ability). When ability is not 'adeterminer of destiny, the student egts on a"birthright" to reaeh his/her goal. Such an attitudeaffe.cts learning, but test taking situations may beeven more sensitive. Sinee this attitude may notaffect a ilong period of learning as strimgly as acontecntrated, sensitive request for demonstration ofskills, t 14e demonstration of skills may be moreseriously lumpered by the non-competitive attitude.

'Therefore. the (est a skills will not rewal the learningwhich ltas taken place despite the notiehalange of thestudent. Some solutions to this might be to:

(1) Make tests more _interesting.(2) Conrince the Ntudent of a e unpetit ire World.(3) Establish a .self-compoquirencss. an attitude often

mlopted by star atheletes.

Withdrawal Due to Previous Failure

Eisenberg( 1967.) confirm,: what many educatorslta%e seen in older children, if ehildren meet withrepeated failure. it IS Mitch more rewarding for themnot to compete at all. \lam children. especiallyminority students. withdraw from a testing ittiation

" sew, face"; In )t to trv and fail without II-% ingsVVIIIS better tban failing despite nne*S be:4i efforts.

nfortuttairk. the child does not discerningly choose%%here he/she 111011 Sneered it effort %sere musten.d.

Eisenberg found t t lower-class children were likely.to give any answer that would end the testing, re-

:,

tirdiess of whether the answer was right or wrong .

te child perceives that failure to exert efforttaints with vagueness ejny conclusions as to ability,thereby saving face by'having lowered the confidenceof statements about the examinee. Generalizing to oursituations, we might say, that the child has workedtoward minimum loss under avoidance strategies.Thc program's evaluation therefore suffers throughmeasurements of low confidence.

The solution seems obvious: in areas of basic-) skills and lime special talent areas, give the child

a taste of suecess. Appropriate level (out-of-grade)testing may help, but most important is the priorclassroom experience. It may be possible to enc )uragestudents to engage in test activities by giving them.successful.and recent experiences with material similarto the forthcoming test.

Purposeless testing, and teacher attitude:

Who in testing has not at sometime heard thecomplaints of teachers when examination time ap-proaches? Students are quick to absorb the sense ofpurposeless tel-taking where results are rarely usecl bythe teacherrwver shared with students.' It is wellrecognized by many test coordinators that teachers'attitudes and overt concerns regarding test results aremajor incentive factors in dudent performance. Ofcourse, the purposes and consequences of the testingshould be explained before testing -- not as an after-thought, which would have little effect on test-takingstrategy.

Convincing teachers and siudents of the purposeserved bv a test is not a small task. eurtIwrmore, theprocess may backfire in special eircumstatwes whereprogram evaluation or screening is a major purpose:It is possible that a vengeful 14tudent may capitalizeon such knowledp to attack a teacher, principal, orOw system. In another situation, revealing the purposeof Title I testing in a suburban New York arca scholdsystem lowered test scores because children wantedto qualif for the special summer program whichinvolved field trip:, games. and reading skill in-struction. Since New Jerse% has provided programsfor under-achieving gifted or talented students. it

wouldn't be surprisingif some gifted students mightti .;,ire to qualify for gifted and talented eompen-sator% education programs bv scoring low on achieve-ment tests. Such attempts van M. dampened It usingother criteria to verif% screening test ,.:cores.

\s long as it sstern Ine: health\ relationsbch+cen staff itnti students, and attrartkr programal term+ I i es, explaining purposes ot testing imd IIIa k i rigthe test purposeful at student and teacher Icelswill probalth increase scores. Some suggestitms are:

3

(I) Make sure teachers receive all interpretative_manuals end order time-saving reports fromcomputerised scoring services. Don't expect'hand laying.

(2) Involve staff in the selection of tests and provideopportanity for criticism of the tests and reportformat. With mandated ltests, allow teachers tohelp construct or declare certain items non-relevant as a district. Obviously this only per-tains in'a criterion-referenced approach.

(3) Teachers must be able to interpret test results.Unfortunately, central office staff rarely havetime to do justice to in-service or conferencesregarding test results. Unless administrators shiftpriorities to allotting more in-service trainingtime 16 test data analysis, we must rely on betterteacher training, clearer report formats, andteacher desire to self-educate. Building com-mittees may help educate' teachers with moreflexibility than district training.

(4) Increase feedback to parents and students witchcomputerized reports and teacher conferences;Purposefulness of any test wanes with delay inthe return of results. Feedback should be ascurrent as possible and in time for decisionspredicated on test results. Such feedback shiruldbe expedited by methods discussed elsewhere inthis Handbook. Pretend, as a test coordinator,that you are processing blood samples fordiagnostic work in a hospital. Schedule testing somails can work for you over weekends andlwlidayk Pre-correction processing might takeplace on a 20-hour work schedule. Ship resultsby air. The added cost is minor compared to theeffort and cost expended in testing.

_Of cours9, in-house or cooperative correction. cangive the best turn-around time on correction. Forcertain lestiy programs. self-scoring sheets orstudent scoring can in, utilized. Calculators andteaching ntachines can offer immediate feedbackwith a record for the teacher.

In LOng Branch. we have had the enwrience ofstudenIN calling our guidance department duringthe summer to .find out results of their PMnningCareer Coals Test Battery. The Battery containsguidance information matched with basic skills(ISSessment. 'This group of students has beentested as juniors in the late spring and in spiteof a ITIOlitYaillnally difficult time to test. thestmlents exhthited interest berfluNe Of chance.for personal benefit. However. we were notadequately staffed to reNp0Iiii thew sulnmerinquiries. 'If anything is In be learned by thisoccunince. it is .to uggest that provision forummer counselittg should be built in OS a .folbm-up 10 Npring teSting NI illilibIN to) increase test

r

performance. Under current regulations, suchfollow-up is not considered fundable undercompensatory education. Yet failure to providesuch foilow-up by funding a summer guidancecounselorprobabl.Nwillincreasethe compensatoryeducation load since students see less purpose inthe test. Relating test scores to career goals (evenif these goals are temporary appeals to manystudents in the upper grades; the appeal is muchgreater than can be aroused through threateningmore homelvork or a poor program evaluation.

(5) Although the main purpose of some testing maybe for program evaluation, stress some personaluse of test data for the student. Durost s (1974)indictment of some Title 1 test-taking strategiesdemonstrates that program evaluators must besensitive to high guessingand to poorly motivatedstudents. As suggested above, using guidancerelated test or establishing guidance related normsmay serve as an effective way to interest studentsin test restuls. Currently most commercial normingdoes not provide interesting norms for t he student.A secretarial student may be interested in howwell he/she does on a grammar section of a testin relation to other commercial students, not inrelation to all students in large cities or in thenortheast. On the other hand, the test coordinatorwould want the more general norm for assessment.Planning career Goa IS (CTB-Mf'Gra up IMallowssuch dual comparisons. When given the oppor-tunity to take extra sectimis of the test. 50% ofthe students'in.ontOistrict used their free timefor more testing.

Techniques used to reduce testing time bydividing items am) ng students may lower motiva-tion if students see no individual consequenceof taking the test. Such approaches as matrixsampling should he carefully monitored.

(6) Provide guides .for student interpretation to re-lieve teachers or guidance counselors of someinterpretation. The Pkuming Career Coals Testhas exemplary student materials. Most mherbatteries hare hand-out materials.

Provide instruct ional program respo Me to demon-strated need.

Include certain results of shoulardized or depart-mcnt al (Tit erion-referempj tests (1S port of astudeat's grade or as extra credit. kmbed certainevaluation or assessment it 4' Ills in normal 011.ss-room activities. Most lest numnal directions harestressed the reduction of an xiel y on (lie part ofstudents. Nssibly the pendulum bps strung Ion).far. many stud enis lack an.1' hint Of a II Xi ci y

4

some even sleep during assessment tests! Cer-tanily some standardized tests (e.g. MLA foreignksnguage tests, Howell Geometry Test, MissouriCollege English Test) are valid enough forparticular courses to p.iarrant credit be awardedtoward a student's grade.

Variety in the testing program:

Providing variety :n the testing program, which astudent erperienees, may have motivational conse-quene.es, although I am not aware of any formalstudies on this factor. Even if the standardizationprograms that generate norms for our tests :luffer frompossible fatigue and boredom (especially "alternateform" norming), it is probably not wise to try toduplicate such negative factors. Test Coordinators cantry to avoid examinee boredom by varying the testingapproach and series. I have heard counselors remarkmore than once, that after four or six years of thesame test, students just don't try to perform on thetest -- even if the questions are different (but of thesame style.) It seems that the title page of the test isenough to disuade some students from tryingieventhough the various levels in a series do have differentquestions. In. New Jersey, the minimum skills testmay provide a break from the yearly administrationof a test series. Some life-skill tests may be utilizedto good motivational-end in providing variety in adistriet's test schedule. Measurement of growth onone scale might have to he delayed ayear or two, butthis may increase the validity of the measurement.

Wade-Hoykin's research at Cornell Universityshows cultural differences in reaction to variety intest gtimuli. Perhaps unsuccessful students look for-ward to trying new test situations in which to provetheir abilities, whereas successful students look at"variety" as a threatening challenge.

11-

Teacher enthusiasm:

If teachers are c(mfident that they c1 r. reacha program improvement goa!, their enthusiasm todemonstrate such mav stimulate students to perform.Impossible program achievement koals may tend todampen teacher, enthusiasm, which leads to a poororientation of students towarA the lest. The nse ofshort-range goals with "front-line workers- is often abetter management technique than ri.vealing long-range goals, or, (Well worse, evaluating the workeron long-range goal 'standards. Teacher elmfidencealso can he increased by adequate pre-test orientation( )1 teachers, even if only -in the area of examiners''instructions. confused teacher during administratiopof the test also leads to a poor orientatn)Il of studentshmarel the test, confused teacher can hardly lwc.pecied Li ShOw 11111(11 cut husiasm for the test

which contributes to such unwanted insecurity orembarassment.

Test-Wiseness

Assuming that teachers and students desire toperfotin, the next concern for the test-coordinator isto allow pertinent skills to be evaluated or assessedwith reliability and validity. The Concern Is Not"ToMeasure Or Compare Non-Relevant Skills. If wewish to measure communication skills, let's nottneasure the ability of the students to pace them-selyes on the test or some other test-taking skill.

Certain criticisms levied against test-wisenesstraining often accuse the school of "teaching to thetest." I am not suggesting rehearsal of test questionsappearing verbatim on the test; nor am 1 iuggestingcramming for the content of a test. What I am suggestinghas been well established in the literature (Eakinset. al., 1976 ; Erickson, 1972; Fenton and Mueller; 1977;Ford, 1973; Maryland 1)( dartmeat of Eat-I-cation,1975; Millman et. al. 1965; and Sabers, 1975);the teaching and acquisition of test-wiseness -- theability to reliably demonstrate the full extent of one'spertinent skills and knowledges through the mediumof a valid test, including the demonstration of masteredand partially devekyed skills.

Without such training, many students who failto achieve minimum competency scores may possesssatisfactory skills in reading and mathematics. Suchfailures to wove competency contribute to the overloadon remed al services. Special funding is allotted toteach realling when. for sotne students, it might bebest to te:meh test-y% isetwss. Concern for eost-effeetive-ness should tirue_ileyelopment and implementation oftest-wiseness units and provision for good testingenvir(mments. The quest for valid instruments in theminimum competency movement may be severelyconfounded due to the variability in test -wisenessand testing en% iromnent (including teacher attitu(les).

Ford (1973) reports that "roaching" (learhingcontent area of the test and eramming) before testshas not been shown to raise scores as much as test-wise-!less stud v wind] avoided instrurtion in the subjectmatter to be tested (Eakins et. al... 1976). Forinstance, Barron's guide flow to Prepare for CollegeEntrance Exa in ina lions( Bro wnst ein and Weiner. l(M9)is --more a maching book as mmpared to Honig( 1973 ). lh mk (.1962), fluff (1961) and Millman (1969),Priyate schools and some public schools mighrhirFord's conclusions in-mind w hen constructing prepa a-I or% courses and selectityg appropriate materialsfor PS \I and SY1' examinations.

ith poorly- defined domains. it is difficult toascertain what an item intends to measure and what is

germane to test-wiseness; to teach the former iscoaching (if done specifically for the many domainsjust prior to 'testing) while to teach the latter iswhat concerns us now. Fenton and Mu.eller (1977)point out that to teach to the domain of the testis legitimate. If done well in advance of A batteryor long evaluation test, such teaching is the essence ofthe instructional program. No one advocates teachingspecific items td be found on the test. Sabers (1975)emphasizes that psychologists, through the AmericanPsychological Association, deemed it essential thatthe examinee be given the strategy to maximize his/hertest score. The following factors are considerationswhen devising a test-wiseness unit.

Should a st ident less on examinations? -

Perhaps the most significant and controversial facetof ttest-wiseness pertains to guessing on ultiplechoice tests. The author's opinion is that w houldnot shy from increasing the pace of students orurging the use of certain techniques so that all answA:rsare completed, even if this may result in some blatantguessing..In fact, some scoring formulas correct forleaving questions unanswered by adding to thenumber of right answers, the chance, score thatmight have been obtained by answering all unansweredquestions. This yields a corrected ram score whichis equal to or higher than the number right. If it isnot possible tot alter iresent correction formulas(due to econotes or inflexibility of either currentcomputer programs or standardized test correctionproce(1ures). physical alterations of answer sheetscan approach the same end. Such credit for un-answered questions gives each examinee his/her maxi-mum benefit from test-wiseness in guessing strategiesas if he/she were verv test-wise. Of course thismethod makes it very clear that assigning meaning toraw scores at or below chance level is an erroneousprocedure except when measuring tilt tendency toguess. Such a statement is true even when the "numberright" scoring formulas arc used.

If we are c(unparing studonts or non-randomgnwps. as is (1( we by scores on norm-referencedtests or with item perfoynance ro,dts on'the NeuJersey Educational sAssinent Peog-ram tests. itwoujd behome us to eliminate contaminant measur( ssuch as t est-w isc guessing.Gking credit for unatisweredquestions eliminates the penalty for students whowere too cautions. withdrew from mmpetition, weremistakcith cautioned not to guess. or exhibitedother unwise test behmiors. Stinlents not needingto guess or lia%ing greater partial knowledge will

.havc increased chances of being correct as comparedto pure guessers',* and will still obtain4 highereores (len though the knowledgeable examinees

ma% not haw their 'scores increased as much 11% the,abovcscoring correction.

I

5 G

1111%1 ose,)1.)111 01 1),11)0o1 S.)10.1s 1.4.)1 J 11119111.1.1

10110101r 'S.)%1!.) .)110): III .1s0.1 011!

'1),)1-I J.)10!111.10.11.01 III Iumil.).%.into\ $:,61 LiuI

ju minis lin -nnod 'anti ploy! 'as01 %al() str.1 111!oII1

11"11 }"s""i' (1 '61 ) Z11!1'")1 I MI!.1. .1" .s.).;1 pi) I

IMP 1s0111(1 111111!1-Is1.1 101.)1!J .10fr111 .rno ii

- !HUI 111(111s 1)111! ...)9!.) 0;1.01 01

ss.)11.)sli-1s.)1 .)7.1111!111111 1)1110i% 11 ;;.1.).st-:111! ).)111!i1.) 1110JJ

I9.)11.)(1 1011 111;1111 11!111 111.)I)111- 11.)11!.\!1010 1.10011

.1.)111110.) i11111.11!(1 Pitwm sin!, 1,4.9 0.1911.)

flft rl!!!101(1(11(1.1 ism!! !!! ig .).m111 .H1 iy!!)%% s(1110111 1111011 $4) suos1.1m11110.) 11!111 .)1(11!1.11!% 0s 1)01111.4

.)(1 111!.) ss.)1111 .)11.)101.11 JILL 1).)1).).)(1,4 1s.)1

JP) sso11111 sI1Ogs,)1111 HI! 11011.)1(Illl).) 1111011 9

.mt or.) (1).4)111.)111 .d1h)1 mg1 IillA(f) .M.11 s,110.4.1.111 I! Jo 110111

1.1111111 .1111 '1S.)1 )11Soll711!11)

1)./S11 .141 0) 1011 s1sal 211!!!mias !to

2.06 4st1rA:.1 pun puounim) III splapo)s pluis!!! psuf wst unit) dloui

I plii.Qpidts poi I utio J01 spitod .)Api Hint tiou.).11.1( I.) iy4J) .91:4011111i111 110 11!111

S111.111111S 0111 1S111 .)M JI 111111 :WM 01 1s.).1.)1111 .)(1

11 St11.1.)1 .11111os(lI! 111 .)10.)s 0111 .1t111.1 so011.s111.)11

patituto Joj snuo(l 'Au! ppr .;,s1110.) Jo .14.1.1MSLIV.iii10.1M

JO 1109.1011 Li 1.11!.111111S 11.1111M SII1111UJOJ 4.11011.1.)1.1( I.)

inuoptinui 1111M W.)1111.).1.10.1 1110111.10J Li 11.)/1s 11!111

1110 9ii!O(1 (2.61) .a.mp.h)uu 41u!.10.)..4 IundJ

Ii (I! aloas Mr.1 .)111 01 pap! .! 14110,-) aitanu.) A.!!

1h)Jam..;4n siontsur p.m!tuo JO tiopiodol(1 411091Mi1Ms) LI!

SU0SI.111d(110.) 111111) 4.1111.1111111s

.103 11.11411 S1SJ1 .111S0M1R111 J1 'A101A:1.1 11410.111

111111 .)10.1.1.ua9p pur suotpoub Iliitsstut tuolj 1tj.nrat1 :E.11.11S SS:ad 0) 11X/1114103113 :111 .1011

11111011S S11101111)44 11)g3,14 arpowlytu JO asto atu, UJ

suousanb Jay° papMati Jo

patuJtjuoa a9 0) -.):Hin($;) Ii Imq astiodsdi Autsstul 10

).);wo-aut !iv lipts pautjap Liana uo sliops.mb 3.10111

annli ,(1100S11 S)S3) TIS011`11111(1 11111,)1 JO M31A1.1 0) pax' Snw Appal )111IM 1.110.1J SOU10.) WaL13(1 `1S.11

1SOUM?1.11 11 sole) uagm s)sai pm! .)!Isoutntp uaamiaq tispl.thystp 01 iryipuj ulaag soop 1!

AAajeJis .1JMS1113 aintalo.uldn Pun !want ia9 tistatunsip ULUJI aaitirli opal 399oupetu pun

tuwatas totAt sa!gapu)s piaNstoaD Ala aAt oa

igmtprpunis Isom ytnt pazulln glinnua2 st qa!qm jo Jatmou '(fLo

'JattnAt Limn!) oui wain! JO (E2,6 I `tiRild)is RO? 1,1,61 6umlualii Ras) tutisal aattaptium) top% pail

oi Omani? iou 11!At jacInd spu, Jutssaut Jo stualgoid

ainuttutp unasatiwyt wapms 0) sayanwddr

pity tiontioalum. ley Palr)S NtiotIs 1! 'ampunf sly Dumas(' lursan9 atti, ptonv !lotto% gus

qnsts411opotosauterypoirtatogtidosonyja,Luani

suo9do uonas sant `oldturxo Joj (22 6 ) 1(JO)U0A

11 S.111P111.111111% .9S00111!!(1 Syr..1 's,)J0.)q 11(1 st.)%.)1

.1.)111i11.) .).)01,.).1 01 sII09(10 .1.)stslli! .10 10(11111111 00 01d111.) sisol .11110s 11:111 1),11011 .)(1 PI11011s 11 *(C2j)

111?111111111) 11111ss,10.11 (1101)111!.1 hl poI1s11(11110.).w .)(I

,).10.)s II O Nil(l IMO ;1111711).4.%001.)I! 110111 pity $4) 11011.4(11HO.) ;101.11111.) .(1 1101ss.)11.11 .).)11111.)

01101111 1).)II1I!ri s11110(1 11111%111!111).)

J(9 1.)s ,)(1 s.)!.11.)11%lis ;11,!Nri-IsJi

.1%.1.P.4.)1) 01 p4!.1111 )11.)1.)jpi) 11111 op !:11111 pun 4..41w) 01 simq.m,).1 I.)!tie pin?

(-1-161 .,;(10.wr 1)11,i1.4(1 )()J tvol ;him% 7111111!.)1()111 Joi Mold :4111 t111 s.i.).Ms111!

.1loyllsiL)III (1) 1).))(1Lpy .110111 UI IILi.) sp401

1).).)ii.u.9.).1-11o!.1.)11.JJ Jo 711O.10.)s Imp .111.1 1 s! It .)!!!!. ;,,r11) 111.).101111) .111) 110!).)1!.).1 111.)11111s 110

1.11!11

111 1).)sr(i (3,}1 sIs.)1 1).).)11.)1.)J.).1-111.1oll 11111! (nr.)) SIS.)1 1).).10.).1.9.).1-11(i1.1o11.1.) 110 posINI J.)!A!!!! ilstIL111,11 42'

-s11) (9 6 t 110-ii1,41 Pm! .i.opfuJ:) 4s,DiNt p(ln sjAr) 101 sa0a1n11s lusssant) lualainu aiatu, Jay

'111.11X.1 IIIMUIX11111 .1111 11.).1.1

SI! .)11s111?.1.1 .1011 s!! io!! si Ilti!ss.)101 J0 111.0 \,1

111111111111111 JIM 1.1JI!,1.1 0) .1.10.10.1./111 110SS.11171 111!111.)1!

. . S.M1111111!X.) .)1110s. I 11111.1.),)1111 .1.)1111111,V) .1111 s,) tIm1

01 1011 S111.41111s J.:1.yr 01 '.1.111N11111 I

pl.)!)!.!

-1,4),) 7ilth.;s.)11`,1 onms 11.).1.5%I.n! qs!t17III!is!!) mtm.st s.).1.%.1.1s 1109.).1.1103 1S.)) 1S0U1 01101111 .)11111111.11! 1011

S.111111011,01 11111.10.)s ..)JotwatiplIA i1'.)9.)11.1(1 01)!!!,iii s! 34) 00.61.)!)

p!!!I .a.)in!!!.) 1)1 1).911($!1)111 11.),1(1 dim! IOW 11111(1A1

.).1(I.)S .)111 JO land tio.! Popitio.).!! .01palm0to! .1!

111.11.1111.11S(11 )11.1111SSOSSI! 11.1AID A WI 110 .1.1111111J

01 .11111 4)1110d .1.10:1S .11111SS0d 7111111111111.1.1 all 1 71011)111!

X!! Poti!")(Io aJois t! pun .!!!!!!!! s.pinpms atif alo.)s Im.5.)!ti.m till toontimi amialajj!!! at!)

s! nut!! Jutssatd tuntupitut 5,11114)!A!!!!!! !Iv .1.1111i11.1 .111). JO pus!' tuntispertu asp 1).)1.)mtio.) ,H1

111110.1 .3.103s 'Llopsonl, (1,)e,) JOJ Jomstit! is.19 71(1!!1.)1.)er! pun ism .1111 Atiuntal )noto!m 1).).%.!!!!;) .)!!

pima .uo.)s titepo;) 's)s.9 aatoti-.) didtilnut 110 'Jas.! .p.)110).)0 11!tt SN.11 .fif) 4.i1M($11

ti) Jayittl :asuas into ut .).mpunlia .1(J)

tiltAt sotlen ituut mu .A1111111.10,1 Jutimis 010.191

Jo )5..)) ay JO ilowiduma )pull umulpanu sit -tpu.p ini.) .1M ing splriaJ suousimb Joj Aultilno.l(I 0101011?) Itlitssingt wanaid 01 itnaimp .0;pm:up:a s! ii

'WV 1 un nap.' OU y!At apputrlqo-spnaloatanuJopad Jo Jooll

imaap!stma J9 !ma aintut)sa =aux= sty 'spa) paauaaajim-uouintla illtAt Tam:Wm xi 1.1133 1.10913)14.1d

-Jaw! )Atm.inciluoa os :)putysa turnuponu 111

mislay* njj oiatqd puss uotpa.uoa amscp ay; aptitoid

-alqnsann Iitsssson2 snoamtapia asp Jo =maid mil saiouts sisintu.sal uotpa.1.10..) iipta-aaquinu atm 01

......

guessing in other experiment's (Diamond and Evans1973 and RoWley "and Raub, 1977). No cleaiempirical date of which the- author is aware showsthat increased guessing would make it hatder tofind significant gains in evaluation studies.

. Rowley. and Traub (1977) concluded that exam-inees having risk-taking personalities lenefit from "donot guess" directions, and that- most examineescan'not distinguish pure guessing from informedguessing. Numbers of 'studies have shown that scoresare higher when guessing or faster pacing is encouraged.When left to their own test taking strategies,, thepersonality characteristics of -students are stronginfluences over guessing (Cross and Frary, 1976;Ditimond and Evans 1973; Durost and Hodges 1974:Ford, 1973; Rowley and Traub, 1977; and, Sherman,1976). The tendency to guess (or not to respondwhen any doubt exists) is probably related tosocio-economic class and seems to be strongly associatedwith groups of. minority students (Sherman, 1976).It should be noted that Maryland's attempt to teachtest-wiseness arose from a suggestion froni an advisorycommittee on *minority relations in MontgomeryCounty.

Should We Urge Guessing When CorrectionFormulas Are Applied? Almost all references agreethat even with correction formulas being.applied forguessing, a person who can eliminate one wronganswer option from the remaining is legitimatelyrakina his/her score by guessing. When correctionformuelas are used (as with College Entrance Examina-tion Board Exams, but rarely with batteries used bydistricts), blind guessing (blackening in the spaces)wastes time -- time which might be better used inreasoning a difficult question. For groups testedwith correction formulas for guessing, averages rarelygo down even with blind guessing. Scores willobviously not go lower if a number right scoringsystem is used. Furthermore, Rowley and Traub(1977) present thesis data using ninth graders whichdemonstrate that 41% of answers claimed to be non-informed guessing were correct guesses, when 25',;would have been at the chance level. They corn hidethat students eaninq distinguish between informedand blatantly random guessing. This suggests thatmisjudgment on the part of a student does no harmwhen it results in guessing when correction formulas areusecl. Severe harm is done when a stmlent neglectsto ums on a test .orreeted b the number rightf(ortnnla. One should not urge blatant guessing whenguessing correction formulas are applied. Ihrwc%ernot much is lost if the student "over-genmilizes-from thi: normal ty pc of school exam.

Where Can I Find Additional Discussion on the

Guessing Question? Oprments to encouraging guessingand Duros and Hodges (1974), Lord (1075). imd

-

Sherman (1976). The reader may wish to* consultthese references prior to establishing a test-wisenessprogram. Thoile advocating guessingt especially withnumbef-rigfit corriction formulas and particidarlywith the possibility to eliminate one wrong answer,include Ford (1973) and Rowley and Traub (1977).All advocate test-wiseness instruction.

What Might One Conclude From The IdeasPresented In This Paper? In summary, on assessmentand evaluation tests, completion of the test does notseem to harm evaluative, group comparisons and mayeliminate some variance due to tesi-wiseness. Ifscoring formulas that add "chance poin ts to raw scores"cannot be found, mechanical blackening in of answersheets can accomplish the same adjustment. Such anadjustment at least puts all students in comparisongroups on the same footing in regard to guessing.Differences in scores will 'then be more heavily de-pendent on the variable intended to be measured. Thestandard deviation of group scores might bc reduced,whereas the effect on individual scores is contro-versial. Empirical studies do not seem to confirmtheoretical models which disregard "educated guessing."

Durost and Hodges' (1974) research desermsscrutiny beyond this palier since it contains dataquite generalizable to needs asfiessment operations inNew Jersey and also to an experimental' variableencouraging students to complete the screeninginstrument. Insightful comments on their researchreveal that test-wise behavior was lacking in manyNew Hampshire youths. with cautions stated aboutthe meaningfulness of much of the test data. Thepaper's data may need reworking before the readeris willing to accept Durost and Hodges' conclusionthat attempts to eliminate guessing would yield moreinformation on such non-diagnostic tests. Theirthoughts on criterion-referenced item /analysis andthe placement of items on a test in order of difficultyshould be carefully evaluateil. They conclude, as domany test manufactizrers, that mathematical cor-rection for guessing has no benefit in reducingguessing or reordering students in performance rank.It does accomplish lower snores in an absolute sense,but ma% also discriminate against certain personalitytypes.

(1) Students Must Be Taught To Pace. Thereappears to be some evidence that "speed reading" atest is a test-wise approach to demonstrating abilitiesto be rneasured. Miller and Weiss (/976) l'oundproviding lime limits on difficult items on tests didnot reduce accuracy. but did increw test takingspeed. The Maryland lest-wiseness syllabus (1975).trains students in adjusting pace according to the

, -44

,

type of sub-test (also see Hallberg, 1971). For non-fiction reading items, it recommends reading the

:,.,question and scanning for the answer.' So we see---t kit itpriroach can even vary by item type (Maryland

State Department of Education, 1975), *much asrefular reading or text-studying behavior& might be

justed for the type of book being read.

tudents shwild be urged not to become dis-turbed if they Ivannot answer every item withdefinite confidence (Sabers, 1975). A student willingto abandon certain questions can speed on to morequestions. Spatial relations tests are an excellentexample of tests on whic h a perm/liven become boggeddown on certain questions. It is not unknown to ex-perience a change of 70 percentile ranks when takingsuch a test. a second time and pacing oneself in amore cursory manner.

Two necessary ingredients for pacie.!7 r. ructionsare practice and being able _to have an cx.!ernal timeilteck. Classroom practice and yearly examinationscan provide test-wiseness to varying degrees in k-12,curricula. Practice on certain forms of items whichwill be encountered on a test does influence pacingmore than counting on generalization. Possibly, how-ever, even SAT scores may eventually begin to rise asstudents are provided more testing experiences undergeneral accountability pressures.

The second factor in /wing is being able to seesome _timing device and to calculate time intervals.The procedure of writing times on the chalkboardor announcing time left is thought by some to betoo disturbing (e.g. 26). .4 silent clock in the room.viewable by all, with finish time written on the board(by diagram for youngsters who can't tell time) is agood solution. It should be noted that many childrencannot afford wrist watches and their schools may nothave operable clocks. Are normative or evaluativeCOM pat*ons not made invalid by such problemsa.ssocinted with socio-economic conditions?

The recent trend to digital clocks may requirethat lest coordinators supplement firm, pieces intesting rooms. _Depending upon what fraction of aminute is viewed on a digital or impulse clock,an examinee's "minute" check mar be over in a fewseeon6. Furthermore. many digital clock .faces areless visible to students than classical, sweep secondwall docks.

(2) Students are Aware of Item Con-struction Cues. Certain groups of teachers (111(1 student smay perpetuate a rwirete on item writing cues because

one f r bothered teaching them how to take atest or how to write items. Diamond et. al. (1976) and

et. al. (1965) deal with test-wiseness

8

-a

construction. One- may wonder how test constructorscan coMmit item writing errors so blatant that such test-wiseness instruction can be beneficial. Districts shouldcarefully assets the worth of teaching how to spotincorrect or correct answers by such factors as lengthof option, matched graphemes in stem and answerand use of ungrammatical alternatives. It is true thatsome test publishers and many teachers do stillcommit such errors.

(3) Provide Psychomotor PractIce for Some TestAnswer Sheets and Students. With some answer sheets,young students can benefit from practice in recordingscore (Sabers, 1975). Some testlyiseness programsprovide "answer entry" practice for several daysprior to testing (Maryland State Department ofEducation, 1975). Test coordinators should chooseanswer recording format and correction serviceswith care. Extra money 'spent in assessing can saveunnecessary remedial expenditures.

Some have discovered thai the rewriting ofhorizontal math problems poses a test of smallmuscle coordinatioa rather than math competency.If conclusions draWn from assessments might beconnected with psychomotor preparation, notationsduring testing* or during test-wiseness preparationcould be used during item analyi is to increase theappropriateness of instruction. llopeftlly,test-wisenessinstruction could have a psychomotor componentto increase the validity of screening tests followingsuch instruction.

(4) Practice With Format of Items and Allow ,

Students to cultivate Familiarity with Directions. Thestudent who understands diretions wt,11 and does nothave to refer to directions during a timed test has a de-finite advantage over the student unfamiliar with item

. format and the directions. Some types of questionsare so involved that weeks of practice are needed.Practice embedded in the normal instructional routineis more efficient than specific units in formatpractire (Eakins, et al., 1976; Ford, 1973: .and,lhiryland State Department of Mucation 1975).

(5) Vrge Students to Check Answers. Althoughunsure guesses mar generally be correct on the firsttry. reasoned answers have proved more Jorrec't whencorrected or checked once. Co ntimuil 'change ofanswers is dangerous. Ford (1973) warns againstpondering over a question at length.

(6) Demonstrate That Eliminating at Least OneAnswer Helps Raise Scares. Ford (1973) recotnmendsreading all options before deciding on one Option.If time is limited. locating at least one wrong optionaml then choosing what looks like a good answer mayprove more efficient. Once again, blank answers or"don't knows** are not useful in i he author's opinion. ,

.10.0

""

Reaction To This Paper

Not all existing programs, not all pertitientjournal articles, have been reviewed for this paper.It is hoped that readers will reply with added sugges-tions for further development of issues discussed here.A-.4eswiseness program developed with suggestionscontained in this paper will do no harm to studentscores, nor will it confound evaluation firactices. Wewill, however, make an effort to cimtinuedisseminationof ,possthle improvements to tests-wiseness programs.

U'e hope some response ;s forthcoming fromtest publishers to concerns expressed in the paper,

wcially in orevising test correction formulas so(list .ts might seieut appropriate te(:hniques.

)tir aim in suggesting test-wiseness courses is toimprow the measurement process, not destroy it.

About the Author....

David F. Engkhardt is Assiaant Superintendentof Schools for the Long Branch Public Schools,.LongBranch, Ntew Jersey. He received his B.S. degree fromYale University, an M.A.T. from Duke University and'an M.D. from Harvard Graduate School of Education.

His areas of specialization are measurement andprogram evaluation, research utilization, and long-range educational planning. Prior to his *presentpositthn he .has worked with the MassachusettsDepartment of Education, served as an educationalconsultant working with -clients on the easternseaboard and has taught biology.

In addition to being -a member of *numerousprofessional organizations including AASA, NJASA,Phi Delta Kappa, AERA and NCME

Small editing correctionshave been made in this copyby the author. Commentsfor the next revision maybe sent to his home;

50 Rosalyn CourtLong Branch, N. J. 07740

9 .

111

. .

References

Brennan, Robert t. The Evaluation of Mastery Test Items. NCERD Project: Pint Report. EdUcationalResources Information Center,(ERIC), ED1092,593, 1974.

".;-.

Brownstein, S. C. & Weiner, M. Barron's How to Prepare for College Entrance Examinations. Woodbury,New York: Barron's Educational'Series, Inc., 1969..

Crehan, K. D. et al. 1977. Developmental Aspects of Test-Witieness. Paper presented at the New Yorkmeeting of the American Educational Research Association,April, 1977. Educational ResourcesInform:Ilion Center (ERIC), ED 137,394, 1977.

Crocker: L. & Benson, J.--I976. Achievement, Guessing, and Risk-taking Behavior Under Norm Referencedand Criterion Referenced Testing Conditions. American Education Research Journal, Summer1976, 13, 20?-215.

Cross, L. & Frary, R.B. A Study of Omitted Responses Under the Conventional Correction for Guessing.Paper presented at the San Francisco meeting of the American Educational Research Asiociation.April, 1976. Educational Resources Information Center (ERIC), ED 120; 231, 1976.

Diamond, J. J. et al. 19 v. Are Inner City Children Test-wise? Journal of Educational Measurement, Spring,1976, 14,-39-45.

Diamond, .1. & Evans, W. The Correction for Guessing. Review of Educational Research, Spring, 1973, 43,181-191;

Hallberg. Al. kacing the Test: English. Miami. Florida: Dade County Public Schools. Educational ResourcesInformation Center (ERIC), ED 064720, 1971.

flonig, F. 1973. Taking Tests and Scoring High: How to Study for Tests nd Pass Them. New York: ARCO,219. Park Avenue, 1973.

llook,J.N. 1967 Testmanship: Seven avs to Raise Your Examination Grades. New York: Barnes and_ _

Noble, Inc., 1967.

Iluff, D. Score: The Strati.,,v of Taking Tests. New York: Appleton - Century - Crofts, Inc., 1961,

Jacobs, S:s. Behavior on Objective Tests Under Theoretically Adequate, Inadequate. and UnspecifiedScoring Rules. Paper presented at the (Reap) meeting 4r-fE YmerreanNssociationApril, 1974. Educational Resources Information Cent(r (ERIC), El) 006312, 1974.

Jongsma. The Effects of Instruction in rrest-takino Skills 1 pon Student Performance on Standardized. .Achiewnient Tests: Final Report. Baton Rouge. I,a.: Loutsiana State Universitv Foum 1011.

_ _ .

Educational Resources information Center (ERI( ), El) 114.108. 1975.

klaas. A.C. Reassessineut_of Standard Error of Measurement. Paper presented zit the \ashingion. D.C.meeting of the American Educational Research Associatimi. pril. I 977). Edurational ResourcesWormatnm Center (ERR). El) 104949.1975.

11:Import. J.. Perle. U. Reinarch. J. Taking Tests and Rela\ing. Film strip and cassette programs. Norwalk.Conn.: \ ev. Dimensions in Education. Inc.. 83 heeler Avenue. 06854. 1976.

-10 -

1 ,t

t..

so

Lord, F,M. Formula Scoring and Number - Right Scoring. Journal of Educational Measurement, Spring,1975, 1,2, 7-11.,

ary land State Department of Education. Improving Student Attitudes and Skills for Taking Tests. OecapionalPaper Series on Accountability, AufliWTaueational Resources-ri .1)--i-ration-renter (ERIC), Elf-128352, 1975.

Miller, T.W. & Weiss, D.J. 1976. Effects of Time Limits on Test Taking Behavior. Office of Naval Research(;rant., University of Minnesota: Office of Naval Research Grant. Educational Resources InformatioaCenter (ERIC), El) 125123, 1976.

Millman, J. Passing Scores and Test Lengths for Domain-Referenced Measures. Review of Educational ,

Research, Spring, 1973, 43, 205-216:

Millman..1.: Bishop. C. & Ebel, R. An Ana,lysi-: of Tr st-Wisencss. Educational and Psychblogleal Measurement1965, 25. 707-726.

Millman, & Pauk, W. How To Take Tests. New York: McGraw Hill Book Cmpany, 1969.

Moore, J. Schultz, R. & Bakr, R. 1966. The Application of a Self-Instructional Technique to Devlop aTest-taking Technique. American Educational Research Journal, 1966, 3 13-17.

New York State Education Department. SPPED cloze Training yanual, Form C82. Educational ResourcesInformation Center (EA1C), El) 127353, t 976.

Oakland. T. Effects of test-Wiseness Mated us on Standardized Test Performance of Pre-school DisadvantagedChiblren. Journal of School Psveholoav 1972. 10. 355-360.

Rowley, G.14. & Traub, R .E. Formula Seoring. Nundwr U ight Scoring, and Test-taking Strategy. Journal oEducational Measurement. Spring, 1977, 14,. 15-22.

-Sabers, D. Test-taking Skills. Tucson. Arizona: Arizona Center for Educational Research and Development.(:( dirge of Education, Lni%ersil% of Arizona. Educational Resources Information Center (ER IC).ED 1333-11. 1975.

Sherman. S.\\ . 1976. Multiple (hoice Test Bias hiemenll l I se of an -1.1)on:t know- Althernatke. Paper.

presented at the San Francisco meeting of the \ mcrican Educational Research Association. \ pril.1976. Educational Resource, Information (enter (ERIC). El) 12182.1.19M.

Shakier. NI., Koehler. h.. N Iloiriptuti. S. 1970. Learning- 'Fest -1\ iscness IA programmed te1,..lournal of1:,ducaliunal Nleasurnment. 1970. I. 217-251,

Stegman. Subjeetke Probabilik .and the \11mmiAriumn 111( )bjectkr Paper presented at theNtn-theastern Eduridnund Research tssuriatiun Cun%(watiun. I" ;. Educational lles(ntruesInformation Center ( I(:). El) 0911.27. I 973.

Uald-trom N V. The Influence pun \chie%ement. Educational andMea,nrement. I 968. 28, 113 120.

\1.1. f Bandwil I ;10..,..int, riutri I t I,,i \ Iiit 1.....timate-. 121.S lit-wan-II

lin, 1971. 7. 1-:i2. I Information (.1.nici (1...1: I(). El ) 0991IT. 1()-1.

;

PUIMICATION$ IN TIIIS'SERIES INCLUDE:

V4)11111111 I. Plirpose 131 Test ing David W. Pearsall

Ahdeolin .1. ConwayV1)11111111 II. TesI Selection and l'Aalualion or Tests

Organizing and Nlanaging aDist rid Xesling Ii ()gram /lobed L. Jackson, 11

\ olunin I Siu ll%I I Ises or StandardizedTest Results Eugene J. Travers

olunui V. Motivatiolt and Test \ iseness David F. Englehardt

Published b the Neu, Jersey Department of Education, 22.5 West Slate Street,Trenton, New Jersey O8b25. The opinion's e%pressed in this publicaition arethose of the invited authors and do not necessarily reflect the position or policyof the Aviv Jerwy Depart/111,dt of Education.

ACKNOWLEDGMENTS

Thauks are etriuted to the Planning Committtx for this publication and to theeditorial re% iess paticl vonsisting of Dr. Curtis Ranks, Prineeton l,niwrsitsDr. Douglas Pcntield, Rutgers UnisersiLy : and, Dr. Paul Vali It. Miller, LehighUniversits .

Appreciation' is also epressed to Ms. \ isian tiohke for preparation of themanuscripts.