how reliable and valid is the japanese language...
TRANSCRIPT
![Page 1: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/1.jpg)
3232
JALT JournAL
How Reliable and Valid is the Japanese Version of the Strategy Inventory for Language Learning (SILL)?
Gordon RobsonShowa Women’s UniversityHideko MidorikawaShowa Women’s University
ThisstudylooksattheinternalreliabilityoftheStrategyInventoryforLanguageLearning(Oxford,�990),usingtheESL/EFLversioninJapanesetranslation.Theresultsof theCronbach’salphaanalysis indicateahighdegreeof reliabilityfortheoverallquestionnaire,butlesssoforthesixsubsections.Moreover,thetest-retestcorrelationsforthetwoadministrationsareextremelylowwithanaveragesharedvarianceof�9.5percentattheitemleveland25.5percentatthesubsectionlevel.Inaddition,theconstructvalidityoftheSILLwasexaminedusingexploratoryfactoranalysis.While theSILLclaimstobemeasuringsixtypesof strategies, the two factor analyses include asmany as �5 factors.Moreover,anattempttofit thetwoadministrationsintoasix-factorsolutionresultsinadisorganizedscatteringofthequestionnaireitems.Finally,interviewswithparticipatingstudentsraisedquestionsabouttheabilityofparticipantstounderstand themetalanguageused in thequestionnaire aswell as theappropriatenessof some items fora JapaneseandEFL setting.TheauthorsconcludethatdespitethepopularityoftheSILL,useandinterpretationofitsresultsareproblematic.
本研究は、Oxford(1990)の外国語学習ストラテジー・インベントリー(SILL)のEFL/ESL用日本語版の内部信頼性及び構成概念妥当性を実験と統計によって検証したものである。クロンバック・アルファ検定による内部信頼性については、インベントリーの全項目は全体としては信頼性が高かったが、6タイプのサブカテゴリーに分類されたストラテジーについては信頼性が低かった。また、インベントリーを用いたテスト・再テストの相関は低く、全項目では平均寄与率19.5パーセント、サブカテゴリーでは25.5パーセントであった。構成概念妥当性検定のための説明的因子分析の結果は、6タイプのストラテジーが15因子に細分化されたこと、さらに、全項目を6因子に分けた結果、それぞれの因子が無秩序に分類される結果となった。最後に、インタビューによって、この実験に参加した被験者学生にインベントリーの各項目の内容理解について確認した結果、日本語がわかりにくく判断しいくい記述、日本のEFLの状況では理解しにくい記述があることが明らかになった。以上のすべてから、SILLの実用的評価にもかかわらず、それを用いること、また、そこから得た結果の解釈には問題が含まれているというのが、本研究の研究者が得た結論である。
![Page 2: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/2.jpg)
3333robson & MidorikAwA
Theuseofself-reportinstrumentstoinvestigatevariousaspectsofindividuallearnerdifferencesisacommonandacceptedprac-tice in thefieldofsecondlanguageacquisitionresearch.Asa
consequence,alargenumberofsuchinstrumentshavebeendevelopedandusedover theyears.These include theAttitude/MotivationTestBattery(A/MTB)(GardnerandLambert,�972),theForeignLanguageClassroomAnxietyScale(FLCAS)(Horwitz,Horwitz,andCope,�986),andtheinstrumentunderdiscussionhere,theStrategyInventoryforLanguageLearning(SILL)(Oxford,�990).However,despitethewideacceptanceanduseoftheseinstruments,issuessuchastheirreliabilityandvalidityareoftenlostintheenthusiasmtofindoutwhatstudentsreallyfeelorbelieve.Althoughagiveninstrumentmayhavebeenrigor-ouslydevelopedandevensubjectedtovariousmeasuresofreliabilityandvalidity,whenitistranslatedintoanotherlanguageorusedinaculturalsettingdifferentfromtheoneoriginallyintended,itmustonceagainberigorouslyexamined,assuggestedbyGriffee(�999).
Thisreportwillpresenttheinitialresultsoftheresearchers’attemptstoprovidereliabilityandvaliditydataontheSILLinaJapaneseuniversitysetting.Thisstudyisgroundedintheresearchers’numerousotherat-temptstovalidateotherJapanesetranslationsofmeasuresofindividuallearnerdifferences,suchasmotivation,anxiety,learningstyles,learningbeliefs,andlearningstrategies.ReliabilityistypicallymeasuredthroughstatisticssuchasCronbach’salphaormultipleadministrationsofatestwiththesamesubjects,bothofwhichareusedhere.Regardingvalidity,althoughinthepastothermethodsofvalidatinghavebeenputforward,recentlyChapelle(�994)andMessick(�989)havepersuasivelyarguedforvaliditytobecondensedintoasingle,generalapproachwherethefocusisontheinstrumentasaconstruct.Asthemeasurestypicallyusedinthistypeofresearchhavebeenself-reportquestionnairesinwhichitemsweregroupedintocategoriesorsubscales,researchershavefa-voredfactoranalyticvalidationforthevariousgroupingsorcategoriesassignedtothequestionnaireitems.Theuseoffactoranalysistoconfirmatheorizedgroupingofitemsisalong-establishedpractice(Guilford&Fruchter,�973),especiallyinthefieldofpersonalityresearch,whereithasbeenusedtovalidateself-reportquestionnairesforover50years(seeforexampleAllport,�937;Guilford,�940;McCrae,�989).Therefore,thiswillbetheapproachtakeninvalidatingthesixgroupsofstrategiesmakinguptheSILL.
What is Reliability and Validity?
Therearevariousapproachestotestingandconfirmingthereliability
![Page 3: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/3.jpg)
3434
JALT JournAL
andvalidityofagivenresearchinstrument.Wecandefinethereliabilityastheproportionofthevariationintestscoresthatistruevariationandnoterror(Bachman,�990;Brown,�988).Typically,whenmeasuringreliability,theitemsonthequestionnairearesubjectedtooneormoretypesofstatisticalmeasurement.ThemostcommonlyemployedstatisticisCronbach’salpha,whichmeasurestheinternalconsistencyofatest.Anotherapproachistoobtainsimplecorrelationsbetweentestitemsofameasurethatisgiventothesamepopulationtwoormoretimes.Thisisreferredtoastest-retestreliability.
Ingeneral,thevalidationofaself-reportinstrumentismuchmoredifficultandinthepastinvolvedseveraldifferenttypesofvaliditysuchasfacevalidity,contentvalidity,constructvalidity,factoranalyticvalid-ity,andcriterion-relatedvalidity.However,inaninsightfularticleMes-sick(�989)pointsoutthatwhileitisimportanttovalidatethemethodofdatacollection,themorecrucialareaistovalidatetheinferences,interpretations,andactionstakenbasedonthescoresderivedfromthedata.Moreover,Chapelle(�994)arguesthat“constructvalidityiscentraltoallfacetsofvalidityinquiry,”andasanongoingprocess,thereisnoonceandforevervalidity(p.�6�).
Fromastatisticalpointofview,thereareseveralwaystoconfirmtheconstructvalidityofaninstrument.Theuseofcorrelationapproachesand factor analysishasbeennotedpreviously. Typically these ap-proachesinvolveusingseveraltestsorquestionnairesthatarebelievedtorepresentaconstruct,suchaslanguage-learningstrategies,andthenconfirmingthevalidityof the itemsthroughhighcorrelations. If thecorrelationsarehighenough,thenwecaninferthattheymeasurethehypothesizedconstruct.Factoranalysiscanbeusedwhenmeasuresforseveraldifferentconstructsarebeingused,suchasformotivation,strategies,andpersonality.Subsequently,theirloadingsondistinctfac-torsconfirmthattheymeasureseparateaspectsoflearnerbehavior.Aseconduseoffactoranalysisistobreakameasureintosubgroupings,suchasthesixhypothesizedpartsoftheSILL,andthenfactorthemtoseeifthesedivisionsarevalid.Evidenceofameasure’svaliditycanalsobeconfirmedexperimentallyorquasi-experimentallythroughrelatedoutcomesusing,forexample,ameasureoflanguagelearningstrategiesandscoresonsomemeasureoflanguagelearningsuchastheTOEFLTest.Thiswouldindicatenotonlythatthemeasurewasvalidlymeasur-ingstrategies,butalsothatsuchstrategieswereuseful.
![Page 4: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/4.jpg)
3535robson & MidorikAwA
The Strategy Inventory for Language Learning (SILL)
TheStrategy Inventory for LanguageLearning (SILL) is a self-reportquestionnaire for determining the frequencyof language learningstrategyuse.Itconsistsof50itemswithfiveLikert-scaleresponsesofneveroralmostnevertrueofme,generallynottrueofme,somewhattrueofme,generallytrueofme,alwaysoralmostalwaystrueofme.Basedonafactoranalysisofanearlier,largerversion,OxfordorganizedtheSILLintosixstrategysubscales:(a)MemoryStrategies(9items),(b)CognitiveStrategies(�4items),(c)CompensationStrategies(6items),(d)MetacognitiveStrategies(9items),(e)AffectiveStrategies(6items),and(f)SocialStrategies(6items).ThequestionnairewastranslatedintoJapaneseaspartoftheJapaneselanguageversion(Oxford,�990/�994)ofOxford’s(�990)LanguageLearningStrategies:WhatEveryTeacherShouldKnow.AlthoughOxforddoesnotdirectlydiscusstheprocessforestablishingthereliabilityandvalidityoftheSILL,anotetoChapterSixexplainsthatanearlier,�2�-itemversionoftheSILLwasfoundtohaveareliabilityof.96basedona�,200-personsampleand.95witha483-personsample.Shethengoesontostatethatthereliabilityof9of�0factorswasfoundtobemoderatetohighwithfiguresof.60to.86,althoughforthe�0thfactoritwasonly.3�.Thisisnotthetypicalwayofreportingtheresultsoffactoranalysis,andifthe�2�-itemversionwasclaimingtomeasuresixstrategytypes,thena�0-factorsolutionishardlyconfirmation.Thenotegoesontostatethatthefifty-itemversion7.0oftheSILLunderdiscussionherewasstillbeingassessedforreli-abilityandvalidity.Thus,whileitwouldseemthatthevariousversionsoftheSILLhaveaprovenlevelofreliability,thisdoesnotsuggestthatthequestionnaireisvalid.AsBachman(�990)hasstated,エheprimaryconcernintestdevelopmentanduseisdemonstratingnotonlythattestscoresarereliable,butthattheinterpretationsanduseswemakeoftestscoresarevalid(p.237).IfatthispointintheSILL’sconstructionitwerefoundtobeunreliable,therewouldbenoneedtoproceed,asanunreliablemeasureissimilarlynotvalid.
Oxford(�996)discussesthepsychometricqualitiesoftheSILL,andin termsof reliability, shecitesWatanabe (�990),wherea JapaneseversionoftheSILLachievedaCronbach’salphareliabilityof.92,andotherstudieswithsimilarreliabilitiesinthe.90range.Followingtheabove-mentionedMessick(�989)andChapelle(�994)approachtotestvalidity,OxfordexaminedanumberofstudieswheretheSILLcorrelatedsignificantlywithvariousmeasuresof language learning. InOxford,Park-Oh,Ito,andSumrall(�993),amultiple-regressionanalysisfoundlowbutsignificantpredictiverelationshipsbetweenstrategiesandfinal
![Page 5: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/5.jpg)
3636
JALT JournAL
testgrades(.20).Takeuchi(�993),alsousingmultiple-regressionanalysiswithlanguageachievementasmeasuredbytheComprehensiveEnglishLanguageTest (CELT), found that fourSILL items(�7,2�,22and32)positivelypredictedlanguageachievementwhilefouritems(6,30,43and49)negativelypredictedlanguagesuccess.Finally,Watanabe(�990)foundlowcorrelationsbetweenSILLitemsandstudents’self-ratingsoftheirownproficiency.Althoughtheseresultsprovidesomemeasureofvalidation,onlyafewSILLitemsareinvolved,andthecorrelationsareextremelylow.
InBrown,Robson,andRosenkjar(�996)anindependentlytranslatedversionoftheSILLwasusedinamultiple,individuallearnerdifferencesstudy.Theoverallreliabilityofthattranslatedversionwas.94withthereliability for the six strategy typesbeing .74 forMemoryStrategies,.84 forCognitiveStrategies, .69 forCompensationStrategies, .88 forMetacognitiveStrategies,.63forAffectiveStrategies,and.73forSocialStrategies.ThefactoranalysisintheBrownetal.(�996)studywasonlyusedtodetermineiftheSILLwasmeasuringsomethingdistinctfromtheothermeasuresofsuchvariablesaspersonality,anxiety,andmo-tivation.Thesixstrategytypeswerefoundtoloadonasinglefactor,whichconfirmedthattheSILLwasmeasuringavariabledistinctfromtheother instruments.The researchersknowofnootherpublishedstudythathasattemptedtoestablisheitherreliabilityorvalidityinthismannerusingaJapaneseversionoftheSILL.
However,atTESOL2000inVancouver,Canada,HsiaoandOxford(2000)presentedtheresultsofamulti-groupconfirmatoryfactoranalysisforan80-itemSILL.Thefactoranalysisplacedonly�7itemsintothesixhypothesizedgroupings,leaving63itemswithnorelationtothesixstrategycategorieshypothesized.The�7itemswereMemoryStrategies(4,5,8),CognitiveStrategies(26,27,28),CompensationStrategies(4�,43),MetacognitiveStrategies(49,53,55),AffectiveStrategies(66,68,69),andSocialStrategies(72,73,74).Ofthese,onlyitems5,27,28,68,and72arethesameasorsimilartoitemsonthe50-questionversionoftheSILLunderstudyhere.
Tosummarize,theSILLappearstoenjoyahighdegreeofreliabilityin its variousversions and the languages inwhich ithasbeenem-ployed.However,thereliabilityhasbeenfortheSILLasawhole,withtheexceptionofBrownetal.(�996),whereseveralofthescaleswereratherlow.Thisstillleavesthequestionofvalidity,whichbasedonthesourcesdiscussedseemsfarfromestablished,andhasledustoaskthefollowingresearchquestions.
![Page 6: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/6.jpg)
3737robson & MidorikAwA
Research Questions
�.HowreliableistheJapaneselanguageversionoftheSILLforJapaneseuniversitystudents?
2.TowhatdegreeistheJapaneselanguageversionoftheSILLvalidforJapaneseuniversitystudents?
Method
Thepresent study isbasedon twoadministrationsof theofficiallytranslatedSILL(Oxford�990/�994)tothesamegroupof�53Japaneseuniversitystudents.Thegroupwascomprisedof��0first-andsecond-yearfemalesand43first-andsecond-yearmalesstudyingataprivatewomen’suniversityandaprivatecoeducationaluniversityinTokyo.TheirEnglishproficiency levelwasapproximately lowintermediate.Thefirstadministrationwasconductedatthebeginningofthespringsemester.Asecondadministrationwasconductedduringthebeginningofthefallsemesterusingaversioninwhichtheorderoftheitemshadbeenrandomized.Therewerenochangesinthemakeupofthegroupofsubjectsforthetwoadministrationsofthequestionnaire.Inaddi-tion,post-administrationinterviewswereconductedwithtenrandomlyselectedstudents,fourmalesandsixfemalestogetfeedbackonwhatthestudentsthoughtaboutthequestionnaire.Theinterviewswerecon-ductedindividuallyinJapanesebytheJapanesenativespeakerauthorofthisstudywitheachoftheinterviewees.Theywerequestionedabouttheirthoughtsoneachofthe50itemsandtheirresponsesweretakendownintheformofnotes.
Analysis
ThedatacollectedfromthetwoadministrationsoftheSILLwerefirstanalyzedforitemstatisticsfollowedbydescriptivestatisticsforthesixpartsaswellastheentireSILL.Thealphalevelforallstatisticaldecisionswassetat .05.BothadministrationswerethenexaminedforinternalconsistencyusingCronbach’salphaforeachofthesixpartsaswellasoverallreliabilityforbothadministrations.Next,eachTimeOneitemwascomparedtoitsidenticalTimeTwoitemusingthePearsoncorrelation.Theresultingcorrelationswerethensquaredtodeterminethedegreeofsharedvariance.Thesquaredvalueofthecorrelationcoefficientcanbeinterpretedastheproportionofsimilaritybetweenthetwoitems(Hatch&Lazaraton,�99�).ThisprocedurewasrepeatedforthesixpartsoftheSILLandfortheentireSILLaswell.Finally,thetwoadministrationswere
![Page 7: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/7.jpg)
3838
JALT JournAL
examinedusingprincipalcomponentanalysis(PCA),whichisatypeofexploratoryfactoranalysis,withvarimaxrotationandeigenvaluessetatone.Thesearethetypicalproceduresforcarryingoutfactoranaly-sis.Asiscommon,loadingsof.30andabovewereconsideredstrongenoughforinclusioninagivenfactor(Hatch&Lazaraton,�99�).IntheinitialuseofPCA,theanalysiswasallowedtoselectasmanyfactorsascouldbefoundwithaneigenvalueover�.00;however,asecondPCAwasrunonbothadministrationsinwhichtheanalysiswasforcedtochoosesixfactorsbasedonOxford’stheorizedgrouping.ScreeplotsforallPCAswerealsocalculated.TheseadditionalprocedureswereconductedtoprovidetheSILLwithasmanyopportunitiesaspossibletosupplysupportforitstheoreticalbasis.Finally,thenotestakendur-ingtheinterviewswereexaminedtodeterminethetypesofdifficultiesthestudentshadunderstandingthequestionnaireitemsandhowtheirdifficultiescomparedtooneanother.
Results
Table�showstheitemsthemselveswiththeirgroupings,themeanoneachitemandthestandarddeviation,withTable2showingthemeansandstandarddeviations for the itemson thesecondadministration.Table3provides thedescriptive statistics for the six subsectionsoftheSILLandtheentireSILLforbothadministrations.Thedistributionsarealleitherpositivelyornegativelyskewedandthosewithskewnessstatisticsat�.0orgreaterareproblematic(Brown,�997).Theseskeweddistributionscanreducethetestreliabilityandareviolationsoftheas-sumptionsofnormalityforthecorrelationstatisticsandfactoranalysis,whichcouldadverselyaffecttheseresults.
Table�:MeanScoresandStandardDeviationsfortheItemsandTheirStrategyTypes,TimeOne(n=�53)
Item Statement Type M SD
� IthinkofrelationshipsbetweenwhatIalready knowand newthingsIlearninEnglish. Memo 2.79 0.942 IusenewEnglishwordsinasentencesoI canrememberthem. Memo 2.56 0.953 IconnectthesoundofanewEnglishword andanimageorpictureofthewordtohelp meremember. Memo 3.02 �.094 IrememberanewEnglishwordandan imageorpictureofasituationinwhichthe wordmightbeused. Memo 2.63 �.�25 IuserhymestoremembernewEnglishwords. Memo 2.4� �.��6 IuseflashcardstoremembernewEnglishwords. Memo 2.�9 �.42
![Page 8: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/8.jpg)
3939robson & MidorikAwA
7 IphysicallyactoutnewEnglishwords. Memo �.80 0.888 IreviewEnglishlessonsoften. Memo 2.66 0.669 IremembernewEnglishwordsorphrasesby rememberingtheirlocationonthepage,onthe board,oronastreetsign. Memo 2.56 �.24�0 IsayorwritenewEnglishwordsseveraltimes. Cog 3.98 0.99�� ItrytotalklikenativeEnglishspeakers. Cog 3.09 �.23�2 IpracticethesoundsofEnglish. Cog 3.40 �.05�3 IusetheEnglishwordsIknowindifferentways. Cog 2.89 0.96�4 IstartconversationsinEnglish. Cog 2.�7 0.86�5 IwatchEnglishlanguageTVshowsspokenin EnglishorgotomoviesspokeninEnglish. Cog 3.25 �.09�6 IreadforpleasureinEnglish. Cog 2.77 0.97�7 Iwritenotes,messages,lettersorreportsinEnglish.Cog 2.�9 �.06�8 IfirstskimanEnglishpassage(readoverthe passagequickly)thengobackandreadcarefully. Cog 3.39 �.09�9 Ilookforwordsinmyownlanguagethatare similartonewwordsinEnglish. Cog 2.39 �.�620 ItrytofindpatternsinEnglish. Cog 2.8��.072� IfindthemeaningofanEnglishwordbydividing itintopartsthatIunderstand. Cog 2.70 �.�822 Itrynottotranslateword-for-word. Cog 2.96 0.9923 ImakesummariesofinformationthatIhear orreadinEnglish. Cog �.97 0.9224 TounderstandunfamiliarEnglishwords, Imakeguesses. Comp 3.44 0.9225 WhenIcan’tthinkofawordduringa conversationinEnglish,Iusegestures. Comp 3.65 �.�626 ImakeupnewwordsifIdonotknowthe rightonesinEnglish. Comp 2.23 �.��27 IreadEnglishwithoutlookingupeverynewword. Comp 3.07 �.0528 Itrytoguesswhattheotherpersonwillsay nextinEnglish. Comp 2.35 0.9929 IfIcan’tthinkofanEnglishword,Iuseaword orphrasethatmeansthesamething. Comp 3.8� 0.9430 ItrytofindasmanywaysasIcantousemyEnglish.Meta 2.60 �.0�3� InoticemyEnglishmistakesandusethat informationtohelpmedobetter. Meta 3.37 �.0�32 IpayattentionwhensomeoneisspeakingEnglish. Meta 3.60 0.9833 ItrytofindouthowtobeabetterlearnerofEnglish. Meta 2.73�.0734 IplanmyschedulesoIwillhaveenoughtime tostudyEnglish. Meta 2.3� 0.8935 IlookforpeopleIcantalktoinEnglish. Meta 2.�9 �.0336 Ilookforopportunitiestoreadasmuchas possibleinEnglish. Meta 2.50 0.9737 IhavecleargoalsforimprovingmyEnglishskills. Meta 2.94 �.2938 IthinkaboutmyprogressinlearningEnglish. Meta 3.09 �.0439 ItrytorelaxwheneverIfeelafraidofusingEnglish.Aff 2.80 �.0740 IencouragemyselftospeakEnglishevenwhen Iamafraidofmakingamistake. Aff 3.07 �.�64� IgivemyselfarewardortreatwhenIdowell
![Page 9: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/9.jpg)
4040
JALT JournAL
inEnglish. Aff 3.43 �.0942 InoticeifIamtenseornervouswhenIam studyingorusingEnglish. Aff 3.08 �.�643 Iwritedownmyfeelingsinalanguagelearningdiary. Aff �.480.8644 ItalktosomeoneelseabouthowIfeelwhen IamlearningEnglish. Aff �.99 0.9945 IfIdonotunderstandsomethinginEnglish,Iask theotherpersontoslowdownorsayitagain. Soc 4.�4 0.8846 IaskEnglishspeakerstocorrectmewhenItalk. Soc 2.65 �.�947 IpracticeEnglishwithotherstudents. Soc 2.24 �.0�48 IaskforhelpfromEnglishspeakers. Soc 2.69 �.2449 IaskquestionsinEnglish. Soc 2.44 �.0950 ItrytolearnaboutthecultureofEnglishspeakers. Soc 3.03 �.2�
Note:ThestatementforeachitemisintheEnglishoriginalfromwhichtheJapanesetranslationwasmade.KeyforStrategyType:Memo=Memory,Cog=Cognitive,Comp=Compensation,Meta=Metacognitive,Aff=Affective,Soc=Social
Table2:MeanScoresandStandardDeviationsfortheItems,TimeTwo(n=�53)
Item M SD Item M SD Item M SD Item M SD
� 2.99 �.�4 �6 3.95 0.89 3� 3.39 �.06 46 2.08 �.042 3.25 �.�3 �7 2.5� �.24 32 3.�5 �.04 47 2.97 �.223 3.52 0.90 �8 3.24 �.�7 33 2.88 �.07 48 3.47 �.044 2.36 0.82 �9 2.22 �.06 34
Table3:DescriptiveStatisticsfortheSILLandSubsections,TimesOneandTwo(n=�53)
Measure M SD Min Max Range Skew
SILL,TimeOne �39.00 24.60 66 207 �4� -.24Memo,TimeOne 22.64 4.66 �� 36 25 -.04Cog,TimeOne 39.95 7.65 �8 6� 43 -.20Comp,TimeOne �8.33 3.60 8 28 20 -.04Meta,TimeOne 25.03 6.34 9 40 3� -.�6Aff,TimeOne �5.84 3.64 7 27 20 .07
![Page 10: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/10.jpg)
4�4�robson & MidorikAwA
Table4givesthereliabilityforthesixpartsandtheoverallreliabilityforbothadministrations.WhiletheSILLasawholeforbothtimesoneandtwohasveryhighreliabilityat.93,severalofthesubsectionsarevery low. Inparticular, theTimeOne reliabilities forMemo,Comp,andAffareunacceptablylow.ThesameistrueforMemo,Comp,Aff,andSocinTimeTwo.Theresultsforthesecondmeasureofreliability,test-retest,areshowninTables5and6.Thedegreeofsharedvariancefortheitemsdoesnotexceed46percentwithsomeaslowas3,4,5,and7percent.Theaverageforalltheitemsisjust�9.5percent.Forthesubsections,thesharedvarianceissimilarlylow,withtheonlyexcep-tionbeingfortheSILLasawholeat58percent.
Soc,TimeOne �7.�8 4.89 6 30 24 .��SILL,TimeTwo �44.58 25.22 63 229 �66 -.26Memo,TimeTwo 26.67 4.94 �� 4� 30 -.29Cog,TimeTwo 37.84 7.9� �6 66 50 .�3Comp,TimeTwo �8.85 3.4� 8 27 �9 -.42Meta,TimeTwo 26.2� 5.7� 9 44 35 -.�6Aff,TimeTwo �7.45 3.73 6 27 2� -.38Soc,TimeTwo �7.54 3.65 6 26 20 -.25
KeyforStrategyType:Memo=Memory,Cog=Cognitive,Comp=Compensation,Meta=Metacognitive,Aff=Affective,Soc=Social
Table4:InternalConsistencyfortheSILLandSubsections,TimesOneandTwo(n=�53)
Measure Alpha Measure Alpha
SILL,TimeOne .93 SILL,TimeTwo .93
Memo,TimeOne .63 Memo,TimeTwo .66
Cog,TimeOne .80 Cog,TimeTwo .83
Comp,TimeOne .67 Comp,TimeTwo .58
Meta,TimeOne .85 Meta,TimeTwo .79
![Page 11: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/11.jpg)
4242
JALT JournAL
Table5:PercentageofSharedVarianceBetweenSILLItems,TimesOne&Two(n=�53)
Items RSquared Items RSquared
� .03 26 .282 .�6 27 .�03 .�6 28 .�04 .2� 29 .�45 .�8 30 .246 .07 3� .�87 .�8 32 .�88 .07 33 .249 .�3 34 .�8�0 .�8 35 .4��� .34 36 .25�2 .26 37 .42�3 .�2 38 .�6�4 .29 39 .24�5 .29 40 .22�6 .29 4� .34�7 .27 42 .08�8 .�4 43 .2��9 .�4 44 .0520 .�4 45 .�42� .27 46 .4622 .07 47 .2423 .�� 48 .2824 .07 49 .0725 .36 50 .04
Table6:PercentageofSharedVarianceBetweentheSILL&Subsections,TimesOne&Two(n=�53)
Measure RSquared
Memo .25Cog .36Comp .�4Meta .35Aff .�7Soc .26SILL .58
KeyforStrategyType:Memo=Memory,Cog=Cognitive,Comp=Compensation,Meta=Metacognitive,Aff=Affective,Soc=Social
![Page 12: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/12.jpg)
4343robson & MidorikAwA
Tables7and8showtheresultsofthefirstPCAswitha�5-factorso-lutionforTimeOneanda�3-factorsolutionforTimeTwo.Wewouldexpectthefactoranalysistogroupitems�through9inonefactor,items�0through23inasecondfactor,items24through29inathirdfactor,items30through38inafourthfactor,items39through44inafifthfac-tor,anditems45through50inasixth.However,theresultsfortheTimeOnePCAshowveryfewitemsloadingtogether.Thegreatestgroupofitemsloadingtogetherisinfactor�4withitems46through49together;however,beyondthis,therearenogreatergroupsofloadingsthanjusttwoorthreeitemstogether.Factoronetakesup23percentofthetotalvariancewiththeotherfactorsaccountingforconsiderablyless,whichisconfirmedbytheeigenvalues.Inaddition,thecommunalities,whichshowthedegreetowhichthefactorsareaccountingforeachitem,arenotparticularlyhighexceptforitems24,25,35and36.AsimilarstateofaffairsisfoundfortheTimeTwoPCA;however,therearenogroupsofloadingsgreaterthantwo,makingtheresultsappearevenlesssystem-aticthanwiththoseoftheTimeOneanalysis.Again,almostallthetotalvarianceisbeingaccountedforbyfactorone.Also,withtheexceptionsofitems�3and�4,thecommunalitiesarenotparticularlyhigh.
Tables9and�0showtheattempttoforcetheSILLintoasix-factor
Table7:PrincipalComponentAnalysis,TimeOne(n=�53)
Item FactorLoadings Factor� Factor2Factor3 Factor4 Factor5Communalities
35 0.85 0.9�36 0.85 0.9�23 0.64 0.4730 0.60 0.5850 0.44 0.6�32 0.72 0.5729 0.66 0.5�4� 0.59 0.533� 0.56 0.6240 0.53 0.5839 0.45 0.5020 0.70 0.579 0.63 0.35�9 0.58 0.442� 0.39 0.458 0.79 0.4934 0.49 0.54�6 0.45 0.57Table7cont...
![Page 13: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/13.jpg)
4444
JALT JournAL
Item FactorLoadings Factor� Factor2Factor3 Factor4 Factor5Communalities
44 0.79 0.38� 0.32 0.38
Eigenvalues ��.54 3.�9 2.40 2.02 �.98PercentofTotalVariance 23.08 6.37 4.8� 4.04 3.95
Item FactorLoadings Factor6 Factor7Factor8 Factor9 Factor�0Communalities
24 0.9� 0.9�25 0.9� 0.9�27 0.43 0.5926 0.79 0.3933 0.63 0.5338 0.56 0.5537 0.52 0.5��2 0.38 0.53�0 0.78 0.3845 0.4� 0.496 0.79 0.4342 0.53 0.35
Eigenvalues �.69 �.60 �.45 �.33 �.22PercentofTotalVariance 3.39 3.20 2.9� 2.66 2.45
Item FactorLoadings Factor�� Factor�2 Factor�3 Factor�4 Factor�5Communalities
�8 0.68 0.4722 0.65 0.50�3 0.37 0.453 0.74 0.464 0.58 0.5�7 0.46 0.4�5 0.46 0.44�7 0.73 0.49�4 0.59 0.65�5 0.58 0.42�� 0.48 0.5948 0.70 0.5746 0.64 0.6547 0.52 0.6343 0.52 0.4649 0.5� 0.6�Table7cont...
![Page 14: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/14.jpg)
4545robson & MidorikAwA
28 0.74 0.422 0.47 0.36
Eigenvalues�.�9 �.�8 �.�2 �.08 �.04PercentofTotalVariance2.38 2.36 2.24 2.�6 2.09
Note:Onlyitemswithloadingsequaltoorover0.30areindicatedinthetable
Table8:PrincipalComponentAnalysis,TimeTwo(n=�53)
Item FactorLoadings Factor� Factor2 Factor3 Factor4 Factor5Communalities
�7 0.72 0.6635 0.7� 0.7626 0.65 0.594� 0.6� 0.5630 0.58 0.52�9 0.53 0.5522 0.46 0.48�2 0.43 0.4�8 0.39 0.5�2� 0.38 0.549 0.69 0.5838 0.64 0.4844 0.62 0.4940 0.56 0.5437 0.55 0.56�0 0.54 0.5�32 0.49 0.55�� 0.48 0.5542 0.66 0.6�28 0.65 0.53�6 0.63 0.6245 0.63 0.4729 0.54 0.5�5 0.38 0.42�5 0.67 0.4247 0.59 0.533� 0.5� 0.55� 0.5� 0.533 0.46 0.494 0.38 0.37�3 0.85 0.9��4 0.85 0.9�
Eigenvalues ��.94 3.02 2.70 2.09 �.89PercentofTotalVariance 23.88 6.04 5.4� 4.�8 3.79
![Page 15: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/15.jpg)
4646
JALT JournAL
Table8cont...Item FactorLoadings Factor6 Factor7 Factor8 Factor9 Factor�0Communalities
2 0.76 0.4236 0.45 0.5346 0.85 0.48�8 0.73 0.5�20 0.52 0.6624 0.49 0.4725 0.42 0.4643 0.77 0.4650 0.49 0.4�49 0.39 0.5�6 0.62 0.387 0.43 0.42
Eigenvalues �.58 �.46 �.39 �.36 �.20PercentofTotalVariance 3.�6 2.9� 2.79 2.73 2.40
Item FactorLoadings Factor�� Factor�2 Factor�3 Communalities
48 0.72 0.3739 0.43 0.6434 0.37 0.4923 0.75 0.5333 0.40 0.7027 0.72 0.3�
Eigenvalues �.�6 �.�0 �.05PercentofExplainedVariance 2.3� 2.20 2.09
Note:Onlyitemswithloadingsequaltoorover0.30areindicatedinthetable.
solution.Herewehaveaclearerpictureofwhythefirstfactoristak-ingupsomuchofthetotalvariance,althoughwithTimeTwo,thereislessofaconcentrationofitemsinthefirstfactor.Nonetheless,theloadingsforbothPCAsshowacombinationofrelatedandunrelateditems fromthesixsubgroups loading together.Figures�and2givevisualrepresentationsoftheeigenvaluesthroughscreeplots,which,ifwecountthenumberoffactorstotheleftofthepointwherethelineturnsstronglytotheright,seemtoindicatethataonefactoranalysisoftheSILLwouldbemostappropriate.
Theinterviewsrevealedsomeveryinterestingproblemsthequestion-
![Page 16: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/16.jpg)
4747
Table9:PrincipalComponentAnalysiswithSixForcedFactors,TimeOne(n=�53)
Item FactorLoadings Factor� Factor2 Factor3 Factor4 Communalities
35 0.79 0.9�36 0.79 0.9��4 0.69 0.6547 0.68 0.6449 0.68 0.6�30 0.67 0.5946 0.67 0.6548 0.60 0.5723 0.55 0.47�7 0.5� 0.4950 0.49 0.6240 0.47 0.5828 0.44 0.42�6 0.43 0.574 0.42 0.5��5 0.39 0.42�3 0.35 0.4526 0.3� 0.3920 0.72 0.572� 0.66 0.45�9 0.59 0.4422 0.49 0.50�8 0.44 0.4739 0.43 0.509 0.43 0.357 0.40 0.423 0.39 0.465 0.35 0.4432 0.69 0.57�� 0.62 0.5945 0.57 0.49�2 0.56 0.5329 0.55 0.5�38 0.5� 0.5533 0.5� 0.533� 0.49 0.6237 0.44 0.5��0 0.39 0.388 0.69 0.4934 0.49 0.532 0.48 0.366 0.44 0.43
Eigenvalues ��.54 3.�9 2.40 2.02PercentofTotalVariance 23.09 6.37 4.8� 4.04
![Page 17: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/17.jpg)
4848
JALT JournAL
Table9cont...Item FactorLoadings Factor5 Factor6 Communalities
43 0.53 0.4644 0.53 0.38� 0.43 0.384� 0.37 0.5324 0.86 0.9�25 0.86 0.9�27 0.44 0.50
Eigenvalues �.98 �.69PercentofTotalVariance 3.96 3.39
Note:Onlyitemswithloadingsequaltoorover0.30areindicatedinthetable.
Table�0:PrincipalComponentAnalysiswithSixForcedFactors,TimeTwo(n=�53)
Item FactorLoadings Factor� Factor2 Factor3 Factor4 Communalities
38 0.63 0.4844 0.58 0.4932 0.57 0.5539 0.57 0.6437 0.57 0.563� 0.56 0.5524 0.53 0.4740 0.52 0.544 0.47 0.3725 0.47 0.46�5 0.40 0.4235 0.68 0.7622 0.66 0.4826 0.65 0.59�7 0.59 0.66�3 0.57 0.9��4 0.57 0.9�6 0.54 0.387 0.5� 0.424� 0.5� 0.5630 0.5� 0.5233 0.44 0.7023 0.39 0.532� 0.39 0.54�0 0.38 0.5��6 0.72 0.6242 0.65 0.6�
![Page 18: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/18.jpg)
4949robson & MidorikAwA
Table�0cont...20 0.65 0.6628 0.64 0.5329 0.6� 0.5�45 0.49 0.47�8 0.47 0.5�5 0.47 0.423 0.37 0.4934 0.37 0.4947 0.59 0.5349 0.57 0.5�36 0.57 0.53� 0.52 0.5348 0.49 0.3750 0.48 0.4�8 0.44 0.5��2 0.32 0.4�
Eigenvalues ��.94 3.02 2.70 2.09PercentofTotalVariance 23.88 6.04 5.4� 4.�8Item FactorLoadings Factor5 Factor6 Communalities
9 0.62 0.58�� 0.58 0.55�9 0.56 0.5543 0.39 0.462 0.69 0.4227 -0.52 0.3�46 -0.42 0.48
Eigenvalues �.89 �.58PercentofTotalVariance 3.79 3.�6
Note:Onlyitemswithloadingsequaltoorover0.30areindicatedinthetable.
naireposedfortherespondents.Themajorityofstudentsinterviewedhaddifficultyunderstandingitems�,5,6,7,�4,�9,20,22,26,43,44,and47.ThemostcommonlycitedreasonfortheirlackofunderstandingwasunfamiliarJapaneseorEnglishexpressions.Thiswasparticularlytrueforitems5,6,22,and43.Anotherreasonrespondentsgavefortheirdifficultyinunderstandingwasthattheycouldnotimaginethesituation.
Discussion
TheresultsreportedaboveprovideahighlevelofreliabilityfortheSILL
![Page 19: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/19.jpg)
5050
JALT JournAL
Figure�:ScreePlot,PrincipalComponentAnalysis,TimeOne(n=�53)
Figure2:ScreePlot,PrincipalComponentAnalysis,TimeTwo(n=�53)
![Page 20: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/20.jpg)
5�5�robson & MidorikAwA
asawhole,whichisproblematic,astheSILLshouldbemeasuringsixdifferent typesof strategies,notonegrand strategy type.Moreover,thealphasforthesubsectionsshowasimilarlylowlevelofreliabilityaswasfoundbyBrownetal.(�996).Onereasonforthiscouldbethenumberofitemsinthesubsections,wherethelongersubsectionssuchasCognitiveStrategieshavehigherreliability.Lengthisanimportantfactorinreliability,aslongermeasurestendtobemorereliable(Bach-man,�990).Moreover,aswasnotedpreviously,allthedistributionsareskewed,whichmustalsobeaffectingthelevelofreliability.Forexample,SocialStrategiesTimeOnehasfairlyhighreliability,butatTimeTwoitdropsto.59.However,thereisalsoanincreaseintheskewbetweentimesone(.��)andtwo(-.25).Nevertheless,theseskeweddistributionscannotfullyexplaintherelativelylowreliabilityastheCognitiveStrate-giessubsectionhasaconsistentlevelofreliabilityfromTimeOnetoTimeTwo,butskeweddistributionsof-.20and.�3.Thetest-retestreli-abilityasindicatedbythepercentageofsharedvariancefortheitems,subsectionsandentiremeasureshowthattheSILLishighlyunreliable.Itisimportanttorememberthatreliabilitycanbemeasuredseveraldif-ferentways,andthatdependenceonasingleapproachcanberisky.Onereasonforthelowfigureshasbeenfoundinotherstudieslookingateitherbeliefsorstrategies(forexampleGaies&Sakui,�999),wherethestudentswerefoundtochangeovertime.Althoughitisdifficulttodeterminetheexactreasonsforchangewithoutconductingextensivepost-administrationinterviews,studentsmayinterpretthequestionsonagivenmeasureinlightoftheircurrentlearningsituationandnotlearn-ingsituationsingeneral.Moreover,theeffectsoftrainingandlearningmustalsobetakenintoaccount.Inaddition,itisimportanttoremem-berthatstrategiesarenotpersonalitytraits,whichhavebeenshowntoremainstableovertimeandacrosssituations(seeAngleitner,�99�).Thus,itishardlysurprisingthatthepercentageofsharedvarianceshouldbesolowbetweenthetwoadministrations.However,thereareotherpossibleexplanationsforthelowlevelsofreliability.Again,theskeweddistributionscouldbeadverselyaffectingtheresults,oritispossiblethatthepopulationsurveyedwastoohomogeneous.Thesubjectsareallfromasinglelanguagebackgroundandculturewithclosesimilaritiesinageandpossiblyeducationalexperience.Theskeweddistributionsarelikelypartoftheexplanation.Nonetheless,theJapaneseversionoftheSILLwasdesignedtoexaminejustthistypeofpopulation.Moreover,theeducationalbackgroundofthisgroupofsubjectsisprobablynotallthathomogeneous.Thestudentsatthewomen’suniversitycomefromawideareanorthandwestofTokyoandattendedbothprivateandpublichighschoolswherethereareeducationaldifferencesfromone
![Page 21: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/21.jpg)
5252
JALT JournAL
schooltothenext.Theco-edschoolsubjectsaresimilarinthisregard.ItalsoseemsreasonabletoexpectthatmostadministratorsorteacherswillusetheSILLundersimilarconditionsinJapan.
ThefactoranalysisresultsdonotconfirmOxford’ssixstrategycat-egoriesevenwhenattempting to force theanalysis intoa six-factorsolution.Infact,theSILLiseithermeasuring�5or�3differenttypesofstrategies,orevenjustoneasindicatedbytheeigenvaluesandscreeplots.Thereareanumberofpotentialreasonsforthis.Thelowreli-abilityisanimportantfactorasisthesizeofthepopulation.HatchandLazaraton(�99�)recommendatleast35subjectspervariableforPCA,whichinthisstudywouldnecessitateannsizeofabout�,750subjects.Withasamplesizeofonly�53,thereisconsiderablelossofstatisticalpower.Nonetheless,other studieswith larger sampleshave shownsimilarresults(Hsiao&Oxford,2000)andbasedonthosefoundhereaswellasinBrownetal.(�996),itwouldseemsafertolimittheSILLtoonegrandlanguagelearningstrategiesfactorinsteadoftryingtobreakitintotheorizedgroups.
Attemptingtolabeleachofthesestrategytypesisverydifficult.Thereseemstobealmostnosystemtothefactorloadings,although,someofthefactorscanbetentativelylabeled.Forexample,TimeOnefactor�4seemstoberelatedtoOxford’sSocialStrategies,whilefactor2con-tainsitemsfromtheAnalyzingandReasoningsubgroupwithinCogni-tiveStrategies.Factor�2seemstobetheMemoryStrategysubgroupApplyingImagesandSounds.ThefactorsolutionforthesecondtimeshowsanevengreatermixingofitemsfromdifferentstrategygroupsalmostnecessitatingacompleteabandonmentofOxford’scategories.However,bylookingatthewording,wecanapplytentativelabels.Forexample,factortwocanbeinterpretedasvariousspeakingstrategies.Inaddition,thereseemtobegroupingsofitemsinbothTimeOneandTimeTwobasedonthetypeofactionexpressedbytheverbinJapa-nese.AnexamplewouldbeTimeOnefactorfour,wherethesubjectsseemtoplaceemphasisonsuchactionsas“review,”“read,”and“plan.”TheattempttoforcetheSILLintosixfactorsforTimeOneresultedinwhatlookslikeaone-factorsolutionincludingsomeitemsfromeachsubsection.IftheseresultshadbeenrepeatedinTimeTwo,therewouldhavebeenanopportunitytosupportaone-factorsolutionbasedonthisdata.Unfortunately,theitemsinthefirstfactordiffer.
Theproblemsstudentshadunderstandingthequestionnairewerepartiallyrevealedbythepost-administrationinterviewsconductedwithaverysmallsample.Thesestudentswereunfamiliarwithsuchexpres-sionsorsituationsaskokoroniegaku(makingamentalpicture)initem4,inotsukau(userhymes)initem5,“flashcards”initem6,andkarada
![Page 22: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/22.jpg)
5353robson & MidorikAwA
dehyエnshite(physicallyactout) in item7.Forexample,duringtheadministrationofthequestionnaire,themajorityofstudentscouldnotreadthecharacterin,whichmeansrhymeinJapanese,anddidnotknowitsmeaningwhenitwasreadtothem.Moreover,otheritemsthatwereincomprehensiblewereonesthatreflectamoreWesternapproachtolearningstrategiesthanonewithwhichJapanesestudentsarefamiliar,suchaswithitems4and7.Inaddition,studentshaddifficultyrelatingmanyof the situationspresented in thequestionnaire to theirownlearning.First,thequestionnairepresumesanESLlearningsituation,wherethesituationinJapanisclearlyEFL.Thus,thesestudentshavefewopportunitiesfortargetlanguageuseoutsideoftheclassroom.Ofthe�0interviewees,3hadexperiencedstudyinginanEnglish-speak-ingcountryandhadfewcomprehensionproblemswiththelearningsituationspresented.However, for the remaining7 students, targetlanguagestudyandusewaslimitedtotheclassroom,library,home,train,ortheirEnglishSpeakingSociety(ESS)meetingsandtheyfoundmanyofthelearningsituationsinthequestionnaireunimaginableorstrange.Moreover,aswasnotedabove,theintervieweesrespondedtoitemsbasedontheircurrentlearningsituationandnotlearningsitua-tionsingeneral.
Conclusion
Thesimplestconclusiononecandrawfromthisinitialattemptatde-terminingthereliabilityandvalidityoftheJapaneselanguageversionoftheSILListhatitisneitherreliablenorvalidbasedonthisstudentsample.AlthoughtheSILLhasshownahighdegreeofinternalreliabilityfortheentirequestionnaire,itclaimstomeasuresixdifferentstrategiesandthusmustbeanalyzedassixdifferentmeasures.Infact,thehighdegreeofreliabilityfortheentireSILL,asnotedabove,isnotnecessarilyagoodthing.Thesubsectionshaveagenerallylowandunacceptablealphalevel.Moreover,thereareseriousquestionsabouthowreliabletheresultsarewhengiventothesamegroupmorethanonceandhowvalidthecategoriesusedtogrouptheitemsonthequestionnaireare.Inotherwords,whiletheSILLmayindeedbemeasuringlanguage-learn-ingstrategies,itdoesnotseemtobemeasuringgroupsofstrategiesinthemannerOxfordhasclaimed,atleastfortheselearners.Itwouldseemreasonable,basedonthehighreliabilityfortheentireSILL,theeigenvalues,andscreeplots, todescribe theSILLasageneralmea-sureoflanguage-learningstrategiesandnotameasureofsixdifferentstrategytypes.Theresearchersbelievethattheseconclusionscanbedrawnbasedonthesedatainspiteofpotentialproblemswithnsize,
![Page 23: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/23.jpg)
5454
JALT JournAL
thepossiblyhomogeneouspopulation,skeweddistribution,andlowreliability.
Asdiscussedpreviously, themethodsOxfordused tovalidateanearlierversionoftheSILLaresomewhatsuspect.Takentogetherwiththelackofestablishedreliabilityorvalidityforthelaterversions(de-spiteclaimsbyYamato,2000,p.�42,tothecontrary),thoseusingtheEnglishversionoftheSILLwillnotbeabletorelyontheresults.More-over,HsiaoandOxford’s(2000)confirmatoryfactoranalysisdoesnotprovidemuchconfidenceeither.CautionaryusebecomesevenmorenecessarywiththeJapanesetranslation,asitisnowanewquestion-naire thathasnotgone througha rigorous reliabilityandvalidationprocess.Theseissuesandproblemsarenotjustaboutstrategies,butrelatetoanyuseofaself-reportquestionnaire.Itshouldbeclearfromthisanalysisthatsimplytakingaquestionnaire,translatingitintoanotherlanguage,administeringittoagroupofstudentsandthenusingthere-sultsformakingeducationalpolicydecisionsareveryunwisepractices.Moreover,anyquestionnairemustreflecttheactuallearningsituationsofthetargetpopulation,theirstrategyuse,thetypeoflanguagewithwhichtheyarefamiliar,andanyculturaldifferencesthatmightaffecttheoutcome.TheresearchershopethatthroughthisinitialattemptatvalidatingOxford’squestionnaireotherresearchersandlanguage-teach-ingprofessionalswilltakeamorecautiousapproachtoquestionnaireuseandinterpretation.
AcknowledgementsThisisamuchrevisedandexpandedversionofapaperdeliveredatAILA99inTokyo.Theauthorswouldliketothankthetwoanonymousreviewersfortheirsuggestionsandvaluablecomments.
GordonRobsonisProfessoratShowaWomen’sUniversity.Hisresearchinterestsincludereadingstrategiesandindividuallearnerdifferences.
HidekoMidorikawa isProfessor at ShowaWomen’sUniversity.Herresearchinterestsincludereadingstrategiesandteachereducation.
ReferencesAllport,G.W.(�937).Personality:Apsychological interpretation.NewYork:
Holt,Rinehard&Winston.
Angleitner, A. (�99�). Personality psychology: Trends anddevelopments.EuropeanJournalofPersonality,5,�56-�7�.
![Page 24: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/24.jpg)
5555robson & MidorikAwA
Bachman,L.F.(�990).Fundamentalconsiderationsinlanguagetesting.Oxford:OxfordUniversityPress.
Brown, J.D. (�988).Understanding research in second language teaching:A teacher’s guide to statistics and researchdesign. London:CambridgeUniversity.
Brown,J.D.(�997).Statisticscorner,questionsandanswersaboutstatistics:Skewnessandkurtosis.Shiken:JALTtesting&evaluationSIGnewsletter,�(�),�6-�8.
Brown,J.D.,Robson,G.,&Rosenkjar,P.(�996).Themotivation,anxiety,learningstylesandproficiencyof Japanesecollege students.UniversityofHawai’iWorkingPapersinESL,�5(�),33-72.
Chapelle,C.(�994).AreC-testsvalidmeasuresforL2strategyresearch?SecondLanguageResearch,�0(2),�57-�87.
Gaies,S.,&Sakui,K.(�999).Symposiumonlearners’beliefsaboutlanguagelearning.Paperpresentedatthe�2thWorldCongressofAppliedLinguistics,Tokyo,Japan.
Gardner,R.C.,&Lambert,W.E.(�972).Attitudesandmotivationinsecond-languagelearning.Rowley,MA:NewburyHouse.
Griffee,D. (�999). Translatingquestionnaires fromEnglish into Japanese:Is itvalid?InA.Barfield,R.Betts, J.Cunningham,N.Dunn,H.Katsura,K.Kobayashi,N.Padden,N.Parry&M.Watanabe(Eds.),OnJALT98:Focusontheclassroom(pp.�76-�80).Tokyo:JapanAssociationforLanguageTeaching.
Guilford,J.P.(�940).AninventoryoffactorsSTDCR,manualofdirectionsandnorms.BeverlyHills:SheridanSupplyCo.
Guilford,J.P.&Fruchter,B.(�973).Fundamentalstatisticsinpsychologyandeducation.NewYork:McGrawHill.
Hatch,E.&Lazaraton,A.(�99�).Theresearchmanual:Designandstatisticsforappliedlinguistics.NewYork:NewburyHousePublishers.
Horwitz,E.K.,Horwitz,M.B.,&Cope,J.(�986).Foreignlanguageclassroomanxiety.TheModernLanguageJournal,70,�25-�32.
Hsiao,T.-Y.,&Oxford,R.L.(2000).Learningstrategyuseandtargetlanguages:confirmatory factor analysis across languages. Paper presented at theannualmeetingoftheTeachersofEnglishtoSpeakersofOtherLanguages,Vancouver,Canada.
McCrae,R.R.(�989).WhyIadvocatethefive-factormodel:JointanalysesoftheNEO-PIandotherinstruments.InD.M.Buss&N.Cantor(Eds.),Personalitypsychology:Recenttrendsandemergingdirections(pp.237-245).NewYork:Springer-Verlag.
Messick,S.(�989).Validity.InR.E.Linn(Ed.),Educationalmeasurement(3rded.,pp.�3-�03).NewYork:Macmillan.
Oxford,R.L.(�990).LanguageLearningStrategies:WhatEveryTeacherShouldKnow.NewYork:NewburyHouse.
![Page 25: How Reliable and Valid is the Japanese Language …jalt-publications.org/files/pdf-article/jj-23.2-art2.pdfconcern in test development and use is demonstrating not only that test scores](https://reader033.vdocuments.us/reader033/viewer/2022042018/5e76330ac90eb627357e1dd7/html5/thumbnails/25.jpg)
5656
JALT JournAL
Oxford,R.L.(�994).Gengogakushusutoratejiiエgaikokugokyoushigashitteokankerebanaranaikoto[LanguageLearningStrategies:WhatEveryTeacherShouldKnow](M.Shishido&N.Ban,Trans.).Tokyo:Bonjinsha. (originalworkpublished�990)
Oxford,R.L.(�996).Employingaquestionnairetoassesstheuseoflanguagelearningstrategies.AppliedLanguageLearning,7,(�&2),25-45.
Oxford, R. L., Park-Oh,Y., Ito, S.,& Sumrall,M. (�993). Factors affectingachievementinasatellite-deliveredJapaneselanguageprogram.AmericanJournalofDistanceEducation,7,�0-25.
Takeuchi,O.(�993).AstudyoflanguagelearningstrategiesandtheirrelationtoachievementinEFLlisteningcomprehension.BulletinoftheInstituteforInterdisciplinaryStudiesofCulture,�0,�3�-�4�.
Watanabe,Y.(�990).Externalvariablesaffectinglanguagelearningstrategiesof JapaneseEFL learners: Effects of entrance examination, years spentat college/university, and stayingoverseas.Unpublishedmaster’s thesis,LancasterUniversity,Lancaster,U.K.
Yamato,R.(2000).Awarenessandrealuseofreadingstrategies.JALTJournal,22(�),�40-�64.
(ReceivedAugust5,2000;revisedMay3,200�)