don't walk: rasch to join the questionnaire trend!
DESCRIPTION
Testing & evaluation (TEVAL) SIG workshop at JALT 2014 (Tsukuba), November 22.TRANSCRIPT
![Page 1: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/1.jpg)
+Don’t walk:
Rasch to
join
the
questionnai
re trend!
Ritsumeik
an
University
Keita
Kikuchi
Kanagaw
a
University
J. W.
Lake
Fukuoka
Women’s
University
![Page 2: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/2.jpg)
+
Get ready to be Rasched…
(This is supposed to be a joke. Laugh.)
Structure of
this
workshop
Ritsumeik
an
University
Keita
Kikuchi
Kanagaw
a
University
J. W.
Lake
Fukuoka
Women’s
University
![Page 3: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/3.jpg)
+
Rasch concepts
Terminology and explanations
Structure of
this
workshop
Ritsumeik
an
University
![Page 4: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/4.jpg)
+
Steps to create a
questionnaire
Constructs, concepts, items, piloting, evaluation,
revision…
Structure of
this
workshop J. W.
Lake
Fukuoka
Women’s
University
![Page 5: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/5.jpg)
+
Demotivation: An example
The flaws of traditional FA and why Rasch can
help questionnaire creation
Structure of
this
workshop
Keita
Kikuchi
Kanagaw
a
University
Teacher
Behavior
GETA
e1
.69
MET
e2
.85
ONE
e3
.75
PRON
e4
.69
EXPL
e5
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.76.74.67
VID
e15
.70
GRAM
e6
Experience
of difficulties
VOCI
e7
VOC
e8
TEST
e9
SELF
e10
.72 .68 .66.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOF
e18
.82
.83.77.78.81
NOTU
e11
.74
demotivation
.74
.68
.69
d1
d2
d3
d4
.66
FRN
e16
.64
MST
e17
.61
.50
GFI=.911
NFI=.905
CFI=.920
RMSEA=.064
SRMR = .055
![Page 6: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/6.jpg)
+
Matthew Apple
Ritsumeikan University
Department of Communication
International Communication Program
Rasch
concepts
![Page 7: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/7.jpg)
+
Rasch terms
Rasch log-odds (logits)
Rasch measures (logit scores)
Infit/Outfit (means sq. and z-score)
Item difficulty / endorsability
Person/item reliability / separation
Construct validity and
unidimensionality (not strictly
speaking Rasch, but…)
Rasch Principal components
analysis (Rasch PCA)
Loadings
Contrasts and Residuals
A probabilistic model
![Page 8: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/8.jpg)
+Logits (log-odds)
The probability of a person correctly
answering an item 50% of the time
![Page 9: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/9.jpg)
+Fit
0.75 to 1.3 logits
0.60 to 1.4 logits
“Within 2 standard deviations of the mean”
“Only Outfit z-scores of 3.0”
![Page 10: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/10.jpg)
+Separation
The number of groups distinguishable by the
measurement instrument
(Wilson, 2005;
Wright, 1996)
The ratio of error-free variance and
observed variance
(Fisher, 1992)
![Page 11: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/11.jpg)
+Item map
Persons and items on the same
linear logit scale
![Page 12: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/12.jpg)
+
The “Line”Items and persons on the same scale
Item-person map (or)
Wright map
Ben Wright
![Page 13: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/13.jpg)
+
The “Line”Items and persons on the same scale
Item-person map (or)
Wright map
![Page 14: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/14.jpg)
+
How Rasch works
for questionnaires
Likert-scale data
Likert-type category data
Questionnaires do not produce
true interval but ordinal data
The steps in the “scale” can be
conceived as thresholds (τ)
named after Thurston, originator
of factor analysis)Refer to Andrich (1977, 1978)
![Page 15: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/15.jpg)
+
How Rasch works
for questionnairesRefer to Andrich (1977, 1978)
![Page 16: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/16.jpg)
+
How Rasch works
for questionnairesRefer to Andrich (1977, 1978)
N
1
2
3
4
5
![Page 17: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/17.jpg)
+
How Rasch works
for questionnairesRefer to Andrich (1977, 1978)
N
1
2
3
4
5
1 + 3 = 4
![Page 18: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/18.jpg)
+
How Rasch works
for questionnairesRefer to Andrich (1977, 1978)
N
1
2
3
4
5
1 + 3 = 4?
SD + N = A?
![Page 19: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/19.jpg)
![Page 20: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/20.jpg)
+
J W Lake
Fukuoka Women’s University
Steps to
create a
questionnaire
![Page 21: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/21.jpg)
Steps in scale development: Issues to
consider (Netemeyer, Bearden, Sharma,
2003)
Step 1: Construct definition and content domain
The importance of clear construct definition,
content domain, and the role of theory.
Construct dimensionality: unidimensional,
multidimensional, or a higher-order construct?
Determine the purpose of the scale:
measurement or correlational analysis or
model building
![Page 22: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/22.jpg)
Step 2: Generating and judging measurement items
Theoretical assumptions about items (e.g., domain
sampling)
Generating potential items and determining the
response format
How many items as an initial pool
Dichotomous vs. polytomous response formats
Item wording issues
The focus on “content” validity in relation to
theoretical dimensionality
Item judging (expert and layperson) --- the focus on
“content” and “face” validity
![Page 23: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/23.jpg)
Step 3: designing and conducting studies to develop
and refine the scale
Pilot testing as an item-trimming procedure
The use of several samples from relevant populations for scale development
Designing the studies to test psychometric properties
Initial item analyses via exploratory factor analyses (EFAs)
Initial item analyses and internal consistency estimates
Retaining items for the next studies
EFA may be useful for correlational analysis or model building
![Page 24: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/24.jpg)
Step 4: Finalizing the scales
The importance of several samples from relevant populations
Designing the studies to test the various types of validity
Item analysis via EFA
The importance of EFA consistency from Step 3 to Step 4
Deriving an initial factor structure—dimensionality and theory
Item analyses and confirmatory factor analyses (CFAs)
Testing the theoretical factor structure and model
![Page 25: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/25.jpg)
+
Keita Kikuchi
Kanagawa University
Demotivatio
n: An
example
Teacher
Behavior
GETA
e1
.69
MET
e2
.85
ONE
e3
.75
PRON
e4
.69
EXPL
e5
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.76.74.67
VID
e15
.70
GRAM
e6
Experience
of difficulties
VOCI
e7
VOC
e8
TEST
e9
SELF
e10
.72 .68 .66.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOF
e18
.82
.83.77.78.81
NOTU
e11
.74
demotivation
.74
.68
.69
d1
d2
d3
d4
.66
FRN
e16
.64
MST
e17
.61
.50
GFI=.911
NFI=.905
CFI=.920
RMSEA=.064
SRMR = .055
![Page 26: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/26.jpg)
Example Study
focused on specific external /internal forces that Japanese high school students may experience which might cause their motivation to be reduced or diminished
administered the questionnaire asking high school students to report what diminished their motivation to study in their high school days, which contained 40 Likert-scale questions (4-points)
analyzed the quantitative data using a confirmatory factor analysis using Amos and Rasch PCA of the residuals using Winsteps.
If you’d like to read this process thoroughly, please locate Kikuchi (forthcoming).
![Page 27: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/27.jpg)
Demotivation
Dörnyei (2001)
Definition of demotivation
“specific external forces that reduce or
diminish the motivational basis of a
behavioral intention or an ongoing action” (p.
143).
I expand this definition and explore
demotivators including both internal and
external forces.
![Page 28: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/28.jpg)
Previous Studies (Dörnyei, 1998)
Based on interviews with 50 secondary school students, he identified following as demotivators, the reason to get demotivated.
1. Teachers’ personalities, commitments, competence, teaching methods.
2. Inadequate school facilities (very big group, not the right level or frequent change of teachers).
3. Reduced self-confidence due to their experience of failure or success.
4. Negative attitude toward the foreign language studied.
5. Compulsory nature of the foreign language study.
6. Interference of another foreign language that pupils are studying.
7. Negative attitude toward the community of the foreign language spoken.
![Page 29: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/29.jpg)
Previous Studies (Kojima, 2004, p.42)
English
Learning
Demotivation
Languag
e Level
Learning
situation
Amoun
t of
study
.58
.86
.71
.46
.41
Gramm
ar
Readin
g
Self-
confidenc
e
Writing Learning
method
Memori
zing
Vocab.
Learner
Level
Listening
problem
Class
atmospher
e
Change
of teaching
style
Teaching
approach
Teacher
personality.89
.77
.89
.89
.85 .90 .92 .80 .83 .85
GFI = 0.906
AGFI = 0.890
RMSEA = 0.052
![Page 30: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/30.jpg)
Previous Studies
Kikuchi (2009)
47 university students
open-ended questionnaires
reflection on high school days
Kikuchi and Sakai (2009)
112 university students
a 35-item questionnaire with a 5-point scale
Sakai and Kikuchi (2009)
676 high school students
a 35-item questionnaire with a 5-point scale
![Page 31: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/31.jpg)
Common demotivation factors Sakai and Kikuchi(2009)
F1: Learning Contents and Materials,
F2: Teachers’ Competence and Teaching Styles
F3: Inadequate School Facilities [Classroom Environment]
F4: Lack of Intrinsic Motivation
F5: Test Scores [Experience of Inferiority]
Kikuchi and Sakai (2009)
F1: Course Books
F2: Inadequate School Facilities
F3: Test Scores
F4: Non-Communicative Methods
F5: Teachers’ Competence and Teaching Styles
Both studies used a principal axis factor analysis using the direct
oblimin rotation
![Page 32: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/32.jpg)
Six Original Constructs Teachers: Teachers’ attitude, teaching competence,
language proficiency, personality, and teaching style
Characteristics of classes: Course contents and pace,
focus on difficult grammar or vocabulary, monotonous and
boring lessons, a focus on university entrance exams and
the memorization of the language
Experiences of failure: Disappointment due to test scores,
lack of acceptance by teachers and others, and feeling
unable to memorize vocabulary and idioms.
Class environment: Attitude of classmates, compulsory
nature of English study, friends’ attitudes, inactive classes,
inappropriate level of the lessons, and inadequate use of
school facilities such as not using audio-visual materials
Class materials: Not suitable or uninteresting materials (e.g.,
too many reference books and/or handouts)
Lack of interest: Sense of English used at schools is not
practical and not necessary. Little admiration toward English
![Page 33: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/33.jpg)
Participants (N=1,266)
![Page 34: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/34.jpg)
MethodMaterials
background questions
a 40-item questionnaire “We would like to study the situations of English study in high schools.
The following statements are possible demotivating factors for English learning. To what extent are these statements true for you? Answer based on your experience.”
Questions are revised from what I used in my previous studies (Kikuchi and Sakai, in-press; Sakai and Kikuchi, 2009).
Example of Items (1=Strongly disagree, 2=disagree, 3=agree, and 4=Strongly agree) Teachers made one-way explanations too often.
The number of students in classes was large.
A great number of textbooks and supplementary readers were assigned.
I lost my understanding of the purpose of studying English.
I could not do as well on tests as my friends.
![Page 35: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/35.jpg)
Results of EFA
An Exploratory Factor Analysis
40 items
principal axis factor analysis with a promax rotation procedure
a four-factor solution
teachers behaviors
class environment
experiences of difficulties
Lack of interest
Only 22 items left to be included in Confirmatory Factor Analysis.
![Page 36: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/36.jpg)
Table 2:Factor Analysis of DemotivationNo. Item descriptions F 1 F 2 F 3 F 4
Factor 1: Experience of difficulties(α = .87)
i16 There were too many vocabularies that I did not understand in reading. 0.81 0.03 -0.14 -0.11
i15 I had difficulty in memorizing words and phrases. 0.80 -0.13 -0.01 -0.08
i13 I got low scores on tests (such as mid-term and final examinations). 0.79 -0.17 0.06 -0.09
i8 I did not understand grammar even though I studied. 0.75 -0.02 -0.19 0.05
i39 I started not to understand the content of the class. 0.71 0.09 -0.09 0.08
i7 I could not do as well on tests as my friends. 0.62 -0.11 0.10 -0.09
i33 I got lost in how to self-study for English lessons. 0.60 -0.02 0.03 0.09
Factor 2: Teacher behavior(α = .84)
i18 I thought that the approach that teacher used was not good. -0.05 0.93 -0.14 0.00
i5 Teachers' explanations were not easy to understand. 0.03 0.86 -0.22 0.02
i17 Teachers made one-way explanations too often. 0.06 0.78 -0.07 -0.02
i6 Teachers' pronunciation of English was poor. -0.10 0.73 0.02 -0.07
i34 I could not get along with teachers. -0.10 0.68 0.12 0.05
i31 The pace of lessons was not appropriate. 0.04 0.63 0.06 -0.03
Factor 3: Class environment(α = .85)
i28 Audio materials (such as CDs and tapes) were not used. -0.08 -0.01 0.82 -0.06
i23 The Internet was not used. -0.12 -0.16 0.81 0.04
i27 Topics of the English passages used in lessons were old. 0.04 0.02 0.71 -0.11
i35 Visual materials (such as videos and DVDs) were not used. -0.03 0.05 0.69 0.04
i10 My friends did not like English. 0.02 -0.05 0.67 -0.07
i40 The number of students in classes was large. -0.03 -0.11 0.65 0.13
Factor 4: Lack of Interest(α = .90)
i3 I lost my understanding of the purpose of studying English. -0.13 0.00 0.01 0.91
i2 I lost my goal to be a speaker of English. -0.08 -0.04 0.03 0.85
i26 I think that I will not use English in my future. 0.03 0.01 -0.10 0.81
i11 I don’t have specific goals for studying English. 0.04 -0.04 0.01 0.79
i24 I lost my interest in English. 0.12 -0.02 0.01 0.71
![Page 37: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/37.jpg)
Method
Analysis
Rasch PCA of the residuals/Confirmatory
factor analysis of these six factors
Rasch PCA factor analysis /Confirmatory
factor analysis of these four factors were
conducted once again…
Conventional factor analysis confirmed
only four factors!
![Page 38: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/38.jpg)
+
Enter the RaschWith Matt
![Page 39: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/39.jpg)
+
Category utility
Measures the distance between
thresholds among the Likert-type
categories (“steps” of the scale)
![Page 40: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/40.jpg)
+
Rasch PCA output
Loading
Measure
Infit means squared
Outfit means squared
Principal components analysis
![Page 41: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/41.jpg)
+
Item fit analysis
Measure
Standard error
Infit Outfit means squared & z
![Page 42: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/42.jpg)
+
Keita Kikuchi
Kanagawa University
Demotivatio
n, Part
Deux: The
Rasched
Teacher
Behavior
GETA
e1
.69
MET
e2
.85
ONE
e3
.75
PRON
e4
.69
EXPL
e5
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.76.74.67
VID
e15
.70
GRAM
e6
Experience
of difficulties
VOCI
e7
VOC
e8
TEST
e9
SELF
e10
.72 .68 .66.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOF
e18
.82
.83.77.78.81
NOTU
e11
.74
demotivation
.74
.68
.69
d1
d2
d3
d4
.66
FRN
e16
.64
MST
e17
.61
.50
GFI=.911
NFI=.905
CFI=.920
RMSEA=.064
SRMR = .055
![Page 43: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/43.jpg)
Results
Rating Scale Instrument Quality Criteria (based on Fisher, 2007)
Criterion Poor Fair Good Very Good Excellent
Item Model Fit Mean-Square < 0.33 - >3.0 0.34 - 2.9 0.5 - 2.0 0.71 - 1.4 0.77 - 1.3
Person and item measurement
reliability <.67 .67-.80 .81-.90 .91-.94 >.94
Variance in data explained by
measures <50% 50-60% 60-70% 70-80% >80%
Unexplained variance in 1st
contrast of PCA residuals >15% 10-15% 5-10% 3-5% <3%
![Page 44: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/44.jpg)
Table 1: Variance in measure explained by each demotivator
construct.
Six Demotivator
Constructs
Variance
explained by
measure
Unexplained
variance by
measure
Unexplained
variance
explained by 1st
contrast
Teachers 57.0% 43.0% 12.3%
Characteristics of classes 45.4% 54.6% 9.7%
Experiences failure 57.1% 42.9% 12.7%
Class environment 40.4% 59.6% 13.3%
Class materials 55.1% 44.9% 10.8%
Lack of interest 62.1% 37.9% 12.8%
![Page 45: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/45.jpg)
No. Item description Logit
Score
Infit
MNSQ
Outfit
MNSQ
Factor
loadings
1. teachers(k=6, Rp=0.99, Gp=11.02)
15 Teachers shout or got angry 0.97 1.42 1.38 0.68
5 Teacher asked us to use accurate grammar 0.54 1.26 1.25 0.68
14 Teachers explanation not easy -0.85 0.79 0.85 -0.63
11 Teachers bad pronunciation 0.16 0.98 0.91 -0.45
40 Teachers’ bad teach method -0.31 0.75 0.72 -0.44
13 Teachers one-way explanation -0.51 0.83 0.84 -0.03
2. Characteristics of Classes (k=9, Rp=0.98, Gp=6.53)
10 Inappropriate pace of lesson -0.10 1.00 1.04 0.58
41 Monotonous class -0.22 0.98 1.04 0.58
1 Rare chance of communication -0.14 1.19 1.24 0.45
2 Focused on translation 0.06 0.81 0.83 -0.57
3 Focused on grammar -0.35 0.96 1.01 -0.56
42 Amount to study for mid-term/final tests -0.32 1.19 1.29 -0.20
6 Required memorizing passages in textbooks 0.26 0.94 0.95 -0.15
43 Amount of handout distributed 0.11 0.94 0.95 -0.13
4 Focused on college entrance exam. Prep. 0.69 0.90 0.89 -0.12
Results –Rasch PCA- Loadings of
Contrasts
![Page 46: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/46.jpg)
No. Item description Logit
Score
Infit
MNSQ
Outfit
MNSQ
Factor
loadings
3. Experiences of Failure (k=6, Rp=0.99, Gp=8.89)
27 Did not do well on tests compared w friends 0.64 1.04 1.06 0.76
8 low scores on school test 0.25 0.84 0.84 0.68
37 Did not understand grammar -0.15 1.07 1.07 -0.64
36 Did not understand class 0.23 1.01 1.01 -0.56
9 Get lost in self-study -0.26 1.05 1.07 -0.12
7 Could not memorize vocabulary and idiom -0.71 0.95 0.96 -0.05
4. Class Environment (k=6, Rp=0.98, Gp=7.40)
31 English being compulsory subject -0.73 1.08 1.20 0.82
26 Too many students in class 0.00 1.05 1.00 0.23
22 Video and DVDs not used -0.41 0.91 0.90 0.05
25 Audio not used 0.04 0.87 0.85 -0.59
29 Friends did not like English 0.67 0.97 1.03 -0.42
23 Internet not used 0.43 1.08 0.99 -0.31
![Page 47: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/47.jpg)
No. Item description Logit
Score
Infit
MNSQ
Outfit
MNSQ
Factor
loadings
5. Class Materials (k=6, Rp=1.00, Gp=7.30)
16 Topics of Passages uninteresting 0.14 1.06 1.09 0.57
20 Topics of Passages old 1.42 0.96 0.95 0.56
35 Unclear answers to Questions 0.22 1.01 1.03 0.41
19 too much reading 0.44 0.97 1.00 0.19
44 Many difficult Vocabulary -1.21 0.98 1.00 -0.56
18 Sentences were difficult to read -0.97 0.98 1.03 -0.55
17 Passages too long -0.04 1.00 1.01 -0.47
6. Lack of Interest (k=5, Rp=0.98, Gp=6.87)
34 No goal for being a person who can use Eng 0.31 0.92 0.89 0.75
39 No need studying English 0.63 0.85 0.84 0.73
33 Lost interest in English -0.30 1.15 1.14 -0.56
32 Lost purpose of study English -0.59 1.06 1.06 -0.42
46 No use of English in the future -0.05 0.97 0.95 -0.31
![Page 48: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/48.jpg)
Figure 1: CFA of 6 factor models of demotivators
Notes.
GFI=800
CFI=808
RMSEA=
.072
teacher
i36e1
.56
i34e2
.73i30e3
.60
i18e4.82
i17e5 .76
i6e6.68
i5e7
.74
lesson
i29e8
i25e9
i22e10
i21e11
i20e12
i19e13
i4e14
.61
.58
.60
.59
.55
.60
.47
i31e15
.59
i32e16
.46
environment
i10 e17
i23 e18
i28 e19
i35 e20
i37 e21
i40 e22
.63
.67
.71
.70
.65
.62
material
i1 e23
i9 e24
i12 e25
i14 e26
i16 e27
i27 e28
.53
.57
.50
.56
.59
.57
interest
i2 e29
i3 e30
i11 e31
i24 e32
i26 e33
.82
.83
.77
.79
.81
failure
i39
e35
i33
e36
i15
e37
i13
e38
i8
e39
i7
e40
.74.68.67.71.67.60
i38 e41
.68
.79
.37
.27
.59
.54
.70
.56
.93
.79
.54
.88
.52
.73
.63
.57
![Page 49: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/49.jpg)
Figure 2.
CFA of
4 factor
models
Notes.
GFI=916
CFI=926
RMSEA=
.062
Teacher
Behavior
GETA
e1
.69
MET
e2
.85
ONEW
e3
.75
PRON
e4
.69
EXPL
e5
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.76.74.67
VID
e15
.70
GRAM
e6
Experience
of difficulties
VOCI
e7
VOC
e8
TEST
e9
SELFS
e10
.71 .67 .66.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOU
e18
.82
.83.77.79.81
NOTU
e11
.75.67
FRN
e16
.64
MST
e17
.61
.46
.51
.44
.53
.23
.35
![Page 50: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/50.jpg)
Teacher
Behavior
GETA
e1
.69
MET
e2
.85
ONE
e3
.75
PRON
e4
.69
EXPL
e5
.77
Environment
AUD
e14
TOPC
e13
INTR
e12
.76.74.67
VID
e15
.70
GRAM
e6
Experience
of difficulties
VOCI
e7
VOC
e8
TEST
e9
SELF
e10
.72 .68 .66.68
Lack of
Interest
NOP
e22
NON
e21
NOG
e20
NOI
e19
NOF
e18
.82
.83.77.78.81
NOTU
e11
.74
demotivation
.74
.68
.69
d1
d2
d3
d4
.66
FRN
e16
.64
MST
e17
.61
.50
GFI=.911
NFI=.905
CFI=.920
RMSEA=.064
SRMR = .055
Figure 3.
My tentative
model of
demotivation
![Page 51: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/51.jpg)
Results –Rasch PCA of four factor
models-
Four Demotivator
Constructs
Variance
explained by
measure
Unexplained
variance by
measure
Unexplained
variance explained
by 1st contrast
Teachers 58.5% 41.5% 10.7%
Experience of difficulties 58.0% 42.0% 12.2%
Class environment 39.9% 60.1% 13.3%
Lack of interest 54.8% 45.2% 15.1%
![Page 52: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/52.jpg)
This is how the poor factor was
working…
![Page 53: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/53.jpg)
An activity for this workshop
Let’s try to make item bank of questionnaire items together for your practice.
Topic is Demotivating factors in English education for communication in Japan.
With your partner, think of constructs first and write items for each construct. How many constructs? How many items for each of them? Remember “the Line”!
Please use the questionnaire items that you have in your handout about demotivating factors in English education in high school English classroom to generate your discussion.
![Page 54: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/54.jpg)
References
Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.
Dörnyei, Z. (2001). Teaching and researching motivation. Harlow: Longman.
Fisher, W. P. (2007). Rating Scale Instrument Quality Criteria. Retrieved November 25, 2007, from http://www.rasch.org/rmt/rmt211m.htm
Kikuchi, K. (2009). Student demotivation in Japanese high school English classrooms: Exploring with qualitative research methods. Language Teaching Research, 13(4), pp.453-471.
Kikuchi, K. (forthcoming). What are possible demotivators in SLA? –An insight from English teaching contexts in Japan. Multilingual Matters
Kojima, S. (2004). English learning demotivation in Japanese EFL students: Research in demotivational patterns from the qualitative research results of three different types of high schools. Unpublished master thesis, Kwansei Gakuin University, Hyogo, Japan.
Linacre, J. M. (1997). Guidelines for rating scales. Retrieved November 25, 2007, from http://www.rasch.org/rn2.htm.
Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures: Issues and applications. Thousand Oaks, CA: Sage Publications.
Sakai, H., & Kikuchi, K. (2009). Japanese learners' demotivation to study English: A survey study? JALT Journal, 31 (2), pp.183-204.
Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum.
![Page 55: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/55.jpg)
![Page 56: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/56.jpg)
![Page 57: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/57.jpg)
![Page 58: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/58.jpg)
Q & A and discussion
![Page 59: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/59.jpg)
The Rasch model
(Rasch, 1960)
![Page 60: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/60.jpg)
The Rating Scale Model
(Andrich, 1978)
![Page 61: Don't walk: Rasch to join the questionnaire trend!](https://reader034.vdocuments.us/reader034/viewer/2022042817/559d6eb61a28abe32b8b46bd/html5/thumbnails/61.jpg)
+Don’t walk:
Rasch to
join
the
questionnai
re trend!
Ritsumeik
an
University
Keita
Kikuchi
Kanagaw
a
University
J. W.
Lake
Fukuoka
Women’s
University
Send future inquires to Keita: