+ Keita Don’t Kikuchi Kanagaw walk: a Rasch to University J. W. Lake join Fukuoka Women’s the Universit questionna y ire trend!
+ Keita Structure Kikuchi Kanagaw of this a workshop University J. W. Lake Fukuoka Women’s Universit y Get ready to be Rasched… (This is supposed to be a joke. Laugh.)
+Structure of this workshop Rasch concepts Terminology and explanations
+Structure of this workshop J. W. Lake Fukuoka Women’s Universit y Steps to create a questionnaire Constructs, concepts, items, piloting, evaluation, revision…
GFI=.911 NFI=.905 CFI=.920 RMSEA=.064 SRMR = .055 + e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 e11 GETA MET ONE PRONEXPLGRAMVOCI VOC TEST SELF NOTU Keita Struct.u 69 .85r.75 .69 .77 .66 .72 .68 .66 .68 .74 Te e acher d1 d3 Experience Behavior
Kikuchi .50 of difficulties .68 demotivation Kanagaw of thisd2 .69 .74 Environment Lack of Interest a d4 works .61 .82 .64 .67 .74 .70 .76 .83 .81 .78 .77 INT h RTOP o C AU p University D VID FRN MST NOF NOI NOG NON NOP e12 e13 e14 e15 e16 e17 e18 e19 e20 e21 e22 Demotivation: An example The flaws of traditional FA and why Rasch can help questionnaire creation
+ Rasch concepts Matthew Apple Ritsumeikan University Department of Communication International Communication Program
+ Rasch log-odds (logits) Rasch measures (logit scores) Infit/Outfit (means sq. and z-score) Item difficulty / endorsability Person/item reliability / separation Rasch terms Construct validity and A probabilistic model unidimensionality (not strictly speaking Rasch, but…) Rasch Principal components analysis (Rasch PCA) Loadings Contrasts and Residuals
+ Logits (log-odds) The probability of a person correctly answering an item 50% of the time
+ Fit 0.75 to 1.3 logits 0.60 to 1.4 logits “Within 2 standard deviations of the mean” “Only Outfit z-scores of 3.0”
+ Separation The ratio of error-free variance and observed variance (Fisher, 1992) The number of groups distinguishable by the measurement instrument (Wilson, 2005; Wright, 1996)
+ Item map Persons and items on the same linear logit scale
+ The “Line” Items and persons on the same scale Item-person map (or) Wright map Ben Wright
+ The “Line” Items and persons on the same scale Item-person map (or) Wright map
+ Likert-scale data Likert-type category data Questionnaires do not produce true interval but ordinal data The steps in the “scale” can be How Rasch works conceived as thresholds (τ) for questionnaires named after Thurston, Refer to Andrich (1977, 1978) originator of factor analysis)
+ How Rasch works for questionnaires Refer to Andrich (1977, 1978)
+ 5 N How Rasch works 4 for questionnaires Refer to Andrich (1977, 1978) 3 2 1
+ 5 N How Rasch works 4 for questionnaires Refer to Andrich (1977, 1978) 3 2 1 1 + 3 = 4
+ 5 N How Rasch works 4 for questionnaires Refer to Andrich (1977, 1978) 3 2 1 SD + N = A? 1 + 3 = 4?
+Steps to create a questionnair e J W Lake Fukuoka Women’s University
Steps in scale development: Issues to consider (Netemeyer, Bearden, Sharma, 2003) Step 1: Construct definition and content domain The importance of clear construct definition, content domain, and the role of theory. Construct dimensionality: unidimensional, multidimensional, or a higher-order construct? Determine the purpose of the scale: measurement or correlational analysis or model building
Step 2: Generating and judging measurement items Theoretical assumptions about items (e.g., domain sampling) Generating potential items and determining the response format How many items as an initial pool Dichotomous vs. polytomous response formats Item wording issues The focus on “content” validity in relation to theoretical dimensionality Item judging (expert and layperson) --- the focus on “content” and “face” validity
Step 3: designing and conducting studies to develop and refine the scale Pilot testing as an item-trimming procedure The use of several samples from relevant populations for scale development Designing the studies to test psychometric properties Initial item analyses via exploratory factor analyses (EFAs) Initial item analyses and internal consistency estimates Retaining items for the next studies EFA may be useful for correlational analysis or model building
Step 4: Finalizing the scales The importance of several samples from relevant populations Designing the studies to test the various types of validity Item analysis via EFA The importance of EFA consistency from Step 3 to Step 4 Deriving an initial factor structure—dimensionality and theory Item analyses and confirmatory factor analyses (CFAs) Testing the theoretical factor structure and model specification Evaluating CFA measurement models Factor model invariance across studies (i.e., multiple-group analysis)
GFI=.911 NFI=.905 CFI=.920 RMSEA=.064 SRMR = .055 + e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 e11 GETA MET ONE PRONEXPLGRAMVOCI VOC TEST SELF NOTU Demot.69i.8v 5 .75 .69 .77 .66 .72 .68 .66 .68 .74 Teac a her d1 d3 Experience Behavior ti.50 of difficulties on: An d2
.68 demotivation .69 .74 Environment Lack of Interest d4 exam .61 .82 .64 .67 .74 .70 .76 .83 .81 .78 .77 I p NTRT l OP e C AUD VID FRN MST NOF NOI NOG NON NOP e12 e13 e14 e15 e16 e17 e18 e19 e20 e21 e22 Keita Kikuchi Kanagawa University
Example Study focused on specific external /internal forces that Japanese high school students may experience which might cause their motivation to be reduced or diminished administered the questionnaire asking high school students to report what diminished their motivation to study in their high school days, which contained 40 Likert-scale questions (4-points) analyzed the quantitative data using a confirmatory factor analysis using Amos and Rasch PCA of the residuals using Winsteps. If you’d like to read this process thoroughly, please locate Kikuchi (forthcoming).
Demotivation Dörnyei (2001) Definition of demotivation “specific external forces that reduce or diminish the motivational basis of a behavioral intention or an ongoing action” (p. 143). I expand this definition and explore demotivators including both internal and external forces.
Previous Studies (Dörnyei, 1998)
Based on interviews with 50 secondary school students, he identified following as demotivators, the reason to get demotivated. 1. Teachers’ personalities, commitments, competence, teaching methods. 2. Inadequate school facilities (very big group, not the right level or frequent change of teachers). 3. Reduced self-confidence due to their experience of failure or success. 4. Negative attitude toward the foreign language studied. 5. Compulsory nature of the foreign language study. 6. Interference of another foreign language that pupils are studying. 7. Negative attitude toward the community of the foreign language spoken. 8. Attitudes of group members. 9. Course books used in class.
Previous Studies (Kojima, 2004, p.42) Teacher .89 personality Learning Languag situation .77 Change e Level of .89 teaching .58 .41 style Teaching English .46 .89 approach Learning Demotivatio Amoun n .71 t of Class atmospher .86 study e GFI = 0.906 Listening Learne problem AGFI = 0.890 r Level RMSEA = 0.052 .85 .90 .92 .80 .83 .85 Gramm Readin Writing Learning Memori Self- ar g method zing confiden Vocab. ce
47 university students
reflection on high school days
Kikuchi and Sakai (2009)
112 university students
a 35-item questionnaire with a 5-point scale
Sakai and Kikuchi (2009)
676 high school students
a 35-item questionnaire with a 5-point scale
Common demotivation factors
Sakai and Kikuchi(2009) F1: Learning Contents and Materials, F2: Teachers’ Competence and Teaching Styles F3: Inadequate School Facilities [Classroom Environment] F4: Lack of Intrinsic Motivation F5: Test Scores [Experience of Inferiority]
Kikuchi and Sakai (2009) F1: Course Books F2: Inadequate School Facilities F3: Test Scores F4: Non-Communicative Methods F5: Teachers’ Competence and Teaching Styles
Both studies used a principal axis factor analysis using the direct oblimin rotation
Six Original Constructs Teachers: Teachers’ attitude, teaching competence, language proficiency, personality, and teaching style Characteristics of classes: Course contents and pace, focus on difficult grammar or vocabulary, monotonous and boring lessons, a focus on university entrance exams and the memorization of the language Experiences of failure: Disappointment due to test scores, lack of acceptance by teachers and others, and feeling unable to memorize vocabulary and idioms. Class environment: Attitude of classmates, compulsory nature of English study, friends’ attitudes, inactive classes, inappropriate level of the lessons, and inadequate use of school facilities such as not using audio-visual materials Class materials: Not suitable or uninteresting materials (e.g., too many reference books and/or handouts) Lack of interest: Sense of English used at schools is not practical and not necessary. Little admiration toward English speaking people
Method Materials background questions a 40-item questionnaire “We would like to study the situations of English study in high schools. The following statements are possible demotivating factors for English learning. To what extent are these statements true for you? Answer based on your experience.” Questions are revised from what I used in my previous studies (Kikuchi and Sakai, in-press; Sakai and Kikuchi, 2009). Example of Items (1=Strongly disagree, 2=disagree, 3=agree, and 4=Strongly agree) Teachers made one-way explanations too often. The number of students in classes was large. A great number of textbooks and supplementary readers were assigned. I lost my understanding of the purpose of studying English. I could not do as well on tests as my friends.
Results of EFA An Exploratory Factor Analysis 40 items principal axis factor analysis with a promax rotation procedure a four-factor solution teachers behaviors class environment experiences of difficulties Lack of interest Only 22 items left to be included in Confirmatory Factor Analysis.
Table 2:Factor Analysis of Demotivation No. Item descriptions F 1 F 2 F 3 F 4 Factor 1: Experience of difficulties(α = .87) i16 There were too many vocabularies that I did not understand in reading. 0.81 0.03 -0.14 -0.11 i15 I had difficulty in memorizing words and phrases. 0.80 -0.13 -0.01 -0.08 i13 I got low scores on tests (such as mid-term and final examinations). 0.79 -0.17 0.06 -0.09 i8 I did not understand grammar even though I studied. 0.75 -0.02 -0.19 0.05 i39 I started not to understand the content of the class. 0.71 0.09 -0.09 0.08 i7 I could not do as well on tests as my friends. 0.62 -0.11 0.10 -0.09 i33 I got lost in how to self-study for English lessons. 0.60 -0.02 0.03 0.09 Factor 2: Teacher behavior(α = .84) i18 I thought that the approach that teacher used was not good. -0.05 0.93 -0.14 0.00 i5 Teachers' explanations were not easy to understand. 0.03 0.86 -0.22 0.02 i17 Teachers made one-way explanations too often. 0.06 0.78 -0.07 -0.02 i6 Teachers' pronunciation of English was poor. -0.10 0.73 0.02 -0.07 i34 I could not get along with teachers. -0.10 0.68 0.12 0.05 i31 The pace of lessons was not appropriate. 0.04 0.63 0.06 -0.03 Factor 3: Class environment(α = .85) i28 Audio materials (such as CDs and tapes) were not used. -0.08 -0.01 0.82 -0.06 i23 The Internet was not used. -0.12 -0.16 0.81 0.04 i27 Topics of the English passages used in lessons were old. 0.04 0.02 0.71 -0.11 i35 Visual materials (such as videos and DVDs) were not used. -0.03 0.05 0.69 0.04 i10 My friends did not like English. 0.02 -0.05 0.67 -0.07 i40 The number of students in classes was large. -0.03 -0.11 0.65 0.13 Factor 4: Lack of Interest(α = .90) i3 I lost my understanding of the purpose of studying English. -0.13 0.00 0.01 0.91 i2 I lost my goal to be a speaker of English. -0.08 -0.04 0.03 0.85 i26 I think that I will not use English in my future. 0.03 0.01 -0.10 0.81 i11 I don’t have specific goals for studying English. 0.04 -0.04 0.01 0.79 i24 I lost my interest in English. 0.12 -0.02 0.01 0.71
Method Analysis Rasch PCA of the residuals/Confirmatory factor analysis of these six factors Conventional factor analysis confirmed only four factors! Rasch PCA factor analysis /Confirmatory factor analysis of these four factors were conducted once again…
+ Enter the Rasch With Matt
+ Category utility Measures the distance between thresholds among the Likert-type categories (“steps” of the scale)
+ Rasch PCA output Loading Measure Infit means squared Outfit means squared Principal components analysis
+ Item fit analysis Measure Standard error Infit Outfit means squared & z
GFI=.911 NFI=.905 CFI=.920 RMSEA=.064 SRMR = .055 + e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 e11 GETA MET ONE PRONEXPLGRAMVOCI VOC TEST SELF NOTU Demot.69i.8v 5 .75 .69 .77 .66 .72 .68 .66 .68 .74 Teac a her d1 d3 Experience Behavior ti.50 of difficulties on, Pad2rt .68 demotivation .69 .74 Env ironment Lack of Interest d4 Deux .61 .82 .64 .67 .74 .70 .76 .83 .81 .78 .77 I : NT RT T OPC Ah UD e VID FRN MST NOF NOI NOGNONNOP e12 e13 e14 e15 e16 e17 e18 e19 e20 e21 e22 Rasched Keita Kikuchi Kanagawa University
Results Rating Scale Instrument Quality Criteria (based on Fisher, 2007) Criterion Poor Fair Good Very Good Excellent Item Model Fit Mean-Square < 0.33 - >3.0 0.34 - 2.9 0.5 - 2.0 0.71 - 1.4 0.77 - 1.3 Person and item measurement <.67 .67-.80 .81-.90 .91-.94 >.94 reliability Variance in data explained by <50% 50-60% 60-70% 70-80% >80% measures Unexplained variance in 1st >15% 10-15% 5-10% 3-5% <3% contrast of PCA residuals
Table 1: Variance in measure explained by each demotivator construct.
e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 e11 Figure GETA METONEWPRONEXPLGRAMVOCI VOC TESTSELFSNOTU 2. .85 CFA of .69 .75 .69 .77 .67 .71 .67 .66 .68 .75 4 factor .44 .23 models Teacher Experience Behavior of difficulties
.51 .46 .53 .35 Environment Lack of Interest Notes. .61 .82 GFI=916 .64 .67 .74 .70 .76 .83 .81 .79 .77 CFI=926 INTRTOPC AUD VID FRN MST NOU NOI NOG NON NOP RMSEA= .062 e12 e13 e14 e15 e16 e17 e18 e19 e20 e21 e22
GFI=.911 Figure 3. NFI=.905 CFI=.920 My tentative RMSEA=.064 model of SRMR = .055 demotivation e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 e11 GETA MET ONE PRONEXPLGRAMVOCI VOC TEST SELF NOTU .69 .85 .75 .69 .77 .66 .72 .68 .66 .68 .74 Teacher d1 d3 Experience Behavior .50 of difficulties .68 demotivation .69 .74 d2 Environment Lack of Interest d4 .61 .82 .64 .67 .74 .70 .76 .83 .81 .78 .77 INTRTOPC AUD VID FRN MST NOF NOI NOG NON NOP e12 e13 e14 e15 e16 e17 e18 e19 e20 e21 e22
Results –Rasch PCA of four factor models-
This is how the poor factor was working…
An activity for this workshop Let’s try to make item bank of questionnaire items together for your practice. Topic is Demotivating factors in English education for communication in Japan. With your partner, think of constructs first and write items for each construct. How many constructs? How many items for each of them? Remember “the Line”! Please use the questionnaire items that you have in your handout about demotivating factors in English education in high school English classroom to generate your discussion.
References Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum. Dörnyei, Z. (2001). Teaching and researching motivation. Harlow: Longman. Fisher, W. P. (2007). Rating Scale Instrument Quality Criteria. Retrieved November 25, 2007, from http://www.rasch.org/rmt/rmt211m.htm Kikuchi, K. (2009). Student demotivation in Japanese high school English classrooms: Exploring with qualitative research methods. Language Teaching Research, 13(4), pp.453-471. Kikuchi, K. (forthcoming). What are possible demotivators in SLA? –An insight from English teaching contexts in Japan. Multilingual Matters Kojima, S. (2004). English learning demotivation in Japanese EFL students: Research in demotivational patterns from the qualitative research results of three different types of high schools. Unpublished master thesis, Kwansei Gakuin University, Hyogo, Japan. Linacre, J. M. (1997). Guidelines for rating scales. Retrieved November 25, 2007, from http://www.rasch.org/rn2.htm. Netemeyer, R. G., Bearden, W. O., & Sharma, S. (2003). Scaling procedures: Issues and applications. Thousand Oaks, CA: Sage Publications. Sakai, H., & Kikuchi, K. (2009). Japanese learners' demotivation to study English: A survey study? JALT Journal, 31 (2), pp.183-204. Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum.
Q & A and discussion
The Rasch model (Rasch, 1960)
The Rating Scale Model (Andrich, 1978)
+ Keita Don’t Kikuchi Kanagaw walk: a Rasch to University J. W. Lake join Fukuoka Women’s the Universit questionna y ire trend! Send future inquires to Keita: firstname.lastname@example.org