john andersontodd rogers don klinger university of victoriauniversity of alberta queen’s...
TRANSCRIPT
John Anderson Todd Rogers Don Klinger University of Victoria University of Alberta Queen’s University
Charles Ungerleider Barry Anderson Victor GlickmanUniversity of British ColumbiaBC Ministry of Education Edudata Canada
Funding
Canadian Education Statistics CouncilCanadian Education Statistics Council
Social Sciences & Humanities Research CouncilSocial Sciences & Humanities Research Council
The project focus
Modeling the relationships of student, school, and home characteristics to
the achievement of learning outcomesin the domains of
reading, writing, mathematics and science
Utilizing hierarchical linear modeling
&
School Achievement Indicators Program
Education Quality & Accountability Office program
Alberta Provincial Language Arts & Mathematics Achievement Tests
BC Foundational Skills Assessment program
datasets
Outcomes
Data issuesGraduate researchFindings
Next Steps
Data issues Complexity of datasets
Problem solving – age 13
Problem solving – age 16
Math content – age 13
Math content – age 16
Student achievement tests
Student Questionnaires
Teacher Questionnaires
Principal Questionnaires
Data issues Organization of assessment program
School-based
First, it should be noted that for both Language Arts and Mathematics, most of the variation in achievement was among students :
• 77.1% in Language Arts
• 75.1% in Mathematics.
class level:
• 15.3% for Language Arts
• 15.7% for Mathematics.
school level
•10.1% for Language Arts
•11.3% for Mathematics.
The Alberta study
__________________________________________________ Test ρ
_________________________________________________
SAIP Math 2001
Problem solving – age 13 0.18Problem solving – age 16 0.15
Math content – age 13 0.19Math content – age 16 0.15
OSSLT
Reading 0.13Writing 0.10
__________________________________________________
PISA average is 0.34 and ranges from .04 to 0.63
Data issues Data integrity
Student Gender Distribution
Gender: Inside questionnaire
Gender/Cover Male Female Total
Male 4,456 1,388 5,166
Female 1,563 4,689 5,589
Total 6,019 6,077 12,096
Data issues Missing Data
SAIP Math
Parental Educational Level (Items 24 a&b)
34% missing on mother
36% missing on father
Parental Vocational Status (Items 25a&b)
53% missing on mother
40% missing on father
Data issues
Large number of variables
Student beliefs about mathematics
Derived variables from Student QuestionnaireDerived variables from Student Questionnaire
•Math is more difficult than other school subjects
•I am not very interested in mathematics
•I learn lots of new things in mathematics
•Math is an important school subject
•Math is important for my future studies
•Many good jobs require math
Derived variables from Student QuestionnaireDerived variables from Student Questionnaire
• You & your parents work on math homework
• You & your parents work on other homework
• In math course we work in pairs or small groups
• In math we use math books & magazines
• In math we had guest speakers or experts
• In math we use computers
• In math we use the internet
• In math we use the computer lab
Instructional supports used by students
Derived variables from Student QuestionnaireDerived variables from Student Questionnaire
Instructional practices In math courses this year. . . • The teacher gives notes• The teacher shows us how to do problems • We participate in math projects • We are taught different ways to solve problems • The teacher assigns homework • We discuss quiz or tests • We work alone on assigned work • We work on exercises from textbook • We study the textbook • The teacher reads from the textbook • Teachers asks questions of students • Students ask teacher questions
Causes of math performance
• To do well in math you need hard work
• To do well in math you need encouragement - teachers
• To do well in math you need encouragement - parents
• To do well in math you need good teaching
Derived variables from Student QuestionnaireDerived variables from Student Questionnaire
Disciplinary climate
In math courses this year. . .
• There is noise or disorder in the classroom
• We lose 5-10 minutes because of disruptions
Derived variables from Student QuestionnaireDerived variables from Student Questionnaire
Graduate researchCSSE 2004
Potential and Pitfalls of Secondary Data Analyses of SAIP data. Todd Rogers & Teresa Dawber, U of AlbertaCSSE 2005: The COLO Project 2005 Graduate Symposium
Student and school indices in SAIP questionnairesCarmen Gress & Shelley Ross, UVic
Correlates of mathematics achievement: a meta-synthesis
Margot English, Shelley Ross, Carmen Gress, UVicIssues and results arising from the HLM analysis of the Ontario Secondary School Literacy Test.
Chloe Soiblelman, Jinyan Huang, Cheryl Poth, &
Don Klinger, Queen’s UniversityFactors that influence writing performance
Jiawen Zhou, University of AlbertaStability of SAIP Factor Analysis: Results from school questionnaire items
Ally Feng, University of Alberta
Findings
No grand general models
Student level (level 1) coefficients
_______________________________________________________________________ Correlate CONTENT PROBLEM
13 16 13 16 _______________________________________________________________________
Student math beliefs .36 .33 .38 .37
Instructional supports -.18 -.22 -.22 -.29
Instructional practices .03 .08 .04 .10
Causes of math -.08 0 -.06 0
Discipline climate 0 0 0 0
Gender 0 -.09 .10 0.7
School level (level 2) coefficients for average school math score
_______________________________________________________________________ Correlate CONTENT PROBLEM
13 * 16 * 13 * 16 * _______________________________________________________________________
Limits to learning -.14 -.21 -.14 -.18
Instructional supports -.12 -.19 -.10 -.19
Causes of math 0 -.22 0 -.22
Discipline climate 0 0 0 -.17Student math beliefs .13 0
.11 0
School climate 0 0 -.04 -.05Parent engagement 0 .05 0 .06Student status .07 .06 .05 0Student achievement .05 0 .05 0Instructional practices .09 0 0 0
Findings Perhaps no grand models, but
As Lindblom (1968, 1990) has pointed out time and again, the desire that models of complex social systems such as public
education have an instrumental use remains an elusive dream.
Models of complex social systems are likely to be, at best, enlightening – allowing incrementally expanding understandings of
complex and dynamic systems such as public schools
(Kennedy, 1999)
The low rho suggests that
• Canadian schools are relatively homogeneous
&
• Most variation in achievement results lie within classrooms and
between students
Findings The specificity of models to grade
and domain suggests that the correlates of learning outcomes have to be considered within the
context of specific learning situations
For example . . . . .
Findings
The SAIP Math models show that
Student attitudes about math are related to achievement
Student dependence is related to math achievement
The views of school principals in regard to instructional impediments are related to
average school math scores
Gender tends to have a much reduced relationship to achievement when other
variables are entered into the model
Climate, Discipline and Parental Involvement – non-operative
Next Steps
Linkage with other assessment programs
Work with other educational partners:
Teachers
Parents
Ministries
Communications
Data collection
Analysis and application