theory and methods in ost research
DESCRIPTION
Theory and Methods in OST Research. Denise Huang. American Educational Research Association Annual Meeting AERA San Diego, CA - April 14, 2006. What do all these mean?. The gold standard for research design and analysis involve experimental randomization. - PowerPoint PPT PresentationTRANSCRIPT
American Educational Research AssociationAnnual Meeting
AERA San Diego, CA - April 14, 2006
Denise Huang
Theory and Methods in OST Research
2 / 28
Experimental Qualitative Evaluation
Purpose New knowledge, Truth To uncover and understand what lies behind any phenomenon
Mission achievement, product delivery
Outcome Generalizable conclusions Emergence of theory Specific decisions
Value Explanatory and predictive power
Rich descriptive dataInsiders’ point of view
Determining worth and social utility
Impetus Curiosity and ignorance Understanding Needs and goals
Conceptual Basis Cause and effect relationships Experiences as a whole, in context, interactive
Means ends processes
Key event Hypothesis testing Observations of multiple, shifting axioms
Assessing attainment of an objective
Classic paradigms Experimental methodCorrelational method
Grounded theory approach System approachObjective approach
Discipline Control & manipulation of variance
Iterations Program planning and management
Criteria Internal and external validity Fit, understanding, generality, and control
Isomorphism and credibility
Functional type Pure and appliedTrue experimental Quasi-experimental
Grounded theoryEthnography,Phenomenon approachCase studies, etc.
Formative – summativeProcess – product
3 / 28
What do all these mean? The gold standard for research design and analysis involve
experimental randomization.
Satisfactory research findings required generalization and replication.
The afterschool population has distinct characteristics
Ethical issues
Self-selective group
Difficulty in obtaining comparison group (consent forms)
Accuracy of data and availability of longitudinal records
High transience
Qualitative strategies are vital tools in finding out the intricate rich data about a program and help to explain, elaborate, and triangulate quantitative findings…
Matching study questions with design and appropriate methodology
4 / 28
Most afterschool studies are evaluation research
The systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs (Strauss & Corbin, 1990).
• Focus on finding and explaining educational /social effects or outcomes from the intervention, and
• Devising instructional strategies that will improve the outcomes.
Formative and summative evaluations
Process and impact evaluations
5 / 28
Different types of evaluation research
Process Evaluations:
Formative studies
• Evaluations that assesses the conduct of the program during the initial design and testing stages with the intent to improve the program
Program monitoring
• Systematic examination of program coverage and delivery- (target population, fidelity, efficiency)
• Identifying successful implementation strategies for program diffusion
6 / 28
Impact Evaluations
Assesses the changes in the well-being of individuals that can be attributed to a particular intervention, such as a project, program or policy (Strauss & Corbin, 1990).
Summative evaluation
• Summative evaluation provides information on the product's efficacy ( it's ability to do what it was designed to do)
• By looking at the intervention group, the evaluator can examine the learning materials and learning process together with the outcomes-- hence the name Summative Evaluation.
Impact Evaluation
• Impact evaluation involves constructing a counterfactual
• Random selection and isolation from interventions are seldom practicable and sometimes ethically difficult to defend.
• Quasi-experimental method is often used.
7 / 28
Which theoretical perspective informed your work, and why did you select this theory?
• Goal setting theory
• Social cognitive theory
• The contextual understanding of the social ecological model
8 / 28
The role that theory play in research
Theory is an invention aimed at organizing and explaining specific aspects of the environment (Pedhazur & Schmelkin, 1991)
• Theory provides the researcher with a “selective point of view-an orientation
• Helps to determine what variable is relevant and which are not relevant
Being a way of seeing, a theory is also a way of not seeing.
9 / 28
How did this theory shape your research methods?
• Mixed methods
• Process/outcome
• SEM
• HLM
10 / 28
Examples: Formative, Process, & Summative
The ASL/KidzLit Formative Study
ASL is a reading and writing project that was based on the theoretical foundation of the literature-based instructions (social cognitive theory)
• How do the LA’s BEST teachers evaluate the ASL project in terms of training, materials, and support?
• To what extent have staff capabilities been enhanced after the ASL training?
• If the staff members teach during the regular school day and in the after-school program, to what extent have they carried the ASL strategies into their regular school day teaching?
• What changes in student attitudes might be associated with ASL? Are there any preliminary indicators that participation in ASL is affecting students?
11 / 28
Teacher survey I-formative evaluation
For the effectiveness of the ASL training, the evaluation based on ASL’s four core components:
• Care
• Talk
• Read
• Connect
To evaluate the ASL project as a whole, teachers’ perspectives were examined:
• ASL materials
• ASL training
• Support for the ASL project
For an affective/motivational outcome the following indicators were examined:
• Students’ enjoyment
• Teacher attitude
12 / 28
The student survey
Based on the social cognitive theory that learning is a social phenomenon and social collaboration is essential in the classroom climate, we investigate on students’:
• Enjoyment of their reading class;
• Perception of the social support in the classroom;
• Perception of their relation with their teachers.
Based on the motivational theory, we look at the students’:
• Attitudes towards reading and writing; and
• Reading and writing self-efficacy.
And students’ perception of the classroom climate:
• The autonomy orientation of their classroom climate; and
• The emphasis on effort allocation.
13 / 28
Qualitative studyExploring the intellectual, social, and organizational capitals at LA’s
BEST
Relationships
Expectations &Social Norms
Trust
Staff & Student Relationships
Staff & Student Relationships
Student Engagement
Student Engagement
Organizational CapitalOrganizational
CapitalOrganizational
Capital
Social CapitalSocial Capital
Social Capital
Intellectual CapitalIntellectual Capital
Intellectual Capital
Internal Networks External Networks Innovation and Staff Development
Competence
Attitudes
Problem-solving& Agility
14 / 28
Quantitative outcome evaluations
Keeping kids in school study
• 4 cohorts of students (all LA’s BEST participants and a stratified random sample of non-participants)
• 3 participation level ( 1,2,3+ years)
• 6-9th grade in 1998-1999 followed through to 2002-2003
• Chi-square analysis
• Cox survival analysis
15 / 28
Comparison of dropout rates for LA’s BEST vs. LAUSD non-participants (3+ Years).
16 / 28
The Afterschool Hours-SEM and HLM
Examining the Relationship between Afterschool Staff-Based Social Capital and Student Engagement in LA’s BEST
• What are LA’s BEST staff perceptions of collective staff efficacy, teamwork, and communication, and the quality of their relationships with students?
• What are student perceptions of their relationships with LA’s BEST staff? To what extent do they value education and have high aspirations for their futures? What are their reported levels of student engagement in LA’s BEST and the day school?
• How are staff-student relationships, teamwork and communication, and collective staff efficacy at the site level, related to student perceptions of their relationships with staff?
• What is the association between student perceptions of their relationship with staff, their value of education, future aspirations, and engagement in the afterschool program and day school?
17 / 28
Hierarchical Linear Modeling
• Examine the relationship between the social capital predictors measured by the staff surveys (i.e., staff-student relationships, collective staff efficacy, and communication and teamwork) and student perceptions of social capital (i.e. staff-student relationship) as measured by the student surveys.
• The demographic variables of gender, grade level, and languages spoken were also included in each model to control for individual student differences.
• A total of 2,270 students and 395 staff from 50 school sites were included in the HLM analysis.
18 / 28
Structural Equation Modeling
Staff/Student Relationships
LAB Engagement
School Engagement
Value of Education
0.82
0.23
0.34
0.74
-0.30
0.41
Future Aspirations
0.87
19 / 28
DOJ study-using propensity matching & cost-benefit analysis
Quasi-experimental design
Longitudinal sampling of academic and juvenile crime data
Advanced multilevel propensity scores methods to establish study samples
Hierarchical growth modeling and survival analysis (multilevel discrete-time hazard)
Students were followed from 1994-1995-2002-2003 school years
Cost-benefit analysis
20 / 28
How did your method(s) contribute to our knowledge of OST, and what were the limitations of the method?
• Better comparison sample—propensity matching
• The importance of “dosage”
• Importance of goal setting
• Importance of employing a continuous improvement model
21 / 28
Why results are inconsistent?
• Linking results to program content
__Design___________________________________________
• Linking results to implementation procedures(program fidelity)
__Treatment________________________________________
• Linking results to program attendance
__Dosage___________________________________________
22 / 28
What does that theory mean for what you would do with children and youth during the out-of-school time and inform program design?
Importance of having
• A theory of change
• A logic model
• A continuous improvement strategy
23 / 28
Why create a logic model?
• They are powerful tools for designing, planning, implementing, and evaluating OST programs
• They set up a plan of how a program is expected to work
• Provide a map of how to achieve goals (cause and effect-program process and outcomes)
• Stimulate clear thinking, preparation, and organization
24 / 28
Example: Theoretical-based logic model designed for the Afterschool Partnership study
Program SupportManagement
Staff /ResourcesPeriodical Evaluation for Continuous
Improvement
Periodic Assessment of
StudentPerformance
Linkage to School Day
Linkage to School Day Research Based
Practice & Strategies
Research Based Practice & Strategies
Content Practice Quality
After School Center Quality
Content Alignedto Standards
Content Alignedto Standards
Set Goals
Align activitiesto goals
Motivation
Engagement Opportunities to Practice
Process
Structure
Content
Process
25 / 28
Example: Outcome based logic model
Linkage with School
ContentAcademic
Design
ContentAcademic
Design
ProcessAfterschool
Program Implementation
ProcessAfterschool
Program Implementation
Attitudes & Skills
Attitudes & Skills
EngagementEngagement
Linkage with Standard
Research-based
Internal Program Structure
External Communication
& Support
Student/Adult Interaction
Parent Satisfaction
Structure Program Structure
Structure Program Structure
Indicators Design & Process Immediate Outcome Expected Outcome Long-term Outcome
Professional Development
AchievementAchievement
26 / 28
Using data for continuous improvement
Plan of ActionPeriodic
Assessment
Analysis of Results
Specification of MonitoringContinuous
Adjustment
Data Based decision Making Process
Theory and Research-
based Programming
Student Engagement
Student Retention
Student Improvement
27 / 28
Looking Forward
Gap in knowledge as we are working to develop the 21st century skills:
• Technology divides
• The contextual environmental
• Social capital and human networking system