© ccsr 5essentials: background and methodology rachel levenstein senior manager for survey research...
TRANSCRIPT
![Page 1: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/1.jpg)
© C
CS
R
5Essentials: Background and methodology
Rachel LevensteinSenior Manager for Survey [email protected]
![Page 2: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/2.jpg)
© C
CS
R
20 years of research in Chicago Public Schools (CPS)- Conversations with practitioners- Wide scan of the literature- Surveys
Organizing Schools for Improvement
Origin of the 5Essentials
![Page 3: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/3.jpg)
© C
CS
R
Surveys of CPS
1999
2001
2003
2005
2007
2009
2011
2012
2013
2014
2015
0
0.2
0.4
0.6
0.8
1
Student and teacher response rates
Students Teachers
![Page 4: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/4.jpg)
© C
CS
R
Initial measure development process conducted in CPS
• Prior research findings indicate a link between construct and student outcomes1. Search existing literature for survey questions
2. Focus groups Draft questions
3. Cognitive interviews Revise questions
4. Pilot survey questionsCheck reliability, revise questions
• Validation study
![Page 5: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/5.jpg)
© C
CS
R
Development of survey questions: Focus groups
• Gather 5-10 practitioners or students together to talk about the concept.
• Moderated by a researcher, using a protocol.
• Identify themes and common nomenclature.
Get ideas for how to write survey questions
![Page 6: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/6.jpg)
© C
CS
R
Development of survey questions: Cognitive Interviews• One-on-one interviews with practitioners,
students• “Tell me how you came to your answer”• “What kinds of programs were you thinking
about?”
Identify problems understanding or interpreting the questions.
![Page 7: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/7.jpg)
© C
CS
R
Development of survey questions: Pilot testing
• Small- or large-scale pilot• Same conditions as the “regular” survey
– May be conducted as part of the “regular” survey
• Analyze responses and calculate reliability.
Identify if the measure holds together (or not).
![Page 8: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/8.jpg)
© C
CS
R
Development of survey questions:Validation study
Program Coherence example: Newman et al., 2001
– Qualitative data collection: (School improvement plans, classroom observations, staff meeting observations, assignment samples)
• Reports highly correlated with survey measure
– Schools that improved on Program Coherence between 1994 and 1997 showed more improvement on ITBS than schools that did not improve—by 1/5 of a school year!
![Page 9: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/9.jpg)
© C
CS
R
Practices change Understanding of constructs change General revisions for clarity Clients outside of Chicago inform some changes
May add, delete, or modify questions every few years.
Changes to content
![Page 10: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/10.jpg)
© C
CS
R
To what extent do you agree or disagree with each of the following statements?
- You can see real continuity from one program to another in this school.
- Many special programs come and go in this school.- Once we start a new program, we follow up to
make sure that it’s working.- We have so many different programs in this school
that I can’t keep track of them all.- (Strongly disagree, disagree, agree, strongly agree)
Changes to content: Program Coherence
![Page 11: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/11.jpg)
© C
CS
R
To what extent do you agree or disagree with each of the following statements?
- You can see real continuity from one program to another in this school.
- Once we start a new program, we follow up to make sure that it’s working.
- Many special programs come and go in this school.- Once we start a new program, we follow up to make sure
that it’s working.- We have so many different programs in this school that I
can’t keep track of them all.- (Strongly disagree, disagree, agree, strongly agree)
Changes to content: Program Coherence
![Page 12: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/12.jpg)
© C
CS
R
How much influence do teachers have over school policy in each of the areas below?- Hiring new professional personnel.- Planning how discretionary school funds should be used.- Determining books and other instructional materials used
in classrooms.- Establishing the curriculum and instructional programs.- Setting standards for student behavior.(Not at all, A little, Some, To a great extent)
Changes to content: Teacher Influence
![Page 13: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/13.jpg)
© C
CS
R
How much influence do teachers have over school policy in each of the areas below?- Hiring new professional personnel.- Planning how discretionary school funds should be used.- Determining books and other instructional materials used in
classrooms.- Establishing the curriculum and instructional programs.- Setting standards for student behavior.
(Not at all, A little, Some, To a great extent)
(No influence, A little influence, Some influence, A great deal of influence)
Changes to content: Teacher Influence
![Page 14: © CCSR 5Essentials: Background and methodology Rachel Levenstein Senior Manager for Survey Research rmlev@uchicago.edu](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649ce65503460f949b363a/html5/thumbnails/14.jpg)
© C
CS
R
Putting a measure into the 5Essentials
• Two years of data before it becomes an Essential
• Fit within 5Essentials framework– Factor analysis