how to use ed360 and other data tools for year end evaluation · •decide what you want to know....
TRANSCRIPT
How to Use Ed360 and Other Data Tools for Year End Evaluation
Presented by: Gina FultonModule #11
May 27, 2020
Learning Objectives Module #11
2
To review and understand evaluation models and frameworks
To conceptualize and gather the data needed for the evaluation
To discuss theory and questions related to program evaluations
To refresh everyone’s memory about the data available in Ed360 for supporting program evaluation
Program Evaluation Definition
“A common definition used to separate evaluation from research is that program evaluation is conducted for decision-making purposes, whereas research is intended to build our general understanding and knowledge of a particular topic and to inform practice.”
-Spaulding, Dean T.
https://www2.ed.gov/about/offices/list/oese/sst/evaluationmatters.pdf
https://www2.ed.gov/about/offices/list/oese/sst/evaluationmatters.pdf
Why Evaluate Your Program?
Let’s use the Title I Program for an example as you’re required to evaluate Title I annually.
Brainstorm and list reasons why you would evaluate your Title I program.
Besides it being the law…
Brainstorm Ideas--Responses
To research if Title I program is effective To determine if we’re meeting the needs of all
students To determine if our resources align to our needs To decide whether there are areas we can improve
upon To determine if some areas of the program are not
working
What other questions might you want to include in your program evaluation? Why did you choose these?
What questions would you ask?
GENERAL IDEAS FOR PROGRAM EVALUATION QUESTIONS:(1) Has the Title I program been effective?(2) What has worked well in the Title I program? (3) What has not worked well in the Title I program(4) How should the Title I program be refined?
The data are analyzed, and the results of these analyses are used as the source of evidence to determine the answers to the four questions.
What are the steps in an evaluation?
• Decide what you want to know. (PURPOSE) • Form your evaluation questions. (What do you want to
know?)• Use a good model, design or evaluation matrix. (Logic
model and program overview slides)• Determine what data you want collected. (Types and
methods)• Collect data throughout the year. (Action plan)• Analyze data. (frequency, % change, aligned to ?)
What are the steps in an evaluation?
• Make connections to what the data is saying through triangulation. (3 or more valid data points/sets)
• Write a summary of program effectiveness. (Narrative style, using a standard template—see Rockwood 5* example.)
• Share findings and recommendations.• Ask for feedback on evaluation. What can others
do to help support the necessary changes?
Sample Evaluation Matrix
Evaluation Objective
Stakeholder Group
Tools Used to Collect Data
When Purpose
To document depth and breadth of activities that support theTitle I program during the 2019-2020 school year.
Certified and non-certified staff members, administrators, any providing services to Title I students.
InterviewSurvey
October during Parent/Teacher Conference weekOr throughoutthe year at different times.
Summative
Questions?15
Pause & Ponder16
Please write down your reflections.What ideas stuck with you?What aspects do you disagree with?What aspects do you want to ponder further?What resonates?How does this perspective differ from previous knowledge?
17
Demonstration of Ed360-Dashboard
18
Academic Overview, Grade 3, Low Income with All Races
19
Academic Overview, Grade 3, Students with Disabilities and All Races
20
Academic Overview, Grade 3, English Learners, All Races
21
KIDS Survey
22
KIDS Results by Readiness Developmental Areas Fall 2018 - 2019 with 2019-2020 School Overview
23
KIDS Readiness Measures by School
24
KIDS, Percentage of Students Demonstrating K Readiness-Male
25
KIDS, Percentage of Students Demonstrating K Readiness-Female
26
KIDS, Percentage of Students Demonstrating K Readiness, All Students, Hispanic or Latino
What data could your collect? Why?
Brainstorm activity: Share and discuss the types of data and the importance of the timeline in data collection.What data have you collected in your district for program measures?Once you collected data, what occurred? Why are the types important? Why is the timeline important?
Types of Data
Quantitative Qualitative Mixed Measures (both) Summative—collected at end of project to measure
outcomes and how those outcomes relate to the overall judgment of program (both quant. & qual.)
Formative—collected during project to measure outcomes and report back to stakeholders to be able to make timely adjustments to program
Tools for Collecting Data
Surveys Interviews Questionnaires Observations Document review Achievement data Student demographic data Student behavioral data
NAEP DATA
https://www.nationsreportcard.gov/profiles/stateprofile?chort=2&sub=MAT&sj=AL&sfj=NP&st=MN&year=2019R3
Data Analysis
Table 1-A Illinois Snapshot for NAEP Mathematics Grade 4 for 2000-2019
https://www.isbe.net/Documents/naep-2019-math-state-4-8.pdf
Triangulate Data
Be sure to include various data sets or points of collection
Check for outliers Make sure there isn’t bias or other factors that
skew your data Make sure you use evidence, grounded in facts—
not person feelings.
Quick reminders from the start of the DT Roadmap with online sessions
District Data Team• Meets monthly• Identifies District:
– Goals– Student Outcome Indicators – District Targets – Adult Action Indicators
• Develops District Strategic Improvement Plan
• Evaluates Programs
District Level Data
Data Analysis Protocol
School Level Data
Data Analysis Protocol
Classroom/Student Level Data
Data Analysis Protocol
Remember, it’s all about practicing the 5-Step process that works for your DT
Reeves’ 5 Steps: Collect & chart data Analyze strengths and
obstacles Establish goals: set,
review, revise Select instructional
strategies Determine results
indicators
Discuss the Process in DT: Consider a problem you face
as a leaders Record data you can
reconstruct from memory or have with you
Set a SMART goal Brainstorm strategies for
moving toward goal List indicators of fidelity and
success
Just remember…
If it is not written down, it doesn’t exist!
And
Document your process, analysis and results to demonstrate program effectiveness and areas of needed improvement annually.
Questions?44
Pause & Ponder45
Please write down your reflections.What ideas stuck with you?What aspects do you disagree with?What aspects do you want to ponder further?What resonates?How does this perspective differ from previous knowledge?
Wrap Up46
All Modules for Building Your Data Team (DT) Roadmap Training Modules can be found on Ed360’s Resource page https://www.isbe.net/ed360
Thanks for your participation.
IMPORTANT READING SUGGESTIONS
https://www2.ed.gov/about/offices/list/oese/sst/evaluationmatters.pdf
Content questions about this presentationGina Fulton: [email protected]
47
Ed360 Outreach [email protected]
Ed360 webpageisbe.net/ed360
Ed360 Logined360.isbe.net
CONTACT INFORMATION