![Page 1: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/1.jpg)
Evaluation:
A focus on Title I, Part A
Julie E. McLeodSupervisor of Federal Program EvaluationHillsborough County Public Schools
Courtney Zmach, Ph.D.Coordinator, Research and Program EvaluationCollier County Public Schools
![Page 2: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/2.jpg)
EvaluationOverview of Program Evaluation
Evaluating Title I, Part A
Ideas/sample template
![Page 3: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/3.jpg)
“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.”
--Albert Einstein
![Page 4: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/4.jpg)
What is Program Evaluation?Process to assess the implementation
and effects of a program
Involves:Carefully collecting informationAnalyzing dataDrawing conclusions about
implementation and/or effects of program
![Page 5: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/5.jpg)
Evaluation vs. Needs AssessmentNeeds Assessment is “a systematic process for determining goals, identifying discrepancies between desired and actual performance, and establishing priorities for action.” http://www.ag.ndsu.edu/evaluation/needs-assessment-1
The evaluation of the implementation and effectiveness of last year’s program can also function as the upcoming year’s needs assessment.
![Page 6: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/6.jpg)
Guiding Principles
http://evaluationtoolbox.net.au/index.php?option=com_content&id=12
Evaluation is not just about demonstrating success, it is also about learning why things don’t work.
Evaluation is not about finding out about everything, but about finding the things that matter.
Evaluation allows you to continually improve your project, both during its implementation and planning new projects.
![Page 7: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/7.jpg)
Conducting a Successful EvaluationTake time to planSet aside adequate resourcesBegin the evaluation during the initial
stages of program implementationPromote participation of stakeholders
in the evaluation
Ensure confidentiality of responsesConsider cultural issues
![Page 8: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/8.jpg)
Program Evaluation Framework
Source: http://www.cdc.gov/eval/framework/index.htm
Engage Stakeholders
Describe the Program
Focus the Evaluation
DesignGather
Credible Evidence
Justify Conclusions
Ensure Use and Share
Lessons Learned
StandardsUtility
FeasibilityProprietyAccuracy
![Page 9: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/9.jpg)
Define the Program & Develop the Evaluation Plan
What are you trying to evaluate?
What questions do you want answered?
What information/data need to be collected to answer these questions?
When/how will collection of information occur?
![Page 10: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/10.jpg)
![Page 11: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/11.jpg)
Types of EvaluationsImplementation - examine the delivery of the program, the quality of its implementation, the organizational context, procedures, personnel, and so forth Outcome - examine the
effects or outcomes of the areas of interest; summarize data by describing, can include cost-benefit analysis
![Page 12: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/12.jpg)
Types of DataQuantitative
• Numerical (test scores, tallies, etc.)
Qualitative• Text-based (open-ended survey responses,
focus groups, interviews)
![Page 13: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/13.jpg)
Framing Your QuestionsFocus of Evaluatio
nEvaluation Questions
Process • How well was the project designed & implemented?
Outcome• To what extent did the project meet the overall needs?• Was there any significant change and to what extent was it
attributable to the project?
Learnings • What worked and what did not? • What were unintended consequences?
Investment• Was the project cost effective?• Was there another alternative that may have represented a
better investment?
What next• Can the project be scaled up? • Can the project be replicated elsewhere?• Is the change self-sustaining or does it require continued
intervention?
http://www.evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=20&Itemid=159
![Page 14: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/14.jpg)
Why do we do it?
How do we do it?
Sample Template
Recommended Reading: Guide for Evaluating Title I, Part A Programs (FLDOE Bureau of Federal Education Programs)
Evaluating Title I, Part A
![Page 15: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/15.jpg)
“Title I Evaluation” in a Nutshell
Annual review of strategies in Schoolwide (or Targeted Assistance) Plan to determine if the program is directly or indirectly contributing to
the desired outcome (increased student achievement)
![Page 16: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/16.jpg)
NCLB Federal Regulations for SchoolsTitle 34, Part 200, Section 200.26(c) – Core elements of a
schoolwide program:
(c) Evaluation. A school operating a schoolwide program must—
(1) Annually evaluate the implementation of, and results achieved by, the schoolwide program, using data from the State's annual assessments and other indicators of academic achievement;
(2) Determine whether the schoolwide program has been effective in increasing the achievement of students in meeting the State's academic standards, particularly for those students who had been furthest from achieving the standards; and
(3) Revise the plan, as necessary, based on the results of the evaluation, to ensure continuous improvement of students in the schoolwide program.
![Page 17: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/17.jpg)
NCLB Federal Regulations for LEAsTitle 34, Part 200, Section 200.30—
Local Review (a) Each LEA receiving funds under
subpart A of this part must: Use the results of the state assessment system to review
annually the progress of each school service to determine if the school is making AYP
Publicize and disseminate the results of its annual progress review to parents, teachers, principals, schools, and the community; and
Review the effectiveness of actions and activities that schools are carrying out under subpart A of this part with respect to parental involvement, professional development, and other activities assisted under subpart A of this part.
![Page 18: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/18.jpg)
Relevant FLDOE Compliance Items Compliance Item AIA-3: The Local Educational Agency (LEA)
shall ensure that schools implementing schoolwide programs conduct a comprehensive needs assessment (CNA) of the entire school, while taking into account the needs of migratory children, which is based on student achievement data related to the state academic content standards and the state academic achievement standards.
Section 1114(b)(1)(A), P.L.107-110; 34 CFR, Part 200, Section 200.26(a)(1)
Compliance Item HIA-1: The Local Educational Agency (LEA) shall annually evaluate the Title I program and report the results in the following areas: the LEA's progress in achieving the objectives in its approved
application; the effectiveness of the project in meeting the purposes of the
program; and the effect of the project on students being served by the project.
EDGAR 34 CFR Part 75 Section 200.26
![Page 19: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/19.jpg)
Benefits of Title I EvaluationProvide information to administrators, project
staff, school personnel, and parents
Examine the operation of the school (as detailed in the program plans):
Implementation of instructional strategiesDegree of parental involvementOther elements that support increased student
achievement
Assist district and school personnel in making decisions about the program to best meet the needs of students
Determine whether the school/program has met its objectives as stated in the SWP (school level) or Project Application (district level).
![Page 20: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/20.jpg)
Engage the Stakeholders Individuals involved in day-to-day operations
TeachersAdministratorsSupport staff
Individuals and groups served by the programStudentsParentsCommunity members
Individuals in a position to make recommendations and/or decisions regarding the programSACSchool planning teamSchool administratorsDistrict personnel
![Page 21: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/21.jpg)
Evaluating Title I has two parts –1.Implementation 2.Outcomes
![Page 22: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/22.jpg)
Implementation Evaluation Have the activities/strategies been
implemented and to what degree?
SWP/SIP (school)Project Application (LEA)
Were activities/strategies implemented with fidelity? If not, why?
What were the barriers to implementation?
Were staff properly trained?
![Page 23: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/23.jpg)
Outcome EvaluationDid the school (or LEA) meet the
goals/objectives of their SWP/SIP (Project Application)?
Did the students meet the state academic achievement standards?
Did achievement increase (especially for those who were furthest from meeting standards)?
For those students who are not proficient, what specific content areas are the most troublesome?
![Page 24: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/24.jpg)
Potential Data SourcesProgress monitoring dataState CRT (FCAT 2.0/FSA)End-of-Course (EOC) ExamsState reading assessment
(FAIR)Kindergarten ReadinessClassroom walk-through
data
Graduation rate
Attendance
Promotion/retention rates
Demographic data
Discipline data
Professional Development participation
Evaluations/Feedback from workshops and meetings
Surveys/Focus Groups
![Page 25: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/25.jpg)
Where to find the data: Florida School Accountability Reports : http://schoolgrades.fldoe.org/ FCAT Demographic Reports: https://app1.fldoe.org/FCATDemographics/ PK-12 Public School Data Publications and Reports: http://
www.fldoe.org/accountability/data-sys/edu-info-accountability-services/pk-12-public-school-data-pubs-reports/index.stml
ACT-SAT-AP Reports: http://www.fldoe.org/evaluation/act-sat-ap.asp Education, Information, and Accountability Services Publications and
Reports: http://www.fldoe.org/eias/eiaspubs/default.asp Performance Profiles:
http://fldoehub.org/PerformanceProfile/Pages/default.aspx School Environmental Safety Incident Reporting (SESIR) System
Reports: http://www.fldoe.org/safeschools/sesir.asp School Public Accountability Reports (SPAR):
http://doeweb-prd.doe.state.fl.us/eds/nclbspar/index.cfm
It is important to keep in mind that some of these websites are updated sporadically. Be sure to check the dates for all of the data you are pulling from these sites.
![Page 26: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/26.jpg)
General PointersWeave federal and state monitoring
requirements with what is wanted/needed at the local level
Determine what you want to learn about your program or what would help most
Incorporate the project application goals, objectives, and strategies
Include tables and graphs to illustrate data
Craft appropriate recommendations for program improvement
![Page 27: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/27.jpg)
Ideas and ExamplesThe ECTAC Evaluation Team has created
“templates” to provide a structure for evaluating the implementation of the program
Need 3 Sample
www.ectacfl.net/Evaluators.html
![Page 29: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/29.jpg)
Evaluation Matrices
![Page 30: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/30.jpg)
Evaluation Template
![Page 31: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/31.jpg)
Sample
![Page 32: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/32.jpg)
Your Turn…
![Page 33: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/33.jpg)
Questions??????
![Page 34: A focus on Title I, Part A Julie E. McLeod Supervisor of Federal Program Evaluation Hillsborough County Public Schools Courtney Zmach, Ph.D. Coordinator,](https://reader036.vdocuments.us/reader036/viewer/2022062321/56649e115503460f94afc59c/html5/thumbnails/34.jpg)
Thank you!Julie McLeod813-272-4445
Dr. Rich Janiak941-255-7535 x258
richard.janiak@ yourcharlotteschools.net
Dr. Courtney Zmach239-377-3581
Jan Mahowski (727) 893-2988 [email protected]