a focus on title i, part a julie e. mcleod supervisor of federal program evaluation hillsborough...
TRANSCRIPT
Evaluation:
A focus on Title I, Part A
Julie E. McLeodSupervisor of Federal Program EvaluationHillsborough County Public Schools
Courtney Zmach, Ph.D.Coordinator, Research and Program EvaluationCollier County Public Schools
EvaluationOverview of Program Evaluation
Evaluating Title I, Part A
Ideas/sample template
“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.”
--Albert Einstein
What is Program Evaluation?Process to assess the implementation
and effects of a program
Involves:Carefully collecting informationAnalyzing dataDrawing conclusions about
implementation and/or effects of program
Evaluation vs. Needs AssessmentNeeds Assessment is “a systematic process for determining goals, identifying discrepancies between desired and actual performance, and establishing priorities for action.” http://www.ag.ndsu.edu/evaluation/needs-assessment-1
The evaluation of the implementation and effectiveness of last year’s program can also function as the upcoming year’s needs assessment.
Guiding Principles
http://evaluationtoolbox.net.au/index.php?option=com_content&id=12
Evaluation is not just about demonstrating success, it is also about learning why things don’t work.
Evaluation is not about finding out about everything, but about finding the things that matter.
Evaluation allows you to continually improve your project, both during its implementation and planning new projects.
Conducting a Successful EvaluationTake time to planSet aside adequate resourcesBegin the evaluation during the initial
stages of program implementationPromote participation of stakeholders
in the evaluation
Ensure confidentiality of responsesConsider cultural issues
Program Evaluation Framework
Source: http://www.cdc.gov/eval/framework/index.htm
Engage Stakeholders
Describe the Program
Focus the Evaluation
DesignGather
Credible Evidence
Justify Conclusions
Ensure Use and Share
Lessons Learned
StandardsUtility
FeasibilityProprietyAccuracy
Define the Program & Develop the Evaluation Plan
What are you trying to evaluate?
What questions do you want answered?
What information/data need to be collected to answer these questions?
When/how will collection of information occur?
Types of EvaluationsImplementation - examine the delivery of the program, the quality of its implementation, the organizational context, procedures, personnel, and so forth Outcome - examine the
effects or outcomes of the areas of interest; summarize data by describing, can include cost-benefit analysis
Types of DataQuantitative
• Numerical (test scores, tallies, etc.)
Qualitative• Text-based (open-ended survey responses,
focus groups, interviews)
Framing Your QuestionsFocus of Evaluatio
nEvaluation Questions
Process • How well was the project designed & implemented?
Outcome• To what extent did the project meet the overall needs?• Was there any significant change and to what extent was it
attributable to the project?
Learnings • What worked and what did not? • What were unintended consequences?
Investment• Was the project cost effective?• Was there another alternative that may have represented a
better investment?
What next• Can the project be scaled up? • Can the project be replicated elsewhere?• Is the change self-sustaining or does it require continued
intervention?
http://www.evaluationtoolbox.net.au/index.php?option=com_content&view=article&id=20&Itemid=159
Why do we do it?
How do we do it?
Sample Template
Recommended Reading: Guide for Evaluating Title I, Part A Programs (FLDOE Bureau of Federal Education Programs)
Evaluating Title I, Part A
“Title I Evaluation” in a Nutshell
Annual review of strategies in Schoolwide (or Targeted Assistance) Plan to determine if the program is directly or indirectly contributing to
the desired outcome (increased student achievement)
NCLB Federal Regulations for SchoolsTitle 34, Part 200, Section 200.26(c) – Core elements of a
schoolwide program:
(c) Evaluation. A school operating a schoolwide program must—
(1) Annually evaluate the implementation of, and results achieved by, the schoolwide program, using data from the State's annual assessments and other indicators of academic achievement;
(2) Determine whether the schoolwide program has been effective in increasing the achievement of students in meeting the State's academic standards, particularly for those students who had been furthest from achieving the standards; and
(3) Revise the plan, as necessary, based on the results of the evaluation, to ensure continuous improvement of students in the schoolwide program.
NCLB Federal Regulations for LEAsTitle 34, Part 200, Section 200.30—
Local Review (a) Each LEA receiving funds under
subpart A of this part must: Use the results of the state assessment system to review
annually the progress of each school service to determine if the school is making AYP
Publicize and disseminate the results of its annual progress review to parents, teachers, principals, schools, and the community; and
Review the effectiveness of actions and activities that schools are carrying out under subpart A of this part with respect to parental involvement, professional development, and other activities assisted under subpart A of this part.
Relevant FLDOE Compliance Items Compliance Item AIA-3: The Local Educational Agency (LEA)
shall ensure that schools implementing schoolwide programs conduct a comprehensive needs assessment (CNA) of the entire school, while taking into account the needs of migratory children, which is based on student achievement data related to the state academic content standards and the state academic achievement standards.
Section 1114(b)(1)(A), P.L.107-110; 34 CFR, Part 200, Section 200.26(a)(1)
Compliance Item HIA-1: The Local Educational Agency (LEA) shall annually evaluate the Title I program and report the results in the following areas: the LEA's progress in achieving the objectives in its approved
application; the effectiveness of the project in meeting the purposes of the
program; and the effect of the project on students being served by the project.
EDGAR 34 CFR Part 75 Section 200.26
Benefits of Title I EvaluationProvide information to administrators, project
staff, school personnel, and parents
Examine the operation of the school (as detailed in the program plans):
Implementation of instructional strategiesDegree of parental involvementOther elements that support increased student
achievement
Assist district and school personnel in making decisions about the program to best meet the needs of students
Determine whether the school/program has met its objectives as stated in the SWP (school level) or Project Application (district level).
Engage the Stakeholders Individuals involved in day-to-day operations
TeachersAdministratorsSupport staff
Individuals and groups served by the programStudentsParentsCommunity members
Individuals in a position to make recommendations and/or decisions regarding the programSACSchool planning teamSchool administratorsDistrict personnel
Evaluating Title I has two parts –1.Implementation 2.Outcomes
Implementation Evaluation Have the activities/strategies been
implemented and to what degree?
SWP/SIP (school)Project Application (LEA)
Were activities/strategies implemented with fidelity? If not, why?
What were the barriers to implementation?
Were staff properly trained?
Outcome EvaluationDid the school (or LEA) meet the
goals/objectives of their SWP/SIP (Project Application)?
Did the students meet the state academic achievement standards?
Did achievement increase (especially for those who were furthest from meeting standards)?
For those students who are not proficient, what specific content areas are the most troublesome?
Potential Data SourcesProgress monitoring dataState CRT (FCAT 2.0/FSA)End-of-Course (EOC) ExamsState reading assessment
(FAIR)Kindergarten ReadinessClassroom walk-through
data
Graduation rate
Attendance
Promotion/retention rates
Demographic data
Discipline data
Professional Development participation
Evaluations/Feedback from workshops and meetings
Surveys/Focus Groups
Where to find the data: Florida School Accountability Reports : http://schoolgrades.fldoe.org/ FCAT Demographic Reports: https://app1.fldoe.org/FCATDemographics/ PK-12 Public School Data Publications and Reports: http://
www.fldoe.org/accountability/data-sys/edu-info-accountability-services/pk-12-public-school-data-pubs-reports/index.stml
ACT-SAT-AP Reports: http://www.fldoe.org/evaluation/act-sat-ap.asp Education, Information, and Accountability Services Publications and
Reports: http://www.fldoe.org/eias/eiaspubs/default.asp Performance Profiles:
http://fldoehub.org/PerformanceProfile/Pages/default.aspx School Environmental Safety Incident Reporting (SESIR) System
Reports: http://www.fldoe.org/safeschools/sesir.asp School Public Accountability Reports (SPAR):
http://doeweb-prd.doe.state.fl.us/eds/nclbspar/index.cfm
It is important to keep in mind that some of these websites are updated sporadically. Be sure to check the dates for all of the data you are pulling from these sites.
General PointersWeave federal and state monitoring
requirements with what is wanted/needed at the local level
Determine what you want to learn about your program or what would help most
Incorporate the project application goals, objectives, and strategies
Include tables and graphs to illustrate data
Craft appropriate recommendations for program improvement
Ideas and ExamplesThe ECTAC Evaluation Team has created
“templates” to provide a structure for evaluating the implementation of the program
Need 3 Sample
www.ectacfl.net/Evaluators.html
Evaluation Matrices
Evaluation Template
Sample
Your Turn…
Questions??????
Thank you!Julie McLeod813-272-4445
Dr. Rich Janiak941-255-7535 x258
richard.janiak@ yourcharlotteschools.net
Dr. Courtney Zmach239-377-3581
Jan Mahowski (727) 893-2988 [email protected]