using individual project and program evaluations to improve the part d programs
DESCRIPTION
Using Individual Project and Program Evaluations to Improve the Part D Programs. Dr. Herbert M. Baum. Session Theme # 3. Part D programs have their own priorities for evidence, which project directors need to support. Objectives. By the end of this presentation you will: - PowerPoint PPT PresentationTRANSCRIPT
Using Individual Project and Program Evaluations to Improve the Part D Programs
Using Individual Project and Program Evaluations to Improve the Part D Programs
Dr. Herbert M. Baum
Session Theme # 3Session Theme # 3
Part D programs have their own priorities for evidence, which project directors need to support
ObjectivesObjectives
By the end of this presentation you will:
• Know the difference between program measurement and program management
• Understand how to use logic models for developing program measures
• See examples of how current program information is being used to enhance the Part D Programs.
The trickle up approach (1)The trickle up approach (1)
• Each OSEP/RTP Program has a logic model• Each OSEP/RTP Program has an approved
series of performance measures.• Each OSEP/RTP Program funds projects
The trickle up approach (2)The trickle up approach (2)
Number and Type of Measures by OSEP/RTP Program
Program Annual Long-term Efficiency
Personnel Development
5 2 1
PTIC 3 2 1
TA&D 3 2 1
Tech and Media
3 2 1
The trickle up approach (3)The trickle up approach (3)
• How does each OSEP/RTP Program know that they are meeting their targets?o Your projects need to provide that
• How does each OSEP/RTP Program use the data to manage?o Keeping project officers informed of where the Program
is relative to the target.o Using the measures to direct new grant applications to
address how they will help OSEP/RTP meet their targets.o By meeting targets OSPE/RTP is in a stronger position to
ask for additional funding.
Performance MeasurementPerformance Measurement
Performance ManagementPerformance Management
Evaluation in Performance ManagementEvaluation in Performance Management
Performance Management StepsPerformance Management Steps
1. Assess – review program purpose and design
2. Plan – set strategic performance targets
3. Measure – measure program performance
4. Analyze – evaluate program results
5. Improve – implement program enhancements
6. Sustain – manage program effectiveness
Evaluation in the Context of PARTEvaluation in the Context of PART
• The Program Assessment Rating Tool (PART) was developed to assess the effectiveness of federal programs, and help inform management actions, budget requests, and legislative proposals directed at achieving results.
• The PART Assess if and how program evaluation is used to inform program planning and to corroborate program results.
• 4 sections of a PART review:
o Program purpose and design (20%)
o Strategic planning (10%)
o Program management (20%)
o Program results (50%)
Evaluation and PART (2)Evaluation and PART (2)
• Question 2.6 “…[a]re independent evaluations of sufficient scope and quality conducted on a regular basis, or as needed to support program improvements, and evaluate effectiveness and relevance to the problem, interest, or need.”
• Question 4.5 asks if “independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results.”
What is a logic model?What is a logic model?
The underlying rationale for the evaluand's design, usually an explanation of why the various components of the program (for example) have been created and what-and how-each of them is supposed to contribute towards achieving the desired outcomes. Some logic models include environmental factors, some do not. Note that we are talking about the alleged ‘theory of operation’ and the evaluation may discover considerable discrepancies between this-the view of the designers and possibly also the managers of the program-and the views of the service deliverers who are the hands-on staff engaged in dealing with the recipients of service or product.”
Simplified Logic ModelSimplified Logic Model
Inputs
What the program needs to accomplish it
outcomes.
Activities
What programs do to accomplish their outcomes.
Outputs
What programs produce to accomplish their
outcomes.
Outcomes
What changes the program expects based on their inputs, activities and outputs. [Short-term, Intermediate, and Long-term (impact)]
*Fully Qualified = Highly Qualified for special education teacher; Qualified for paraprofessional/aide; Fully Certified for administrator/coordinator, for related or supportive services in a school setting, or for teacher, related services, or supportive services in early intervention, early childhood.
OSEP OSEP Personnel Development Program - Logic ModelPersonnel Development Program - Logic ModelA blueprint to enhance understanding of the ProgramA blueprint to enhance understanding of the Program
Goal: To improve results for children with disabilities and their familiesGoal: To improve results for children with disabilities and their families
INPUTSINPUTS OUTPUTSOUTPUTS OUTCOMESOUTCOMES
Program Investments
Activities Participation Short Term Long Term
Project Officers
Funding
Evidence-Based &
Best Practices
Research
Program &
Grants Policy
Technology
Process MeasuresProcess Measures Outcome MeasuresOutcome Measures
Intermediate
Increased supply of fully qualified personnel* with awareness and knowledge of EBP & best practices
Increased collaboration - SEAs, IHEs, LEAs & lead agencies
Increased training opportunities
Increased placement of fully qualified*personnel
Improved personnel development infrastructures
Increased retention of fully qualified* personnel in workforce – schools & programs, educational & lead agencies, & IHEs.
Grantees
Faculty
Students in IHEs
SEAs & LEAs
Lead Agencies
Practitioners Administrators
Children
Families
Train personnel
Redesign &build models & networks for collaboration
Develop and disseminate resources
CONTEXT
Federal Law & Regs
Time
Develop priorities &manage competitions
Monitor grants
Build models & networks for collaboration
Measure 1.1 Measure 1.1 (Annual)(Annual)The percentage of Special Education Personnel Preparation projects that incorporate evidence-based practices in the curriculum.
HomeHome
Measure 1.2 Measure 1.2 (Long-Term)(Long-Term)The percentage of scholars completing Special Education Personnel Preparation funded training programs who are knowledgeable and skilled in evidence-based practices for infants, toddlers, children, and youth with disabilities.
Measure 2.1 Measure 2.1 (Annual)(Annual)
The percentage of Special Education Personnel Preparation funded scholars who exit training programs prior to completion due to poor academic performance.
HomeHome
The percentage of low incidence positions that are filled by personnel who are fully qualified under IDEA.
Measure 2.2 Measure 2.2 (Long-Term)(Long-Term)
HomeHome
The percentage of Special Education Personnel Preparation funded degree/certification recipients who are working in the area(s) for which they were trained upon program completion and who are fully qualified under IDEA.
Measure 2.4 Measure 2.4 (Annual)(Annual)
The percentage of degree/certification recipients who maintain employment for 3 or more years in the area(s) for which they were trained and who are fully qualified under IDEA.
Measure 2.5 Measure 2.5 (Long-Term)(Long-Term)
HomeHome
Example from the Personnel Development Program
Example from the Personnel Development Program
Measure 1.1 of 2 (Annual)
The percentage of Special Education Personnel Preparation projects that incorporate evidence-based practices in the curriculum.
Example from the TA& D ProgramExample from the TA& D Program
Measure 1.1 of 1 (Long-term)
The percentage of school districts and service agencies receiving Special Education Technical Assistance and Dissemination services regarding scientifically- or evidence-based practices for infants, toddlers, children and youth with disabilities that implement those practices.
