Download - Statistical Modeling for Education Planning [email protected] URBPL 5/6020 / April 19, 2007
Statistical Modelingfor Education Planning
[email protected]/financeURBPL 5/6020 / April 19, 2007
Who We Are
Utah State Office of Education– Staff to the State Board of Education
Financial and Business Services Division Finance and Statistics Section
What We Deal With
Populations– Students– Staff– Schools
Finance– Minimum School Program (MSP) Budget & NCLB Allocations– Financial Reporting & Auditing– Property Tax
Operations– School Facilities– Student Transportation– Safety
How We Do It
Acquire– Data
Allocate– Money to local education agencies according to
data
Audit– For accuracy of data and appropriateness of
expenditures
Analytics Cycle
Population (Enrollment) Projections– How many people do we need to serve?
Fiscal Impact Analysis– How much will service options cost?
Formula Allocation– How do we get the right amount to the right place?
Compliance Audit– How well did service providers follow the rules?
Program Evaluation– How well did we serve the population?
Enrollment Projections:Institutional Context
Common Data Committee– Legislative Fiscal Analyst– Governor’s Office of Planning and Budget– Utah State Office of Education
Current Work– By county (then allocate to districts and adjust for charter schools)– In October– Single year (to next October)– Agreed upon figures for legislative session
Future Plans– Multiyear project with GOPB using REMI for Baseline 2008
Enrollment Projections:Model
Cohort ProgressionParticipation Ratio
– Kindergarten subset
Enrollment Projections:Data
Historical Variables– Total Enrollment (Current and Prior Years)– Grade 12 Enrollment (Current and Prior Years)– Kindergarten Enrollment (Current Year)– Births (by Month, 4- and 5-Years Prior)
Intermediate Variable– Projected Kindergarten Enrollment
Enrollment Projections:Formula
Formula Element
EY+1
= EY Base Population
+ (BY-4 * (KY / BY-5 )) - GY) Cohort Progress.
+ (EY - EY-1) - (KY - GY-1) Implied Migration
Fiscal Impact Analysis:Example
HB 222 (2002)– “Make recommendation on … the ideal size of
schools districts in this state …”
Optimization Problem Cost Function
– Relates expenditures per student to enrollment (main cost driver) controlling for academic achievement (output measured in quality)
Fiscal Impact Analysis:Design
Sample: Cross section of 40 Utah school districts
Data: Superintendent’s Annual Report, 2000-01
Model: Y = m + (b1X + b2X2) + b3Z + e
Procedure: OLS regression
Fit (adj R2): .29
Predictors: Enroll Enroll2 Lexile
Coeff (b): -.138 .0000016 -665
Sig (p): .01 .05 .03
Fiscal Impact Analysis:Results
Empirical Cost Function– exp = .0000016enr2 - .138enr – 665lex + 6,468
Differentiated and Set Equal to Zero– 0 = .0000032enr - .138
Solution is Optimal Size– enr = .138/.0000032 = 43,125 students
Fiscal Impact Analysis:Politics (TTC, SL Tribune 2/23/04)
Columbia University professor’s critique:– “I’d be happy to go with the [USOE] analysis rather than the
fiscal analyst’s, which is opaque to the point of incomprehensibility”
Fiscal analyst’s defense:– “Anybody’s guess is as good as the next person’s”
Opponents’ critique:– “Foes have long accused the fiscal analyst’s office of
working the numbers to achieve a favorable outcome” Fiscal analyst’s concession:
– “At the outset, the intention is to have it come out in a positive way so there’s not a cost”
Fiscal Impact Analysis:Ethics
Substantive claims must be warranted by evidence
Production of evidence must be based on transparent procedures
Allocation Formulas:Minimum School Program (1)
“The purpose of this chapter [Utah Code 53A-17a] is to provide a minimum school program (MSP) for the state in accordance with the constitutional mandate.”
“It recognizes that all children of the state are entitled to reasonably equal educational opportunities regardless of their place of residence in the state and of the economic situation of their respective school districts or other agencies.”
NOTE: Overriding concern with equity; adequacy is another issue of growing legal importance, but its operationalization is very unclear.
Allocation Formulas:Minimum School Program (2)
“It further recognizes that although the establishment of an educational system is primarily a state function, school districts should be required to participate on a partnership basis in the payment of a reasonable portion of the cost of a minimum program.”
NOTE: Utah sources of revenue (FY 2006):– State 55% from income tax– Local 36% from property tax– Federal 9% from who knows where
Allocation Formulas:Minimum School Program (3)
“Each locality should be empowered to provide educational facilities and opportunities beyond the minimum program and accordingly provide a method whereby that latitude of action is permitted and encouraged.”
NOTE: Local school boards can impose several additional property taxes for specified educational purposes.
Allocation Formulas:Budgeting for Basic Program (1)
Majority of funding is based on “Prior Year + Growth” formula
Prior year is Average Daily Membership (ADM)
Growth is percentage difference between projected Fall Enrollment and current year Fall Enrollment
Hold harmless in case of negative growth
Allocation Formulas:Budgeting for Basic Program (2)
Result is number of Weighted Pupil Units (WPUs), a quantification of the basic service which each local education agency is obligated to provide
Legislature sets monetary value of WPU every year
Total WPUs times WPU $ value determines basic appropriation
Allocation Formulas:Budgeting for Basic Program (3)
If LEA property tax revenue cover its obligation, then buck stops there; otherwise, state pays balance from income tax revenue
In practice, all LEAs need some state assistance and appropriations often fall short, so funds are prorated according to WPUs
Since FY 2001, K-12 funding has approximately kept pace with inflation
Allocation Formulas:Categorical Programs
In addition to the basic program, the Legislature has established dozens of categorical programs to address particular concerns
Cost drivers of categorical programs can be quite complex — “Special Education Add On” is an especially striking example of what can happen when trying to reconcile competing interests through a funding program
Allocation Formulas:Categorical Program Example (1)
Per WPU, which is the greater of the average of Special Education (Self Contained and Resource) ADM over the previous 5 years (which establishes the foundation [hold harmless] below which the current year WPU can never fall) or prior year Special Education ADM plus weighted growth in Special Education ADM.
Allocation Formulas:Categorical Program Example (2)
Weighted growth is determined by multiplying Special Education ADM from two years prior by the percentage difference between Special Education ADM two years prior and Special Education ADM for the year prior to that, subject to two constraints:
Allocation Formulas:Categorical Program Example (3)
Special Education ADM values used in calculating the difference cannot exceed the prevalence limit of 12.18% of total district ADM for their respective years.
If this measure of growth in Special Education exceeds current year growth in Fall Enrollment, growth in Special Education is set equal to growth in Fall Enrollment (incidence limit).
Allocation Formulas:Categorical Program Example (4)
Finally, growth is multiplied by a factor of 1.53.
This weight is intended to account for the additional cost of educating a special education student.
However, the weight is not based on an empirical analysis of the cost of special education relative to "regular" education.
An Australian Approach:Victoria’s Principles (1)
Preeminence of Educational Considerations– Elimination of disparities reflecting historical and
political decisions for which there is no current or future educational rationale
Cost Effectiveness– Relativities among allocations should reflect
knowledge of efficient ways of achieving school and classroom effectiveness
An Australian Approach:Victoria’s Principles (2)
Fairness– Schools with the same mix of learning needs
should receive the same total resources; this requires accurate and comprehensive information on those student characteristics which best predict academic achievement
Transparency– Basis for allocations should be made public and
readily understandable by all with an interest
An Australian Approach:Victoria’s Principles (3)
Subsidiarity– Decisions on resource allocation should be made
centrally only if they cannot be made locally Accountability
– A school that has authority to make decisions on how resources will be allocated should be accountable for the use of the resources, including educational outcomes in relation to learning needs
An Australian Approach: Simple Budget Structure (1)
Core Funding– Grade Level– School Size
Student Disadvantage– Disabilities– Special Learning Needs– English as Second Language– Rurality and Isolation
An Australian Approach: Simple Budget Structure (2)
Facilities (operation & maintenance) Administration Costs outside of control of schools
– e.g., Transportation to and from school
“Priority” Programs– Money for politicians of the day to play with
An Australian Approach:Special Learning Needs
Sample (83 schools; 7,233 students) Hierarchical linear & Structural equation modeling Demographic index to predict achievement:
– Poverty (qualified for education welfare payment)– Parental occupation (skill level)– Language spoken at home (other than English)– Family composition (two parent, one parent, none)– Aboriginality (= Alaska Native or American Indian)– Transience (recently changed schools, = Mobility)
An Australian Approach:Reference
Hill, Peter W. (1996). Building equity and effectiveness into school based funding models: An Australian case study. 18p.
http://nces.ed.gov/pubs97/97535i.pdf
Compliance Audit:Purpose
Provide reasonable assurance that local education agencies are correctly applying State Board of Education rules in accounting for their students
Statistical summaries from individual data files serve as written management assertions
Auditors follow agreed upon procedures
Compliance Audit:Sampling
Efficient auditing depends on selection of sample appropriate to purpose
For example, if you want to adjust statistics based on audit, you need a probability sample
The right sample size then depends on:– Variation in the population– Risk you are willing to take of being wrong
Sample Size: The Price of Precisionpop = 80,000; mean = 154; sd = 25
90% 95% 99%
1% 703 1001 1718
5% 29 41 71
10% 8 11 18
Program Evaluation:Regression with Treatment as Dummy
“In the actual practice of applied social science, the most common mode of causal inference, the most common quasi-experimental design …” (Cook & Campbell)
Crucial to valid interpretation:– Specification of correct theory (of nonrandom
selection process) as represented by equation– (Near) perfect measurement of variables
Program Evaluation:Recommendation
Consider path analysis as extension of regression:– Explicit theory of how program works as a causal
mechanism in form of path diagram Consider multiple indicators of each variable:
– Use factor analysis to obtain composite measure representing only common variance
In short, poor man’s structural equation modeling
Program Evaluation:Path Diagram Example
Program Evaluation:Bibliography
Fitzpatrick, Sanders & Worthen (2004) Program Evaluation: Alternative Approaches and Practical Guidelines
– LB2822.75 .W67 Patton (1997) Utilization Focused Evaluation
– H62.5.U5 P37 Mohr (1995) Impact Analysis for Program Evaluation
– H97 .M64 Cook & Campbell (1979) Quasi Experimentation: Design and
Analysis for Field Settings– H62 .C5857
Scriven (1991) Evaluation Thesaurus– AZ191 .S37
Some Education Data Issues
Is a Navajo living in a hogan homeless? Kanab is on the urban fringe of which city? Who decides the racial identity of a student? When is a person who leaves school without
graduating not a dropout? Is being in a single parent family a reliable
indicator of being at risk of low academic performance?
Highly Impacted Schools Criteria:Factor Analysis
% of Enrollment Median Loading Median Loading
Ethnic Minority .93 .95
Limited English .91 .93
Free Lunch .80 .86
Single Parent -.49 -
Mobile -.60 -
R2 55% 82%
Data Sources
Digest of Education Statisticshttp://nces.ed.gov/programs/digest
NCES Tables & Figureshttp://nces.ed.gov/quicktables
USOE Assessement, Accountability & Divisionhttp://www.schools.utah.gov/eval
Utah State Superintendent’s Reporthttp://www.schools.utah.gov/finance/other/AnnualReport/ar.htm