analysis and communication of us news rankings using monte carlo simulations: a comparison to...
Post on 19-Dec-2015
213 views
TRANSCRIPT
Analysis and Communication of US News Rankings using Monte Carlo Simulations: A Comparison to Regression Modeling
Presented by Chris Maxwell
Purdue University
AIR 2010
Introduction
What changes in submitted data most influence our US News rankings?
• Identify key data elements• Provide realistic expectations of future rank
This presentation will focus on the US News graduate program in education rankings
Results will also be presented for graduate business and national universities rankings
Import into Excel and use ordinary linear regression (OLS) to model the US News score:
Initial Analysis
Started with US News data from website:
OLS Problems
Variable rejections
Multicollinearity
Model variability - which model is “right” ?
Counterintuitive results
OLS Problems (continued)
Models can be extremely accurate, but communication of results becomes very problematic
Is there another way to model the score using the same data?
US News Methodology
US news scores are z-score based:• (observation - mean)/standard deviation
In general, each institution’s z-scores are:• multiplied by the US News weight• totaled• the highest total is scaled to 100
Not all calculation details are known and some data are missing
Monte Carlo Simulation
Can a US News-type equation be simulated that calculates the US News scores?
•18 unknowns, but 50 observations…
The equation framework is input into an iterative Excel VBA program
Reasonable ranges are defined for the 18 unknown standard deviations and “means”
Monte Carlo Simulation (continued)
For each iteration (~40,000) in a run:•Randomly chose all unknowns•Compute score for each institution•Rescale so top score is100•Compute sum of squared errors
The best-fit equation is saved, algebraically rearranged, and compared to regression
Refine the model and repeat the process
Model Comparisons
Graduate Education 2009: Regression Monte Carlo
Intercept -34.0 -35.8Peer Survey 8.45 8.80
Superintendent Survey 8.70 9.48GRE Verbal 0.053 0.033
GRE Quantitative 0.000 0.013Acceptance Rate -11.50 -9.54
Student/Faculty Ratio 0.002 -0.182Doctorates Produced/Faculty -2.73 -2.25
Research Funds (millions) 0.500 0.496Funds/Faculty (thousands) 0.026 0.027
R Squared 97.6% 97.4%
Model Comparisons (continued)
Graduate Business 2009: Regression Monte Carlo
Intercept -145 -145Peer Survey 8.91 8.36
Recruiter Survey 6.04 5.78Undergraduate GPA 17.5 17.4
GMAT score 0.071 0.074Acceptance Rate -2.11 -2.04
Starting Salary 1.7E-04 1.9E-04Employed at Graduation 10.8 10.2
Employed at 3 months 27.5 27.6
R Squared 99.96% 99.95%
Model Comparisons (continued)
Graduate Education 2010: Regression Monte Carlo
Intercept -38.1 -36.6Peer Survey 11.2 10.7
Superintendent Survey 7.96 8.52GRE Verbal 0.024 0.021
GRE Quantitative 0.016 0.016Acceptance Rate -7.40 -8.97
Student/Faculty Ratio -0.420 -0.343Doctorates Produced/Faculty -1.79 -2.13
Research Funds (millions) 0.417 0.420Funds/Faculty (thousands) 0.028 0.027
R Squared 99.2% 99.1%
National Universities 2009: Regression Monte Carlo
Intercept -96.8 -86.1Peer Survey 11.3 11.1
Graduation Performance 10.4 10.3Classes under 20 13.5 12.6
Classes 50 and over -6.50 -5.06Student/Faculty ratio 0.065 -0.054
Full-time Faculty 0.05 1.41SAT 25th 0.014 0.006SAT 75th -0.001 0.006
High School Rank 10.1 11.7Acceptance Rate -2.74 -2.84
Faculty Terminal Degrees 8.80 8.13Graduation Rate 19.0 16.3
Retention Rate 5.82 7.04Alumni Giving 19.1 16.2
IPEDS Finances ratio (log) 11.15 8.76Faculty Salaries 4.2E-05 7.5E-05
R Squared 99.6% 99.5%
National Universities 2009 (stdevs): Actual Monte Carlo
Peer Survey 0.7 0.5Graduation Performance 7% 11%
Classes under 20 14% 11%Classes 50 and over 6% 9%
Student/Faculty ratio 4.3 4.2Full-time Faculty 10% 16%
SAT 25th 140 143SAT 75th 134 153
High School Rank 27% 12%Acceptance Rate 21% 12%
Faculty Terminal Degrees 9% 8%Graduation Rate 18% 22%
Retention Rate 9% 13%Alumni Giving 10% 7%
IPEDS Finances ratio (log) 0.24 0.26Faculty Salaries $16,700 $21,100