honors research colloquium presentation slides
Post on 13-Aug-2015
96 Views
Preview:
TRANSCRIPT
Producing Valid College Football Rankings in
Reasonable TimeMark Edwards, Department of Mechanical Engineering
Jonathan A. Shumaker, Department of Chemical Engineering C. Richard Cassady, PhD, Department of Industrial Engineering
5th Annual FEP Honors Research Symposium 4/20/13
2
NCAA Football
NCAA Football Bowl Subdivision: 124 colleges/universitieso each team plays approximately 12 games against other FBS
teamso only 9% of possible matchups occur
determining the FBS champion: one game, 2 teamso 4-team playoff beginning in 2014o more than just a game: $172M on the table ($450M in 2014)
effectively ranking the teams is critical, but difficulto 124! possible rankingso what to consider in the rankingso maintaining fairness in the rankingso conflicting results
Edwards, M. and J. Shumaker 5th Annual FEP Honors Research Symposium
3
Current Ranking System
Bowl Championship Series rankingso two opinion polls: 1/3 eacho six computer rankings: 1/3
BCS flawso opinion polls: bias, who is actually voting, weird ballotso computer rankings: secrecy, data errors, design bias
annual controversy
Edwards, M. and J. Shumaker5th Annual FEP Honors Research
Symposium
5th Annual FEP Honors Research Symposium
4
Our Platform
4-team playoff increases need for a better ranking systemo selection committee will need guidanceo harder to distinguish between #4 and #5 than #2 and
#3 no such thing as an unbiased approach
o what counts should be clearly stated a computer-based system should be used
o humans cannot simultaneously process results of 650-800 games
o computer can apply biases consistently across all teams
Edwards, M. and J. Shumaker
5th Annual FEP Honors Research Symposium
5
Our Approach: CMS+
define the ranking problem mathematicallyo quadratic assignment problemo inputs: degree of victory, relative distanceo output: who gets ranked where
degree of victory: how we compare pairs of teamso head-to-head? margin of victory? date? location?o transitive results? conference champions? schedule
strength? relative distance: how far apart are positions in
the rankingo bell curve
Edwards, M. and J. Shumaker
5th Annual FEP Honors Research Symposium
6
Solving the QAP
large QAP are impossible to solve to guaranteed optimality
CMS+ uses a two-stage heuristico stage 1: genetic algorithm
• uses “survival of the fittest” concepts to convert good rankings into better rankings
• simulate 100,000 generations of rankingso stage 2: local search
• seek pair-wise switches in the best GA solution that improve solution quality
initial GA population is randomo replicate the heuristic 20 times
Edwards, M. and J. Shumaker
5th Annual FEP Honors Research Symposium
7
CMS+ Issues
What to include in degree of victory
o Objective 1: to investigate what factors should be included (and how to include them) in degree of victory (in partnership with T. Dodson and A. McElhenney)
Long time required to solve the QAP
o Objective 2: to investigate the impact of reducing the number of GA generations and heuristic replications on solution quality
Edwards, M. and J. Shumaker
8
Objective 1: Degree of Victory (DOV)
identify possible factors to include define ways to quantify these factors collect data from past seasons (1998-2011) generate alternative rankings using various
combinations of the identified factors
Edwards, M. and J. Shumaker5th Annual FEP Honors Research
Symposium
5th Annual FEP Honors Research Symposium
9
DOV: Factors Considered
winners and losers of regular season games four optional factors
o game location• winning away from home generates a bonus
o common opponents• which team won more games against common opponents
o conference champions• if both teams are in the same conference, did one win the
conferenceo media perception
• which team is ranked higher in the Associated Press opinion poll
Edwards, M. and J. Shumaker
5th Annual FEP Honors Research Symposium
10
DOV: Quantifying the Factors
Team A receives 15 DOV points over Team B ifo game location is considered and Team A defeated Team
B at Team B’s home field• 7.5 points for a neutral site victory
o common opponents are considered and Team A has more victories than Team B against their common opponents
o conference champions are considered, Team A and Team B are in the same conference, and Team A is the conference champion
o media perception is considered and Team A is ranked above Team B in the Associated Press opinion poll
Team A receives d DOV points over Team B if Team A defeated Team Bo d = 100 – 15*(# of optional factors considered)
Edwards, M. and J. Shumaker
5th Annual FEP Honors Research Symposium
11
DOV: Data Collection
Collected data from past seasons (1998-2011)1
Validated data with two parallel effortso T. Dodson, A. McElhenneyo B. Wiles, industrial engineering senior
4 optional factors (off/on) results in 16 DOV sets for each yearo generated in Microsoft Excel using a Visual Basic for
Applications (VBA) macro
Edwards, M. and J. Shumaker
1Data collected from: http://homepages.cae.wisc.edu/~dwilson/rfsc/history/howell/http://espn.go.com/college-football/conferenceshttp://www.collegepollarchive.com/football/ap/seasons.cfm?seasonid=1998
5th Annual FEP Honors Research Symposium
12
DOV: Rankings
16 DOV sets for 2010 used as test cases CMS+ executed for each set
o Shown below with comparison to BCS Rankings from 2010
Edwards, M. and J. Shumaker
BCS Rankings DOV 05(AP Rank)
DOV 16(Everything on)
Fitness 246025 196313
1 Auburn Oregon Oregon
2 Oregon Texas Christian Auburn
3 Texas Christian Boise State^ Texas Christian
4 Stanford Auburn Boise State^
5 Wisconsin Ohio State Michigan State
6 Ohio State Nevada^ Wisconsin
7 Oklahoma Wisconsin Virginia Tech^
8 Arkansas Stanford Nevada^
9 Michigan State Michigan State Stanford*
10 Boise State Utah^ Ohio State
5th Annual FEP Honors Research Symposium
13
Objective 2: Run Time (RT)
define test cases for the QAP using past seasons experiment with various combinations of the
number of GA generations and the number of heuristic replications
assess the impact of reduced run time on solution qualityo in partnership with T. Dodson and A. McElhenney
Edwards, M. and J. Shumaker
5th Annual FEP Honors Research Symposium
14
RT: Experimental Plan
Edwards, M. and J. Shumaker
experiments decrease in generations and replicationso Based on the original parameters
number of replications did not have an affect on run time
fitness and rankings never changed number of generations is the only factor that
alters run time
5th Annual FEP Honors Research Symposium
15
RT: Results
Edwards, M. and J. Shumaker
In the 4,500 to 2,500 range of total generations, the lowest time value can be achieved. Beyond this in either direction, the time starts to increase.
5th Annual FEP Honors Research Symposium
16
RT: Impact on Solution Quality
for any given DOV set and a corresponding rankingo fitness can be computed as described by CMS+o larger fitness implies a better solution/ranking
T. Dodson and A. McElhenney concluded that fitness is normally distributed
they provide estimates ofo the average fitness and the standard deviation of fitnesso the number of standard deviations by which the CMS+
solution exceeds the averageo the number of the 124! rankings that have a better
fitness value
Edwards, M. and J. Shumaker
5th Annual FEP Honors Research Symposium
17
RT: Impact on Solution Quality
Fitness remained constant for all generation amountso Reduction of generations without having to worry about
loss of quality Changing replication amount had no affect on
fitnesso Also had no affect on time
Time was able to be reduced to as low as about 25 secondso At 3,500 generations
• More or less generations meant longer time• Optimal range is around 3,500 generations
Edwards, M. and J. Shumaker
5th Annual FEP Honors Research Symposium
18
Conclusions
objectives allowed for CMS+ to be improvedo DOV research created varying factors o Run Time research found ideal specifications within the
program CMS+ system can be improved with future
researcho More parameters for point allocation
• Only used four parameters while there are many more parameters that could be used
o Adjust other factors within the CMS+ program to see how they affect run time and quality• Only used two parameters, total generations and
replication amount• Could change population size
Edwards, M. and J. Shumaker
top related