![Page 1: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/1.jpg)
Are we still talking about diversity in classifier
ensembles?Ludmila I Kuncheva
School of Computer ScienceBangor University, UK
![Page 2: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/2.jpg)
Are we still talking about diversity in classifier
ensembles?Ludmila I Kuncheva
School of Computer ScienceBangor University, UK
Completely irrelevant to your Workshop...
![Page 3: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/3.jpg)
Let’s talk instead of:
Multi-view and classifier ensembles
![Page 4: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/4.jpg)
classifier
feature values(object description)
classifier classifier
class label
“combiner”
A classifier ensemble
![Page 5: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/5.jpg)
feature values(object description)
class label
a neural network
classifier
combiner
classifier
ensemble?
![Page 6: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/6.jpg)
classifier
feature values(object description)
class label
class
ifie
rcl
ass
ifie
rcl
ass
ifie
rcl
ass
ifie
rcl
ass
ifie
rcl
ass
ifie
r
ensemble?
a fancy combin
er
![Page 7: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/7.jpg)
classifier?
a fancy feature
extractor
classifier
feature values(object description)
classifier classifier
class label
“combiner”
![Page 8: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/8.jpg)
a. because we like to complicate entities beyond necessity (anti-Occam’s razor)
b. because we are lazy and stupid and can’t be bothered to design and train one single sophisticated classifierc. because democracy is so important to our society, it must be important to classification
Why classifier ensembles then?
![Page 9: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/9.jpg)
combination of multiple classifiers [Lam95,Woods97,Xu92,Kittler98]classifier fusion [Cho95,Gader96,Grabisch92,Keller94,Bloch96]mixture of experts [Jacobs91,Jacobs95,Jordan95,Nowlan91]committees of neural networks [Bishop95,Drucker94]consensus aggregation [Benediktsson92,Ng92,Benediktsson97]voting pool of classifiers [Battiti94]dynamic classifier selection [Woods97]composite classifier systems [Dasarathy78]classifier ensembles [Drucker94,Filippi94,Sharkey99]bagging, boosting, arcing, wagging [Sharkey99]modular systems [Sharkey99]collective recognition [Rastrigin81,Barabash83]stacked generalization [Wolpert92]divide-and-conquer classifiers [Chiang94]pandemonium system of reflective agents [Smieja96] change-glasses approach to classifier selection [KunchevaPRL93]etc.
fanciest
oldest
oldest
![Page 10: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/10.jpg)
combination of multiple classifiers [Lam95,Woods97,Xu92,Kittler98]classifier fusion [Cho95,Gader96,Grabisch92,Keller94,Bloch96]mixture of experts [Jacobs91,Jacobs95,Jordan95,Nowlan91]committees of neural networks [Bishop95,Drucker94]consensus aggregation [Benediktsson92,Ng92,Benediktsson97]voting pool of classifiers [Battiti94]dynamic classifier selection [Woods97]composite classifier systems [Dasarathy78]classifier ensembles [Drucker94,Filippi94,Sharkey99]bagging, boosting, arcing, wagging [Sharkey99]modular systems [Sharkey99]collective recognition [Rastrigin81,Barabash83]stacked generalization [Wolpert92]divide-and-conquer classifiers [Chiang94]pandemonium system of reflective agents [Smieja96] change-glasses approach to classifier selection [KunchevaPRL93]etc.
Out of fashionOut of fashion
Subsumed
Subsumed
![Page 11: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/11.jpg)
Congratulations!The Netflix Prize sought to substantially improve the accuracy of predictions about how much someone is going to enjoy a movie based on their movie preferences.On September 21, 2009 we awarded the $1M Grand Prize to team “BellKor’s Pragmatic Chaos”. Read about their algorithm, checkout team scores on the Leaderboard, and join the discussions on the Forum.We applaud all the contributors to this quest, which improves our ability to connect people to the movies they love.
classifier
feature values(object description)
classifier classifier
class label
combinerclassifier ensemble
![Page 12: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/12.jpg)
cited 7194 times
by 28 July 2013
(Google Scholar)
classifier
feature values(object description)
classifier classifier
class label
combinerclassifier ensemble
![Page 13: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/13.jpg)
Saso Dzeroski
David Hand
S. Dzeroski, and B. Zenko. (2004) Is combining classifiers better than selecting the best one? Machine Learning, 54, 255-273.
David J. Hand (2006) Classifier technology and the illusion of progress, Statist. Sci. 21 (1), 1-14.
Classifier combination? Hmmmm…..
We are kidding ourselves; there is no real progress in spite of ensemble methods.
Chances are that the single best classifier will be better than the ensemble.
![Page 14: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/14.jpg)
Quo Vadis?
"combining classifiers" OR "classifier combination" OR "classifier ensembles" OR "ensemble of classifiers" OR "combining multiple classifiers" OR "committee of classifiers" OR "classifier committee" OR "committees of neural networks" OR "consensus aggregation" OR "mixture of experts" OR "bagging predictors" OR adaboost OR (( "random subspace" OR "random forest" OR "rotation forest" OR boosting) AND "machine learning")
![Page 15: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/15.jpg)
time
visi
bilit
y
naiv
e eu
phor
ia
asymptote of reality
slope of enlightenment
trough of disillusionment
peak of inflated expectations
Gartner’s Hype Cycle: a typical evolution pattern of a new technology
Where are we?...
![Page 16: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/16.jpg)
1990 1995 2000 2005 20100
0.05
0.1
0.15
0.2
0.25
0.3
0.35
per m
il of
pub
lishe
d pa
pers
on
clas
sifie
r ens
embl
es
time
IEEE
TSM
C
IEEE
TPA
MI
NN ML
IEEE
TPA
MI
IEEE
TPA
MI
ML
IEEE
TPA
MI
ML
JASA
ML
IJCV
PRIE
EE T
PAM
IIE
EE T
PAM
IJA
E PPL
PPL JT
BCC
(6) IEEE TPAMI = IEEE Transactions on Pattern Analysis and Machine Intelligence
IEEE TSMC = IEEE Transactions on Systems, Man and CyberneticsJASA = Journal of the American Statistical Association
IJCV = International Journal of Computer VisionJTB = Journal of Theoretical Biology
(2) PPL = Protein and Peptide LettersJAE = Journal of Animal Ecology
PR = Pattern Recognition (4) ML = Machine Learning
NN = Neural NetworksCC = Cerebral Cortex
top cited paper is from…
application paper
![Page 17: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/17.jpg)
1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 20120
500
1000
1500
2000
2500
3000
3500
4000
4500
num
ber o
f cita
tions
time
[ML] Bagging predictors
[IEEE TPAMI] On combining classifiers
[ML] Random forests
[IJCV] Robust real-time face detection
![Page 18: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/18.jpg)
International Workshop on Multiple Classifier Systems2000 – 2013 - continuing
![Page 19: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/19.jpg)
Combiner
Features
Classifier 2
Classifier 1
Classifier L
…
Data set
A Combination level• selection or fusion?• voting or another combination method?• trainable or non-trainable combiner?
B Classifier level• same or different classifiers?• decision trees, neural networks or other?• how many?
C Feature level• all features or subsets of features?• random or selected subsets?D Data level
• independent/dependent bootstrap samples?
• selected data sets?
Levels of questions
![Page 20: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/20.jpg)
50 diverse linear classifiers 50 non-diverse linear classifiers
![Page 21: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/21.jpg)
Number of classifiers L1
The perfect classifier• 3-8 classifiers• heterogeneous• trained combiner(stacked generalisation)
• 100+ classifiers• same model• non-trained
combiner(bagging, boosting, etc.)
Large ensemble of nearly identical classifiers - REDUNDANCY
Small ensembles of weak classifiers - INSUFFICIENCY?
?
Must engineer diversity…
Strength of classifiers
How about here?• 30-50 classifiers• same or different models?• trained or non-trained
combiner?• selection or fusion?• IS IT WORTH IT?
![Page 22: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/22.jpg)
Number of classifiers L1
The perfect classifier• 3-8 classifiers• heterogeneous• trained combiner(stacked generalisation)
• 100+ classifiers• same model• non-trained
combiner(bagging, boosting, etc.)
Large ensemble of nearly identical classifiers - REDUNDANCY
Small ensembles of weak classifiers - INSUFFICIENCY
Must engineer diversity…
Strength of classifiers
• 30-50 classifiers• same or different models?• trained or non-trained
combiner?• selection or fusion?• IS IT WORTH IT?
![Page 23: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/23.jpg)
classifier
feature values(object description)
classifier classifier
class label
“combiner”
A classifier ensemble
one view
![Page 24: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/24.jpg)
classifier
feature values(object description)
classifier classifier
class label
“combiner”
A classifier ensemble
multiple views
feature values(object description)
feature values(object description)
![Page 25: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/25.jpg)
1998
![Page 26: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/26.jpg)
“distinct” is what you call
“late fusion”
“shared” is what you call
“early fusion”
![Page 27: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/27.jpg)
EXPRESSION OF EMOTION - MODALITIES
facial expression
posture
behaviouralphysiologic
al
peripheral nervous system
central nervous system
EEG
fMRI
Galvanic skin response
blood pressure
skin to
respiration
EMG
speech
gesture
interaction with
the compute
r
pressure on mouse
drag-click speed
eye tracking
fNIRS
pulse rate
pulse variation
dialogue with tutor
![Page 28: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/28.jpg)
Data Classification Strategies
modality 1
modality 2
modality 3
(1) Concatenate the features from all modalities
(2) Feature extraction and concatenation
(3) Straight ensemble classification
ensemble
“early fusion”
“late fusion”
“mid-fusion”
And many combinations thereof...
![Page 29: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/29.jpg)
Data Classification Strategies
modality 1
modality 2
modality 3
(1) Concatenate the features from all modalities
(2) Feature extraction and concatenation
(3) Straight ensemble classification
ensemble
“early fusion”
“late fusion”
“mid-fusion”
We capture all dependencies but can’t handle the complexity
We lose the dependencies but can handle the complexity
![Page 30: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/30.jpg)
Multiview early and mid-fusion
Ensemble Feature Selection
By the ensemble(RANKERS) For the ensemble
Decision tree
ensembles
Ensemblesof different
rankers
Bootstrapensemblesof rankers
Randomapproach
Systematicapproach
Uniform (Random subspace)
Non-uniform
(GA)
Incrementalor iterative
Feature selection
Multiviewlate fusion
Greedy
Greedy
![Page 31: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/31.jpg)
Multiview early and mid-fusion
Uniform (Random subspace)
Non-uniform
(GA)
Incrementalor iterative
Feature selection
Greedy
Greedy
![Page 32: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/32.jpg)
This is what I think:
1. Deciding which approach to take is rather art than science
2. This choice is, crucially, CONTEX-SPECIFIC.
![Page 33: Are we still talking about diversity in classifier ensembles? Ludmila I Kuncheva School of Computer Science Bangor University, UK](https://reader030.vdocuments.us/reader030/viewer/2022032604/56649e5c5503460f94b54d7a/html5/thumbnails/33.jpg)
Where does diversity come to this?
Hmm... Nowhere...