what’s the purpose of assessing higher education’s learning outcomes
TRANSCRIPT
WHAT’S THE PURPOSE OF
ASSESSING HIGHER EDUCATION’S
LEARNING OUTCOMES?
Dirk Van Damme
OECD/EDU/IMEP
WHAT IS IT?
• Objective: test whether it is possible to
– measure what undergraduate students know and
can do upon graduation …
– … across diverse countries, languages, cultures
and types of institutions
• For
– Generic skills
– Economics
– Engineering
Assessing Higher Education Learning
Outcomes (AHELO) Feasibility Study (2008-12)
•
•
Observers
Bahrein
Brazil
Saudi Arabia
Singapore
•
•
• •
• •
• •
• •
•
• •
•
•
•
•
•
•
•
• •
A range of geographic, linguistic and cultural
backgrounds involved
•
Generic Skills
Colombia
Egypt
Finland
Korea
Kuwait
Mexico
Norway
Slovak Republic
United States
(CT, MO, PA)
Engineering
Abu Dhabi
Australia
Canada (Ontario)
Colombia
Egypt
Japan
Mexico
Russian Fed.
Slovak Republic
Economics
Belgium (Fl.)
Egypt
Italy
Mexico
Netherlands
Russian Fed.
Slovak Republic
• Data collection from February to June 2012
• 17 countries involved in 25 strand replications
• Data collected from over
– 23,000 students
– 4,900 faculties
– 270 institution coordinators
• 1,000 test sessions and 20,000 computers involved
• Scoring completed in June 2012
• Analysis of results and findings
• Final reporting 2012-13
Quick facts on AHELO Feasibility Study
• The main conclusions of the meeting were:
– The revised proposal should frame the assessment of higher education learning outcomes around transversal skills, and build any disciplinary assessments and contexts around those.
– Individual student results and feedback will be critical to institutional and student engagement and, therefore, the assessment should extend to all eligible students of participating entities, rather than a sample only.
– To report results from the assessment both in absolute terms (bottom-line results) as well as relative to the socio-economic and institutional context of individuals and universities (analytic value added).
AHELO Main Study – London meeting with
interested countries on 11 Feb 2015
6
• The main conclusions of the meeting were:
– To devote greater attention to the development of intermediate outputs (frameworks, instruments and methodologies), not only to ensure that participating entities obtain short-term value for their investments, but also to facilitate the important dialogue with the academic community and other stakeholders.
– The AHELO Main Study should be governed by national or subnational government authorities, but the governing body should be complemented with a strong Academic Advisory Group from among participating institutions that would oversee much of the substantive development of the assessment. The governing body would be made up of those countries or subnational entities that contribute to the development and financing of the project, while recognising that not all participating entities may be able to implement the assessment from the outset.
AHELO Main Study – London meeting with
interested countries on 11 Feb 2015
7
• The AHELO Scoping Paper has been revised
accordingly and published as
EDU/EDPC(2013)17/REV3
• A new funding scheme has been developed.
• Countries are invited to express their willingness
to participate in Year 1 (2016) of the AHELO Main
Study on the basis of this proposal and to do so
before the end of May 2015.
AHELO Main Study – where are we now
8
SHOULD IT BE DONE?
• Higher education is a rapidly expanding and globalising system with extremely low levels of systemic transparency
• Lack of internally generated transparency leads to various forms of external mechanisms, most importantly rankings and consumerist approaches to quality
– Often not providing right incentives to improve teaching and learning
• Various national attempts to measure learning outcomes in higher education, but lack of international, comparative approach ≈ PISA
Background
1. Globalisation of demand: what are students
actually buying?
2. Outcomes and impact: what do employers and
wider society get?
3. Doubts on the value of HE qualifications:
selection, credentialism or learning gain?
4. Growing dissatisfaction with the way rankings
are ‘ordering’ the HE system and distribute
rewards and incentives
Drivers
• Explosion of global demand in emerging
countries
– In what exactly are the emerging middle classes
investing enormous resources?
• Global imbalance between demand and supply
of supposed academic excellence
– Imbalance which cannot be met by international
mobility (4.5m now, expected to increase to 7.2m
in 2020) and e-learning/MOOCs
1. Globalisation
Global expansion & redistribution of qualifications Global distribution of tertiary educated 25-34 y-olds in 2013 and 2030
14
United States, 13.7%
China, 17.8%
Russian Federation,
10.9%
Japan, 6.9%
India, 11.4%
Korea, 3.9%
Mexico, 3.0% France,
2.6%
Germany, 2.0%
United Kingdom,
2.9%
Indonesia, 4.3%
Spain, 2.2%
Canada, 2.1%
Brazil, 3.0%
Turkey, 1.7%
Other, 11.7%
Share in academic graduates 2010
United States 43.2%
United Kingdom 13.8%
Netherlands 6.0% Germany
4.3%
Canada 4.3%
Australia 4.3%
Switzerland 3.5%
France 3.0%
Japan 2.5%
Sweden 2.6%
Korea 2.2%
Hong Kong 2.0%
Other 8.4%
Share in academic excellence THEWUR 2012
Global distribution of academic
graduates and academic excellence
• Economic outcomes
– Employment
– Earnings
• Social outcomes
– Social capital, interpersonal trust
– Health
– Political participation
– Volunteering
– Anti-social behaviour
2. Persistent high impact of HE on economic
and social outcomes
Employment benefits very significant Employment rates among 25-64 year-olds, by educational
attainment (2012)
20
30
40
50
60
70
80
90
100
Icel
and
No
rway
Swit
zerl
and
Swed
en
Ger
man
y
Net
her
lan
ds
Au
stri
a
Den
mar
k
Latv
ia
Bra
zil
Slo
ven
ia
Isra
el
Luxe
mb
ou
rg
Po
lan
d
Bel
giu
m
Au
stra
lia
Fran
ce
Fin
lan
d
Ch
ile
Un
ited
Kin
gdo
m
New
Ze
alan
d
Cze
ch R
epu
blic
OEC
D a
vera
ge
EU 2
1 a
vera
ge
Ru
ssia
n F
eder
atio
n
Esto
nia
Po
rtu
gal
Can
ada
Mex
ico
Un
ited
Sta
tes
Slo
vak
Rep
ub
lic
Irel
and
Jap
an
Hu
nga
ry
Ital
y
Spai
n
Ko
rea
Turk
ey
Gre
ece
% Below upper secondary education Upper secondary or post-secondary non-tertiary education Tertiary education
Social outcomes – interpersonal trust Proportion of adults reporting that they trust others, by educational
attainment (2012)
0
10
20
30
40
50
60
70
Slo
vak
Rep
ub
lic
Esto
nia
Ital
y
Cze
ch R
epu
blic
Fran
ce
Ko
rea
Ger
man
y
Jap
an
Irel
and
Po
lan
d
Englan
d/N
.…
Ave
rage
Flan
der
s (B
elgi
um
)
Can
ada
Spai
n
Au
stri
a
Un
ited
Sta
tes
Au
stra
lia
Fin
lan
d
Net
her
lan
ds
Swed
en
No
rway
Den
mar
k
%
Below upper secondary education Upper secondary or post-secondary non-tertiary education Tertiary Education
Social outcomes – trust in political institutions Individuals with higher level of education more likely to believe
they have a say in government
0
10
20
30
40
50
60
70
80
Czech R
ep
ub
lic
Italy
Spa
in
Esto
nia
Germ
any
Ja
pa
n
Slo
vak R
epu
blic
Irela
nd
Cana
da
Kore
a
Eng
land
/N.
Irela
nd
(UK
)Austr
ia
Avera
ge
Fla
nde
rs (
Be
lgiu
m)
Austr
alia
United
Sta
tes
Neth
erla
nds
Sw
ede
n
Denm
ark
Fin
land
Norw
ay
%
Below upper secondary education
Upper secondary or post-secondary non-tertiary education
Tertiary Education
• What do we know about the value of higher
education’s qualification in terms of knowledge
or skills?
3. Knowledge & skills value of higher
education qualifications
Formal education vs skills
Formal education vs skills
200 220 240 260 280 300 320 340 360 380
Spain
England (UK)
England/N. Ireland (UK)
Ireland
Italy
Korea
Canada
Poland
United States
Northern Ireland (UK)
Australia
Estonia
Average
France
Denmark
Norway
Slovak Republic
Germany
Japan
Sweden
Austria
Netherlands
Flanders (Belgium)
Czech Republic
Finland
Numeracy scores of tertiary educated adults of 25-34y old
95th percentile mean score tertiary 25-34y
24
Numeracy equivalent of tertiary qualifications Proportion of 25-64 year-olds scoring at PIAAC numeracy level
4 and 5, by educational attainment of the population (2012)
0
5
10
15
20
25
30
35
40
Japa
n
Fin
land
Ne
therl
and
s
Sw
ede
n
Austr
alia
No
rwa
y
Fla
nde
rs (
Belg
ium
)
Engla
nd
(U
K)
Engla
nd
/N. Ir
ela
nd
(UK
)
Un
ite
d S
tate
s
Czech R
epu
blic
OE
CD
avera
ge
Pola
nd
Ca
nad
a
No
rth
ern
Ire
lan
d (
UK
)
Austr
ia
Germ
any
Irela
nd
Fra
nce
De
nm
ark
Esto
nia
Slo
vak R
epub
lic
Kore
a
Ru
ssia
n F
ed
era
tion
Spain
Ita
ly
Below upper secondary education Upper secondary or post-secondary non-tertiary education Tertiary education
Literacy equivalent of tertiary qualifications
25
• Using the THEWUR database on the Top 200
universities, how dynamic is the HE system?
– Teaching – research – citations
• Current input and reputation measures do not
provide the right incentives for improving or
innovating teaching & learning
4. Structure and hierarchy in HE and the
associated incentives system
Institutional moves in the THEWUR Absolute moves
0
5
10
15
1 51 101 151
Teaching Research Citations International
Poly. (Teaching) Poly. (Research) Poly. (Citations)
Where are the top20 performers?
0%
25%
50%
75%
100%
0
5
10
15
20
Teaching Research Citations International Relative share of top performing uni (any scale) per country (right axis)
• Is it possible?
– To improve our understanding of what students actually ‘learn’ in higher education
– To exchange reputations with empirically grounded observations of quality of teaching & learning
– To gradually transform the field on which credentials are traded into a more level playing field
– To provide better information to students and employers about the quality of teaching & learning experiences
– To develop feedback loops to improve teaching and learning
– To reward and incentivise institutions that significantly improve their teaching & learning environments
– To re-confirm the value of teaching as part of the university’s mission next to research
Conclusion: what are the value-propositions?
CAN IT BE DONE?
• Strong preference among interested countries
for generic skills
– Generic academic skills are at the core of the
mission of universities
– Specialised, disciplinary knowledge and skills is
more different
– Overlap with established systems or practices on
disciplinary level
– Strong preference among employers as well
Disciplines or generic skills
• How to respect and value institutional heterogeneity and diversification?
• Developing various scales?
– Research skills
– Complex reasoning
– Creative thinking
– Problem-solving
– Social and emotional skills
– Communication skills
– Employability and entrepreneurship skills
– Etc…
Various scales
• Orientation towards assessment with clear
benefits for students
• Countries want a census-type of assessment
• Necessity to provide something of value to
universities
– Balancing accountability and improvement
functions
Who benefits: student, institution, country?
• Real value-added assessment considered to be
the best option, but at the same time unrealistic
• Two approaches
– Absolute scores at individual, institutional and
country level
– Analytical value-added scores based on
regressions to background variables at
institutional and country level
Absolute or value-added
