developing a critical interventions framework
DESCRIPTION
In early 2013, the authors were commissioned by DIICCSRTE to develop a critical interventions framework for student equity in higher education. To answer the seemingly simple question of whether we as a sector were on track in achieving our national social inclusion goals, we must review the current student equity makeup of the sector, and determine how effective our equity initiatives are. The first part of that question was relatively easy to answer. However, finding clear, rigorous evidence of program efficacy from the literature was much more difficult. In this presentation, I will discuss the critical interventions framework and the difficulties with uncovering evidence of effectiveness as opposed to the theoretical strength of an initiative, and briefly discuss how the framework might be used in the future.TRANSCRIPT
Critical Interventions FrameworkDr Ryan Naylor, CSHE, University of Melbourne 05/02/2014
Developing a
Curtin University is a trademark of Curtin University of TechnologyCRICOS Provider Code 00301J
Project Background
• Commissioned project for DIICCSRTE, completed by Dr Ryan Naylor, Dr Chi Baik, Professor Richard James
• Are we on track in achieving national social inclusion goals?
1. Where are we?
2. How effective are our current initiatives? (What appears to work well? What doesn’t?)i. Is it possible to generate a typology of equity initiatives to allow consolidation of research evidence?
ii. Is there evidence in the literature or from HEPPP evaluations to support their efficacy?
Caution!
• Some necessary simplifications had to be made in creating the typology and fitting the literature to it
• “Intervention” is a contentious term with troubling associations
• New coalition government changes the policy context
What has had the biggest effect on equity?
• Uncapping/deregulation of volume of undergraduate places?
• National target for low SES participation (and associated Mission Based Compacts)?• HEPPP funding?• Wider societal trends in community beliefs about the value of undertaking higher education, entry requirements, eligibility for participation?• Efficacy of equity initiatives depends on underlying factors such as these• Many variables, highly inter-related
Where are we?A quick look at the numbers
Since 2007, there has been an explosion in domestic student numbers
2007: approx. 722,000 domestic students
2011: approx. 888,000 domestic students
= An increase of 23% over 4 years, oran annual growth rate of 5%
This level of growth is unprecedented in Australian HE
Gains in participation share have been made…
Participation Ratio (2011)
…But they have been relatively modest and not universal
Participation Ratio (2011)
It is difficult to improve equity during growth periods
Stud
ents
with
a d
isab
ility
Indi
geno
us s
tude
nts
Stud
ents
from
a N
on E
nglis
h Sp
eaki
ng
Back
grou
nd
Wom
en in
Non
-Tra
ditio
nal A
reas
Low
SES
(pos
tcod
e m
easu
re)
Low
SES
(CD
mea
sure
)
Regi
onal
Rem
ote
Tota
l
0
2
4
6
8
10
12
Tota
l
Average grow
th rate (2008-2011) (%)
But some groups whose share has historically been stable have
increased their participation share
Stud
ents
with
a d
isabi
lity
Indi
geno
us st
uden
ts
Stud
ents
from
a N
on E
nglis
h Sp
eaki
ng B
ackg
roun
d
Wom
en in
Non
-Tra
ditio
nal
Area
s
Low
SES
(pos
tcod
e m
easu
re)
Low
SES
(CD
mea
sure
)
Regi
onal
Rem
ote
Tota
l
0
2
4
6
8
10
12
Tota
lA
verage growth rate (2008-2011) (%
)
This change has not been uniform – low SES
Access rate (2011) (%)
Change since 2007 (%)
A 4.25 -0.01
B 6.92 2.38
C 7.10 -0.39
D 15.14 -0.14
E 20.49 -1.38
F 32.82 0.66
G 33.83 2.88
Sector average 16.87 0.75
• No correlation between access rate (2011, 2007) and change• Complex factors – different geographical contexts, access policies,
etc• Traditional strong performers didn’t do better
Most groups appear no less likely to succeed
Tota
l
Most groups (bar 2) are no less likely to succeed
Tota
l
May not have seen full effects yet – not all students from DDS cohort have moved through the system yet (early days yet!)
For most groups, the key problem continues to be access
Equity group Participation Retention Success
Students with a disability 0.48 0.96 0.93
Indigenous students 0.55 0.85 0.81
Students from Non English Speaking Background
0.82 1.04 0.97
Women in Non-Traditional Areas 0.35 1.01 0.99
Low SES (postcode measure) 0.67 0.98 0.97
Low SES (CD measure) 0.62 0.97 0.96
Regional 0.64 0.98 0.99
Remote 0.39 0.91 0.94
Participation and access are (should be?) key focal points in student equity and social inclusion
Equity group Participation Retention Success
Students with a disability 0.48 0.96 0.93
Indigenous students 0.55 0.85 0.81
Students from Non English Speaking Background
0.82 1.04 0.97
Women in Non-Traditional Areas 0.35 1.01 0.99
Low SES (postcode measure) 0.67 0.98 0.97
Low SES (CD measure) 0.62 0.97 0.96
Regional 0.64 0.98 0.99
Remote 0.39 0.91 0.94
This is not to argue they don’t need support once enrolled• less academically well prepared students from any background• Indigenous students
For most groups, the key problem continues to be access
How effective are our current initiatives:The Critical Interventions Framework
The Critical Interventions Framework
• Is it possible to create a typology of equity initiatives?• Can we find sufficient evidence in the literature to point to the efficacy of particular types of initiatives?• Can we identify the initiatives that are most effective?• We don’t know for sure, so we had to make some guesses
The Critical Interventions Framework
• Is it possible to create a typology of equity initiatives?• Yes
• Can we find sufficient evidence in the literature to point to the efficacy of particular types of initiatives?• Often no
• Can we identify the initiatives that are most effective?• No. There simply isn’t enough evidence.
The equity initiative terrain across a notional student lifecycle
The Critical Interventions Framework (one small section)
The Critical Interventions Framework (one small section)
The Critical Interventions Framework (one small section)
• We all have intuitions about what works • Attempted to base framework on evaluative science, not intuition• Unfortunately, the science is largely not there, so some estimation and judgement was involved
The Critical Interventions Framework (one small section)
All things considered, what did we rate highly?
High
1B. Later-year outreach (Years 10-12)
2B. Bridging/foundation programs
2D. Scholarships
4C. Student services provision
5A. Monitoring student completion rates
Very High
1D. School curriculum enhancement/support
2A. Pathway/articulation programs
2C. Alternate selection criteria and tools
3A. First year orientation/transition support
• Not intended to narrow or homogenise people’s efforts
Red - rated very high; green – rated high
Do we have good evidence?
High Quality of Evidence
1B. Later-year outreach (Years 10-12) Limited
2B. Bridging/foundation programs Some (from US)
2D. Scholarships Strong (needs-based, not merit)
4C. Student services provision Varies depending on service
5A. Monitoring student completion rates N/A
Very High
1D. School curriculum enhancement/support Some (from US); strong for need
2A. Pathway/articulation programs Mixed
2C. Alternate selection criteria and tools Strong
3A. First year orientation/transition support Strong (from US)
Red – strong evidence
Green – some evidence
Blue – may be impossible to get evidence
How are we spending our HEPPP funding?
High Proportion of HEPPP funding
1B. Later-year outreach (Years 10-12) 14
2B. Bridging/foundation programs 7
2D. Scholarships 14
4C. Student services provision 15
5A. Monitoring student completion rates N/A
Very High
1D. School curriculum enhancement/support 1
2A. Pathway/articulation programs 3
2C. Alternate selection criteria and tools 2
3A. First year orientation/transition support 10Total: 66%
• This is purely descriptive, not normative!• Not all initiatives require the same amount of funding
Where to next?
• Written for national policy purposes – not meant to function at the institutional level • We need better evidence of program efficacy (and this is where institutions come in)
o Detailed, rigorous and published evaluationso To enable a sector-wide conversation about
equity initiatives• How can the CIF be used in an institutional context? Is this typology helpful?
Where to next?
• Funding sought to research: o Process and attitudinal factors affecting program
evaluationo Resources for embedding evaluation into core
business• First year experience survey• Collaborations with NCSEHE staff