understanding learning gain and why this might matter to you

CONSULTING www.coronyedwards.co.uk

CoronyEdwards

UnderstandingLearning Gain

and why this might matter to youJisc Student Experience Experts’ Group

Birmingham23 June 2016


CoronyEdwards

Session outline and objectives1. To clarify what learning gain means in the context of UK HE

2. To consider some drivers for the current interest in LG, including TEF

3. To introduce the types of measure used to gauge LG, and the challenges associated with developing these measures

4. To consider how learning technologies could be deployed to provide robust data for learning gain measures.


CoronyEdwards

Definitions of learning gain“an attempt to measure the improvement in knowledge, skills, work-readiness and personal development made by students during their time spent in higher education” HEFCE Learning Gain website

“the ‘distance travelled’, or the difference between the skills, competencies, content knowledge and personal development demonstrated by students at two points in time”

RAND report: McGrath et al, 2015, p. ix)


CoronyEdwards

Attempts to measure learning gain are not newBut, as noted in the RAND report (2015, p. ix)

“Debate on the measurement of the learning students gain in higher education has been prominent internationally, but the concept has not been studied extensively in the English higher education context.”


CoronyEdwards

Drivers for measuring learning gain“There is growing interest around the world in measuring how much students in higher education learn …

Amongst other factors, such measures can be seen as important to the debates about the quality and impact of higher education, how we evidence the value of investment in it, and how we evidence students’ skills acquisition for employers.” RAND report: McGrath et al 2015, p. xi


CoronyEdwards

The need to measure impact

Rich, 2015. pp 3 - 4

“… if universities are to command public investment, then a public good has to be served and observed.”

“… what has been lacking is any measure of the actual value added by universities to students and to the economy.”

“Without equations to demonstrate impact, it is hard to measure the public good and, in an austere world driven by econometrics, what is hard to measure is hard to fund.”


CoronyEdwards

Transparency for studentsThe desire to improve career prospects is the primary reason students cite for going to university (Kandiko Howson, 2012)

‘Many higher education applicants … imagine that the relationship between a course of study and a career is mechanistic, that higher education is about direct preparation or training for a job’ Rich, 2015 p. 18

“Higher education cannot reasonably offer [employment] guarantees to students. We need to shift the way we frame the student experience, to be more transparent about what they will really gain from it.” Rich, 2015 p. 18


CoronyEdwards

Learning Gain and the Teaching Excellence Framework• Good teaching seen as a means to increasing graduates’ earning

potential, and also their overall contribution to the economy and to society (the public good) (BIS 2015)

• Proposed TEF2 will provide ‘better understanding of the range of skills and knowledge [students] bring from their course’ (BIS 2015 p. 21)

• BIS envisage incorporating evidence of learning gain as part of TEF2 (BIS 2016 pp. 11, 12, 16 and 30)

Department for Business, Innovation and Skills, May 2016


CoronyEdwards

RAND report 2015: 5 main groups of measure• Standardised tests: measure the skills students acquire; generic or specialised

• Grades: measure the progress in students’ achievement by comparing the difference between grades at selected points in time

• Self-reporting surveys: ask students to report the extent to which they think they have gained knowledge and developed skills; run at a number of points throughout a degree programme

• Other qualitative methods: e.g. PDPs; encourage students to reflect on their learning, acquired skills and skills gaps; stimulate productive discussions between students and their tutors

• Mixed methods: draw on a range of tools and indicators to track improvement in performance, (e.g. through a combination of grades, student learning data and student surveys); can be used to predict students’ performance. See Wabash study for a comprehensive example https://www.youtube.com/watch?v=N4OKKsW-YdQ

https://www.youtube.com/watch?v=N4OKKsW-YdQ

https://www.youtube.com/watch?v=N4OKKsW-YdQ


CoronyEdwards

RAND report 2015: types of LG measure• Proxies, e.g. engagement, experience ratings, employment (including

HMRC data) ‘could prove efficient since they exploit existing metrics’ but validity is questionable

• Direct measures include ‘before’ and ‘after’ grades; standardised tests (generic & discipline specific); student self-report surveys (again, validity concerns)• e.g. UKES pilot included 12 items on students’ perceived skills development;

NSS contains one question on personal development


CoronyEdwards

Unintended consequences and caveats• Gaming by institutions

• Discouraging learning itself

• Quality is relative and must take into account the diversity of missions and goals in HE

• Quality of learning is more than just learning gain, and quality of HE is more than just quality learning

• Skills and competencies may not be comparable across disciplines• Contextual and demographic characteristics are a factor, but difficult to determine which

• Learning gain should not be the sole determinant of the quality of HE (cf TEF proposals).

RAND report: McGrath et al 2015 pp. 11, 12


CoronyEdwards

Challenges of developing LG measures which…

• Have a clearly stated purpose• Identify the dimensions of analysis (personal development, generic skills etc)• Consider complexity of LG and diversity of students, HEIs, state mission etc• Adopt methods which have comparable inputs and outputs• Are practical, can be implemented effectively and are cost-effective• Remain aware of unintended consequences, for example, need to prevent ‘gaming’• Achieve buy-in from HEIs and other stakeholders• Acknowledge the limits of the measures• Validate methods, for example by controlling for factors which could impact on

validity, such as sociodemographic or neighbourhood effects


CoronyEdwards

UEL/Roehampton/BrunelFocus on academic skills and engagement with learningSame measures being trialled across all 3 HEIs

Academic Behavioural Confidence (ABC) scale to test students; also to predict grades and where students grades will lie in relation to cohort“A psychometric means of assessing the confidence that undergraduate university students have in their own anticipated study behaviours in relation to their degree programme” (Sander and Sanders 2009:19)

Where are digital capabilities?


CoronyEdwards

UEL/Roehampton/Brunel• A ‘Need for Cognition’

scale (drive to learn) may also be used

• May also use NSSE or UKES in future• Also looking at developing

a new tool to measure employability confidence


CoronyEdwards

Lincoln https://www.adc.uk.com/assessment/products/dilemmas-situational-judgement-test/graduate/

Where are digital capabilities?

https://www.adc.uk.com/assessment/products/dilemmas-situational-judgement-test/graduate/

https://www.adc.uk.com/assessment/products/dilemmas-situational-judgement-test/graduate/


CoronyEdwards

Russell Group consortium example 1Career planning / readiness• Using existing methodology

developed by Bob Gilworth (Leeds)

• Formative tool embedded in annual registration process

• Students pick one of 10 statements which they feel most closely reflects their current career thinking (3 x ‘Decide’, 3 x ‘Plan’, 3 x ‘Compete’ and 1 ‘sorted’)

http://careerweb.leeds.ac.uk/info/13/career_planning




CoronyEdwards

Russell Group consortium example 2Graduate attributes

• ‘Realise2’ questionnaire

• Blue chip organisations (eg Ernst & Young) use this as part of their graduate recruitment process to test for a set range of graduate attributes (strengths)

“Through 180 questions, 60 strengths are accurately assessed according to the three dimensions of Energy, Performance and Use – with each user receiving their unique, personalised Profile, revealing their realised strengths, learned behaviours, weaknesses and unrealised strengths.”http://www.cappeu.com/R2StrengthsProfiler

http://www.cappeu.com/R2StrengthsProfiler


CoronyEdwards

Discussion1) Is your HEI using or developing measures of learning gain or learner analytics? If

so, briefly tell others in your group what is being measured and how2) How are, or could, learning (or other) technologies be used to support the robust

measurement of learning gain? Consider:a) Student engagement data sources: already recorded / available but not

recordedb) Administration of tests: existing and new

3) Which other professional and academic departments will you need to collaborate with in order to do this?

4) Are you ready for it?


CoronyEdwards

Thank you for your participation!

If you would like a longer version of this presentation, please contact me via my website


CoronyEdwards

References• Britton, J. et al (2016) How English domiciled graduate earnings vary with gender, institution attended, subject and socio-economic

background, IFS Working Paper W16/06. Nuffield Foundation.

• Cacioppo, J. T., & Petty, R. E. (1984). The need for cognition: Relationships to attitudinal processes. In R. P. McGlynn, J. E. Maddux, C. Stoltenberg, & J. H. Harvey (Eds.), Social perception in clinical and counseling psychology. Lubbock, Texas Tech University Press. [see also http://www.liberalarts.wabash.edu/ncs/ for overview article]

• Collini, S. (2012) What are universities for? Penguin, London

• Fulfilling our Potential: Teaching Excellence, Social Mobility and Student Choice. Department for Business, Innovation & Skills, 2015 https://www.gov.uk/government/consultations/higher-education-teaching-excellence-social-mobility-and-student-choice

• Greatbatch, D & Holland, J. (2016) Teaching Quality in Higher Education: Literature Review and Qualitative Research May 2016 Department for Business, Innovation & Skills https://www.gov.uk/government/publications/teaching-quality-in-higher-education-literature-review-and-qualitative-research

• Teaching Excellence Framework Technical Consultation for Year Two. Department for Business, Innovation & Skills, 2015 https://www.gov.uk/government/consultations/teaching-excellence-framework-year-2-technical-consultation

• Kandiko Howson, C. (2012) Student expectations and perceptions of higher education. Kings College London and Quality Assurance Agency

http://www.liberalarts.wabash.edu/ncs/

http://www.liberalarts.wabash.edu/ncs/

https://www.gov.uk/government/consultations/higher-education-teaching-excellence-social-mobility-and-student-choice

https://www.gov.uk/government/consultations/higher-education-teaching-excellence-social-mobility-and-student-choice

https://www.gov.uk/government/publications/teaching-quality-in-higher-education-literature-review-and-qualitative-research



https://www.gov.uk/government/consultations/teaching-excellence-framework-year-2-technical-consultation

https://www.gov.uk/government/consultations/teaching-excellence-framework-year-2-technical-consultation


CoronyEdwards

References contd.• Learning Gain: HEFCE Policy Guide http://www.hefce.ac.uk/lt/lg/

• McGrath, C. H. et al. (2015) Learning gain in higher education. RAND Corporation, Santa Monica, CA http://www.rand.org/pubs/research_reports/RR996.html (The HEFCE ‘RAND report’)

• Meltzer, D. (2002) ‘The relationship between mathematics preparation and conceptual learning gains in physics: A possible “hidden variable” in diagnostic pretest scores’. American Journal of Physics. 70:12, 1259-68

• Morrison, B, (2005) Evaluating learning gain in a self-access language learning centre Language Teaching Research. 9:3 267-293

• Rankin, E. & Dale, L. (1969) ‘Cloze Residual Gain - A Technique for Measuring Learning Through Reading’ in Schick, G. & May, M. Eds. The Psychology of Reading Behavior; Eighteenth Yearbook of the National Reading Conference. National Reading Conference, Milwaukee

• Rich, J. (2015) Employability: degrees of value. HEPI Occasional Paper 12. Higher Education Policy Institute, Oxford http://www.hepi.ac.uk/2015/12/10/employability-degrees-value/

• Sander, P & Sanders, L. (2009) Measuring academic behavioural confidence: the ABC scale revisited. Studies in Higher Education, 34:1, 19-35

• Thorndike, E. L.; Bregman, E. O.; Tilton, J.; Woodyard, E. (1928) Adult Learning. Macmillan, Oxford (Abstract at http://psycnet.apa.org/index.cfm?fa=search.displayRecord&uid=1929-00108-000)

http://www.hefce.ac.uk/lt/lg/

http://www.rand.org/pubs/research_reports/RR996.html

http://www.hepi.ac.uk/2015/12/10/employability-degrees-value/

http://psycnet.apa.org/index.cfm?fa=search.displayRecord&uid=1929-00108-000