what can we know (and claim to have learned) from the...

What can we know and learn from the R&D literature in higher education? Dai Hounsell Aarhus University, 4 April 2013

What Can We Know (and Claim to Have Learned)

from the Higher Education R&D Literature?

Dai Hounsell

University of Edinburgh

EXPERIENCES AND REFLECTIONS ON REVIEWING STUDIES OF WRITTEN FEEDBACK TO STUDENTS


FUNDAMENTALS OF FEEDBACK��What forms does / can feedback take?

pro forma written comments exemplars exams guidance feedforward traditional collaboration on-display learning peer audio past questions screencast whole-class clickers in-class assignments cumulative editing using feedback well elective self co-revision e-feedback redrafting reviewing progress criteria dialogue supervision interaction new briefing involvement faster feedback model answers training video online


Experiences and reflections on

reviewing studies of written feedback to students


SOURCES OF TENSION


CONTINGENCY & GENERALISATION IN REPORTING FINDINGS: An illustration

1.  The majority of students are satisfied with feedback

Krause, K., et al. (2005). The First Year Experience in Australian Universities: Findings from a Decade of National Studies. Melbourne: University of Melbourne, CSHE




2.  A minority of students find feedback unsatisfactory






3.  Most of the students were satisfied with the feedback they got 4. Two out of five first-year students are/were dissatisfied with

the provision of feedback on their set work






3.  Most of the students were satisfied with the feedback they got

4. Two out of five first-year students are/were dissatisfied with the provision of feedback on their set work

5. Two out of five first-year Australian students, research has shown, are dissatisfied with the provision of feedback on their set work.

6.  Three successive, quinquennial, national surveys of first-year students in Australian universities have consistently found that two out of five of the students surveyed were dissatisfied with the provision of feedback on their set work.



INTRODUCTION & RATIONALE ��

Enhancement and innovation as contingent

“Part of the difficulty in developing a better understanding of the student experience is the assumption that research has to produce particular types of large-scale, generalisable finding. And yet research into learning ... increasingly points to the need to understand phenomena in context; to recognise that situations differ, and are specific, and that specific problems need particular answers.”

(Haggis, 2006, our emphasis added)

"The call for practice to be based on evidence has resonated round a wide range of professions. Sometimes the evidence base is 'hard', and robust generalisations are justified on the basis of adherence to strict experimental criteria. However, in higher education and in the social sciences generally, the evidence base is more contingent. Educators cannot simply apply evidence without taking into account the circumstances in which it was created and in which they operate: they have to make evidence-informed professional decisions regarding their practices."

(Yorke & Knight, 2007, our emphasis added)


SOURCES OF TENSION


GENRES, PURPOSES, AUDIENCES & CONTEXTS

“How these myriad forms of publications are viewed – and even more crucially, how they are valued – depends to a substantial extent on one's professional standpoint and the kind of use to which one intends to put the literature. In traditional educational research and scholarship, it is the systematic empirical study which has tended to be prized, and through it the pursuit of generalisation across individuals, levels of study, subject areas, settings and even cultures. Conversely, [...] practice-focused reports and reflections by teachers and other practitioners are likely to be judged of much lower merit, or even disregarded, on the grounds that they offer evidence that is anecdotal, they lack academic rigour, and they are restricted to very specific settings. To the mainstream subject teacher, however, this 'hierarchy of evidence' (Greenhalgh, 2001) can appear lopsided or inappropriate.”

(Hounsell et al., 2007, p.14)


Although systematic review can take various forms (see e.g. Davies, 2002), the predominant approach, meta-review, is derived from medicine, and foregrounds:

• inclusion/exclusion procedures based on research-focused criteria

• prioritises/excludes studies which do not conform

to a 'standard' methodology [in medicine, randomised controlled trials]

• focus on synthesis of quantitative outcomes through a common metric (e.g. effect size)

META-REVIEW & SYSTEMATIC REVIEW


THE GENRE DIMENSION

A Typology of Genres

Commentary/Opinion Piece

Compendium(of evolving practices)

Empirical Study

Enhancement Project

Guide to Professional

Practice

Guidelines

Review of the

Literature

Account of Practice

Theory/Conceptualisation

EvaluationPolicy

Document


THE GENRE DIMENSION ��(from Hounsell et al., 2007)

Research-oriented literature

empirical studies

theory/conceptualisation

enhancement projects

reviews of the literature

evaluations

Practice-focused literature

accounts of practice

guidelines

guides to professional practice

compendia of evolving practices

commentary/opinion pieces


SOURCES OF TENSION


FOCUS OF THE META-REVIEW

v  written feedback comments to students in higher education on their assignments and assessments

v  includes both ‘in-text’ (annotations/marginal/on-script) comments and ‘overall’ (summary/cover-sheet) comments

v  excludes analyses of comments from studies lacking an authentic disciplinary focus and not grounded in a real course setting

Ø  …. the snares of the universalising tendency … and a contrary approach that is, by intention, alert to contingency


AIMS OF META-REVIEW

a.  identify the corpus of salient studies b.  develop inclusion criteria which specify minimum

requirements for the conduct and reporting of an empirical study

c.  review the empirical studies which meet those threshold requirements …. and make reference where valuable [and with caveats) to those which do not

d.  relate the findings to : –  what is known about how students perceive, engage with

and make use of feedback comments –  the guidance literature on effective commenting

see next slide


ESTABLISHING A CORPUS OF RELEVANT STUDIES . . . and the ‘false idol’ of comprehensiveness

WHAT ARE THE OPTIONS ?

• comprehensive searches of major bibliographic databases [using search-strings of descriptors] and citation indexes

• meticulous search through high-relevance journals

• 'snowballing' – by following-up promising references whenever relevant items are

retrieved

– by going to authors' websites

– by looking further in journal websites where one or more promising items were located

• tapping into personal/professional networks


TYPES OF STUDIES


STUDIES BY LEVEL OF ��HIGHER EDUCATION

Undergraduate 50 +

Master’s 6 +

Doctoral 7 +

u The studies identified span all levels of higher education:

u  No single study identified thus far compares comments across levels of higher education


STUDIES BY DISCIPLINE / ��SUBJECT AREA

Ø  the studies identified extend across the subject range

Ø  (inevitably) dearth/absence of quantitative subjects such as maths & stats, and professional subjects with a significant clinical, studio or practicum dimension

Ø  relative over-representation of subjects and courses concerned with language and writing:

q  English as a second language (ESL)

q  English for Academic Purposes (EAP)

q  (in US studies) ‘rhetoric and composition’


Case Examples


CASE EXAMPLE 1: Feedback to Master’s Students��(Hyatt, 2005)

Subjects & institutions

Master’s programme in Education, UK university

Approach • analysis of feedback commentaries on 60 assignments, each of 6000 words

Coding of feedback

Adopts linguistics perspective Seven categories ~ a mix of content, function and procedural requirements

NOTES

Helpfully gives rich examples of each of the coding categories


CASE EXAMPLE 1: DISTRIBUTION OF CODING CATEGORIES

category gloss % Phatic establishing good relations 3

Developmental aiding student with subsequent work 24

Structural organisation of the work 9

Stylistic use of academic language 28

Content its accuracy & appropriateness 32

Methodological* research design or analysis 5 Administrative referring to course-related issues .2

* only in research-based assignments


CASE EXAMPLE 2: Doctoral Feedback (Bitchener et al. 2011)

Subjects and institutions

Humanities, sciences/maths, commerce in 6 NZ universities

Approach • questionnaires & interviews with 54 staff & 180 students • analysis of sample of ‘on-script’ feedback – 351 comments from 14 scripts.

Frame of reference

Linguistics : genres and academic discourse

NOTES

For the majority of these doctoral students, feedback comprised written comments on draft + face-to-face supervision meeting Iterative commenting on drafts was less typical in sciences/maths


CASE EXAMPLE 2: Doctoral feedback ~ FINDINGS

focus range per script (n=14)

Content 1 -12 Cohesion/coherence 0 - 7 Requirements 0 - 6 Linguistic accuracy & appropriateness [e.g given claims made]

0 - 33


CASE EXAMPLE 2: Doctoral feedback ~ FINDINGS

•  Role of PGR feedback in inducting doctoral students in to a disciplinary discourse community

•  Feedback tailored to the individual needs of students: “Such feedback takes into account a range of individual student characteristics (e.g. prior learning background; learning style and preference) and the stage that the thesis has reached”

•  Feedback strongly ‘feedforward’, with built-in check for supervisors on the effectiveness of their feedback:

"by the extent to which issues are addressed in the next draft” (p. 43)


CASE EXAMPLE 3: Feedback to Undergraduates��(Brown & Glover 2005/2006)

Subjects & institutions

Biological & physical sciences, Open & Sheffield Hallam Universities, UK

Approach • analysis of 5015 in-text + overall comments on 147 students’ assignments (112 at OU, 35 at SHU).

• interviews with SHU tutors

Coding of feedback

Coding tailor-made, from content analysis

Focused on analysis of type and purpose of feedback

NOTES

Two later follow-up studies using modified coding scheme: • of feedback comments in Technology (Walker) • of feedback comments in Languages (Fernandez-Toro, Truman & Walker, in press)

Important commenting differences between subjects emerged


CASE EXAMPLE 3: FINDINGS

•  The biggest category [60/70%] of the science tutors' comments at both universities was the science content. –  About half of this was 'omissions'-focused, with rather less attention

relatively to 'errors', requests for 'clarification', or 'irrelevance'.

•  Far smaller proportions were devoted to feedforward (skills and explicit links to future works or tasks), and to motivational comments (mostly praise — use of negative language was rare)

•  “Skills weaknesses were signalled but not how they might be corrected, and interventions designed to encourage further learning were comparatively rare.”

•  Few if any explanatory comments given, in all types of assignment


POSTSCRIPT: SUBJECT DIFFERENCES

WALKER (2009): 3095 COMMENTS ON 106 TECHNOLOGY ASSIGNMENTS

Comment analysis

• two-fifths of comments were on content, one-fifth on skills development, one-third motivating

• only 1 in 10 were of the explanation type

FERNANDEZ-TORO ET AL (2012): 4000 COMMENTS ON 72 SPANISH-LANGUAGE SCRIPTS

Comment analysis

– compared to Technology tutors, language tutors comments more concerned with skills development than content

– a lower proportion of corrections than in Technology, but a higher proportion for indicating errors and providing explanations


Analyses across the

Corpus


A DIVERSITY OF COMMENTING FUNCTIONS

recommending • correcting • explaining

instructing • suggesting • advising • criticising

questioning/probing • cautioning

praising & encouraging • alerting to • conversing

evaluating quality • justifying the mark • requesting


FEEDBACK FUNCTIONS: EMERGING CATEGORIES [an example of how my functions relate to others' categories]

•  correction factual, acknowledging a weakness, omission/missing pieces instruction, imperative, editing, revision, off task

•  evaluation

validation, confirmation, affirmation/negation, highlighting strengths/ calling attention to weakness,analysis, evaluation in relation to an ideal answer, general impression, overall assessment, holistic, overall judgment


FEEDBACK FUNCTIONS: EMERGING CATEGORIES


Shiftening boundaries,

widening ripples


... MORE TYPES OF STUDIES TO INCLUDE ...?


Unsettled/unsettling Questions


QUESTIONS YET TO BE CLEARLY RESOLVED

a.  Does level of study influence the kinds of comments made?

b.  (How) is volume of comments related to the grade awarded?

c.  Can less be more ?

d.  How do feedforward comments differ from post-hoc feedback ?

e.  How does verbal feedback differ from written feedback ?

f.  What evidence-informed guidance can be offered on giving effective written feedback to university students ?

what can we know (and claim to have learned) from the...

Documents