A case study of one institution’s approach to institutional research

Penny JonesElizabeth MaddisonUniversity of Brighton

Preliminaries: definition and purpose

‘Self-study is about collective reflective practice carried out by a university with the intention of understanding better and improving its own progress towards its objectives, enhancing its institutional effectiveness, and both responding to and influencing positively the contact in which it is operating. As such, self-study is intimately linked to university strategy, culture and decision-making – with an emphasis on each of the collective, reflective and practical components of this definition’

From ‘Managing Institutional Self-Study’ by David Watson and Elizabeth Maddison, 2005

University of Brighton

>21,000 students; >2,000 staff; >£135m turnover

>5,500 awards 2007 submitted 287 staff in 16 RAE units of

assessment highly distributed (five sites; UCH; four partner

colleges) joint medical school (first graduates July 2008) major funding from HEFCE, TDA and NHS

University context

national debate on and requirements for accountability

HEFCE, TDA, NHS, PSBs etc Better Regulation ‘single conversation’ CUC Pls guidance

the Accountable Institution Project(HEFCE-funded; 3 universities)

University context

1999 no real analytic capacity problematic HESES return 2000 first data analyst appointed on fixed term

contract 2008 two permanent data analyst posts plus one

part-time survey post about to be filled continuous improvement in data quality 2008 clean data audit from HEFCE

University context

2007 ‘basket of indicators’ approved by Board of Governors as basis for their own monitoring of institutional performance against Corporate Plan and reporting for HEFCE

significant time series including student retention; surveys of student finance; why chose Brighton / decliners

targets for Faculties (e.g. research grants bid and won; research student completions; commercial income)

Critical success factors in IR at Brighton senior management commitment; SU involvement data quality improvement and sustained effort real examples where data is informing practice and

decision-making, and / or identifying questions to be addressed

feeding in at key moments (e.g. ‘what we know about what students think’)

expectation that Heads know the ‘facts’ about their Schools; will investigate / challenge / respond / change practice

Using a data framework in an effective wayFrom ‘Managing Institutional Self-Study’ by David Watson and Elizabeth Maddison, 2005.

integrate the data cycle with the committee cycle, including Board of Governors

focus on Brighton’s objectives and practices focus on performance indicators identified in corporate plan and

assessing them in appropriate ways keep it well organised and managed to fulfil internal and external

requirements ensure it supports risk management

The data framework at the University of Brighton

Challenges

timeliness of analysis data quality – and understanding when/where data does

not have to be perfect balancing analysis for information only with analysis to

support and/or challenge decision making to improve the quality of analysis over time, and with

changing requirements data literacy – communicating analysis using different

modes to provide appropriate access to different users

1. The Retention Report – an example of analysis well integrated into university cycles

SO

N

D

JFM

A

MJ

JA

Student Cycle

Analysis Cycle

Committee CycleRegistration

HESES Return07/08

RETENTION REPORT – Student cohort 06/07

HESA return06/07

HESAPerformance

indicators

StudentRetention Review Group

SeniorManagement

Team

Academic StandardsCommittee

Board of

Governors

Budget agreed for retention issues

HESA return07/08

Addressing data literacy

• Report on the web• Hard copy of the report sent out to key customers• Lunch time seminar tailored to attendees• An offer of one to one sessions with analyst

Withdrawals survey

2. The National Students Survey – using incomplete data and other challenges

results published at JACS subject level do not map to internal schools and faculties.

data only published at ‘department‘ level if threshold of 10 or more met.

an example of the complexity…

The complexity

BA Hons Social Science (30)

BA Hons Criminology and Sociology (47)

BA Hons Criminology and Social Policy (18)

BA Hons Health and Social Care (13)

BA Hons Sociology and Social Policy (11)

BA Hons Criminology and Applied Psychology (77)

BA Hons Applied Psychology and Sociology (36)

BA Hons English and Sociology (22)

Sociology (116)

Social Policy (192)

Others in SubjectsAllied to Medicine

(74)

Psychology (113)

English Studies (54)

SCHOOL A -‘departments’ (with number of respondents)

JACS Level 3

SCHOOL C

SCHOOL D

SCHOOL E

SCHOOL F

SCHOOL B

Unidentified Respondents

from departments

> 10 respondents

The NSS – the challenge continued…

difficult to ask academics to be accountable for data where we are unsure who the respondents making up the data are

why it matters… Unistats website resolution this year – NSS willing to provide JACS

mapping to make unpicking the results easier. increase response rates – more data at a lower level good example of difficulty in balancing analysis for info

only and for challenge

3. The ‘dashboard’ – improving analysis over time

new corporate plan 2007-2012 opportunity to improve high level analysis

provided to senior management and Board of Governors

undertook comparator group analysis and researched dashboard techniques

resulting UoB Dashboard the challenges

TensionsTension to be managed Desirable feature to be realised

Internal versus external drivers Self-study needs to be ‘an integral part of the normal process of governing and managing the institution by the governing body and the other bodies within the university responsible for managing its academic and administrative affairs’ (CUC 2002: 9)

Evaluation aimed at audit/ assessment and that aimed at reflection/ learning

Self-study should focus on reflection and learning. Otherwise, there is a danger of distorting behaviour and/or doing it only to satisfy external audiences who may have different objectives and values. It must be ‘bottom up’ and must engage as many staff as possible; universities also need to engage in external discussion about the nature of evidence and external requirements, in order not only to understand these but to help shape them appropriately

Perfecting ‘technical’ measures and systems at the expense of ‘political’ sensitivity and staff engagement

Universities need tools that are intelligible and staff to use them intelligently, as well as an appropriate specification for the task. However, there are weaknesses in some of the tools currently available. Universities need to develop their own toolkits to meet their own purposes in the light of their own objectives

Manageability and desirability Recognition that systems and people have limitations: perfect rationality is not the objective. Resources for self-study need to be proportionate and driven by institutional needs. Supporting staff - to ‘do’ self-study, understand the outputs or act upon the knowledge gained - is crucial. Clear responsibility for each of these aspects is essential

From ‘Managing Institutional Self-Study’ by David Watson and Elizabeth Maddison, 2005

Still to do herd the plethora of people involved in data analysis

and evaluation (practitioners and academics; quantitative and qualitative)

bring together data to give complete perspective on each School (e.g. NSS; clearing %; Retention; student and staff data; student complaints / appeals)

clearer processes and timetable (revisiting data cycle and framework)

reduce reinvention review external frameworks (e.g. CSR) align/dialogue between ‘IR’ and academic HE

research interests improve level of analysis (school; course; subject)

Still to do agree definitions (research; ‘third stream’) continuous attention to data quality and for

collecting, using and reporting on data inter-institutional comparisons contribute to national debate (e.g. metrics for

community engagement) technical capacity market intelligence is ‘good enough’ ‘good enough’? continuous attention to ‘so what’? avoid spurious veracity