utilising benchmarking to inform decision-making at the …aair.org.au/app/webroot/media/pdf/aair...

Offic

e o

f th

e D

VC

(S

&E

)

Utilising benchmarking to

inform decision-making at

the institutional level: A

research informed process

Dr Sara Booth

Student Evaluation, Review and Reporting Unit (SERRU)

http://www.utas.edu.au/student-evaluation-review-and-reporting-

unit/

http://www.utas.edu.au/student-evaluation-review-and-reporting-unit/












Presentation Overview

Types of benchmarking

International trends in benchmarking

2009-2012 UTAS benchmarking activities

Lessons learnt



Benchmarking Definitions

Benchmark: surveying to denote a mark on a survey peg but recently

acquired a more general meaning as a reference or criterion against

which something can be measured (Jackson, 2011)

Benchmarking ‘first and foremost, [is] a learning process structured so

as to enable those engaging in the process to compare their

services/activities/products in order to identify their comparative

strengths and weaknesses as a basis for self improvement and/or self

regulation’ (Jackson & Lund, 2000).

Types: implicit/explicit; independent/collaborative; internal/external;

vertical/horizontal; inputs/process/outputs; quantitative/qualitative, self-

referencing against standards or expectations (Jackson, 2011).

Managerial, pragmatic tool research process to inform

decision making


US in early 1990s: defining benchmarks and benchmarking surveys (Epper,1999)

UK early to mid 1990s: introduction of benchmarking

UK 2000 : subject benchmarking and external examiner process (Epper, 1999)

Australia 2000: Benchmarking: A manual for Australian Universities (Mackinnon,

Walker & Davis, 2000)

Europe 2000 onwards: setting performance targets –move from numerical data to

a focus on university processes (Swahn, 2004)

Australia 2007-2008: ACODE & Australian Quality Assurance Agency (AUQA)

Cycle 2 recommendation: What explicit benchmarking has there been to compare

standards?

2010 onwards: benchmarking associated with league tables (Burquel & van Vught,

2010)

International Benchmarking Trends


Desktop-review of Cycle 2 AUQA audits (Booth, 2011)

Types of benchmarking proposed by AUQA auditors;

Benchmarking data (e.g. CEQ, GDS, ISB, student load, research performance,

international services, finance, equity etc.)

Standards-based benchmarking

Sector benchmarking

Whole-of-institution benchmarking

Discipline-specific benchmarking

3 stages of benchmarking development:

1. Early implementation: Urgently consider the development and implementation of a

benchmarking framework; processes and partnerships

2. Further refinement and alignment: Beginning to develop benchmarking processes and

partnerships; but further refinement is required

3. Establishment: Established benchmarking frameworks, processes and partnerships

and extensive use of external reference points and benchmarking

Benchmarking in Australia


2009-2012 UTAS: 7 benchmarking activities:

1. Sector Benchmarking

2. Process and Academic Standards Benchmarking

3. Information Benchmarking

4. Process and Outcomes-based Benchmarking

5. Standards Benchmarking

6. Discipline-Specific Benchmarking

7. International Institutional Research Benchmarking

UTAS Benchmarking Activities


2009- Pilot project with UOW- Academic Transition Support:

AUQA Cycle 1 Audit recommendation

UTAS in 2008: 22, 600 students; 3 regional campuses, faculty structure and

discipline

UOW in 2008: 24,413 students; 5 regional campus, faculty structure and

discipline areas

Sector Benchmarking: occurs when benchmarking partners in the same sector

make comparisons either as a whole organisation or an aspect of the organisation

(Stella & Woodhouse, 2007)

Aims of Pilot Project:

Develop knowledge and experience in the benchmarking process

Compare current processes and practices

Identify areas for improvement and areas of best practice

Benchmarking Activity 1: Sector

Benchmarking


Benchmarking methodology based on ACODE (2007)

Performance Indicator: identify key performance areas that would

indicate realisation of good practice (ACODE, 2007, p.6)

10 Performance indicators (PI):

1. Aligned plans and policies are in place and implemented

2. Planning and delivery of programs/activities are coordinated

3. Programs/activities are informed by recognised pedagogical principles

4. Processes are in place and used to support students at their point of need

5. Programs/activities are promoted to, accessible to and used by students

6. Programs/activities meet student needs

7. Programs/activities are resourced

8. Professional development and support is available to staff, accessed and informs practice

9. The effectiveness of programs/activities is monitored and evaluated

10. Evaluation of feedback and results is integrated into planning for continuous improvement

purposes

Benchmarking Activity 1: Performance

Indicators


Performance Measures: statements that represent progress toward good practice

(ACODE, 2007, p.6)

5 point scale

1. Not at all

2. Limited

3. Moderate

4. Considerable

5. Comprehensive

Benchmarked AUSSE (2008) survey

Self-review Faculty/Centre workshops

Institutional self-review workshop to validate at the institutional level

Peer-review workshops

Benchmarking Activity 1: Performance

Measures


UTAS:

UTAS is comparable in performance standards with UOW on academic transition support-many

examples of good practice

11 areas for improvement (UTAS) (resourcing/staff development)

UTAS rated extremely well when comparing AUSSE results (2008)

Staff contracts in PASS were short-term (UTAS)

UTAS and UOW:

Increased understanding of benchmarking

Lack of evidence

Challenged with supporting equity groups such as students with English as a second language;

students with learning disabilities

Collaborative Actions:

Define role for first-year coordinators; develop a first year transition framework; develop administrative processes

for at risk students; comparison of library programs; professional development on first year transition; comparison

of International Services

Benchmarking Activity 1: Findings


Assessment Policies and Processes (Jan 2010-Nov 2010):

UTAS, UOW and Deakin

Reference Points: ACODE and ALTC Project Teaching Quality Indicators Project (Davies,

2008)

3 Performance Indicators:

Assessment purposes, processes and expected standards of performance are clearly

communicated and supported by timely advice and feedback to students

Assessment practices and processes are fair, reliable and valid and produce marks and

grades that represent the standards achieved by students

Assessment policies and procedures are developed, implemented, reviewed and

improved in accord with policy principles

Performance Measures:

Ratings changed to encourage conversation between staff – Yes; Yes, but; No, but; No

Evidence was necessary to support ratings

Benchmarking Activity 2: Process and

Academic Standards Benchmarking


UTAS, UOW and Deakin used different methodologies

UTAS’ chief research instrument was an online survey to staff (n=336) which used the benchmarking

framework but rephrased into questions for more meaningful responses

7 postgraduate students and 1 Research Fellow analysed survey data and validated findings with

each Faculty through interviews

Self-review process by Associate Deans (Learning & Teaching) at institutional level

Benchmarking of Assessment processes with Academic Senate Chairs

UTAS Findings

UTAS demonstrated that the implementation of criterion-reference assessment (CRA) was critical in

aligning learning outcomes, graduate attributes and assisted in demonstrating student achievement

standards

Associate Deans (L&T) lacked time and resources to drive quality improvement

4 factors affected the timely return of feedback- working in an online environment, moderation, large

student cohorts, multi-campus sites

27 recommendations – Assessment Benchmarking Working Party

Collaboration

Online subject outline templates; best practice case studies on group work; online training manual for

assessment; UTAS Graduate Attributes Project; Deakin Faculty Learning and Teaching Funding Model;

Postgraduate Research

Benchmarking Activity 2: Benchmarking

Methodology & Findings


Institutional information benchmarking exercise (Nov 2010- Jan 2011):

Inform development of Benchmarking Policy and Procedure

Survey of UTAS staff perceptions of benchmarking

Findings:

Clarify distinction between benchmarks and benchmarking

Library, ITR and Support Services strong in benchmarking (functional)

Benchmarking does not include membership on committees, staff born

overseas, visiting scholars, conferences, informal conversations, networking

groups

Information benchmarking informed policy and procedure;

Data comparison process and more investigative process

Benchmarking Activity 3: Information


Benchmarking Project on HDR programs (Mar 2011- Mar 2012):

UOW, Deakin, UTAS

Performance Indicators:

Policy and Governance

Selection and Admission Processes

Student Learning Outcomes

Supervision

Examination Processes

Academic Support-including mentoring and career placement

Non-academic support and resourcing

Monitoring student performance and feedback data

Data comparison:

Student load, post-graduate coursework, HDR EFTSL, staff FTE

Activity 4: Process & Outcomes-based

Examples of good practice- All

Comprehensive policy; multiple modes of communication of policies

and procedures to staff; orientation and induction processes; annual

review processes and examination processes

Examples of areas for improvement-All

More systematic use of data for quality improvement; monitoring

English language entry requirements; clearer articulation of learning

outcomes aligned to AQF; more systematic English language

support; HDR student career development

Collaboration

Fast track supervision program (Deakin); Head of Postgraduate

Studies (UOW) and GRC (UTAS) position descriptions; Brazilian

Consortium to build international HDR collaborations

External Peer-Review: Edith Cowan- ALTC HDR BPF

Activity 4: Process & Outcomes-based


National Teaching Standards Framework (2012):

Led by Macquarie- Phase 2 of project in testing online version of the TSF (Sachs &

Kosman, 2011) ; 12 institutions

Complete self-review- institutional methodology; validity of TSF

Bachelor of Education (Primary); Faculty of Arts (first year)

UTAS framed performance indicators as questions

Learnt software methodology for benchmarking/standards

ALTC Inter-University Peer Review Moderation Project (2012):

Led by UWS (Krause et al, 2010)

UOW, UTAS, Deakin – chosen for strong benchmarking relationship

1st semester, 2012

Economics, Journalism, History and Nursing

Project ongoing- identification of unit level moderation processes for benchmarking

Activities 5 & 6: Standards & Discipline-

Specific Benchmarking


HEA Benchmarking Project on Promotions Policies and Processes

(2012-2013):

HEA-funded research project

Leicester, Newcastle upon Tyne (UK); UOW and UTAS (Aust)

Aim:

To produce resources to guide and improve academic promotion policies and

processes; External evaluator, HEA representative; surveys/interviews; promotions

data

6 Performance Indicators: Planning and Policy; Leadership and Culture; Decision-

Making Structures and Processes; Application Procedures; Training and Support;

Outcomes, Evaluation and Review

Self-Review; Peer-Review Workshop

Dissemination;

Satellite Event- Universities Australia Higher Education Conference 2013

Workshop- UK (TBC)

Research outcomes focused

Activity 7: International Institutional

Research Benchmarking

Lesson 1: Benchmarking has to be translated into institutional research

Identify issue(s) benchmarking is trying to solve (Longden & Yorke, 2009)

Matrix used in institutional research adapted for benchmarking

Key Lessons Learnt


Direction of Focus

Internal Formative

External Summative

Administrative Staff/ Managers

1. Information

- Benchmarking of Data only

2. Presentation

- Sector Benchmarking - Ranking

Faculty Senior Executives

3. Research for

Improvement

- Standards Benchmarking - Projects informed by research

4. Educational

Research - Standards Benchmarking - Projects informed by research and validated by external reference groups

Org

an

isa

tio

na

l R

ole

s

(Longden and Yorke, 2009; adapted from Volkewein 1999); Booth (2012)

Lesson 2: Benchmarking has to become the way things get done at universities

(Hossler, Kuh & Olsen, 2001, p. 212)

Benchmarking is justified by driving operational outcomes that improve processes

Institutional process for organisational improvement and resourcing

UTAS:

Reflected in Strategic and Faculty Plans

Establishment of Student Evaluation, Review and Reporting Unit (SERRU)


Key Lessons Learnt

Lesson 3: A rigorous methodological and theoretical approach is essential

9 Phases in Benchmarking Methodology

1. Determine areas to benchmark

2. Identify benchmarking partners

3. Determine type, and level, of benchmarking

4. Prepare benchmarking framework and documentation

5. Design benchmarking process

6. Implement process

7. Review results

8. Communicate and report results and recommendations

9. Implementation of improvement strategies (Booth, Melano, Sainsbury & Woodley, 2011)

Develop theoretical framework from literature (for e.g. Assessment, HEA

Promotions Project)

A lack of a theoretical framework for benchmarking distinguishes effective from ineffective efforts

(Moriarity & Smallman, 2009)


Key Lessons Learnt

Lesson 4: Benchmarking becomes a Community of Practice

To be successful there has to be bottom-up empowerment (Ellis & Moore, 2006)

Collaboration and openness (Sciulli, Smith & Ross, 2009)

Shared conversation and a form of peer development (Leppisaari et al, 2011)


Key Lessons Learnt

Australian Council on Open, Distance and e-Learning (ACODE) (2007) ACODE benchmarking for e-learning in universities and

guidelines for use.

Booth, S. (2011) Benchmarking policy development. Internal policy paper. University of Tasmania.

Booth, S., Melano, A., Sainsbury, H., Woodley, L. (2011) Articulating and comparing standards through benchmarking of assessment.

Paper presented at the Australian Universities Quality Forum: Demonstrating Quality Conference- 29 June- 1 July 2011, Melbourne.

Burquel, N., & van Vught, F. (2010) Benchmarking in European Higher Education: A step beyond current quality models. Tertiary

Education and Management, 16 (3), pp. 243-255.

Davies, L. (2009) Assessment reform & the quality context: Tensions & synergies. Refereed Conference Paper AUQF 2009 Internal

& External Quality Assurance: Tensions & Synergies. 1-3 July 2009, Alice Springs, Australia.

Ellis, R.A., & Moore, R.R. (2006) Learning through benchmarking: Developing a relational, prospective approach to benchmarking

ICT in learning and teaching. Higher Education, 51, pp. 351-371.

Epper, R.M. (1999) Applying benchmarking to higher education: Some lessons from experience. Change: The Magazine of Higher

Learning, 31(6), pp. 24-31.

Hossler, D., Kuh, G.D., & Olsen, D. (2001) Finding fruit on the vines: Using higher education research and institutional research to

guide institutional policies and strategies. Research in Higher Education, 42 (2), pp. 211-221.

Jackson, N. (2001) Benchmarking in UK HE: an overview. Quality Assurance in Education, 9(4), pp. 218-235.

References


Jackson, N. & Lund, H. (eds.) (2000) Benchmarking for higher education, Society for Research into Higher Education & Open

University Press, UK.

Krause, K-L., Scott, G., Alexander, H., Campbell, S., Carroll, M., Deane, L., Nulty, D., Pattison, P., Probert, B., Sachs, J., & Vaughn,

S. OLT (ALTC) Project (2010) Inter-university peer review and moderation of coursework project.

Leppisaari, I., Vainio, L., Herrington, J., & Im, Y. (2011) International e-benchmarking: flexible peer development of authentic learning

principles in higher education. Educational Media International, 48(3), pp. 179-191.

Longden, B., & Yorke, M. (2009) Institutional research. Perspectives: Policy and Practice in Higher Education, 13 (3), pp. 66-70.

McKinnon, K.R., Walker, S.H., & Davis, D. (2000) Benchmarking: A manual for Australian universities. Department of Education

Training and Youth Affairs, (DETYA) Higher Education Division

Moriarity, J.P., & Smallman, C. (2009) En route to a theory of benchmarking. Benchmarking: An International Journal, 16 (4), pp.

484-503.

Sachs, J., & Kosman, B. (2012) Teaching standards project, online development and pilot project: Final report. OLT (ALTC Project),

Macquarie University.

Stella, A., & Woodhouse, D. (2007) Benchmarking in Australian higher education: A thematic analysis of AUQA audit reports.

Melbourne, VIC: Australian Universities Quality Agency.

Yorke, M. (2004) Institutional research and its relevance to the performance of higher education institutions. Journal of Higher

Education Policy and Management, 26 (2), pp. 141-152.


References

utilising benchmarking to inform decision-making at the …aair.org.au/app/webroot/media/pdf/aair...

Documents