institutional collaboration in benchmarking and peer ... · institutional collaboration in...

51
Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision Dr Sara Booth University of Tasmania and Council of Private Higher Education (COPHE)

Upload: others

Post on 02-Aug-2020

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision

Dr Sara Booth University of Tasmania and

Council of Private Higher Education (COPHE)

Page 2: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

INSERT FACULTY NAME IN FOOTER 2

OVERVIEW

• Importance of context in quality enhancement and quality assurance

• New Zealand and the South Pacific higher education quality context

• Australian higher education quality context

• Definitions of benchmarking• Case study: University of Tasmania • 5 snapshots of institutional

benchmarking projects• Peer review of assessment • National support mechanism for peer

review of assessment • Peer review portal• What are the next steps for HE

institutions in Ireland to assure the quality of provision?

Page 3: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

3

Page 4: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Discussion paper (2016)Options for a sixth cycle of academic audit for New Zealand/South Pacific universities– 2015 External Review of AQA recommended AQA, consider, in

consultation with universities and other stakeholders, how cycle 6 might be more focussed

– Methodological shift in Cycle 5 was that evidence played a significant role in self reviews and … the willingness [opportunity] to be truly self critical could have been more fully developed

– Develop strategies for more specific benchmarking of actual practices and conduct workshops for universities in self evaluation

– AQA is not a regulator, places high levels of emphasis on peer review

– Role of Ako Aotearoa in quality enhancement projects

– Quality improvement and more importantly quality enhancement 4

Page 5: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Booth model [2016] [based on ACED Model]

5

Enhance Enhance Enhance Enhance Assure Assure Assure Assure

Audit preparation

Self-reviewby each institution

Identification of areas for improvement institutionally as well as sector challenges

External referencing and benchmarking with national/international comparators on these identified areas

Peer review and calibration workshop

Final Report with recommendations

Composite/Both assurance and enhancement activities may require different methods/timelines and could run on different timelines

Page 6: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

6

Page 7: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

National University of Samoa: Proposed Benchmarking Project, actions and responsibilities 2017

Key actions

Consider a Vice-Chancellors Consortium to identify strategic priorities in improving the quality of HE in the Pacific

Establish an international Pacific HE Quality Network for benchmarking and peer review of assessment• Establish database of network participants • Develop Framework for benchmarking best practice• Develop online database/clearinghouse of good practice• Secretariat support on a rolling process.eg. National University of Samoa to be first secretariat for

Network• Identification of strategic priorities for the Network e.g. benchmarking literacy and numeracy in

primary education

Establish a collaborative agreement between all participating benchmarking institutions

Liaise with funding agencies to fund this proof-of-concept project for the Pacific and New Zealand HE institutions

HE institutions consider ways of building institutional capacity and awareness in benchmarking and peer review of assessment to support programme review and accreditation, for e.g. a workshop on benchmarking and peer review of assessment

7

Page 8: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

8

• 179 higher education providers

43 universities

New Higher Education Standards Framework (HESF) (2015) Implementation January 1st, 2017 Massive change implications in terms of benchmarking, peer review of assessment and external referencing

Page 9: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Higher Education Standards Framework (HESF) (2015)

9

7 Domains Relevant Standards

Domain 1: Student Participation and Attainment

1.4.1 The expected learning outcomes for each course of study are specified, consistent with the level and field of education of the qualification awarded, and informed by national and international comparators.

Domain 5: InstitutionalQuality Assurance

5.3.1 All accredited courses of study are subject to periodic (at least every seven years) comprehensive reviews that are overseen by peak academic governance processes and include external referencing or other benchmarking activities.

5.3.2 A comprehensive review includes the design and content of each course of study, the expected learning outcomes, the methods for assessment of those outcomes, the extent of students’ achievement of learning outcomes, and also takes account of emerging developments in the field of education, modes of delivery, the changing needs of students and identified risks to the quality of the course of study.

5.3.4 Review and improvement activities include regular external referencing of the success of student cohorts against comparable courses of study, including:a) analyses of progression rates, attrition rates, completion times and rates and, where applicable, comparing different locations of delivery, andb) the assessment methods and grading of students’ achievement of learning outcomes for selected units of study within courses of study.

Page 10: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Definitions of benchmarking

10

Page 11: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Definitions

CURRICULUM & Quality 11

Benchmarking

… a structured, collaborative, learning process for comparing practices, processes or performance outcomes. Its purpose is to identify comparative strengths and weaknesses, as a basis for developing improvements in academic quality. Benchmarking can also be defined as a quality process used to evaluate performance by comparing institutional practices to sector good practice. (TEQSA, 2014)

Page 12: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Benchmarking (Booth, 2013)

CURRICULUM & QUALITY 12

Direction of Focus

Internal Formative

External Summative

Administrative Staff/ Managers

1. Information

- Benchmarking of Data only

2. Presentation

- Sector Benchmarking - Ranking

Faculty Senior Executives

3. Research for

Improvement

- Standards Benchmarking - Projects informed by research

4. Educational

Research - Standards Benchmarking - Projects informed by research and validated by external reference groups

Org

anis

atio

nal R

oles

Calibration (2015)

Page 13: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Case study: University of Tasmania

13

Page 14: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Institutional Context: University of Tasmania

CURRICULUM AND QUALITY 14

• Cycle 1 and Cycle 2 AUQA Audits • Benchmarking Strategy 2016-

2018.• Academic Quality and Standards

Committee approved three benchmarking priorities for 2016: • Governance of third party

arrangements;• Course Approval,

Accreditation and ReviewProcesses; and

• Technology EnhancedLearning and Teaching.

• Benchmarking Policy and Procedure

• Benchmarking Projects and Reports

• Online benchmarking tool • Contract Research

Page 15: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

UTAS: Benchmarking and peer review

15

2009 Academic transition support

Side of my deskCycle 1 AUQA audit

No evidenceData collection over email No policy No procedure No follow up on actions in Final Report

Event not a process

2 universities

2010Assessment Policies and Processes

Whole of institution

6 PhD Students to analyse the data

Evidence patchyGraduate attributes missing in Science

Bottom up approach

Final Report commended by Quality CommitteeRec 1: L&T Dashboard was implemented $750KActions followed up by faculties 3 universities

2011 Assessment Policies and Processescontinued

2012HEA Promotions Policies and Processes

Final ReportExemplar by HEA Report Teaching Resources Benchmarking Resources Teaching Performance Expectations 4 unis

Higher Degree Program

Final ReportInformed significant review of Graduate Research OfficeOnly 2 PhD students had been evaluatedSouth American partnership with UOW/Deakin 3 unis

2013 Maritime Engineering Capstone Units Internationalpartnership for benchmarking coursesEvaluation activities patchy UTAS Online benchmarking tool 5 unis Faculty of EducationAligned to accreditation 13 units literacy, numeracy early childhoodPracticums, professional studies 5 unis

Page 16: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

16

2013 University Department of Rural Health

Final Report Over 20 recommendations including research outcomes Based on partnership agreement and accreditation process2 unis

HDR Course Outcomes and EmployabilityBenchmarked PREQ survey Used DDOGS Framework as guideOLT project 5 unis

Pathway Partnerships Pathway processes and partnerships Partnership framework for UTAS 5 unis

2014Clinical Simulation, Alzheimer's, clinical placement, health literacy

Actions followed up by faculties Clinical Simulation Strategy Informed accreditation for nursing 2 unis

International Student Experience and Student Employability

Student employability survey ISB Informed restructure of DVC and GlobalEngagement 2 unis

2015International student employability and mobility Global Think Tank SummitFinal ReportIncreased emphasis on employers/links to local government Recommendations to Department of Education and Universities AustraliaResearch consultancy, HERDC points 10 unis, 4 countries

Ako Aotearoa Benchmarking Project on Teaching Quality, Student Success, Curriculum Quality, assessment, support for academic staffAko Aotearoa commissioned projectFinal Report and institutional outcomesInformed audit reports for New Zealand UniversitiesResearch consultancy, HERDC pointsSubmitted to the Productivity Commission 7 unis, 3 countries

2016 Quality-Curriculum Review and Third Party Arrangements

In progress

Strategy and Schedule for Institutional BenchmarkingConsultancy 5 unis

Council of Private Higher Education (COPHE) Student support Attrition,retention,Completion Student support processes QILT/Department of Education Sector Report/WorkshopConsultancy 31 private providers

Page 17: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

UTAS: Benchmarking and peer review

INSERT FACULTY NAME IN FOOTER 17

Problems that have arisen: Finding partners to benchmark with, aligning to other HE strategic initiatives Finding peer reviewers/reviewees Time it takes to sign MOU/Collaborative agreements Lack of evidence Screaming academics-How dare you do this to me? Don’t you know that I am the

best in Australia! Issues in change management Difficulty in benchmarking against comparators and finding similarities and

differences Security of data over email Sheer enormity of data over email Language and definitions in peer review and benchmarking Scheduling reviews Monitoring reviews, following up with academics that haven’t completed their

reviews Manually reporting on benchmarking and reviews Follow up actions and responsibilities Resourcing- one person Costs for organising a benchmarking project Seen as an administrative quality exercise, not recognised as research

Page 18: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Benchmarking Process

CURRICULUM AND QUALITY 18

Who is preparing the institutional context statements?

What data and self review information will be shared?

Where and when will the peer review take place?

How long will the peer review workshop take?

Who will coordinate the peer review? How will the benchmark partners

contribute to the workshop? What evaluation strategies will be used?

Page 19: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

5 snapshots: Benchmarking projects

19

Page 20: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Snapshot 1: HEA Benchmarking Project

CURRICULUM AND QUALITY 20

1. University of Wollongong, Aust2. University of Tasmania, Aust 3. University of Leicester, UK 4. Newcastle University, UK

Page 21: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Snapshot 2: Ako Aotearoa Benchmarking Project

21

1. Auckland University of Technology

2. Lincoln University3. Birmingham City University4. The Arts University

Bournemouth5. Swinburne University6. University of Tasmania7. Victoria University

• Repeated recommendation in Academic Audit reports conducted by the Academic Quality Agency for New Zealand’s Universities (Cameron, 2015)

Page 22: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Snapshot 3: International Student Employability & Mobility Benchmarking Project 2015-2016

22

Australia China Business Council (ACBC)Australian Collaborative Education Network (ACEN)Australian Government Department of Education and TrainingCPA AustraliaFederal Group, TasmaniaHobart CityHigh Commission of CanadaHigher Education Academy (HEA)Higher Education Services (HES)NavitasStornaway, TasmaniaSt Ann’s Homes, TasmaniaTasmanian Government and Department of EducationTertiary Education Quality and Standards Agency (TEQSA)The Australian Industry Group (Ai Group)The International Education Association of Australia (IEAA)University of AucklandUniversities Australia (UA)Universities New Zealand (UNZ)

1. Edith Cowan University (ECU), Australia

2. Massey University, New Zealand3. Memorial University of

Newfoundland, Canada4. Plymouth University, United

Kingdom 5. Swinburne University of

Technology, Australia 6. Ulster University, Ireland7. University of Otago, New Zealand8. University of Portsmouth, United

Kingdom 9. University of Tasmania, Australia

10. University of Wollongong, Australia

Page 23: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Snapshot 4: Council of Higher Education Private Providers (COPHE)

23

The aims of this national benchmarking project are to: 1. Compare first year transition

support practices and pathways with HE institutions;

2. Compare student cohort data [retention, completion, attrition] in first year courses/programs/papers;

3. Compare student experience data in first year [ for e.g. item student support in Student Experience Survey (SES) or other comparative survey data]

Four key outcomes for this benchmarking project include:

1. National workshop [16 & 17 June, 2016].Day 1 is the AIR SIG Forum which ACER has kindly invited COPHE members to presentations from TEQSA, QILT, a session on private providers and universities; and a SIG Group for Private Providers. Day 2 will be hosted by Tabor Adelaide and will focus on presentations from the Department of Education on HEIMS data and student cohort analysis data and introduction to benchmarking project.

2. Peer review workshop: Areas of good practice, areas for improvement and areas for sharing [late October, 2016];

3. Final Report with individual recommendations for each institution [November, 2016];

4. Report to the COPHE Board and TEQSA.

Page 24: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Performance Measure 1.1: Aligned policies and organisational structures in course approval [1.1a and 1.1i]

Sector Gaps

Sector Strengths

Discussion: Areas of good practice, areas for improvement/ or further development and areas for sharing Recording: Change ratings if necessary; add institutional andnational recommendations

SAMPLE

Step 2Calibration and validation process

Peer Review Workshop

Step 3Record changes and recommendations

Peer ReviewWorkshop

Step 4Identify actions and accountabilities

Final Report

Step 1Self review process

Self Review Report

24

Page 25: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

1.1a Aligned plans, policies and organisational structures

What institutional plans and policies are in place to ensure support for students transitioning in the first year of study?

Rating No No, but Yes, but Yes1. Adelaide Central School of

Art2. Adelaide College of Divinity3. Australian College of

Christian Studies 4. Christian Heritage College 5. Australian College of

Theology6. Christian Heritage College7. Tabor College of Higher

Education8. The Tax Institute

1. Academies Australia 2. Adelaide College of Ministries 3. Alphacrucis College 4. Australian Institute of Business 5. Australian Institute of Management6. Campion College 7. Eastern College8. John Paul II Institute 9. Montessori Institute 10. Sydney College of Divinity

1. Australasian College of Health and Wellness2. Australian College of Physical Education3. Engineering Institute of Technology 4. Excelsia College5. International College of Management Sydney 6. Le Cordon Bleu Australia7. Macleay College8. Marcus Oldham College9. Martin and Endeavour Colleges 10. Moore Theological College 11. Perth Bible College 12. Photography Studies College13. Universal Business School Sydney14. UOW College

02468

10121416

1.1a 1.1b

Num

ber o

f res

pons

es

1.1 Ratings

Ratings for Performance Measure 1.1

No

No, but

Yes, but

Yes

29

Page 26: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Areas of good practice Areas for improvement/further development

• Defined plans and policies: We have a significant number of plans and policies related to students transitioning. We also have particular staff roles that are dedicated to supporting students and first year students in particular - it is part of their job

• Support for students transitioning in their first year of the degree is integral to achieving student retention, progression and success. The overarching rationale is derived from the Student Engagement Strategy, the PSC Student Consultation and Support Policy, the Teaching and Learning Plans,

• Institutional Plans outlining the UOW College planning framework: UOWC Strategic Plan; UOWC Business Plan; UOWC Annual Operating Plan; and UOWC Education Strategy. Institutional Policies in Place: Admissions Policy; Academic Consideration Policy; Assessment Policy; Attendance Policy; Course Progress Policy; Credit for Prior Learning Policy; Deferment Suspension and Cancellation Policy; Feedback Policy; Fees and refund Policy; Graduate Qualities Policy; Homestay Fees and Refund; International Student Transfer; Between Providers Policy; Student Academic Integrity Policy; Student Conduct Policy; Student Disability Policy; and Student Grievance Policy.

• Strategic plan, has a focus on 'student experience' but this is overarching to all students and not focused on first-year students and their support

• Institutional programs in place to support all students during the course of their studies. The programs are not necessarily documented in formal policies or plans.

• Has no specific policies exist for students transitioning in their first year of study, only generic policies that apply to all students.

• Has good intentions and multiple initiatives to help ensure support for students transition into the MBA (but no written document or plan articulating how the various initiatives together provide evidence of appropriate support for students transitioning into study

• A large percentage of the student body study online. There is no formal transition support policy and plan.

• Looking to develop an overarching student experience strategy. Work on this strategy has not yet commenced.

• Development of policy outlining required components of an orientation program is an action item on the Thresholds Standards 2015 Work Plan. ACT’s e-Learning Panel has begun work on this.

1.1a Support processes, pathways and programs:What institutional plans and policies are in place to ensure support for students transitioning in the first year of study?

30

Page 27: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional
Page 28: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Curriculum and quality, division of students and

education 28

Snapshot 5 : Quality Benchmarking Project: OBJECTIVES1. To compare institutional course approval and accreditation, review and third party

arrangement processes across 5 universities; and2. Identify areas of good practice, areas for improvement/development and areas for

sharing.

OUTCOMES• Agreed benchmarking template on processes for course accreditation, approval and

review and third party arrangements; • Ethics proposal and approval through the University of Tasmania to undertake this

national project;• Self-Review Report and institutional recommendations submitted on the UTAS

online benchmarking tool; • Training for institutional coordinators in coordinating a benchmarking session at

the peer review workshop; • Peer review workshop: Areas of good practice, areas for

improvement/development and areas for sharing [14-15 November, 2016] • Final Report with institutional recommendations for the academic governing bodies

of each institution and case studies [January, 2017];

Page 29: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

CURRICULUM AND QUALITY, DIVISION OF STUDENTS AND EDUCATION 29

1.3f Professional accreditation process

Is there an online process for accreditation? If so, how does this work?

Rating No No, but Yes, but Yes1. Griffith2. Curtin

1. QUT2. UTAS

0

1

2

3

4

1.3a 1.3b 1.3c 1.3d 1.3e 1.3f

Num

ber o

f Res

pons

es

1.3 Ratings

Ratings for Performance Measure 1.3

No

No, but

Yes, but

Yes

Page 30: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Areas of good practice Areas for improvement/further development

• QUT: The CTRP (Course Transformation and Reaccreditation Plan) is an online tool which is used to capture professional accreditation information. It is not yet integrated with other systems however a range of current activities occurring in the university may better support this online into the future, e.g. PIMMS, Compass.In the past QUT Business School Teaching and Learning team have implemented an ‘Electronic Base Room’ toassist with onsite accreditation. The ‘Electronic Base Room’ includes electronic course information which pertains to the design, implementation and experience of delivering all QUT Business School courses. Other faculties through documents and site visits.

• Curtin: No. Some accreditation bodies have an online submission process (e.g. Nursing). AKARI supports the approval process of curriculum changes arising from Accreditation activities. COMMENTARY: The university has clearly outlined policies and procedures that cover Course Approval (CAP) processes. CAP is well supported by centrally developed templates and a dedicated team that work with Schools. A new curriculum management system that provides online support for course approval and workflow was launched 14 November 2016. Subsequent releases focus on curriculum mapping and accreditation modules, and unit outline builder and repository functions. The university has clearly outlined policies and procedures that cover Comprehensive Course Review (CCR) and Annual Course Review (ACR) processes. The CCR is well supported by centrally developed templates and a dedicated team that work with Faculties to establish the schedule or reviews, and negotiate around school priorities, staffing and accreditation activities.

• Recommend continued improvement in this space. The ACR process is about to be supported by a BI Dashboard tool. The ACR is template will also include section on monitoring of Action plans subsequent to accreditation of CCR. There will be online reporting of the ACR. Review of Course Approval and Quality Manual planned for 2017.

• Addition of policy on Course accreditation

• Clarification of reporting of CCR so that it is visible at ULTC and Courses Committee.

• Formal introduction of Action plans to monitoring reports from CCR and Accreditation at University Courses Committee (introduced 2016)

• Finalisation of priority course approval process

ACTIONS UNDERWAY 2016

• Need to establish clear link between Accreditation / Comprehensive Course Review and Annual Course Review processes(in train 2016)

• Improved implementation and support of ACR (in train 2016)

• Better monitoring of CCR and ACR at Faculty and Centrally (in train 2017)

• Priority Course Approval process – initiated, consultation to implement in 2017

1.3f Professional accreditation process : Is there an online process for accreditation? If so, how does this work?

Curriculum and quality, division of students and education 30

Page 31: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

31

Having two days to talk about excellence in teaching in such depth with such openness was an inspiration. For me, when I think of the last few days: all the hallmarks that OLT aspires to. I did have a deep belief that collaboration, particularly across institutions and internationally, is what builds innovation and leadership for innovation and I saw all that here today.

[Ako Aotearoa Benchmarking Project, 2015]

Page 32: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Peer Review of Assessment

32

Page 33: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

• Theme 1: Increase Australians’ university participation

• Theme 2: Develop a globally engaged university sector

• Theme 3: A powerful research and innovation system that drives economic and social progress

• Theme 4: Efficiency, investment and regulation

University actions

• Introduce external peer moderation of assessment standards

• Integrate technologies to support teaching and enhance the student experience

An Agenda for Australian HE 2013-2016: A smarter Australia

43

Page 34: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Definition of peer review of assessment

‘The practice of colleagues providing and receiving feedback on one another’s unit/subject outlines, assessment tasks and marking criteria to ensure that assessment is aligned to intended learning outcomes and includes a calibration process to ensure comparability of achievement standards and an opportunity for professional learning’ (Booth et al, 2015).

44

Page 35: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

35

Verification(e.g.. QVS, IRU)

Moderation (eg. LaTS)

Calibration & Double Blind (eg. AMA)

Primary intent QA (& QE) QA (& QE) QA & QE

Scope Selected final UoS Selected final UoS Selected degree standards/TLOs

Disciplines Multiple (11) Multiple (12) Accounting

Level Bachelor Bachelor Bachelor + Master

Reviewers 1 academic 1 academic per partner 2 anonymous academics +/or professionals (3rd potentially)

Standards Implicit in reviewers Implicit in reviewers Explicit (agreed nationally 2010)

Calibrated Not explicitly Not explicitly Yes by workshops

Products viewed Tasks (inputs) & outputs in final unit of study

Tasks (inputs) & outputs in final unit of study

Tasks (inputs) & outputs in degree evidencing standards

Data selection Stratified Stratified & de-identified Random & de-identified

Sample outputs 12=3 per grade 4=1 per passing grade Minimum 5 per agreed standard

Reviews Manual submission files & aggregation

Manual submission files & aggregation

Online submission & auto aggregation

Authority Institutional Institutional Disciplinary

Comparing Peer Review Models

45

Page 36: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Context setting: Peer review of assessment

Building capacity for peer review and evaluation of practice

Discipline Scholar Networks and Threshold Learning Outcomes projects

Quality Verification System (QVS), Innovative Research Universities (IRU), Academic Calibration Process; Achievement Matters (Watty et al., 2014); Inter-University Moderation Project (Krause et al., 2014); External Examiner System (UK)

Fitness of purpose and fitness for purpose of assessment: Emeritus Prof Geoff Scott: Peer review of program level outcomes (2015). Also builds on Scott’s (2014) work on networks

Peer Review of Assessment Network (Booth, et al., 2015): national support mechanism for peer review of assessment; feedback also pointed to other forms of peer review

Ewan, C. & Freeman, M. (2015) Found evidence of improved assessment practices with the development of threshold learning outcomes (TLOs); the establishment of networks; and the important role Deans Councils play in leading efforts on academic standards. Yet, they also found three noticeable gaps: 1) the absence of non-self-accrediting and private providers in these academic quality projects; 2) the lack of an evidence base for quality assurance; and 3) the lack of external referencing.

46

Page 37: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

National support mechanism for peer review of assessment

37

Page 38: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

National Support Mechanism for Peer Review of Assessment

Levels Networks Leadership Policy Resources

Sector University Networks

UTAS/collaboration

Strong, deft leadership of networksExternal reference group

Good practice principles in peer review

Online Peer Review Portal National clearinghouse of good

practice in assessment (Flip Curric website)

Training for the Peer Review PortalNational workshops/Forum[Higher

Ed Services/UTAS]

HE institution

Institutionalnetworks, mission groups such as IRU

Clear roles for differentplayers, AD L&T, Quality Managers, Course/Program/Discipline Coordinators

Academic governance,

accreditation, course review, assessment, reward &recognition

Institutional register of trained peer reviewers Integrate with other forms of peer reviewInternal grant process Consideration of paying an honorarium PD workshops/including sessional

staff

49

Page 39: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Support Dimensions

Networks Leadership Policy Resources

Discipline Deans Councils (AD L&T Networks) and accreditation bodies

Leadership of networkDiscipline coordinators,

Chairs and Executives of Australian Deans Councils and other Academic Committees and Societies

Accreditation and industry requirements

College of Peers process, e.g.. Annual Forum on calibration,alignment to TLOs

Individual University networks and peer support networks

Course/Program/Disciplinary Coordinators and peers

Alignment to TLOs,course mapping, assessment, grading, calibration and learning resources; workload allocations; reward and recognition

Feedback from other disciplinary/cross-disciplinary peers

National Support Mechanism for Peer Review of Assessment

50

Page 40: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Aligning Accreditation to Peer Review of Assessment

Year 1 Year 2 Year 3

Year 4 Year 5

Accreditation Peer reviewCalibration

- Peer reviewCalibration

Reaccreditation

Organisational Support and

Resources for External Assessors

Training for External Assessors

Peer Review Calibration Workshop

Report to accreditation body and HE institution Australian Council of Engineering Deans

(ACED)

51

Page 41: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Feedback from Higher Ed Services Workshops on Peer Review of Assessment May-July, 2016

• Sector readiness for online peer review portal and numerous templates • Regional series of workshops• Sector readiness for professional development workshops on peer review

and accreditation• Sector support for certificates of participation• More details on costs of collaboration for HE institutions undertaking peer

review• Ability to contact numerous reviewers • Consolidated access point for peer review resources• Online network like Linked in-if looking for collaborative networks• Online discussion forums, video or SMS contact-for quick chat to verify any

questions

52

Page 42: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Feedback from Higher Ed Services Workshops on Peer Review of Assessment May-July, 2016

PEER REVIEW PORTAL • Webinars and online training for peer review and calibration• Industry representatives provide feedback• Ability to monitor both admin and academic hours • Monitor number of reviews status • Scheduling of reviews • Ability to upload multiple mode documents • Intuitive and user friendly • Reporting of outcomes of the process of peer review,

accreditation and calibration• Downloadable at course, faculty and institutional level

53

Page 43: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Collaboration with Higher Ed Services, Professions Australia and the University of Tasmania 2017

54

Universities Australia and Professions Australia: Joint Statement of Principles for Professional Accreditation (2016) • Universities Australia Conference (February, 2017)On Tuesday February 28, HES, the University of Tasmania and Professions Australia will host an afternoon probing professional accreditation and peer review of assessment in the lead up to the 2017 Universities Australia annual conference.

• 2017 Peer Review of Assessment Workshops (March-April, 2017)The full day workshops in Melbourne, Brisbane, Fremantle and Sydney are designed to:

• Support the sector as it implements the new Higher Education Standards Framework (2015)

• Promote innovation in different models of peer review in learning and teaching• Demonstrate a peer review of assessment portal• Provide opportunities for networking and connecting with colleagues• Provide access to resources to support peer review and assessment.

• National Assessment and Review Summit (19-20th September, 2017)

• TEQSA Conference and Higher Education Compliance and Quality Forum [29th

November- 1 December, 2017)

Page 44: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Peer Review Portal

55

Page 45: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Peer Review Portal

Page 46: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional
Page 47: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Online Peer Review Portal

http://nagnqs.axshare.com/#c=2

Register your expression of interest to know more:Peerreviewportal.com

Proof of Concept

Phase 1: Peer review of assessment inputs/outputs [Feb, 2017] Phase 2: Curriculum review [May, 2017]Phase 3: Benchmarking [later in 2017]

57

Page 48: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

INSERT FACULTY NAME IN FOOTER 48

Page 49: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

Faculty of Education : Proposed Draft Plan: Master of Teaching

60

The units suggested for external review in 2017 and 2019 reflect a focus of accrediting bodies on literacy and numeracy in teacher education and involve both our primary and secondary specialisations, representing all four semesters of the course.

Year 2016 2017 2018 2019 2020

Focus

PlanningPlan peer review of assessment activity

and consider potential

benchmarking partners

CalibrationA college of peers is engaged in the first round of external referencing and peer

review of assessment

AdjustmentFeedback is analysed and

generates ideas for revision where appropriate

ModerationA college of peers is engaged in the

second round of external referencing and peer review of assessment

ConsolidationFeedback informs actions

to fine-tune the course ahead of the next

accreditation cycle as part of ongoing improvement

Participants

Faculty leadership and University in consultation with the MTeach team

EMT511 Foundations of EnglishEMT611 English Curriculum and

Pedagogy

EMT510 Foundations of Literacy: Processes and Practices

EMT515 Approaches to English TeachingEMT610 Teaching, Literature, Culture

All MTeach unit coordinators

EMT521 Teaching Primary Mathematics 1EMT620 Teaching Primary Mathematics 2

EMT520 Personal and Professional Numeracy

EMT525 Teaching the 7-12 Mathematics Curriculum

EMT625 Grade 7-12 Students as Mathematics Learners

All MTeach unit coordinators

Page 50: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

61

• What are the next steps for Irish HE institutions on benchmarking, peer review of assessment and external referencing to assure the quality of provision?

Page 51: Institutional Collaboration in Benchmarking and Peer ... · Institutional Collaboration in Benchmarking and Peer Review: Assuring Quality of Provision ... Final Report and institutional

INSERT FACULTY NAME IN FOOTER 51