towards a diversified review model for fe colleges first and second thoughts with no formal status!...

Towards a diversified Towards a diversified review model for FE review model for FE

collegescolleges

First and second thoughts with First and second thoughts with no formal status!no formal status!

Dr Iain MacRobert and other HMI Dr Iain MacRobert and other HMI thinking out loud!thinking out loud!

Aims and objectives: carried Aims and objectives: carried forward from the current forward from the current modelmodel

• Support and develop quality improvement in Support and develop quality improvement in colleges to provide the colleges to provide the best possible best possible experience for the learnerexperience for the learner

• inform the council's funding allocations and inform the council's funding allocations and policy developmentpolicy development

• provide assurances of value for money for provide assurances of value for money for public funderspublic funders

• provide information to the public on the quality provide information to the public on the quality of the colleges and educational programmesof the colleges and educational programmes

Aims and objectives: in Aims and objectives: in addition addition • Inform prospective students of the quality of the Inform prospective students of the quality of the college's educational programmes and servicescollege's educational programmes and services

• provide fast and detailed information for college provide fast and detailed information for college managers and staff to improve the quality of the managers and staff to improve the quality of the programmes and services they provideprogrammes and services they provide

• provide a breadth and depth of review that is provide a breadth and depth of review that is proportionate to the level of confidence that the proportionate to the level of confidence that the college has effective quality management in place college has effective quality management in place for all significant aspects that impact on the quality for all significant aspects that impact on the quality of the students' experienceof the students' experience

• ease the external review and audit burden on ease the external review and audit burden on colleges.colleges.

Three key assumptionsThree key assumptions

• Not all Elements QIs are of equal value - Not all Elements QIs are of equal value - some (eg ELD, Guidance and support, QA some (eg ELD, Guidance and support, QA and QI) are key independent variables that and QI) are key independent variables that determine the quality of other elements.determine the quality of other elements.

• A key output is student achievement for A key output is student achievement for which retention and attainment data often which retention and attainment data often provide a reliable proxy.provide a reliable proxy.

• That the key process is learning (and all That the key process is learning (and all that facilitates it including guidance, that facilitates it including guidance, teaching, teaching, materials, supportmaterials, support etc.) etc.)

Two phases: three levels of Two phases: three levels of reviewreview• First phase (December to March)First phase (December to March)

– Core review (all colleges)Core review (all colleges)

• Second phase (April to June)Second phase (April to June)– proportionate to the strengths and proportionate to the strengths and

weaknesses identified during the Core weaknesses identified during the Core ReviewReview

– proportionate to the size and complexity proportionate to the size and complexity of the collegeof the college

First phase: Core (diagnostic) First phase: Core (diagnostic) reviewreview• Obtain documentation from college - similar Obtain documentation from college - similar

to and no more than that currently to and no more than that currently requested:requested:– Context statementContext statement– Strategic planStrategic plan– Operational plans for all teaching departments Operational plans for all teaching departments

and student support sectionsand student support sections– Recent reports by other bodies carrying out Recent reports by other bodies carrying out

external evaluations of relevant provisionexternal evaluations of relevant provision– Self-evaluations and/or reviews for all educational Self-evaluations and/or reviews for all educational

provision and student support functions produces provision and student support functions produces routinely by the collegeroutinely by the college

First phase: Core (diagnostic) First phase: Core (diagnostic) reviewreview

• Obtain documentation from college (continued):Obtain documentation from college (continued):– Targets and KPIs (including PIs) Targets and KPIs (including PIs) for all areasfor all areas– College procedures for planning, line management, College procedures for planning, line management,

internal communication, programme reviews, internal communication, programme reviews, student guidance and support, SD&CR, student guidance and support, SD&CR, accommodation and accommodation and resourcesresources etc.. etc..

– Records of programme reviews, Records of programme reviews, reviews of services reviews of services for studentsfor students, student guidance and support, staff , student guidance and support, staff development etc..development etc..

– and completed questionnaires from a sample of and completed questionnaires from a sample of taught, taught, flexible, distance and online learning flexible, distance and online learning studentsstudents

First phase: Core (diagnostic) First phase: Core (diagnostic) reviewreview• Evaluate documentary evidence (college Evaluate documentary evidence (college

review team)review team)– against B Elements ELD against B Elements ELD (including flexi learning (including flexi learning

strategies and resources),strategies and resources), Access and inclusion, Access and inclusion, Guidance and support, QA and QI, and A Element Guidance and support, QA and QI, and A Element Student achievement (revised and extended) Student achievement (revised and extended)

– identifying identifying potentialpotential key strengths and key strengths and weaknessesweaknesses

– using 3 reviewers for 5 days (15 days) with a using 3 reviewers for 5 days (15 days) with a senior member of college staff (Depute / senior AP senior member of college staff (Depute / senior AP to assist with interpretation)to assist with interpretation)


• Visit the college (college review team):Visit the college (college review team):– to gather additional evidence to confirm, to gather additional evidence to confirm,

deny or modify key strengths (and good deny or modify key strengths (and good practice) and weaknesses provisionally practice) and weaknesses provisionally identified from documentation (3 reviewers identified from documentation (3 reviewers for 2 days [6 days] but up to a maximum of for 2 days [6 days] but up to a maximum of 3 reviewers for 5 days [15 days])3 reviewers for 5 days [15 days])

and and


• Visit the college (subject review team): Visit the college (subject review team): – to evaluate Learning and teaching process to evaluate Learning and teaching process

(revised to include first-line guidance QIs, (revised to include first-line guidance QIs, learning resources - T&L materialslearning resources - T&L materials, , classrooms, workshops, and formative classrooms, workshops, and formative assessment) sampling viable subject areas assessment) sampling viable subject areas with:with:

– 5 reviewers for 5 days (25 days) normally 5 reviewers for 5 days (25 days) normally comprising an HMI MI, 3 subject specialists comprising an HMI MI, 3 subject specialists and one flexible learning reviewer.and one flexible learning reviewer.


• Visit the college (subject review team): Visit the college (subject review team): – evaluating:evaluating:

•viable subject area with highest retention and viable subject area with highest retention and attainment ratesattainment rates

•viable subject area with lowest retention and viable subject area with lowest retention and attainment ratesattainment rates

•subject area with largest volume of provisionsubject area with largest volume of provision

•a cross-curricular sample of flexible (open, distance, a cross-curricular sample of flexible (open, distance, work-based, community-based, online etc.) provisionwork-based, community-based, online etc.) provision

•or other subject areas broadly meeting these criteria or other subject areas broadly meeting these criteria , as identified in discussion between the MI and , as identified in discussion between the MI and college senior managerscollege senior managers

Issues that we need to address Issues that we need to address in relation to in relation to Learning and Learning and teachingteaching• We tend to confuse learning outcomes with We tend to confuse learning outcomes with learning processeslearning processes

• Grades awarded at present tend to give Grades awarded at present tend to give administrative and planning QIs equal weight administrative and planning QIs equal weight with learningwith learning

• The order of QIs and prompts does not prioritise The order of QIs and prompts does not prioritise learninglearning

• Reviewers looking at more student work over Reviewers looking at more student work over time would help to evaluate time would help to evaluate progressprogress in in learning.learning.

Issues that we need to address Issues that we need to address in relation to in relation to Learning and Learning and teachingteaching• The role of assessment in learning and The role of assessment in learning and teaching needs to be more fully recognised.teaching needs to be more fully recognised.

• QIs need to be applicable to all forms of QIs need to be applicable to all forms of learning (classroom, workshop, community-learning (classroom, workshop, community-based, work-based, resource-based, based, work-based, resource-based, flexible, distance, open, online and blended) flexible, distance, open, online and blended) and address learning materials accordingly.and address learning materials accordingly.

• The The Teaching and learningTeaching and learning element needs element needs to be revised and extended.to be revised and extended.

Perhaps a new Element A5: Perhaps a new Element A5: Learning, teaching and student Learning, teaching and student outcomesoutcomes

• A5.1A5.1 Learning processLearning process

• A5.2 A5.2 Teaching processTeaching process

• A5.3 A5.3 Student (or learning) Student (or learning) outcomesoutcomes

• With a single evaluation gradeWith a single evaluation grade

First phase: Core reviewFirst phase: Core review

• Draft report in bullet points Draft report in bullet points identifying strengths, weaknesses identifying strengths, weaknesses and MPsfA (7 days - one per reviewer)and MPsfA (7 days - one per reviewer)

• Issue draft report within 15 working Issue draft report within 15 working days to principal and senior college days to principal and senior college management, present findings and management, present findings and 'agree' volume and scope of the 'agree' volume and scope of the subsequent follow through reviewsubsequent follow through review


Criteria for determining the scope of Criteria for determining the scope of follow through reviews could be the levels follow through reviews could be the levels of confidence that:of confidence that:– educational provision is well designed, planned and educational provision is well designed, planned and

managed, and responsive to community, student, managed, and responsive to community, student, employer, economy and stakeholder needsemployer, economy and stakeholder needs

– programmes and programmes and all services to students are all services to students are evaluated systematically, major weaknesses are few evaluated systematically, major weaknesses are few and being addressed, and improvements being madeand being addressed, and improvements being made

– the quality of learning and teaching is being the quality of learning and teaching is being evaluated and improvedevaluated and improved


Criteria (continued) for determining the Criteria (continued) for determining the scope of follow through reviews could be scope of follow through reviews could be the levels of confidence that:the levels of confidence that:– demanding but realistic targets (KPIs) are being set demanding but realistic targets (KPIs) are being set

and largely achievedand largely achieved– student retention and attainment is high across student retention and attainment is high across

almost all subject areas and good or very good in the almost all subject areas and good or very good in the subject areas reviewedsubject areas reviewed

– learning, teaching and learning, teaching and related student support is related student support is good or very good in the subject areas reviewedgood or very good in the subject areas reviewed

Second phase: Second phase: Follow through reviewFollow through review

Proportionate. For example:Proportionate. For example:– Exploring in more detail the causes of strengths and Exploring in more detail the causes of strengths and

weaknesses without extending the scope of the Core weaknesses without extending the scope of the Core reviewreview

– Extending the evaluation of Learning and teaching to Extending the evaluation of Learning and teaching to 6 subject areas (3+1+3)6 subject areas (3+1+3)

– Evaluating against other A and B Elements as required Evaluating against other A and B Elements as required (based on the evidence from the Core review)(based on the evidence from the Core review)

– Extending the evaluation of Learning and teaching to Extending the evaluation of Learning and teaching to 7 or 8 subject areas7 or 8 subject areas

– Evaluating against all A and B elements.Evaluating against all A and B elements.

ReportsReports• For college managers and staff: strengths, weaknesses, For college managers and staff: strengths, weaknesses,

grades and MPsfA in bullet point format and FE jargon grades and MPsfA in bullet point format and FE jargon issued to the principal within 15 working days of the issued to the principal within 15 working days of the reviewreview

• Core and follow through report from c4K to c11K words Core and follow through report from c4K to c11K words in the subject section and c 3.5K to c8K to in the in the subject section and c 3.5K to c8K to in the college sectioncollege section

• For public accountability: a report of up to 2K words (for For public accountability: a report of up to 2K words (for a large follow through) in plain English (similar to our a large follow through) in plain English (similar to our current summaries which answer the key questions)current summaries which answer the key questions)

• For prospective students of the college: 400 to 600 For prospective students of the college: 400 to 600 words in plain English to be published by the collegewords in plain English to be published by the college

QuestionsQuestions

• Is a Is a Core reviewCore review a suitable transparent model for a suitable transparent model for deciding on the differentiated review to follow?deciding on the differentiated review to follow?– Can HMIE deliver it? Will colleges buy into it? Will Can HMIE deliver it? Will colleges buy into it? Will

SFEFC be happy with it?SFEFC be happy with it?

• What are the implications for a differentiated What are the implications for a differentiated second phase?second phase?– How will this impact on HMI-college relationships?How will this impact on HMI-college relationships?– What other major issues will this throw up?What other major issues will this throw up?

towards a diversified review model for fe colleges first and second thoughts with no formal status!...

Documents