enqa seminar:first external evaluations of quality assurance agencies – lessons learned panel...
TRANSCRIPT
ENQA seminar: “First external evaluations of quality assurance
agencies – lessons learned”
Panel discussion: Practicalities and challenges of self and
external reviews: perspective of the expert team
Michael Gerhard Kraft (secretary of the review of AQU Catalunya and
academic staff member FH Council (FHR))
2
Review Process (flow chart)
ToR SE-report written evidence+
ENQA briefing package
Telephone conference
RT draws uplines of inquiryand timetable
Site visit: further evidence, verifying
written evidence
Report: template / draft / amendments by the agency
Final reportand state-
ment of theAgency sub-
mitted to the ENQA Board
3
Lessons learned
Terms of reference should state clearly the assignment of the review team (RT) (sole vs. multiple purpose review)
Focus should be on the ESG, when additional areas for evaluation are being asked for, one should bear in mind that they might bind/require additional resources and a certain composition of the RT (e.g. teaching staff and research assessment of the Spanish agencies)
Conclusions of the RT should distinguish between the different areas of evaluation but should take the consistency of the whole report into consideration
Heavy workload particularly for the Chair and the Secretary
4
Email correspondenceand some of my handwriting
Lessons learned
5
Lessons learned
Quality and comprehensiveness of the self evaluation report is important for the preparation of the RT (principles of self-criticism, objectiveness and openness / not window-dressing!)
The agency presented a comprehensive evidence list on the internet, from where the documents were well accessible • but it is important to prioritise: maybe the agency should
state, which documents it considers particularly important as evidence for its findings in the self-evaluation report.
Good overview on the higher education system in which the agency under review operates is important in order to appreciate national peculiarities
6
Lessons learned
Take your time! • Timetable of the site-visit should provide for sufficient
time to reflect and discuss findings of the interviews and to work on the draft report
It’s all about communication• useful to have a preparatory telephone conference, if
it is not possible to meet in advance• ENQA telephone conference was helpful in addition to
written statements of the ENQA Board (clarifying issues)
Drafting the report• Methodological structure of “description, analysis and
conclusion” is very useful
7
Lessons learned
Findings and experiences of RT should be made available to other expert panels and be taken into account for improvement of the ENQA briefing package
• ENQA has already reworked the briefing package (especially on how to interpret the ESG)
One should keep in mind that although the whole undertaking is about ENQA membership, it should be aimed at improvement in character and not control
• guiding “principle of trust” Important to distinguish between aspects which are
under the control of the agency and those which are due to national legislation
• should be clearly distinguished in the review report
8
ESG, their nature and interpretation
Scope of the ESG for the review• “The membership provisions of ENQA correspond to part 3 of
the ESG incorporating also part 2, in Criterion 1 of the membership provisions.” (ENQA Briefing Package)
How to interpret the ESG?• Principle vs. rules based approach• Respecting the diversity of national contexts• Generic principle vs. specific requirement• One has to be cautious in adapting them:
- ESG 3.7: student involvement (“as appropriate”)- ESG 3.6: “Independence”: has implications for the areas of: financing,
composition of the agency, conflict of interest, QA outcomes, reports, etc.- Publication policies
Clear interpretation of the ESG by the ENQA Board should be communicated to the RT• however, a certain scope for interpretation will always remain
9
ESG, their interpretation and improvement
Question of compliance?• Provide the RT with a notion of what
- compliance- substantial compliance- partial compliancemeans.“Any areas of total non-compliance were unacceptable, and that some
element of compliance would be necessary against all criteria in order to provide an overall judgment based on ‘substantial’ compliance. Each case would be examined on its merits and at a local basis.” (ENQA Board)
Reflexive handling of the ESG Self critical, evaluative approach also necessary for the
development and improvement of the ESG as well as their interpretation
Thank you very much for your attention!