1 combining evaluation and accreditation cultures caty duykaerts (aeqes), teresa sanchez,...
TRANSCRIPT
1
Combining evaluation and accreditation cultures
Caty Duykaerts (AEQES), Teresa Sanchez, Jean-Claude Arditti, Pierre Fleischmann, Bernard Remaud (CTI)
A collaboration for the joint evaluation and accreditation of civil and bio engineering programmes in the French Community of Belgium
2
I. Institutional background 1. AEQES 2. CTI II. Origin ans scope of the collaboration
III. Phases of the collaboration 1. Preparatory phase 2. Design of the collaboration 3. Execution phase
IV. Development and main outomes of the preparatory phase 1. Assessing the feasability of the collaboration 2. Collaboration agreement between AEQES and CTI
V. Conclusions and questions for the audience
3
AEQES brief history & legal framework
From 1998 to 2002First evaluations of University programmes
conducted by the CRef AEQES 1 : Act of November 2002
All components of higher educationEvaluation of study programmes
Self-assessment reports, peer reviews and confidential reports AEQES 2 : Act of February 2008
A more independent agency, evaluation of programmes based on a 10-year planning, self-assessment reports, external reviews, publication of results and
follow-up proceduresAgency review every 5 years (ESG compliance)
ENQA full member since september 2011
I. Institutional background II. Origin and scope of the collaborationIII. Phases of the collaborationIV. Development and main outomes of the
preparatory phaseV. Conclusions and questions for the
audience
4
AEQES methodology
Key features
formative QA evaluation process (no accreditation) programme-based quality assurance scope : 1st and 2nd cycle degrees (Bachelor and Master) all similar programmes are evaluated simultaneously, system-wide
analysis- Self evaluation reports based on a reference list of indicators No formal effects in terms of institution fundings or authorisation no rankings, no scores
I. Institutional background II. Origin and scope of the collaborationIII. Phases of the collaborationIV. Development and main outomes of the
preparatory phaseV. Conclusions and questions for the
audience
5
Presentation of CTII. Institutional background II. Origin and scope of the collaborationIII. Phases of the collaborationIV. Development and main outomes of the
preparatory phaseV. Conclusions and questions for the
audience
6
Presentation of CTII. Institutional background II. Origin and scope of the collaborationIII. Phases of the collaborationIV. Development and main outomes of the
preparatory phaseV. Conclusions and questions for the
audienceFull member since 2005
Founding Member 2005. Mutual Recognition Agreements
(OAQ, NVAO)
Founding member 2000 (one of the 7 institutions that can
deliver the EUR-ACE label)
• Since 2010 (European Registry of quality agencies)
7
Origin
In the FCB: all HEIs must be evaluated / engineering programmes make no exception
October 2009: official demand of the four universities concerned for a joint mission AEQES/CTI
Twofold objective behind: Evaluation of the programmes according to AEQES ten-year
plan (compliance with AEQES decree requirements) Accreditation of the programmes according CTI’s criteria
(providing access to the « admission » by the French governement and to EUR-ACE label)
I. Institutional background II. Origin and scope of the collaborationIII. Phases of the collaborationIV. Development and main outomes of the
preparatory phaseV. Conclusions and questions for the
audience
8
Scope
Four universities concerned Catholic Universitiy of Louvain (UCL) Free University of Brussels (ULB) University of Liège (Ulg) (including Faculty of Agricultural Sciences
of Gembloux – FUSAGx) University of Mons (UMons)
Four (4) engineering programmes (BA + MA) in the field of agronomic sciences and biological engineeringFifteen (15 BA + MA) in the field of engineering sciences
Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
9
ScopeInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
10
Phases of the collaboration
Preparatory phase / December 2010Two purposes
To assess the feasibility of the collaboration
To agree on a set of common principles
Signature of a formal collaboration agreement/January 2011Design of the collaboration / from January to December
2011Execution phase / from January 2012 to September 2013
Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
11
Assessing the feasibility
Three main issues
1) Compatibility of two national QA frameworks
2) Compatibility of evaluation / accreditation methods and procedures
3) Applicability of CTI’s accreditation framework to engineering programmes in the FCB
Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
12
1) Compatibility of the two national frameworks
The two national legal frameworks posed no major barriers to the collaboration However, one difficulty detected: difference between the two periodic calendars
Assessing the feasibility
Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
13
2) Compatibility of evaluation/ accreditation methods and procedures put in place by the two agencies
Methodology: Use of the comparison tools and procedures developed by
ECA in TEAM project (as a basis for reflection) Organisation of several work meetings to perform a comparative analysis of standards and procedures Inclusion of observers of the two agencies in a CTI and an
AEQES mission (visit)
Assessing the feasibility
Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
14
2) Compatibility of evaluation/ accreditation methods
Conclusions:Global compatibility with regard to principles and conception. However detection of a number of implementation differences regarding:
- the dynamics and objectives of the HEI’s self-evaluation phase- the balance between quantitative and qualitative criteria- the role of the panel members during the site-visits- the attitude of the HEI’s with regard to the accreditation / evaluation
process
Assessing the feasibility
Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
15
3) Applicability of CTI’s accredidation criteria to engineering programmes in the FCB.
A number of criteria identified as « mandatory ».- A strong and broad basis in fundamental sciences- A guarantee of efficiency and short term adaptation to a
professional activity- Business culture and economic, social, human,
environmental ethcis awareness- Communication skills and international awareness
Assessing the feasibility
Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
16
The agreement
The main outcome of the preparatory phase has been the establishment of a formal agreement signed by the presidents of the two agencies and covering:
- The objectives and the scope of the collaboration- The general organisation of the project- The organisation of the site visits- The composition of the panel- The main outcomes of the mission (productions of several
reports)
Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
17
Conclusion up to now
Benefits of the collaboration from now on:- Agreement on a certain set of basic principles- Setting of a basic work frameworkIllustrative and interesting exercice for both parties.Opportunity to:- confront procedures and methods- bring to the surface the underlying principles behind two apparently
similar proceduresThis step by step collaboration has built a space of trust and
confidence
Institutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
18
Conclusion up to nowInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
Evaluation vs accreditation (1) - Accountability and quality enhancement in a single exercice ?- Accreditation interviews are more « punchy »- Audit report ?
Final « product » for the evaluationWork document to the accreditation committee
- Oral restitution at the end of the audit ?No problem for the evaluationAmbiguous for the accreditation
19
Conclusion up to nowInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
Evaluation vs accreditation (2) Differences in the processes
- The common process ends with the publication of the report with the recommendations- For the accreditation, the report is presented and discussed by the (CTI) committee, which may :
Put different weights to the recommendationsAdd new ones (?)Decide accreditation (Y/N, duration) for each programDecide EUR-ACE labels and/or « titre d'ingénieur »
20
Conclusion up to nowInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
Evaluation vs accreditation (3) Differences in the processes
- The follow-up of recommendations may have not the same objectives
21
Conclusion up to nowInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
First audit vs routine auditThis joint audit was the first for the belgian engineering faculties. In France, HEI's have about 15 years experience of periodic accreditation
Then HEI's are less prepared and suspiciousThe agency (AEQES) has committed very substantial ressources : prepatory meetings, number of days on site, size of the teams.
22
Conclusion up to nowInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
Cultural differencesThe French-speaking Belgians are soft speaking and they dislike disputes The French like controversy and are more direct
Some deans felt uncomfortable during interviews
23
Conclusion up to nowInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
CTI is familiar to international accreditation and to distinguish the core of the standards from the elements linked to the local contextButDecisions for HEI's a few kilometers apart on both sides of the border must be consistent.
24
Open questionsInstitutional background Origin and scope of the collaborationPhases of the collaborationDevelopment and main outomes of the preparatory phaseConclusions and questions for the audience
What are the fundamental differences between an evaluation and an accreditation approach?
To what extent can an agency conciliate the sometimes confronted objectives of the accountability and quality enhancement in a single exercice?
When accreditation becomes a routine : what changes in the procedures , should we shift more from program to institution assessment ?
25
.
Thank you
www.aeqes.be/
www.cti-commission.fr