ecvet pilot projects 2nd generation - 2nd seminar ii... · ecvet pilot projects 2nd generation -...

ECVET Pilot projects 2nd generation - 2nd seminar Approaches to the Assessment of Units of Learning Outcomes

Rome, 26-27 April 2012

Synthesis and Evaluation

08 June 2012

Synthesis and Evaluation Report Rome Seminar

June 5th, 2012 i

Contents

1 Introduction .........................................................................................................3

2 Day 1 ....................................................................................................................4 2.1 Official opening ........................................................................................................................ 4 2.2 Assessment of (units of) learning outcomes and ECVET ........................................................ 4 2.3 ECVET pilot projects: Approaches to the assessment of learning outcomes - Workshop

sessions ................................................................................................................................... 5 2.4 Approaches to the Assessment of Units of learning outcomes - Outcomes of the

workshop sessions - Harvesting Session in plenary .............................................................. 13

3 Day 2 .................................................................................................................. 14 3.1 ECVET implementation landscape ........................................................................................ 14 3.2 Definition of Units of learning Outcomes in the context of lifelong learning - Common note

based on the progress made by the pilot projects ................................................................. 15 3.3 Progress to date: Challenges and Open questions ............................................................... 15 3.4 Closing remarks - upcoming dates and events ...................................................................... 16

4 Evaluation of the seminar ................................................................................... 17 4.1 Expectations regarding this seminar ...................................................................................... 17 4.2 Objectives of the seminar....................................................................................................... 18 4.3 Working methods ................................................................................................................... 19 4.4 Organisation of the seminar ................................................................................................... 19 4.5 Topics for discussion at the next seminar .............................................................................. 19 4.6 Working methods at the next seminar ................................................................................... 20 4.7 Participants’ comments .......................................................................................................... 20 4.8 Conclusions ............................................................................................................................ 21

Annex 1 Agenda ................................................................................................... 22


June 5th, 2012 3

1 Introduction

As part of the project cooperation activities on European level, representatives from the eight

ECVET pilot projects 2011-2013 met in Rome on 26-27 April 2012 for their second joint

seminar. The main aim of this seminar was to present the projects’ different approaches to

the assessment of units of learning outcomes and discuss them on Day 1. After working

together for a year, the seminar also provided an opportunity for the pilot projects to discuss

the main challenges they are currently facing in peer groups on Day 2.

This report summarises the outcomes of the discussions of both days in separate sections.

In a third section, the outcomes of an online-survey undertaken after the seminar are

presented. The evaluation shows that the participants are overall satisfied with the

organisation and facilitation of the seminar. However, as to subjects and working methods,

some ideas for improvement were presented that should be further discussed.

The Agenda to the seminar is presented in an Annex.


June 5th, 2012 4

2 Day 1

2.1 Official opening

Roberta Menoni and Vytaute Ezerskiene, project managers at EACEA, opened the seminar

by highlighting the aims of the seminar and the expected feedback from pilot projects to the

further development and implementation of ECVET. Erik Hess, Policy officer at the European

Commission, DG EAC, was not able to attend the seminar due to other obligations, but sent

his regards via the GHK team.

In an introduction to the main topics, it was stated that

■ On Day 1, the seminar would focus on approaches of assessment of units of learning

outcomes. After an introductory presentation on the topic, the focus would be on the

ECVET pilot projects own approaches to the assessment of units of learning outcomes.

In the workshop sessions the focus would be on commonalities and transferability

between the pilot projects. After discussing lessons learnt during a harvesting session,

the first day would be concluded;

■ On Day 2, the seminar would start with an overview of on-going ECVET processes at

European and national level. After that, a document summarising experiences of pilot

projects with the definition of units of learning outcomes would be presented. The last

session on Day 2 would be dedicated to work in small groups on the main challenges

and open questions the projects face after one year of common work.

2.2 Assessment of (units of) learning outcomes and ECVET

In an introductory presentation, Daniela Ulicna (GHK) highlighted that the assessment of

units of learning outcomes is an important step in the ECVET process:

■ Assessment is a precondition for validation and recognition;

■ Hence, assessment is a precondition for credit transfer (validation and recognition of

something that has been learnt and assessed in one context (programme/qualification) in

another context) and credit accumulation (previously awarded credit is considered

achieved and not taught or assessed again);

■ In the focus of assessment, credit transfer (and subsequent accumulation) means that

sometimes the institution that assesses - or the competent institution in charge of

assessment – is different from the one from which the learner will ultimately obtain a

qualification. Subsequently this has implications for quality assurance of the qualification

awarded;

■ Key issues about assessment in ECVET are: Ensure the assessment concerns the

learning outcomes defined; ensure the assessment is reliable and valid even if it is done

by another institution/ competent authority than ‘my own’; ensure recording results and

evidence of assessment so that another institution/competent authority can validate and

recognise it;

■ There are several arguments why it makes more sense to assess units of learning

outcomes rather than whole qualifications or every single learning outcome: Assessing

every learning outcome would put much stress on learners and mean a lot of work for

competent institutions; while assessing the whole qualification (as it is practice in holistic

systems) complicates transfer and/or accumulation of units;

■ ECVET partnerships need to discuss which concrete tools and approaches should be

used to ensure mutual trust. Assessment grids, assessment criteria and the use of the

learners´ records can help to conduct assessment within a partnership;

■ Outside partnerships, the hosting institution (mobility context), respectively the

institutions and stakeholders to recognise the learning outcomes (lifelong learning

context) is responsible to clarify what information will be required to recognise

candidates’ learning outcomes. Learners´ records need to be designed that provide

transparent information and support recognition.


June 5th, 2012 5

2.3 ECVET pilot projects: Approaches to the assessment of learning outcomes - Workshop sessions

2.3.1 Intention of the workshops

After the initial presentations, workshop sessions were held to discuss the approaches to the

assessment of learning outcomes in the pilot projects and to identify commonalities and the

transferability of these approaches.

The aim of the seminars was to facilitate synergy between the pilot projects and to exchange

ideas on the topic of assessment of learning outcomes. After the workshops, the participants

should

■ Have an overview on the methods and solutions developed;

■ Understand the development process and the reasons for the decisions made by the

projects;

■ Get an idea on the main challenges faced by the projects (with regard to assessment);

■ Be able to formulate common recommendations on assessment methods and how to

develop them for future projects working on ECVET in the context of lifelong learning.

All projects were asked to prepare a 15-minute presentation on their approaches to

assessment; addressing

■ The method developed: Please describe the objective intended and the assessment

method developed. Who will assess? According to what criteria? How have they been

set up? What were the reasons for your decision?

■ The development process: Please describe the process that led to the development of

that method. How did you get to that solution? Whom did you have to involve? How did

you come to an agreement? What were the reasons for your decision?

The projects were asked to focus on challenges, open questions, and lessons learnt. If

available, please bring material and examples like assessment grids etc.

The projects will be asked to participate with one representative to each workshop, so that

each project has the chance to get to know all approaches.

2.3.2 Summary of the presentations in the workshops

Table 2.1 gives an overview on the content of the presentations in the workshops. The

presentations have also been uploaded to the pilot projects website. 1

1 http://www.ecvet-projects.eu/Seminars/SeminarDetail.aspx?id=46

http://www.ecvet-projects.eu/Seminars/SeminarDetail.aspx?id=46


June 5th, 2012 6

Table 2.1 Approaches to the assessment of learning outcomes

Approach to assessment Methods Challenges

2get1care,

Germany

Step1:

Analysis of assessment standards given by the

occupational laws; analysis of the current assessment

procedures within the maxQ-schools; consultation of

experts regarding methods of assessment.

Step2:

Identification of methods ensuring the ECVET-compliant

assessment of learning outcomes within the training

courses.

Written exam with performance and case studies;

OSCE - Objective Structured Clinical Examination Since the project is in very early

stages as to its approach to

assessment, the challenges are

not yet clear

CO.L.O.R,

Italy

Within the COLOR context, assessment is considered as

an important step to enable validation and recognition of

learning outcomes:

▪ A transparent and shared assessment system

associated with learning outcomes descriptions is

proving an effective means to build mutual trust and

agreements among regional authorities;

▪ Regional authorities need to build capacity in the field of

transparency and standardisation of qualification

descriptions and assessment, in order to narrow the

gaps between local systems (= greater systemic

coherence)

▪ Assessment - via its systematic set of procedures and

criteria - is a key vehicle to reduce the gaps between the

value of formal and non-formal learning value and

perception thereof.

▪ Start from qualification based on occupational

standards

▪ Identifying and exchanging regional best

practice

▪ Learning from international examples

▪ Managing two sectoral WGs with national and

regional sectoral experts (led by two Regions)

Sharing basic conceptual elements (in line with

EQF-ECVET definitions and technical

specifications and national-regional level

provisions);

▪ Selecting a national standard based

qualification (construction worker) as a pilot

qualification to test a shared methodology to

describe ECVET units, including assessment

aspects;

▪ Defining a shared methodology to reference the

“local” qualifications to the national pilot

qualification (matrix); Exchanging assessment

methodologies and tools starting with a good

practice in Piedmont (in the field of formal

▪ Promoting an ECVET-oriented

“culture” and introducing

practices in regions with sub-

optimal systems (little awareness

of concepts of transparency,

flexibility, etc.);

▪ Introducing ECVET innovative

practices in systems “reluctant” to

change

▪ Interfacing with regional officials

since technical know- how is

typically outsourced

Lessons learnt:

▪ The key to making progress

seems working on minimum

standards: realistic, achievable

and rewarding for each

participating competent authority

(= no system left behind)

▪ More transparent assessment

practices in the formal context are


June 5th, 2012 7

learning) and in Scotland (in the field of non-

formal learning) Selecting the assessment

elements applicable both in a non-formal and

formal contexts;

proving to be an effective starting

point to promote shared

agreements for the assessment

and recognition of non-formal

pathways (= promoting innovation

by improving existing practices)

CPU-

Europe,

Belgium-fr

A spiral structure is used; units have predetermined order.

Assessment : will take place at the end of each unit, it will

not take place at traditional periods (such as Christmas

and June);

From September 2011, 60 schools are testing this system.

Teachers have accepted to test the process because they

know that the current system is not working. Results are

encouraging because students succeed. In 2013 it will be

mainstreamed to the whole system.

Portfolio containing qualification certificate (the list

of units valued > this will allow transfer to another

institution. It describes units in a clear language.

The qualification certificate is only given at the end.

Test to assess the learning outcomes at the end of

each unit is organised.

Several teachers are responsible for

a part of learning outcomes, but the

assessment must be implemented

on a broader level.

Part of the training should be

conducted in companies; they

should be introduced to the new

approach too.

Easy Metal,

Germany ▪ Assessment is seen as a way to enhance transparency

and to create trust in learning results acquired in other

contexts (pre-vocational system/dual system);

▪ Assessment of units of learning outcomes: for each

learning outcome, test criteria are defined;

▪ The competent institutions (chambers) do not want to be

involved in assessment as it is too time consuming. But

they could be involved in quality assurance, set

standards. That would be a strong argument to trust the

results, the way learning outcomes have been

assessed;

▪ Content of assessment is derived from qualifications in

metal industry.

EASY Metal aims at recognising

learning outcomes acquired in pre-

vocational courses.

The key element is to create trust in

the learning outcomes acquired in

the pre-vocational system, to

ultimately credit some of the learning

outcomes. Assessment can help to

create trust in learning outcomes.

Memoranda of Understanding have

little importance because there are

not really any partnerships between

institutions.

ESyCQ,

Germany

Elaboration of test questions and assessment scenarios for

the selected qualifications;

Implementation of these test elements into the CEMES

platform; the test procedure is implemented with the help of

the CEMES platform, an online-based multilingual

The first step is an online multiple-choice-test,

followed by open questions that also need to be

answered online.

The results of both tests serve as starting point for

the third step: “the personal assessment”


June 5th, 2012 8

competence rating system;

Development of evaluation rules and regulations.

evaluated by assessors. Each step may be used

separately or in combination according to

requirements.

ICARE,

Italy

Assessment within the ICARE project will be done on the

basis of competence matrices / assessment profiles which

have been developed within the project with help from

professionals.

Note: Two assessment profiles are missing at the moment

(babysitter and dental assistant), but these will be

developed as a next step.

Professional profiles will be developed with other countries

too. These will be tested / implemented in other countries,

so mutual trust between countries in respective sectors can

be built.

Note: 100 ECVET points will be allocated to one year

training professions: Socio-assistance assistant / care

operator/dental assistant;

Allocation of 50 ECVET points to six-month training

professions: Babysitter & Family Assistant;

30 ECVET points to one unit; different number of ECVET

sub-points to single learning outcomes;

Assessment is done by written tests and via

simulation of situations.

Tests are planned to check both knowledge and

skills by putting the person in an ‘almost real

situation’ (all the phases of a day with the care

demander are simulated).

Assessment template: a general question, visual

support and specific action are tested. The

template contains closed questions, which have to

be answered / fulfilled etc.

Assessment of key competences

related to personal attitudes and

behaviour

MEN

Europe2

The project aims at deepening the regulatory, operational

and staff working conditions enabling an extension of the

‘shared assessment’ practices in France

Certificate Assessment Difficulties due to…

▪ Mobility organisation

▪ Languages, fundings, different

labour laws and regulations,

organisation tasks, pupils’

motivation, compatibility of

mobility periods with the whole

training pathway, preparation

tasks etc.

▪ the reciprocity principle

concerning a formative

assessment

2 Since the MEN-project representative had to cancel her attendance at short notice, a short version of the presentation had been given by Daniela Ulicna (GHK).


June 5th, 2012 9

▪ the reciprocity principle

concerning pupils exchanges and

the content of mobility periods,

which implies: building of

‘common objects’ that are difficult

to articulate with the national

qualifications, in terms of LO

identification and assessment

modalities

▪ The reciprocity principle impacts

the training organisation:

planning, duration…

▪ Regulations concerning diplomas

▪ Units ‘size’, concepts,

assessment modalities.

VET-CCS,

Malta

All learning outcomes / units of learning outcomes are

assessed.

Assessment shows the evidence of what the learner has

achieved. Certification depends on all learning outcomes

being achieved.

A range of assessment methods are used.

Learning outcomes are broken down into grading

criteria to make sure they are assessable.

Criteria clarify what the students are expected to

achieve, ensuring that assessors are grading at the

same level.

A formative approach to assessment is adopted.

▪ Integrating assessment in the

teaching process is time

consuming for teachers.

▪ Assessment tasks: a range of

methods should be used, more

than what is used today.

▪ For some qualifications: might

have to stick to traditional

assessment methods

▪ Inadequacies of traditional

assessments in current learning

outcomes approach.

▪ Culture change: need for capacity

building, training of training staff,

change of ‘teaching culture’ and

mentalities;


June 5th, 2012 10

2.3.3 Reflection of the workshop discussions

After the individual presentations; small groups were built within each workshop. The small

groups worked out recommendations to the role of assessment in the process of credit

transfer and accumulation discussing a range of questions. The results were presented by

participants and the GHK team in the plenary after the two workshop sessions.

Below, the outcomes of the discussions and presentations are presented in a synoptic

manner, grouped according to guiding questions.

2.3.3.1 Assessment and credit transfer and accumulation

Why is assessment important for credit transfer and accumulation?

Assessment is important

■ To support lifelong learning;

■ To create trust in the training and in its results in the respective educational system;

■ To identify the competences achieved in another country or at the workplace to enable

recognition. E.g. for migrant workers where there are no agreements between

educational systems;

■ To valorise achievements and to show progress of learners.

What general characteristics should the assessment process satisfy in order to support

credit transfer and accumulation?

■ Transparent descriptions of the learning outcomes that are assessed are a prerequisite;

■ Clear and transparent assessment standards, processes and methods are a pre-

condition;

■ It is not necessary to set up a one-to-one match between units of learning outcomes +

assessment situations; however, a clear / transparent way to show the relationship is

needed.

■ Concepts, methods and terminology of assessment should be shared among partners;

■ Existing methods to assess competencies and learning outcomes need to be adapted to

the learning outcomes approach;

■ From the formative aspect of assessment; remedial measures are very important; e.g. in

contexts where policies try to avoid drop-outs from learning.

2.3.3.2 Assessment methods

What conditions do you think are crucial to develop assessment methods that support credit

transfer and accumulation?

■ Assessment is the basis for mutual trust;

■ Every assessment methodology should be transparent and have a clear link to a quality

assurance system. E.g. in Malta, different ‘levels’ within each EQF level have been

defined: Pass level (learner has acquired basic skill), higher level of achievement (more

complex situations), distinction (great level of autonomy etc). Each unit can be achieved

at different levels;

■ In the project CPU Europe a similar approach is used: Three levels of achievement of

one unit have been defined, linked to the level of autonomy;

■ In the care sector, assessment structures are organised around working processes (what

a person does during the day with patients). Topics like "health" and "safety" have to be

assessed together with other competencies in concrete situations;

■ Assessment should happen where the learning happens.

What are the particular reasons for designing a process of assessment in a specific way?

■ The intention of assessment will have an impact on the assessment method;

■ Assessment procedures consist of a set of different measures appropriate to the specific

case;

■ Assessment has to be considered in relation to the type of learners and the learning

pathway (assessment after training; assessment of people coming from labour market);


June 5th, 2012 11

■ The range of assessment, the assessment of all units or all learning outcomes have to

be decided depending on the final aim of a learning process;

■ Special training for assessors, guidelines and training on the use of assessment

methods for assessors are needed;

■ The respective contexts and situations have to be clearly described and explained to be

able to discuss assessment instruments and results.

What is the result of assessment? Does it lead to the award of a unit or a qualification? Or is

it formative assessment? Is it related to any regulations?

■ The final result of assessment depends on federal / national / regional regulations; these

should be highly transparent to achieve transferability.

How is the assessment documented?

■ One possibility of documentation are assessment grids that focus on the most important

and relevant key indicators.

2.3.3.3 Key actors

When carrying out assessment: Who are the key actors? Who assesses? According to

which criteria, set out by whom?

■ Key actors of assessment are institutions, companies or chambers at national, regional

and federal level – according to national rules and regulations for validation and

recognition;

■ In many cases institutions are assessing (sometimes with support of companies);

■ Assessment criteria are developed based on existing professional profiles by institutions

together with professional organisations or companies;

■ Assessments can be done by teams of teachers / trainers or single assessors;

■ Practitioners / experts from sectors can be involved in the design of assessment

processes.

Which recommendations can be given to key actors (stakeholders, VET-providers,

assessors, as to concrete ways of assessment and assessment processes?

■ The assessment concept, methods and assessment criteria should be defined in parallel

with the definition of units and the writing of learning outcomes or when curricula are

designed;

■ Assessment criteria and assessment methods and instruments should reflect the

complexity of learning outcomes;

■ National standards and national regulations should be taken into account and explained

to partners;

■ Some projects need to work in the context of existing regulations;

■ E.g. in Germany, certificates given for one unit will play no role in the final assessment.

Therefore VET providers and Chambers of Commerce have to adapt ECVET

specifications to the German system. Chambers are key partners in such projects (they

define assessment methods in the workplace, they can validate informal learning);

■ Tools such as databases can help teachers and VET providers to implement reliable

assessment methods. E.g. the CEMES platform can help to create assessment tests

(multiple choice).

What other recommendation do you have?

■ Guidelines that provide standard rules for assessment would be helpful for users;

■ Assessment tools need to be tested, then readjusted based on the feedback provided by

practitioners. This is an interactive, long-term process;

■ In some sectors, transversal competencies are essential. So assessment methods have

to be adapted to these competencies (e.g. communicating with patients in hospitals).

Such competencies can be included in assessment grids (e.g. ICARE).


June 5th, 2012 12

2.3.3.4 Assessment of transversal skills/generic key competences

The assessment of transversal skills / generic key competences has been an additional

discussion point between several pilot projects. In the following the discussion is

documented:

How can transversal skills and generic key competences be assessed? Should they be

assessed separately?

The discussion evolved around challenges related to the assessment of key competences,

general learning and transversal skills.

In Malta, experimentation in that area is undertaken: Some units are purely technical while

others are described including transversal/general competences. At lower levels of a

qualification, Malta is experimenting “embedded learning” or qualifications where learners

are weak at transversal skills (such as mathematics, languages, but also communication

skills). The measures target learners who have failed at secondary school, so in order to

ensure successful learning the teaching methods need to be altered. The learners are

enrolled in vocational courses mainly to achieve vocational skills. However, general learning

is integrated in the curriculum; general teachers and vocational teachers draft lesson plans

together, e.g. integrate mathematics in practical/vocational courses. Consequently the

learners acquire basic numerical skills related to specific works tasks/vocational skills. That

system has been in place for two years now and is quite successful. Two teachers work

together and design the training together. The view of Malta is that some transversal/general

skills (math) can be taught embedded into vocational course, other transversal competences

cannot.

The project CPU Europe has adopted a similar approach including key competences into

occupational profiles. Generic competencies can be assessed in vocational units. E.g. basic

numeracy skills are assessed while the learner is resolving a problem about how much

fertiliser can be used in a certain context.

In the ESYCQ project transversal skills are assessed together with other competencies, but

in the unit description they are described separately. E.g. competencies around ‘teamwork’

are included in occupational activity.

Since several projects target the group of migrant workers, assessment standards for

language/communication skills have also been discussed: How will migrants be assessed for

communication/ language skills? Should the assessment standards be different?

According to the Maltese project, a standard is the same for everybody. It cannot be

adjusted for a certain target group. But not each specific target group needs to reach “higher”

level to get a qualification. They can reach a high level technically, and “pass” the level for

language. If migrants want to progress further, they will need a higher level of mastery.

The topic of the weight of the units related to key competences has also been discussed. In

the COLOR project key competences common to all qualifications have been defined.

Generic skills should be integrated into vocational training. At the end of 3 years, students

achieve formal certification. They also need certification for key competences. The challenge

for them is that they want to use these units outside formal education pathways. They need

to find a way to define the weight of the technical unit in the whole qualification, so that it can

be recognised on its own, as autonomous, partial qualification, outside of formal pathways.

This is related to the difficulty to attribute points.


June 5th, 2012 13

2.4 Approaches to the Assessment of Units of learning outcomes - Outcomes of the workshop sessions - Harvesting Session in plenary

After the workshop sessions participants were asked to present the lessons learned in

plenary. Table 2.2 gives an overview on the outcomes.

Table 2.2 Lessons learned

What did you learn

about assessment

and ECVET that

you did not know

before coming

here?

▪ Suggestion of different forms / types of learning and training;

▪ Possibilities for the improvement of assessment

▪ The need to get stakeholders other than VET institutions more

directly involved in the assessment validation process;

▪ Different methods / approaches of assessment

▪ Testing of several units in one day of assessment on the basis

of a work situation (matrices of learning outcomes);

▪ Different approaches to assess key competences;

▪ A learning unit is not always identical with a unit of learning

outcomes;

▪ Assessment standards can be fixed independently of learning

situations, target groups etc.;

▪ There are viable web-based solutions for assessment to draw

from new insights into the different approaches (starting from

vocational profiles vs. curricula-oriented);

▪ Deeper understanding of the size of a learning unit;

▪ ECVET allows recognising the learning outcomes that workers

have, instead of focussing on the competences workers should

have.

Lessons learned in

general

▪ Everybody deals with very similar questions;

▪ There is still a lot of work to be done at EU level to harmonise

the "vocabulary" used;

▪ It was interesting to observe that we ECVET specialists still have

difficulties to use "ECVET language";

▪ I gained a broader view on different projects;

▪ Clearer view of some structural differences and problems of

"our" ECVET-project’

▪ ECVET can also be a good tool in order to motivate learners /

workers’;

▪ Different cultures in different countries and every culture gives a

specific form to the assessment methods.


June 5th, 2012 14

3 Day 2

In the morning of day 2 Daniela Ulicna (GHK) presented the on-going and current activities

around the testing and implementation of ECVET on European and national level. After that,

a common note on the description of learning outcomes prepared by GHK/3s was discussed.

Then small working groups discussed identified challenges and tried to identify possible

solutions. The day ended with closing remarks about upcoming events.

3.1 ECVET implementation landscape

In the following, an overview on the topics presented is given. The presentation can be

reviewed on the ECVET pilot projects website.3

ECVET Recommendation – defined basic principles of ECVET (technical specifications)

■ Developed based on demand from EU Ministers in charge of VET;

■ Written based on several years of work of analysis and consensus building;

■ Adopted by the EU Parliament and the Council;

■ Gives the basis for the implementation process, but of course countries decide whether,

how and when to implement ECVET.

The European Commission / EACEA:

■ Support the testing process – funding for pilot projects and analysis of their results;

■ Coordinates development of guidance materials and their dissemination;

■ Coordinates exchange of practice between countries;

■ Cedefop – monitors progress and reports on it;

■ EACEA – oversees the work of the pilot projects and of ECVET promoters.

ECVET Users’ Group (2 representatives from all countries)

■ Develops guidance;

■ Oversees the cooperation on ECVET implementation;

■ Note: As part of the monitoring activities about the pilot projects, GHK regularly reports to

the Users’ Group about the progress of the work of the pilot projects. Examples from the

pilot projects work are also used to illustrate aspects of using ECVET in the User’s guide

that is prepared by then User’s group.

ECVET Team

■ Designs mutual learning on ECVET for competent authorities, policy makers but also

providers;

■ Provides training on ECVET;

■ Manages the ECVET Network.

Further initiatives

■ National teams of ECVET experts provide information about ECVET within the countries;

■ NetECVET: A Network of National Agencies of the LLP work with Leonardo projects (and

ECVET pilot projects) to develop a toolbox for using ECVET for transnational mobility

exchanges of learners (mainly IVET).

■ National level testing initiatives: FINECVET in Finland, DECVET in Germany.

3 http://www.ecvet-projects.eu/Seminars/SeminarDetail.aspx?id=46

http://www.ecvet-projects.eu/Seminars/SeminarDetail.aspx?id=46


June 5th, 2012 15

3.2 Definition of Units of learning Outcomes in the context of lifelong learning - Common note based on the progress made by the pilot projects

During the seminar the note on "descriptions of learning outcomes" prepared by Karin Luomi-Messerer (3s) and Daniela Ulicna (GHK) was discussed and it was agreed that it will be revised based on the feedback and further input from participants.

The main feedback to the note was that the title ‚guidelines‘ could be rather confusing. The pilot projects do not want to create they impression thatb they provide general gyuidelines for the implemenattation of ECVET. Hence, the note should rather be called ‚experiences with the definition of units of learning outcomes‘.

The adapted version will be sent to participants ahead of the next seminar.

The participants agreed that a similar note on assessment should also be prepared, based on the results of the common work process in this seminar.

3.3 Progress to date: Challenges and Open questions

On Day 1, participants were also asked to note the main challenges and questions they face

at the moment on cards and put them up on a pin-board. The questions were clustered by

GHK; 4 main challenges were identified:

■ Challenge 1: ECVET requires a change of culture (LO, assessment) in institutions – how

can this change be supported? How can stakeholders and institutions be convinced of

added value?

■ Challenge 2: How can companies be supported in the process of assessing the learning

outcomes/units that are achieved on the workplace?

■ Challenge 3: What is the best approach to integrate transversal competences/ soft skills

but also general education into units of LO – together with professional KSC?

■ Challenge 4: How can assessment be carried out in reliable and valid manner (which will

be recognised) without it being too burdensome and costly?

Ad-hoc work groups were built according to the main interests of participants. Three

challenges were discussed in more detail.

The results are presented below.

Challenge 1: ECVET requires change of future in institutions – how can this change be supported? How can stakeholders be convinced?

The participants proposed the following measures for supporting changes in institutions.

■ Examples of good practices could influence institutions from the bottom, since they could

motivate people in institutions to adopt a similar approach;

■ On institutional level, "representatives of ECVET" could disseminate the idea of ECVET.

These "representatives of ECVET" should take part in all meetings on institutional level

(e.g. between Italian regions) and bring in the advantages of ECVET for mobility / for

lifelong learning etc.;

■ A concrete (national) ECVET toolkit (in national languages) could disseminate useful

tools and templates;

■ Market(ing): Persons / institutions who / which are concerned or who / which could be

concerned by ECVET should be identified and motivated. This could be done by

focussing on certain target groups - e.g. drop-outs / early school leavers;

■ Certain target groups could be involved by systems of "vouchers" that are already

implemented in some countries. People can finance (parts of) their (further) education by

the use of these vouchers. So the possibility of the use of vouchers could be combined

with descriptions of units of learning outcomes or allocated ECVET points, for example;

■ International exchange allows learning from each other and gives the possibility to see

other / new models.


June 5th, 2012 16

Challenge 3: What is the best approach to integrate transversal competences / soft skills but also general education into units of learning outcomes – together with professional knowledge / skills / competence?

■ Key competences (e.g. maths, languages) should be defined as separate units of

learning outcomes, especially for qualifications allocated to lower levels;

■ A journal on activities (diary of activities, portfolio) could be written by learners;

■ Basic outcomes could be defined at specific levels, but they do not need to be taught

separately.

Challenge 4: How can assessment be carried out in reliable and valid manner (which will be recognised) without it being too burdensome and costly?

■ Establish a guidebook for assessors;

■ Involvement of key stakeholders;

■ Establish peer-groups;

■ Share examples and practices.

3.4 Closing remarks - upcoming dates and events

The second day ended with the announcement of upcoming dates and events.

The next event concerning ECVET will be the ECVET Forum organised by the ECVET team

for the ECVET network: "Moving ahead by working together", 31/05 - 01/06/2012, Brussels;

http://www.ecvet-team.eu/en

In case the projects have not yet registered to become members of the ECVET network,

they were invited to do so to make sure they receive information about events like this.

The next ECVET pilot projects seminar will take place on 22-23 October 2012 in Berlin

(Monday-Tuesday). Each project should be represented by two to three project team

members.

All documents and presentations can be found at: http://www.ecvet-projects.eu/default.aspx

http://www.ecvet-team.eu/en

http://www.ecvet-projects.eu/default.aspx


June 5th, 2012 17

4 Evaluation of the seminar

This part of the report summarises the responses to an online-evaluation. The evaluation

questionnaire was designed as a web-based survey and was sent to participants via email

on 30 April 2011.

The objective of this survey was threefold:

■ Evaluate their satisfaction with regard to this seminar and the discussion held;

■ Evaluate their satisfaction with administration and organisation of the seminar; and

■ Explore with representatives of the 2nd generation their interests in specific topics and

subjects to discuss and methods to apply during the next pilot projects’ seminar.

10 responses were received; representing 59% of the number of participants having

attended the workshop (organisers and facilitators excluded).

Compared to the evaluation surveys sent out after 6th, 7th or 8th seminars in the ECVET

Pilot projects 1st generation, the response rate is higher – those rates were respectively

40%, 30% and 45% of participants in the seminars. This might be due to the fact that the

group is still relatively new and the working methods and cooperation patterns are not as

established as they were with the 1st generation after several years of collaboration; so it

may be fair to assume that participants took the chance to give feedback in order to have a

say in how the future working process will be shaped.

The number of respondents varies per question. The exact response rate to each of the 8

questions of the survey is indicated in the descriptive part of the analysis below.

4.1 Expectations regarding this seminar

The first question in the survey asked the respondents to list their three main expectations as

to this seminar. 7 people answered that question.

Table 4.1 below highlights that the main expectation of most respondents was discussing

‘methods of assessment’ - in line with the announced focus of the seminar.

The second and third expectations mentioned are less homogeneous. However, it can be

noted that the following ideas were quoted by several respondents: ‘Discuss leaning

outcomes’ and ‘exchange about challenges, solutions and similarities among the pilot

projects’, and about the guidelines prepared.

Table 4.1 Participants’ expectations

1st

expectation

Respondent 1 Discuss the different assessment methodologies used

Respondent 2 Discuss about assessment with the other ECVET projects.

Respondent 3 Learn about other assessment methods

Respondent 4 Learn more about different approaches to assessment

Respondent 5 Learn more about different approaches to assessment

Respondent 6 Get to know ECVET project partners and project approaches

Respondent 7 Have a methodological discussion relating to the assessment of learning

outcomes

2nd

expectation

Respondent 1 Discuss how these reflect the assurance of the learning outcomes pre-

defined

Respondent 2 Receive information and recommendations about assessment.

Respondent 3 Discuss units of learning outcomes

Respondent 4 Identify commonalities to create synergies with relevant projects


June 5th, 2012 18

1st

expectation

Respondent 5 Exchange on different challenges

Respondent 6 Get feedback on project solutions

Respondent 7 Exchange with other projects on methodological aspects of the project

implementation

3rd

expectation

Respondent 1 Describe Comparability of Qualifications (the outcomes) between the

involved pilot-projects

Respondent 2 Explain the choices about assessment in Belgium and receive opinions

Respondent 3 Discuss the transferable learning outcomes

Respondent 4 Exchange views on the Guidelines

Respondent 5 Exchange points of view on the Guidelines document

Respondent 6 Improve project solutions

Respondent 7 Analyse in depth challenges within projects

In a second question we asked respondents if their expectations listed in question 1 had

been achieved. As can be seen in Table 4.2 below, respondents’ first expectation was fully

achieved for 71% and partially achieved for 29%. The respondents’ second expectation was

still fully achieved in 57% of cases and partially achieved for 43% of respondents. Finally,

86% of respondents think that their third expectation was at least partially achieved whereas

this third expectation was not at all met for 14% of respondents.

Table 4.2 Answers to question 2

Fully Partially Not at all

1st

expectation 71.40% 28.60% 0%

2nd

expectation 57.10% 42.90% 0%

3rd

expectation 0% 85.70% 14.30%

Globally, participants in the seminar answered that their first two expectations were clearly

achieved and their third expectation even partially achieved in most cases.

4.2 Objectives of the seminar

The next question listed the objectives of the seminar that were announced in the invitation

and background document. 7 answers were received.

Table 4.3 shows that most participants found that the majority of objectives of the seminar

were fully or partially achieved:

■ 3 objectives were fully achieved for 85.7% of respondents;

■ 2 other objectives were fully achieved for 57.1% of respondents;

■ 2 objectives were partially achieved for 42.9% of respondents;

■ 2 other objectives were partially achieves for 14.3% of respondents.

Table 4.3 Reached objectives

Fully Partially Not at all

Provide information about the role of assessment in the context of lifelong learning

85.7% 14.3%

Explore the pilot projects approaches to assessment 57.1% 42.9%

Work towards common recommendations on assessment of learning outcomes acquired in different learning contexts

14.3% 57.1% 28.6%


June 5th, 2012 19

Provide an overview on progress made to date by the pilot projects

57.1% 42.9%

Provide an opportunity to discuss challenges and open questions

85.7% 14.3%

Facilitate exchange and discussion between projects 85.7% 14.3%

4.3 Working methods

The forth question was about the working methods of the seminar, such as the amount of

information input made available and the time spent per session of the seminar (i.e. projects

presentations, plenary sessions, smaller groups sessions, informal discussions among

participants). Respondents were asked to give their opinion about the methods by choosing

between “appropriate”, “too much”, “not enough” or “don’t know”.

As detailed in Table 4.4 below, four out of the five categories to be rated scored high – i.e.

most or all respondents found the amount of input or time made available was appropriate.

However, 71% respondents considered that the time allocated to small groups working

sessions was not sufficient. 14% of respondents thought that too much emphasis had been

given to presentations of the projects.

Table 4.4 Appropriate working method of the seminar

Appropriate Too much Not enough Don’t know

Amount of input of information 100% 0.0% 0.0% 0.0%

Time allocated to projects presentations 71.4% 14.3% 14.3% 0.0%

Time allocated to work in plenary 85.7% 0.0% 14.3% 0.0%

Time allocated to work in small groups 28.6% 0.0% 71.4% 0.0%

Time allocated to discussions between projects 71.4% 0.0% 28.6% 0.0%

4.4 Organisation of the seminar

Eight participants gave us their view about the arrangements and facilities of the seminar

which took place in Rome. The table below (Table 4.5) shows that 100% respondents were

fully satisfied with the organisation during the seminar; and that 87.5% of respondents were

fully satisfied with the administration made before the seminar as well as the chosen venue

and its facilities.

Table 4.5 Arrangements made for the seminar

Fully satisfied Partially Not at all

Administration before seminar 87.5% 12.5% 0%

Seminar organisation during the two days 100% 0% 0%

Venue and facilities 87.5% 12.5% 0%

4.5 Topics for discussion at the next seminar

The participants were asked to rate which topics they would like to see discussed at the next

seminar in October 2012. The scale ranged from “very important” to “not important” or even

“don’t know”.

Table 4.6 below shows that among the 10 respondents, a large majority consider that among

the three offered topics, three are either important or very important. The topic ‘ECVET

related documentation’ does not score as high as the other topics.


June 5th, 2012 20

Table 4.6 Discussion topics for the next seminar

Very important

Important Less important

Not important

Don’t know

The ECVET-related documentation (Memorandum of Understanding, Learning Agreement, Transcript of Records) in the context of lifelong learning

20% 40% 40% 0% 0%

Validation and recognition of learning outcomes acquired in a different learning context

80% 10% 10% 0% 0%

ECVET points 60% 30% 0% 10% 0%

Specific challenges of national implementation

30% 60% 10% 0% 0%

4.6 Working methods at the next seminar

Respondents were asked to identify the appropriate working methods for the various working

sessions to be chosen for the next seminar. 10 participants answered this seventh question.

From the table below (table 4.7), it appears that all suggested working methods are

considered appropriate or even very appropriate by the majority of respondents.

However, one respondent out of nine answered that ‘general information about ECVET in

the context of lifelong learning’ and ‘instruction about how to work with ECVET’ are not

appropriate methods to be used in the next seminar.

Table 4.7 Working methods for October seminar

Very appropriate

Appropriate Less appropriate

Not appropriate

Don’t know

General information about ECVET in the context of lifelong learning

22.2% 44.4% 22.2% 11.1% 0%

Instruction about how to work with ECVET

22.2% 66.7% 0% 11.1% 0%

Presentations of projects' work in context to the seminar topic

30% 70% 0% 0% 0%

Discussions in plenary 40% 60% 0% 0% 0%

Small group discussions 70% 30% 0% 0% 0%

Peer counselling 30% 50% 20% 0% 0%

Work on recommendations based on projects' experiences

44.4% 44.4% 11.1% 0% 0%

4.7 Participants’ comments

Several specific comments were provided which are helpful for the organisation of further

seminars. The comments can be clustered as follows:

Working methods and techniques:

Participants provided interesting ideas as to how the presentation of projects results can be

made more visible.

‘It would be a good idea if every project had to present the template used for qualifications

description in terms of ECVET (i.e.: documenting/highlighting the learning outcomes of a

particular study unit (if not for all study units compiling a whole qualification), the ECVET


June 5th, 2012 21

points allocated, the assessment and teaching methodologies used etc...so that we could all

see how each of us are actually interpreting the ECVET features in real practise.’

‘In relation to project presentations, it is suggested (for the forthcoming seminars) to organise

a sort of poster session where all projects will be able to provide an overview of their

achievements. Posters should be placed in a common space and should be available

throughout the whole seminar duration (in this way each person will have enough time to

read the posters and eventually organise a plenary discussion - question time - the last day

of the seminar’.

They also pointed out that the methods should leave enough time for discussion:

‘The techniques used are varied, but sometimes the discussion is interesting and the

techniques don't must be more important than the topic.’

Language skills of participants

Several participants pointed out that a lack of language skills hindered the discussions and

the flow of information on some occasions – specifically in small group work. This should be

taken into account when thinking about working methods:

‘The fact that projects were presented in different groups (and depending on the language

command of presenters).’

‘Linguistic abilities of participants (with particular regard to presenters) should be further

taken into account.’

‘It is suggested to avoid that the group is split into too small groups (where the quality of

work highly depends on language command of people).’

Developing common guidelines on the ECVET technical specifications

Two participants felt that the work on common guidelines was not well framed.

‘Instructions on how to review the Guidelines were not clear. Our group understood that we

were expected to submit amendments and not on general remarks. This created some

confusion. In addition some members of the group felt frustrated as some proposals made

by our group were not well received.’

‘Not enough time and Instructions on how to review the guidelines were not clear (how

specific or broad had to be the expected feedbacks).’

4.8 Conclusions

Overall, the feedback to the seminar is positive. The participants know what to expect from

the seminars and their expectations are met. Organisation and administration of the

seminars are well received.

As to working methods, the overall mix of methods applied seems to be appreciated.

Adjustments can be made as to some details like working on grids and templates on how to

better present the participants results.

An issue are the language skills of projects representatives. The working methods chosen

(small group work and direct interaction rather than plenary work) largely depend on the

ability of the participants to interact. Since it has been noted by several participants that

discussion in small groups isn’t really fruitful if the information flow is hindered by missing

language skills, methods have to be considered that rely less on oral communication and

direct interaction. A few interesting suggestions have been made (templates, poster sessions

etc.).

The participants’ comments will be taken into account when planning the next seminars.


June 5th, 2012 22

Annex 1 Agenda

26-27 April 2012, Rome

ECVET pilot projects 2nd generation –

Approaches to the Assessment of Units of Learning Outcomes / One year on: Progress to date

AGENDA

DAY 1 – 26 April 2012

9.30 – 10.00 Registration

10.00 – 10.15 Opening – Plenary Session

Welcome by Education Audiovisual and Culture Executive Agency

Roberta Menoni and Vytaute Ezerskiene, EACEA

10.15 – 10.30 Check-in: Who is in the room? What are the participants’ own experiences with the subject?

10.30 – 11.00 Approaches to the assessment of learning outcomes acquired in different learning contexts

Presentation by Daniela Ulicna, GHK & Karin Luomi-Messerer, 3s

11.00 – 11.15 Introduction to Day 1 and 2 Work Programme - Anette Curth, GHK

11.15 - 11.30 Coffee Break

11.30 – 13.00 Group divides into two groups (1 project representative per group)

ECVET pilot projects: Approaches to the assessment of learning outcomes

2 presentations in each group (20 minutes per presentation)

40 minutes discussion – with a focus on commonalities and transferability

Facilitation: Workshop 1 – Karin Luomi-Messerer/Sonja Lengauer, 3s

Workshop 2 - Anette Curth/Cecile Mathou, GHK

13.00 – 14.00 Lunch break

14.00 – 15.30 ECVET pilot projects: Approaches to the assessment of learning outcomes

2 Project presentations in each group (20 minutes per presentation)

40 minutes discussion - with a focus developing recommendations

Facilitation: Workshop 1 – Karin Luomi-Messerer/Sonja Lengauer, 3s

Workshop 2 - Anette Curth/Cecile Mathou, GHK

15.30 – 16.00 Coffee break

16.00 – 17.00 Approaches to the Assessment of Units of Learning Outcomes

Outcomes of the workshop sessions - Harvesting Session in plenary

Presentation of main lessons learnt and main recommendations

Facilitation: Daniela Ulicna, GHK

DAY 1 – evening time

Participants are invited to enjoy Rome at their own leisure


June 5th, 2012 23

DAY 2 – 27 April 2012

9.00 – 9.15 Introduction to the day

Anette Curth, GHK

9.15 – 10.45 Definition of Units of Learning Outcomes in the context of lifelong learning

- Common note based on the progress made by the pilot projects

Discussion of a paper prepared by GHK

Facilitation: Karin Luomi-Messerer, 3s

10.45 - 11.15 Coffee break

11.15 – 12.30 Progress to date: Challenges and Open questions

Peer counselling sessions

Work in small groups on the main challenges and open questions

Facilitation: GHK/3s team

12.30 – 13.00 Closing remarks

Upcoming dates and events

Administrative matters

Facilitation: GHK/3s team

13.00 Departure

ecvet pilot projects 2nd generation - 2nd seminar ii... · ecvet pilot projects 2nd generation -...

Documents