concluding reflections on guidelines for the use of (achieved… lucien... · curriculum iqa nqf...

PLE

The Hague

30 October 2015

Lucien Bollaert NVAO executive

Concluding Reflections on Guidelines for the use of (Achieved) Learning Outcomes

NVAO Peer Learning Event

• Conclusion of the PLE after so many questions?

• Bringing together the various strands that have

been spun during two days

• & the common understanding

• Looking ahead to drawing up a document of

Guidelines/Points of Attention/Topics/Examples

for the Assessment and Demonstration of

Achieved Learning Outcomes ≠ strait jacket 2

PLA LO concluding reflections

3


“The guidelines explain why the standard is important and

describe how standards might be implemented.

ESG (2015)

4


They set out good practice in the relevant area for

consideration by the actors involved in quality assurance.

ESG (2015)

5


They set out good practices in the relevant area for

consideration by the actors involved in generating quality

and quality assurance.

ESG (2015)

6





Implementation will vary depending on different contexts.”

ESG (2015)

7





Practice will vary depending on different contexts.”

ESG (2015)

HELPFUL IN PRACTICE!

8


START WITH THE QUESTION “WHY ?”

THEN ASK “HOW ?”

FINALLY ASK “WHAT ?”

Simon Sinek (2009)

9


• Why do we need (intended) LOs?

• Why do we need to assess them?

• Why do we need quality (assurance)

management?

• Why do we want quality?

• Why are we in (higher) education?

• Why do we want a better society?

• Why do we need a better world?

• Why do we want a better life? 10


ESG : anchor point? meeting place?

• LEARNING outcomes upgraded in ESG

• Standard 1.2 Design and approval of programmes :

“(…) The programmes should be designed so that they

meet the objectives set for them, including the intended

learning outcomes.

The qualification resulting from a programme should be

clearly specified and communicated, and refer to the

correct level of the national qualifications framework for

higher education and, consequently, to the Framework

for Qualifications of the European Higher Education

Area.”

11


g

e

n

e

r

i

awards c c

s

p

e

c

i

f

i

c

EQF (LLL)

QF EHEA

disc

spec

LO

disc

spec

LO

disc

spec

LO

disc

spec

LO

HEI LO

STP

LO

STP

LO

STP

LO

STP

LO

STP

LO

Key LO

c

LO

c

LO

c

LO

c

LO

c

LO

NQF

i n

t

(i)

n t

H

E

I

S

T

P

T


Learning outcomes : need of common language

need of common understanding

13

LO = what a learner is supposed to

know & be able to do after a successful

study

Make of the LO LIVING things that are formulated by

the stakeholders concerned, passionately shared by

the whole team and shaping the teaching, learning

and assessment practices & formats accordingly !

competence = ability to integrate

knowledge, skills & attitude to be

successful in a certain context


ESG : anchor point? meeting place?

• LEARNING outcomes upgraded in ESG

• Only the guidelines (1.3) cover the achieved learning

outcomes

• Standard 1.3 Student-centred learning, teaching and

assessment :

“(…) Considering the importance of assessment for the

students’ progression and their future careers, quality

assurance processes for assessment take into account

the following: (…)

The assessment allows students to demonstrate the

extent to which the intended learning outcomes have

been achieved. (…)”

14


15

LEARNING

OUTCOMES

MISSION

STRATEGY CURRICULUM IQA

NQF

EQA

IQF

LEARNING

FORMATS ASSESSMENT

FORMATS

LEARNING

ENVIRONMENT

teacher/

coach

ECTS

MOBILITY

RECOGNITION

profiling

marketing

EXEMPTION

RECOGNITION


LOs as meeting place

with different perspectives & languages

HRM

professional needs

employment

EHEA

reforms

Current implementation • Intended LOs have become accepted as a possible most

valid concept in (student-centred) learning (in practice &

process)

• LOs can be at the centre of the quality of T & L (added value

between input = incoming LO & output = achieved LO

• … and thus of QA, both internal & external

• LOs should be linked to international & national QFs

• Discipline LO can be considered

• HEI LOs are good to profile the HEI (pitfall of marketing)

• Course LOs should be linked with LOs of study programme

• Defining good intended LO is still an „art‟

• should combine all levels & stakeholders in practical

reflection & cooperation/co-creation (communication!)

16


• BE MORE THAN SMART

LO should be explicit & measurable or assessable

in order to design the curriculum and QA

LO should be specific, objective, achievable,

useful, relevant & standard-setting and timely

Active verb + type + topic + level (+ context)

Make the LO living (dialogue), i.e. forming

learning in team!

Congruency between teaching, learning &

assessment formats


Keep room for experiment, creativity & failure

(student-centred co-creation & assessment)

course design & QA

Duclan Kennedy (2007)

Identify aims & objectives

Write LOs using guidelines

Develop a teaching & learning strategy

to enable students to achieve the LOs

Design assessment method to test

If LOs have been achieved

If necessary modify module content &

Assessment in light of feedback

P

D

C

A


KEEP

THINGS IN

MOTION

STUDENT

TEACHER LEARNING

environment

congruence between LO, learning & assessment LO T & L forms assessment

cognitive lecture end of course exam

tutorials multiple choice

discussions essays

laboratory practical assessment

fieldwork

affective clinical work clinical practice

seminar presentation

peer group project work

presentation co- or peer-

psychomotor assessment

overall assessment

Kennedy (2007)

Demonstrate

Knowledge

Comprehension

Application

Analysis

Synthesis

Evaluation

Integration of

beliefs, ideas & attitudes

Acquisition of physical

skills


Current implementation • Large variety in Europe in the use of ALOs :

• Assessment of integration of knowledge, skills (&

attitudes or behaviour) is not easy

• The use of internal & external evaluators

• Not always part of procedure & IQA.

• EQA : Netherlands and Flanders: ALOs are (most)

important criterium.

Drawbacks: great emphasis in peer review on results;

debate on autonomy of the institution - who is „in

charge‟ of/‟owns‟ LOs & evaluating achieved LOs.

• In Sweden the question whether the ALOs as related

to the NQF are reflected in the final thesis are subject

of EQA.

• In most countries the EQA assessment is evaluation

on meta-level – not the final thesis, but the

methodology of assessing the intended LOs (Austria).

20


axis : learning outcomes (LO)

øLO only intended intended LO intended LO intended LR

no link with NQF + NQF + NQF + NQF

programme prog. & learning

formats

achieved LO

subject EQA

intended & intended LO intended LO intended LO

achieved LO subject EQA subject IQA prog. & learning

subject EQA & assesment f

21


Guidelines

A need for „guidelines‟ ≠ checklist : • There are many different systems and solutions in

Europe

• Need of common language(s)/understanding

• Guidelines should be based on good practices in

European Higher Education

• Results from PLE should be taken into account

• Guidelines should suggest possible solutions for

common problems and pitfalls (holistic approach)

• Defining good intended LOs not a topic in

Guidelines, although...

22


Topics for the guidelines (1)

• 1 (intended) LOs should be formulated in such a

manner that their achievement can be assessed in a

valid manner.

• Good practices

• Project at Dutch HBO

• Pragmatic, holistic approach at Glasgow Caledonian vs

methodological or semantic discussions

• Involve the stakeholders/actors, primarily teachers &

students

• AuCom Model Saxion

23



• 1 (intended) LOs should be formulated in such a manner

that their achievement can be assessed in a valid

manner.

• Pitfalls and obstacles

• ALOs can be formulated too generic and cannot be

assessed in a valid manner. How to address & assess

attitudes?

• They are too much tied to a workplace so that they cannot

be assessed within the programme. Too many, too detailed.

(IBMS)

• LOs should be pro-active towards changing demands in

society. Updating them is crucial. Conformity to NQF-levels

can restrict innovation as manuals & squeezing rubrics.

• ALOs should not become a bureaucratic item, but a living

tool in the classroom and assessment.

24



• 2. Setting up assessment for assessing ALOs

• Good practices

• “protocol” or Self-Evaluation Tool pilot Dutch HBO (“workable”)

• Variety of various formats (congruency)

• Co-assessment with students & external examiners


• Examination and examiners should follow trend to student-

centred learning and not stick to assessing the curriculum.

• Examination and assessment is not structured and uses invalid

methods and criteria.

• External examiners are not used.

• How to assess attitudes? Heavy assessment of competences.

• Should EQA duplicate the assessment of individual students

or (meta-)judge the system of assessment?

25


Topics for the guidelines (3) • 3. Linking learning outcomes to the levels of the National

Qualification Frameworks.

• Good practices

• Dublin descriptors

• ECTS manual (2015)

• Tuning disciplines

• Pitfalls and Obstacles

• Calibrating the differences between ALOs at the levels of

short cycle, bachelor and master (VU mastermind).

• The more professional a programme is, the more specific the

LOs will be. What is the room for

autonomy/experiment/flexibility when LOs are very specific?

• Generic, transversal competences are too generic while not

linked to the sector-specific competences – they should be

transferable instead of transverable (OECD).

26


21st c. LO or competences

(ability to integrate knowledge, skills and attitudes to

come to solutions in specific contexts)

1. Teamwork skills 67%

2. Sector-specific skills 62%

3. Communication skills 60%

4. Computer skills 60%

5. Ability to adapt & act in new situations 60%

6. Good reading/writing skills 59%

7. Analytical & problem-solving skills 53%

8. Decision-making skills 46%

9. Good with numbers 40%

10. Foreign language skills 33%

Gallup for DG Education & Culture (2010)


28

Innovative teaching & learning processes

OECD


New competences needed Communication, problem-solving, creativity, team-work

Research skills, both academic as well as „mode 2‟

(applied) and mixtures

Inter-disciplinary but with skill and attitude to go deep into

a particular discipline

Meta-cognition

Willingness to change, risk-taking, entrepreneurial

„Global competences‟ (e.g. computer skills)

Visionary & inspirational leadership

HOW YOU ENGAGE IN THE WORLD

= ATTITUDES! CHARACTER! PERSONALITY

(see HRM in industry and business)


More or other knowledge & skills are not enough!

30


Topics for the Guidelines (4)

• 4. Achieved learning outcomes can integrate various

competences

• Good practices

• Profession-related programmes & assessment/assignment

• The need of transferable & transverable competences


• How to include transversal competences in LOs? This

requires a good approach to the formulation of LOs so that

they are not too specific for a particular phase in the learning

process and are fit for life long learning.

• This also requires a link between the transversal

competences and the sector-specific in the teaching-learning

process and the assessment formats.

31


Topics for the Guidelines (5) • 5. Achieved learning outcomes should be relevant to

the context of the programme & the professional

fields. (as quality & QA are contextual)

• Good practices

• The “protocol”/Self-Evaluation Tool in Dutch hbo

• Personal competencies category in VU‟s mastermind


• Different domains require specific approaches; in some

domains only the programme is prescibed (ED), in

others only the LOs/competences (professional labels).

• Too much restriction or only programme-related

prescriptions: European Directives (“contents of learning

instead of learning”).

• Different language by each stakeholder.

• How to address attitudes & character in LOs?

• Up-date and pro-activeness of LOs towards the

professions and LLL.

32


Topics for the guidelines (6) • 6. QA of achieved learning outcomes

• Good practices

• Dutch guidelines for assessing final projects

• Practice of taking samples of final projects

• Ghent University database towards enhancement &

accountability


• QA should not restrict innovation (of ALOs);

• Peer review should reflect changing demands in society and

be pro-active towards innovation.

• QA can stifle innovation if institutions/programmes „play it safe‟

(window dressing/adaption vs quality culture).

• Re-assessment vs meta-assessment

• Study programme & institutional level.

• Ownership of quality (triangle where quality is generated)

and QA.

33


vision & mission (strategic) policy results iLO

LO new students study programme achieved LO

use standards

& indicators

influenced by

development phase

influence on

development

phases

input output processes

Quality Management System

added value = transformation

underlying principles

underlying (hidden) values

methodology

tool tool tool tool tool


some helpful sources :

Kennedy, D. (2007), Writing and Using Dumont, H., Istance, D. & http://ec.europa.eu/education/tools/

Learning Outcomes A Pratical Guide, Benavides, F. (2010), The docs/ects-guide_en.pdf

UCC Cork Nature of Learning - Using

Research to Inspire Practice,

OECD, Paris


http://ec.europa.eu/education/tools/

http://ec.europa.eu/education/tools/


available from http://ecahe.eu/home/services/publications/

36


http://ecahe.eu/home/services/publications/

http://ecahe.eu/home/services/publications/


http://www.unideusto.org/tuningeu/

37




http://www.unideusto.org/tuningeu/images/stories/Publications/PHYSICS_FOR_WEBSITE.pdf

http://www.unideusto.org/tuningeu/images/stories/Publications/OCCUPATIONAL_THERAPY_FOR_WEBSITE.pdf

http://www.unideusto.org/tuningeu/images/stories/Publications/EUROPEAN_STUDIES_FOR_WEBSITE.pdf

http://www.unideusto.org/tuningeu/images/stories/Publications/CHEMISTRY_FOR_WEBSITE.pdf

Discipline-specific LO PROCESS : VLUHR+NVAO+AKOV

QAA‟s subject benchmark statements

38


After the meeting they realized the new manager

suffered from severe adaptive difficulty

after the meeting they realized the new manager

suffered from severe adaptive difficulty


To which segment do you belong after this PLE?

The realisation of differences is a good start!

Bring together what is broken


THANKS www.nvao.net +31 70 312 2300 [email protected] [email protected]

41

concluding reflections on guidelines for the use of (achieved… lucien... · curriculum iqa nqf...

Documents