The Use of ICT for the Assessment

of Key Competences

Irish Presidency Conference

Better Assessment and Evaluation to Improve Teaching and Learning

19-20 March 2013

Christine Redecker Yves Punie

IPTS: Part of Joint Research Centre of

the EC: 7 Research Institutes across

Europe

Mission: “to provide customer-driven

support to the EU policy-making

process by developing science-based

responses to policy challenges that have

both a socio-economic as well as a

scientific/technological dimension”

IPTS- Who we are

Future Learning Spaces in 2020

2006 2007 2008 2009 2010 2011 2012 2013

Learning 2.0 in formal E&T

Learning 2.0 in informal settings

Creativity & Innovation in

schools in EU

eLearning in the EU10 MS

- Digital Competence

- Active Ageing & Learning

Tell-Net Teacher Networking

Review ICT

Impact

IPTS Studies on ICT for Learning

The Future of Learning

ICT for the Assessment of Key Competences

Digital Competence

Mainstreaming Creative Classrooms

Open Education in Europe

1to1 Learning initiatives in Europe

Mapping Technologies for Learning

DC Follow up

Where we stand

Multiple Choice

Short Answer

Generation 1

Automated administration

and/or scoring of

conventional tests

Adaptive Testing

Generation 2 Generation 4

Intelligent Tutors

Automated Feedback

Personalised

feedback and

tutoring

Learning 2.0

Generation 3

Continuous

integrated

assessment

Data Mining & Analysis

Behavioural tracking

Generation Re-invention Transformative strategy: use technology to

change test formats

Peer assessment

Collaborative multimedia learning

environments

Online collaboration Virtual laboratories

Games

Simulations

Virtual Worlds

Replication of complex

real life situations within a

test

Bu

nd

erso

n, In

ou

ye

& O

lsen

(1

98

9).

Th

e fo

ur

gen

erat

ion

s o

f co

mp

ute

rize

d e

du

cati

on

al

mea

sure

men

t. I

n R

. L

. L

inn

(E

d.)

, E

du

cati

on

al m

easu

rem

ent

(Th

ird

ed

., p

p. 3

67

-40

7).

New

Yo

rk:

Mac

mil

lan

Ben

net

t, R

. E

. (2

01

0).

Tec

hn

olo

gy f

or

Lar

ge-

Sca

le A

sses

smen

t. I

n P

. P

eter

son

, E

. B

aker

& B

.

McG

aw (

Ed

s.),

In

tern

ati

on

al E

ncy

clo

ped

ia o

f E

du

cati

on

(3

rd e

d.,

Vo

l. 8

, p

p. 4

8-5

5).

fo

rd:

Els

evie

r.

1990 1995 2000 2005 2010 2015 2020 2025 Efficient testing Personalised learning

Learning Analytics

Computer-Based Assessment (CBA) Technology-enhanced learning

Current use of ICT in assessment in schools

Central recommendations on the use of ICT in pupil assessment in compulsory education in primary and general

secondary education (ISCED 1, 2, 3), 2009/10

Source: Eurydice (2011). Key Data on Learning and Innovation through ICT at School in Europe 2011, Education, Audiovisual and Culture

Executive Agency.

There is scope for improvement

1st Generation CBA: Problem-Solving

http://www.worldclassarena.org/v5/flash/09_y

ear_old/backtofront.swf

World Class Arena Sample Test: Problem

Solving (9 years)

Examples

http://www.worldclassarena.org/v5/flash/13

_year_old/pyramids.swf

Problem Solving 13 years

Generation Re-Invention: Problem-Solving

Examples

1st Generation CBA: Situational Judgement

EPSO sample situational judgement test questions; http://europa.eu/epso/discover/prepa_test/sample_test/index_en.htm

Examples

Generation Re-Invention: e-tray exercise

EP

SO

sam

ple

ass

essm

ent

centr

e e-t

ray e

xer

cise

htt

p:/

/euro

pa.

eu/e

pso

/dis

cover

/pre

pa_

test

/sam

ple

_te

st/i

nd

ex_

en.h

tm

Examples

Future eAssessment Strategies

Multiple Choice

Short Answer

Generation 1

Automated administration

and/or scoring of

conventional tests

Adaptive Testing

Generation 2

Generation 4

Intelligent Tutors

Automated Feedback Personalised

feedback and

tutoring

Learning 2.0

Generation 3

Continuous

integrated

assessment

Data Mining & Analysis

Behavioural tracking

Generation Re-invention Transformative strategy: use technology to

change test formats

Computer-Based Assessment (CBA) Technology-enhanced learning

Peer assessment

Collaborative multimedia learning

environments

Online collaboration Virtual laboratories

Games

Simulations

Virtual Worlds Replication of complex

real life situations within a

test

Bu

nd

erso

n, In

ou

ye

& O

lsen

(1

98

9).

Th

e fo

ur

gen

erat

ion

s o

f co

mp

ute

rize

d e

du

cati

on

al

mea

sure

men

t. I

n R

. L

. L

inn

(E

d.)

, E

du

cati

on

al m

easu

rem

ent

(Th

ird

ed

., p

p. 3

67

-40

7).

New

Yo

rk:

Mac

mil

lan

Ben

net

t, R

. E

. (2

01

0).

Tec

hn

olo

gy f

or

Lar

ge-

Sca

le A

sses

smen

t. I

n P

. P

eter

son

, E

. B

aker

& B

.

McG

aw (

Ed

s.),

In

tern

ati

on

al E

ncy

clo

ped

ia o

f E

du

cati

on

(3

rd e

d.,

Vo

l. 8

, p

p. 4

8-5

5).

fo

rd:

Els

evie

r.

Learning Analytics

1990 1995 2000 2005 2010 2015 2020 2025 Efficient testing Personalised learning

Towards Generation 3: Datamining

Signals http://www.itap.purdue.edu/tlt/sig

nals/signals_final/index.htm

Similar: GPS at Northern Arizona

University

http://resourceconnect.nau.edu/gr

ade-performance-status.asp

Examples

Towards Generation 3: Behaviour Tracking

Example: SNAPP

SNAPP is a software tool that

allows users to visualize the

network of interactions

resulting from discussion

forum posts and replies.

Teachers can rapidly identify

patterns of user behaviour.

SNAPP can extract all user

interactions from various

commercial and open source

learning management

systems, such as BlackBoard

and Moodle.

Examples

Towards Gen 4: Intelligent Tutoring Systems

GnuTutor (http://gnututor.com/)

OpenSource version of AutoTutor https://www.youtube.com/watch?v=k6HDEEnYuIA&feature=player_embedded

AutoTutor https://www.youtube.com/watch?v=aPcoZPjL2G8

Examples

Generation 1 Generation 2

Generation 4

Personalised

feedback and

tutoring

Learning 2.0

Generation 3

Continuous

integrated

assessment

Generation Re-invention

Computer-Based Assessment (CBA) Technology-enhanced learning

Collaborative multimedia learning

environments

Bu

nd

erso

n, In

ou

ye

& O

lsen

(1

98

9).

Th

e fo

ur

gen

erat

ion

s o

f co

mp

ute

rize

d e

du

cati

on

al

mea

sure

men

t. I

n R

. L

. L

inn

(E

d.)

, E

du

cati

on

al m

easu

rem

ent

(Th

ird

ed

., p

p. 3

67

-40

7).

New

Yo

rk:

Mac

mil

lan

Ben

net

t, R

. E

. (2

01

0).

Tec

hn

olo

gy f

or

Lar

ge-

Sca

le A

sses

smen

t. I

n P

. P

eter

son

, E

. B

aker

& B

.

McG

aw (

Ed

s.),

In

tern

ati

on

al E

ncy

clo

ped

ia o

f E

du

cati

on

(3

rd e

d.,

Vo

l. 8

, p

p. 4

8-5

5).

fo

rd:

Els

evie

r.

- mainstream use for high stakes testing

- further development of innovative test formats

- increased ease of use at school level

- trend towards automated scoring

→ cultural change:

increased acceptance of CBA

Automated administration

and/or scoring of

conventional tests

Adaptive Testing

Replication of complex real life

situations within a test

- Multimedia collaborative learning

environments support key competence

development.

- Peer and portfolio assessment evolve

naturally.

-Data mining and analysis will make

integrated assessment and tutoring possible

in the future.

We are in a transitional period: Moving from the “Testing

Paradigm” to the (future) “Tutoring Paradigm” in which explicit testing could become obsolete

Technological trends

1990 1995 2000 2005 2010 2015 2020 2025 Efficient testing Personalised learning

Key Questions

1. Which ICT tools, formats, environments, devices etc. are most suitable for the assessment of which Key Competences?

2. How can ICT be used to support competence-based assessment strategies that adequately capture the skills and attitudes dimensions of all eight Key Competences and their transversal nature?

Assessing Key Competences

F = Formative Assessment

D = Diagnostic Assessment

S = Summative Assessment

1 Communication in the mother tongue DS FD FS

2 Communication in foreign languages DS FS

3a Mathematical competence DS FD FDS FDS

3b Basic competences in science and technology FS FDS

4 Digital competence FDS

5 Learning to learn FS FDS

6 Social and civic competences FS

7 Sense of initiative and entrepreneurship FS

8 Cultural awareness and expression FS FS

CB

A

ePo

rtfo

lio

s

Pee

r

Ass

essm

ent

Sel

f

Ass

essm

ent

Qu

izze

s /

sim

ple

g

am

es

Vir

tua

l w

orl

ds

ga

mes

Sim

ula

tion

s

Inte

llig

ent

Tu

tors

etc

.

Focus Areas

•PART 1

•INSPIRATION

Mother Tongue Literacy

1 Communication in the mother tongue CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

Matching spoken and written language

http://www.familylearning.org.uk/sight_word_games.html

Foreign language learning

2 Communication in foreign languages CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

htt

p:/

/ww

w.n

ickjr

.de/

gam

es/1

07

-sw

iper

s-bu

chst

abie

r-buch

Mathematical games 3a Mathematical competence CBA (DS) Quizzes / simple games (FD) Simulations (FDS) Inteligent tutoring systems (FDS)

Mathematical games using simulation for secondary school education

Source: http://hotmath.com/games.html.

5 Learning to Learn CBA (F) Peer Assessment (FS) Self Assessment (FS)

http://spark.uts.edu.au/

Example:

SparkPlus

Peer Assessment for Collaborative Work

• http://www.youtube.com/watch?v=Tp8v2UBbmN4

www.moodle.org

Moodle is a free web

application that educators can

use to create effective online

learning sites.

Moodle Statistics Registered sites: 77,213 Countries: 229 Courses: 7,116,466 Users: 66,349,750 Teachers: 1,292,731 Enrolments: 56,306,274 Forum posts: 118,413,272 Resources: 63,891,942 Quiz questions: 147,057,434

Peer Assessment using moodle

ePortfolios

1 Communication in the mother tongue CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

2 Communication in foreign languages CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

8 Cultural Awareness and expression ePortfolios (FS) Virtual worlds / games (FS) http

://sites.google.co

m/site/ep

ortfo

lioap

ps/o

verv

iew/m

od

el-k-1

2

ePortfolios

ePortfolios

1 Communication in the mother tongue CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

2 Communication in foreign languages CBA (DS) Quizzes / simple games (FD) ePortfolios (FS)

8 Cultural Awareness and expression ePortfolios (FS) Virtual worlds / games (FS)

http

://sites.go

ogle.co

m/site/ep

ortfo

lioap

ps/o

verv

iew/lev

els

Level 1 - ePortfolio as Storage

Collection:

Creating the Digital Archive (regularly – weekly/monthly)

A Focus on Contents & Digital Conversion

Level 2 - ePortfolio as Workspace

Collection + Reflection:

Immediate Reflection on Learning & Artifacts in Collection (regularly)

A Focus on Process & Documentation of Learning

Level 3 - ePortfolio as Showcase

Selection/Reflection/Direction (each semester? End of year?)

A Focus on Product & Documentation of Achievement

→ A (formative/summative) assessment approach on the basis of artefacts

→ Focus on production and reflection

A game approach to arts

8 Cultural Awareness and expression ePortfolios (FS) Virtual worlds / games (FS)

http://pheon.org; http://pheon.org/intro.html;

http://americanart.si.edu/multimedia/games/pheon.cfm.

Pheon is a multimedia scavenger hunt that tests players’ ingenuity through

exploring the collection of the Smithsonian American Art Museum, making

objects, and texting through the galleries

ePortfolios meet Virtual reality

Science: Molecular Workbench

3b Basic competences in science and technology CBA (S) Virtual Worlds / games (FS) Simulations (FDS)

http://mw2.concord.org/public/part2/diffat/page3.cml

3a Mathematical competence CBA (DS) Quizzes / simple games (FD) Simulations (FDS) Inteligent tutoring systems (FDS)

Simulations for mathematics

Sample task from US future New State Tests

(2014/15)

http://www.k12center.org/rsc/pdf/a-sea-change-in-assessment-letter-size.pdf

Cognitive Tutor (Carnegie)

3a Mathematical competence CBA (DS) Quizzes / simple games (FD) Simulations (FDS) Inteligent tutoring systems (FDS)

http://www.youtube.com/watch?v=RR4HPrGlUz

8&feature=related

Science: Quest Atlantis

3b Basic competences in science and technology CBA (S) Virtual Worlds / games (FS) Simulations (FDS)

Example: Quest Alantis Quest Atlantis (QA) is an international learning and teaching project that uses a 3D multi-user environment to immerse children, ages 9-16, in educational tasks. QA combines strategies used in the commercial gaming environment with lessons from educational research on learning and motivation.

Citizenship 6 Social and civic competences CBA (DS) Peer/Self Assessment (FDS) Virtual Games / Simulations (FDS)

Educational game, designed to help teach concepts in citizenship, geography, and media.

Developed by Serious Games International, it has detailed lesson plans and assignments for students.

Example: Child labour in

Bangladesh:

http://www.youtube.com/wat

ch?v=mf9Ju86IVYw

Global Conflicts http://www.globalconflicts.eu/

Examples

•PART 2

•CRITICAL INSPECTION

Digital Competence

4 Digital competence CBA (S) Simulations (FDS)

Internet and Communication Core Certification (IC3) exam sample question

http://www.certiport.com/Portal/common/htmllibrary/Sales-Video-Demos/demo_ic3.html

→ Focus on knowledge

CBA for Digital Competence

Digital Competence: IC3

4 Digital competence CBA (S) Simulations (FDS)

Inte

rnet

and C

om

munic

atio

n C

ore

Cer

tifi

cati

on (

IC3)

exam

sam

ple

ques

tion

htt

p:/

/ww

w.c

erti

port

.com

/Port

al/c

om

mon/h

tmll

ibra

ry/S

ales

-

Vid

eo-D

emos/

dem

o_ic

3.h

tml

→ Focus on operational skills

CBA for Digital Competence

Entrepreneurship Questionnaire

7 Sense of initiative and entrepreneurship CBA (DS) Peer/Self Assessment (FDS) Virtual Games / Simulations (FDS)

? https://www.careeringahead.

com.au/survey/Entrepreneurs

hip/Generic/questions.asp

→ Focus on personality/disposition as a static fact

Entrepreneurship Questionnaire

Conclusions

ICT offers powerful, engaging assessment formats, in particular:

CBA ePortfolios Self / Peer

Assessment Quizzes /

simple games Virtual worlds/ games Simulations

Basic literacy

(foreign/mother tongue)

Basic mathematical

competence

Learning

to Learn

Creative writing

(foreign/mother tongue)

Cultural awareness and

expression

Scientific competence

Digital

competence

Social and civic

competences

Educational software

Mathematical

competence

CBA lends itself to simplified, knowledge-based assessment

→ Pedagogy comes first! Careful selection of question formats

ICT Assessment approaches are underdeveloped for:

Learning to Learn Sense of initiative and entrepreneurship Social and civic competences

Caveat:

The Paradigm Shift

Technology

1990 1995 2000 2005 2010 2015 2020 2025

Focus on: Efficient testing Focus on: Personalised learning

Era of Computer-Based Assessment (CBA) Era of Technology-enhanced learning

Pedagogy Efficient

Testing

Validity of

Scores

Quick

Feedback

Authentic

Contexts

CBA

(Gen.1)

Adaptive

Tests

(Gen.2)

Automatic

Scoring

Gen. Re-

Invention

Learning

2.0

Immersive

Learning

Learning

Analytics

Intelligent

Tutors

? ? ? ?

→ Primacy of Pedagogy:

Pedagogical guidelines for intergrated assessment

• Thank you very much for your attention!

• Christine Redecker

• Yves Punie

• Christine.Redecker@ec.europa.eu • Yves.Punie@ec.europa.eu

http://ipts.jrc.ec.europa.eu/publications/pub.cfm?id=5719

further information ……

http://is.jrc.ec.europa.eu

Newsletters: http://is.jrc.es/pages/newsletter.html

Publications: http://is.jrc.ec.europa.eu/pages/Publications.html

Contact: jrc-ipts-is-NEWS@ec.europa.eu