PARTICIPATORY AND CONSTRUCTIVISTIC

EVALUATION OF ELEARNING QUALITY

A CASE STUDYPatrizia Ghislandi – Juliana Raffaghelli

University of Trento

5th International Conference of Education, Research and Innovation

Madrid, 19-21 November 2012

A) eLearning evolution and the problem of evaluating quality

From Standards to a culture of Quality

B) Reconceptualizing Quality of eLearning

Fourth Generation Evaluation: understanding the epistemological approach to evaluation of eLearning quality.

The context of the research: evaluating quality inside the eLFO12 Learning Community

C) A Case Study

The Methodological Approach

Tools for a participatory evaluation of eLearning quality

Results

D) Conclusions and Debate!

eLearning evolution and the problem of evaluating quality

Traditional eLearning: delivery of online contents

Evolution: Open, networked Learning

…The evolution of quality has been influenced by industrial models

○ Predefined Standards○ Measurements on finished

products○ Quantitative Evaluation○ Intangible aspects (relation,

communication) in a second place for the evaluation tools

○ The need to accountable to external referees

○ The gap between research and educational practice

○ A vision of evaluation as certification, objectivistic quantitative paradigm

4

Reconceptualizing Quality of eLearning

Diverse Cultures of Quality are underpinned by diverse values: Exceptional/Original: the

value is on the uniqueness

Distinctiveness: not for all

Excellence: The highest levels of performance

Fitness for Purpose: doing what has been planned

Inclusive: all people can participate 5

Reconceptualizing Quality of eLearning

6

Q UNESCO Quality for all

EFQUEL –UE-

SLOAN-C MODEL –USA-

CENTRO VIRTUAL PARA EL DESARROLLO DE ESTÁNDARES DE CALIDAD PARA LA EDUCACIÓN SUPERIOR A DISTANCIA EN AMÉRICA LATINA Y EL CARIBE

ISO/IEC 19796

Sistematic ApproachesAccess?Excellence?InnovationInclusiveness?

Who says what QeLHE is?

7

4- Diverse Meanings

PedagogicalOrganizationalEconomicalTechnological

Methodological

1-Diverse PerspectivesThe teacherThe StudentThe institutionThe external evaluators

3- Diverse times of intervention for the analysis

in itinereex anteex post

2- Diverse Methods of Analysisbenchmarking guidelinesstandard

Qualitative EvaluationQuantitative Evaluation

Q

Quality is a Complex issue

5-Diverse Levels of Analysis

IndividualGroupinstitutionalSocio-cultural

Reconceptualizing Quality of eLearning

Quality is not an intrinsic, universal value

It is very much about the methodology of

evaluation, And the substantial

epistemological principles and values underlying the process of evaluation

Fourth Generation Evaluation (Guba & Lincoln, 1989) Understanding the epistemological approach to evaluation of

eLearning quality

Four generations of Evaluation:

1) Measurement2) Observation3) Judgement

There is a wise, well informed, objective, aseptic, external EVALUATOR

There are unaware, badly informed, too emotionally engaged EVALUATED PEOPLE (and the processes/products coming from them

4) Joint Reflection

and Transformation

There is an expert, open to share, committed EVALUATOR

There are committed, willing to learn, EVALUATED

Understanding the epistemological approach to evaluation of eLearning quality

Four generations of Evaluation (Guba&Lincoln, 1989)

1) Measurement 2) Observation 3) Judgement

There is a wise, well informed, objective, aseptic, external EVALUATOR

There are unaware, badly informed, too emotionally engaged EVALUATED

4) Joint Reflection

and Transformation

There is an expert, open to share, committed EVALUATOR

There are committed, willing to learn, EVALUATED

Quantitative Methods

External Evaluation

Qualitative Methods

Self- Peer Evaluation

External, agreed, frameworks of reference (EFQUEL, 2007)

A case studyThe eLFO12 Learning Community

Defining our Case Study

The decisive factor in defining a study a case study is the choice of the individual unit of study and the setting of its boundaries (…)the individual unit may be studied in a number of ways, for instance qualitatively or quantitatively, analytically or hermeneutically, or by mixed methods. This is not decisive for whether it is a case study or not; the demarcation of the unit’s boundaries is. (Flyvbjerg, 2011, p. 301)

Defining our Case Study

The decisive factor in defining a study a case study is the choice of the individual unit of study and the setting of its boundaries (…)the individual unit may be studied in a number of ways, for instance qualitatively or quantitatively, analytically or hermeneutically, or by mixed methods. This is not decisive for whether it is a case study or not; the demarcation of the unit’s boundaries is. (Flyvbjerg, 2011, p. 301)

eLFO12: eLearning course for training the trainers of operators of social and health care

a) the educational process and strategies to improve it b) an eLearning setting

c) a professional learning community composed by operators of health and social services willing to improve their skills as trainers

Moodle platform interoperable with Web 2.0 tools

Socio-constructivist approach to the adoption of educational Technologies

Self-Peer Evaluation:

Trainers’ LogAdAstra RubricsInterviews

Committement University of Trento –UNITN- /Direction of Health Care (Province of Trento) –PAT-

The Process

Rubrics

The learner’s log-book

exploring and

constructing eLearning

Quality

Interviews

•The log-book as part of learning activities to collect live reflections on the learning process.

•A suite of rubrics as meta-evaluation instruments to support reflection on teaching and learning quality

•Interviews with participants to deepen on the emerging perceptions, ideas and reflections in the former instruments.

Becoming aware of values and practices for quality

Making visible the invisible Transform

The Log-book Having introduced the course as a “socio-

constructivist” oriented, the tutors asked the students to be active and collaborative ( a quality value)

it was registered during the initial phases of the course, an important amount of discontent with the approach: to much “chaos”

A lesson on the pedagogy of socio-constructivist approach

Reflections and discussion Becoming more participant and enganged: aware

of the importance of collaboration

The AdAstra Rubrics A set of rubrics, elaborated by the University of Trento research unit

(Ghislandi & Pedroni, 2011), on the basis of SLOAN-C MODEL

Sloan-C pillars and the connected organization of the adASTRA rubrics .

Sloan-C Pillars Rubric adASTRA

Learning effectivenessRubric Analysis, Design,

Syllabus, Community of Practice, Screencast, Portfolio

Student satisfaction Rubric Feedback Students

Faculty satisfaction Rubric Feedback Teacher

Cost effectiveness Not developed

Access Rubric Accessibility

Table 1 - Sloan-C pillars and the connected organization of the adASTRA rubrics .

The AdAstra Rubrics Dimensions of Quality Analized

Dimension DescriptionOrganization Analyses how the several course activities planned and scheduled

are implemented consistently.Resources Analyses how the resources offered for learning are considered by

learners, in the sense of completeness and coherence. Teaching Methods and Strategies

Analyses how the teaching methods and strategies implemented are considered in the sense of effectiveness (the achievement learning goals as well as to maintain learners’ motivation).

Comunication Analyses how the several synchronous and asynchronous instruments of communication are adopted (the usage is coherent with the learning goals and requirements)

Collaboration Analyses how the students interact between them to achieve learning goals

Technologies Analyses how the technological arrays are used, in the sense of the facilitation of learning.

Assessment Analyses how the assessment is implemented, particularly taking into account the coherence between the type of assessment activities and what the course offered to learners.

. The rubrics where implemented through 2 meetings and across 2 months of the course (1 month devoted to the learning design and 1 month to the delivery of the exercise module of peer learning), one at the beginning

and another at a closing phase, being the researchers available to discuss with teachers and students if doubts arisen on the concepts driven.

Interviews: reflecting on quality evaluationParticipatory meta-evaluation: the rubrics as instruments for the reflection on course quality (excerpt on 1 of 4 dimensions)

Dimension Results from Interviews (*)

(*) the students transcriptions in Italian have been translated into English by the research team

Coherence

“It is really important to have something written that helps you to avoid mistakes when planning a course, and also to reflect on what had been done previously (…) to solve problems and to make changes” [eL5]

“the level of detail has to be correlated with the dimension of course quality that you are to analyze, and this was perfectly represented into the rubrics” [eL4]

“Diversely from the student side rubrics, the teacher side rubrics lack of items that recall how the teacher has reacted to critical issues, how did she feel with regard to those critical issues, so the teachers subjective perspective is missed somehow” [eL3]

“It could have been introduced some question regarding the methodology adopted in the design phase or even better questions that allowe to establish if the teacher has followed coherently a pedagogical approach” [eL1]

“The student rubric misses something regarding the relationships and the collaborative work…or this should be more evident….these represents a positive factor for the groups and single learning don’t they? [eL5].

Discussion & Conclusions

eLFO12 Case Study: a participatory approach to the evaluation of Quality

Discussion of Results (1) The three moments of evaluation showed how the

evaluation process worked progressively in addressing understanding and learning towards the values/concepts of the ELFO12 group

The breakthrough in this process was the moment in which the learners were given the opportunity to revisit (self-evaluate) their own learning in the context of socio-constructivist theoretical framework

The rubrics where considered coherent, complete, effective, though rather less efficient. But this meta-evaluation of the tools was given in the context of utility and use of the same: a socio-constructivist and technologically mediated learning activity/course.

Discussion of Results (2)

The quality was evaluated not in the sense of external parameters of performance but in terms of the coherence and authenticity of the course learning model and the learners awareness and engagement within it. It is crucial to point out that both steps or “moments” of quality evaluation.

CONCLUSIONS The selection of qualitative methods,

a phenomenological approach based on narrative self-evaluation, peer-evaluation and meta-evaluation,

emphasizes the interest on processes and on the empowerment of learners AS COMMITTED EVALUATED This logic studies the topic within its context, uses an emerging design

that accounts for reality as subjective and multiple, lessen the distance between “official” evaluators and participants (Denzin & Lincoln, 2011).

As a result,

the evaluation process encompasses a transformational (participatory/innovative)

opportunity for the engaged individuals/ institutions.(Creswell, 2007; Mertens, 2009).

CONCLUSIONS this transformational perspective is the kernel of

a quality learning culture:

a human group that take part of a learning experience as a deep, reflective experience, connected to the own professional/ personal identity ○ For which purposes do I learn? What can I do with this

learning?not just for accomplishing activities, recalling

information, and obtaining credentials (course diploma).

We call this become insiders of the culture of quality.

Thank you for your interest!

For communications:

patrizia.ghislandi@unitn.it