evaluation of the pedagogical aspects of the project

Evaluation of the Pedagogical Aspects of the Project

Contents of the presentation

• Some information about the participating students.

• Objectives of the evaluation – presentation of some results from the questionnaires.

• Critical Thinking

• Motivational and Self-Regulated Learning

• Interest

• Collaborative Learning

• Technical Skills

• Some general guidelines for the teachers.

General Information about the Students

School N Mean Age Std. Deviation

EA (Greece) 27 14.15 0.36

VHS (Denmark) 18 18.00 1.37

BGS (Austria) 12 16.58 0.51

FVS (Germany) 16 16.31 0.48

Total 73 15.97 1.72

The mean ages (in years) of the participating students are:

F(3, 69) = 93.7, p< .001


The students were asked whether they had participated in a European project in the past.

01020

30405060

7080

EA VHS BGS FVS Total

yes

no

Total

School Yes No Total

EA (Gr) 7 20 27

VHS (Dk) 1 16 17

BGS (Au) 1 11 12

FVS (De) 0 15 15

Total 9 62 71

x2(3) = 7.37, p= .061, ns


The students were asked whether they have (or had) communication with students from foreign countries

010

2030

4050

6070

80

EA VHS BGS FVS Total

yes

no

Total

School Yes No Total

EA (Gr) 12 15 27

VHS (Dk) 1 15 16

BGS (Au) 0 12 12

FVS (De) 2 13 15

Total 15 55 70

x2(3) = 14.5, p= .002

General objectives of the evaluation

According to the general objectives of the WebTV project, the pedagogical framework was designed to support activities and processes that are critical for facilitating intentional and thoughtful learning.

Guided through these objectives, the evaluation will focus mainly on the following aspects:

Critical thinking: Critical thinking and inquiry learning are skills essential for the engagement of the students in such a demanding project as the Web TV.

Critical thinking was checked by the items of some scales in questionnaire 5 (scale 4 - Critical Thinking, scale 5 - Metacognitive Self-Regulation).

Some notes about Questionnaires 2, 3, 4, and 5

Before presenting the results from the questionnaires I must mention that:

Each scale of the questionnaires was checked by three items the least.

All single items of the above questionnaires are rated in a scale from 1 to 7.

A reliability analysis test was conducted in each domain-specific group of items to determine whether some of the items should be excluded from the statistical analysis.

The results from the statistical analysis refer to the items that remained after the reliability tests.

Learning Strategies (Q. 5)

Cognitive and Metacognitive Strategies: Critical Thinking (scale 4)

Critical thinking refers to the degree to which students report applying previous knowledge to new situations in order to solve problems, reach decisions, or make evaluations.

Cognitive and Metacognitive Strategies: Critical Thinking

School N Mean St. Dev.

EA (Greece) 26 4,55 1,08

VHS (Denmark) 18 4,30 0,54

BGS (Austria) 12 4,22 0,96

FVS (Germany) 16 4,52 1,20

Total 72 4,42 0,97

Alpha = .6908 F(3, 68) = .46, p= .713, ns

Cognitive and Metacognitive Strategies: Metacognitive Self-Regulation (scale 5)

Metacognition refers to awareness, knowledge, and control of cognition. Questionnaire 5 is focused on the control and self-regulation aspects of metacognition, not the knowledge aspect.


Cognitive and Metacognitive Strategies: Metacognitive Self-Regulation

School N Mean St. Dev.

EA (Greece) 26 4,43 0,66

VHS (Denmark) 18 4,33 0,46

BGS (Austria) 12 3,90 0,74

FVS (Germany) 16 4,23 1,25

Total 72 4,27 0,81

Alpha = .7322 F(3, 68) = 1.22, p= .308, ns

Motivational and Self-regulated learning: It is important for teachers to facilitate the active participation of the students to the multi-faceted topics used for the Web TV activities in ways that make it possible for the students to develop self-monitoring and regulation skills.

Motivational and self-regulated skills of the students were checked by almost all the scales of questionnaires 5 and 3.

Questionnaire 3 (Motivational beliefs)

This is a modified version of part A of the MSLQ (Motivated Strategies for Learning Questionnaire) developed by Paul Pintrich. This questionnaire contains 26 single items which refer to five scales:

1) Intrinsic Goal Orientation,

2) Extrinsic Goal Orientation,

3) Task Value,

4) Control of Learning Beliefs and

5) Self-Efficacy for Learning and Performance.

Motivational Beliefs (Q. 3) Value Component: Intrinsic Goal Orientation (scale 1)

Goal orientation refers to the student’s perception of the reasons why he/she is engaging in a learning task. Intrinsic goal orientation concerns the degree to which the student perceives him/herself to be participating in a task for reasons such as challenge, curiosity and mastery.

Value Component: Extrinsic Goal Orientation (scale 2)

Extrinsic goal orientation complements intrinsic goal orientation and concerns the degree to which the student perceives him/herself to be participating in a task for reasons such as grades, rewards, performance, evaluation by others and competition.

Motivational Beliefs (Q. 3)

Value Component: Task Value (scale 3)

Goal orientation refers to the reasons why the student is participating in the task (“Why am I doing this?”). Task value differs from goal orientation in that task value refers to the student’s evaluation of the how interesting, how important and how useful the task is (“What do I think of this task?”).



Int. G. O Ext. G. O Task V.

M S.D M S.D M S.D

EA (Greece) 5,51 0,81 4,81 1,54 5,58 0,92

VHS (Denmark) 4,69 0,90 5,32 0,88 5,00 0,71

BGS (Austria) 4,46 1,19 4,13 1,05 4,56 0,98

FVS (Germany) 4,79 1,13 4,08 1,18 4,58 0,89

Alpha .65 .72 .80

F(3, 69) 4.54 3.71 6.12

Sig .006 .016 .001

Expectancy Component: Control of Learning Beliefs (scale 4)

Control of learning refers to students’ beliefs that their efforts to learn will result in positive outcomes. It concerns the belief that outcomes are contingent on one’s own effort, in contrast to external factors such as the teacher.


Expectancy Component: Self-Efficacy for Learning and Performance (scale 5)

The items comprising this scale assess two aspects of expectancy: expectancy for success and self-efficacy. Expectancy for success refers to performance expectations, and relates specifically to task performance. Self-efficacy is a self-appraisal of one’s ability to master a task. Self-efficacy includes judgments about one’s ability to accomplish a task as well as his/her confidence in his/her skills to perform that task.



Con. Lear. Beliefs Self-Efficacy

M S.D M S.D

EA (Greece) 5,57 1,02 5,29 0,79

VHS (Denmark) 5,19 0,78 4,87 0,68

BGS (Austria) 5,06 1,27 3,98 0,93

FVS (Germany) 4,69 1,15 4,87 1,09

Alpha .59 .85

F(3, 69) 2.50 6.35

Sig .067 ns .001

Questionnaire 5 (Learning Strategies)

This is part B of the MSLQ questionnaire (Motivated Strategies for Learning Questionnaire) developed by Pintrich, R.R., & DeGroot, E.V. (1990). This questionnaire consists of 50 items which refer to 9 domains:

1) Rehearsal,

2) Elaboration,

3) Organization,

4) Critical Thinking,

5) Metacognitive Self-Regulation, 6) Time and Study Environment,

7) Effort Regulation,

8) Peer Learning, and

9) Help Seeking.

Cognitive and Metacognitive Strategies: Rehearsal (scale 1)

Basic rehearsal strategies involve reciting or naming items from a list to be learned. These strategies are best used for simple tasks and activation of information in working memory rather than acquisition of new information in long-term memory. These strategies are assumed to influence the attention and encoding processes, but they do not appear to help students construct internal connections among the information or integrate the information with prior knowledge.


Cognitive and Metacognitive Strategies: Elaboration (scale 2)

Elaboration strategies help students store information into long-term memory by building internal connections between items to be learned. Elaboration strategies include paraphrasing, summarizing, creating analogies, and generative note-taking. These help the learner integrate and connect new information with prior knowledge.


Cognitive and Metacognitive Strategies: Organization (scale 3)

Organization strategies help the learner select the appropriate information and also construct connections among the information to be learned. Organizing is an active, effortful endeavor, and results in the learner being closely involved in the task. This should result in better performance.



Rehearsal Elaboration Organization

M S.D M S.D M S.D

EA (Greece) 4,98 1,04 4,85 1,24 4,83 1,21

VHS (Denmark) 4,21 0,92 4,50 0,56 4,25 0,93

BGS (Austria) 4,56 1,32 4,28 0,85 4,65 1,01

FVS (Germany) 4,00 1,35 4,60 1,18 3,98 1,25

Alpha .72 .77 .69

F(3, 68) 3.02 0.95 2.23

Sig .036 .422 ns .092 ns

Resource Management Strategies: Effort Regulation (scale 7)

Self-regulation also includes students’ ability to control their effort and attention in the face of distractions and uninteresting tasks. Effort management is self-management, and reflects a commitment to completing one’s goals, even when there are difficulties or distractions.


Resource Management Strategies: Time and Study Environment (scale 6)

Besides self-regulation of cognition, students must be able to manage and regulate their time and their study environments. Time management involves scheduling, planning, and managing one’s study time.

Time & Study En. Effort Regulation

M S.D M S.D

EA (Greece) 4,65 1,11 4,35 1,16

VHS (Denmark) 4,21 0,55 4,36 0,64

BGS (Austria) 4,44 0,77 4,56 0,88

FVS (Germany) 4,65 1,04 4,59 1,38

Alpha .71 .65

F(3, 68) 0.97 0.26

Sig .410 ns .857 ns


Resource Management: Help Seeking (scale 9)

Another aspect of the environment that the student must learn to manage is the support of others. This includes both peers and instructors.


Resource Management: Peer Learning (scale 8)

Collaborating with one’s peers has been found to have positive effects on achievement. Dialogue with peers can help a learner clarify things and reach insights he/she may not have attained on his/her own.

Peer Learning Help Seeking

M S.D M S.D

EA (Greece) 3,74 1,36 4,60 1,29

VHS (Denmark) 4,19 0,94 4,58 1,03

BGS (Austria) 4,15 0,86 4,47 0,93

FVS (Germany) 4,27 0,88 4,96 1,13

Alpha .52 .44

F(3, 68) 1.03 0.52

Sig .386 ns .669 ns


Interest:

The participation of the students in such a project emphasizes cognitive reconstructing, by changing the cognitive division of labor between teacher and student. When a student takes responsibility for higher cognitive activities, it enables him or her to go to a deeper level of the learning process.

This kind of meaningful and close relationship towards the learning tasks also may help students increase their interest.

Interest was checked by questionnaire 2.

Questionnaire 2 (Interest)

This questionnaire consists of a few introductory open questions and of 20 single items which refer to four domains:

1) Interest in Collaboration,

2) Interest in Technology,

3) Interest in Specific Tasks, and

4) Anxiety in Specific Tasks.

Interest (Q. 2)

Inter. Collabor. Inter. Technol.

M S.D M S.D

EA (Greece) 6,19 0,70 5,50 1,83

VHS (Denmark) 4,95 1,16 4,48 1,09

BGS (Austria) 4,72 1,00 4,88 1,54

FVS (Germany) 4,82 1,21 5,84 1,08

Alpha .85 .94

F(3, 66) 10.58 2.64

Sig .000 .056 ns

Inter. Spec. Task Anxiety Sp. Task

M S.D M S.D

EA (Greece) 5,76 0,98 3,19 1,33

VHS (Denmark) 4,93 1,31 4,24 1,31

BGS (Austria) 4,55 1,06 3,32 0,99

FVS (Germany) 4,48 1,60 3,18 0,64

Alpha .86 .73

F(3, 66) 4.93 3.19

Sig .004 .029

Interest (Q. 2)

Collaborative learning: The project aims to encourage the collaboration amongst students.

Our aim is to evaluate collaboration not only as a skill but also investigate how collaboration can enhance learning.

Collaboration among students was checked by questionnaire 4.

Questionnaire 4 (Collaborative learning)

This is partly a short version of the “Inventory of Competence and Knowledge Sharing” developed by Kai Hakkarainen. This questionnaire consists of 26 items which refer to six domains:

1) Peer Learning (by Paul Pintrich),

2) Knowledge Sharing in a Class,

3) Knowledge Building Culture,

4) School Support for Knowledge Sharing,

5) Trust in School, and

6) Progressive Problem Solving.

Collaboration and Knowledge Sharing (Q. 4)

Knowledge Sharing in a Class (scale 2)

Our aim is to investigate how strongly one’s class community is committed to knowledge sharing. Are fellow students willing to share all their skills and knowledge?

Knowledge Building Culture (scale 3)

Is a student engaged in producing ideas with his/her fellow students and in collaborative knowledge building?

Peer Lear. Knowl. Shar. Knowl. Buil.

M S.D M S.D M S.D

EA (Greece) 3,96 1,46 4,52 0,91 4,23 1,26

VHS (Denmark) 3,76 0,96 4,24 0,78 3,86 0,93

BGS (Austria) 4,56 1,27 4,26 0,87 4,07 0,76

FVS (Germany) 3,70 1,03 3,64 0,81 3,53 0,55

Alpha .62 .69 .47

F(3, 62) 1.03 3.61 1.73

Sig .385 ns .018 .171 ns


School support for knowledge sharing (scale 4)

To what extent is the school committed to support knowledge sharing?


Trust in School (scale 5)

To what extent does trust prevail in school so as to make knowledge sharing possible?

Progressive Problem Solving (scale 6)

Is the student ready to put a lot of effort and work continuously at the edge of his/her competence?

School Supp. Trust Sch. Prog. Pr. Sol.

M S.D M S.D M S.D

EA (Greece) 4,82 1,40 4,44 1,88 4,35 1,15

VHS (Denmark) 4,18 0,50 5,29 0,76 4,10 0,79

BGS (Austria) 3,84 0,37 5,19 1,40 3,81 1,04

FVS (Germany) 3,66 0,57 5,05 0,78 3,75 1,06

Alpha .79 .80 .43

F(3, 62) 5.49 1.46 1.30

Sig .002 .235 ns .281 ns


Technical skills: The activities which will take place, concerning the use of new technologies, should lead to the acquisition of knowledge related to the procedures needed for a movie production, video skills, video-conferencing, internet search, bulletin board.

The acquisition of technical skills is checked by questionnaire 6.

Questionnaire 6 (Technical skills)

It consists of 18 partly multiple items, respectively of 105 single items. These items are divided into five parts:

a) Description of the Usage of ICT (frequency, duration, contents),

b) Mastery of ICT Tasks,

c) Utility of the ICT,

d) Effects to the Persons and Towards the Relations with Others, and

e) Acquisition of Technical Skills.

Guidelines for the teachers

Besides the questionnaires, it is essential to collect systematically information about the way the Web TV project is set up and develops in each school. For that purpose, it is important the participating teachers to use pre-structured learning logs during the school-level implementations. The participating teachers could write down how the project develops, what goes well and what problems they face during the implementation of the project.


What’s more, qualitative self-reports, made by students and teachers about their experiences during the project, video analysis and summary of information kept in the logs (by teachers) will be used to give additional information to some specific questions, like: • Active participation of the students in the process of acquisition of knowledge,

• Interactions between students, collaborative learning,

• Interaction between students and teachers,

• Impact on teachers,

• Attitude towards ICT.


After the completion of the project, teachers will be asked to answer some questions (questionnaire 7) about their experiences of the Web TV project, such as: 1) what went right or wrong in the activities, 2) the possible reasons for a failure to accomplish a task, 3) about the collaboration of students both among them and with other people (with teachers, experts etc) 4) what attracted the students’ interest, 5) if they observed any changes of the students’ attitude towards courses (perhaps, towards the ones related to the Web TV project), e.g. did the project motivate the students to study "science" or "history" etc, or has it influenced the way they study (their learning strategies)?


During the implementation of the project:

It is very important teachers to encourage students to engage in inquiry learning by selecting sufficiently multi-faceted topics that provide apple opportunities for engaging in deepening inquiry for solving complex and authentic problems.

It also appears to be important to develop scenarios of collaborative movie building and define activities critical from this perspective. It will be essential to work for achieving a shared understanding of the pedagogical value of Web TV.

evaluation of the pedagogical aspects of the project

Documents

linear association9

linear association6

general objectives

studentsthe students

participating students

pearson chisquare7

pearson chisquare14

likelihood ratio8