miguel martins, senior lecturer – marketing department elearning celebration 22 july 2007...

Miguel Martins, Senior Lecturer – Marketing Department

eLearning Celebration22 July 2007

Improving performance on a Marketing

module through the use of

Formative Computer-Assisted Assessment



This paper was first presented at the

2nd European Conference onThe First Year Experience

Gothenburg University, 9th - 11th May 2007



Work in progress



The delivery mode at MK1004 – Marketing Principles follows very much the traditional lecture model.

Students have two in-class examinations.

The first one is made of two parts: a MCQ (Multiple Choice Questions) test, consisting of 30 questions, and five open questions.

The MCQ test accounts for 25% of the final student grade for this module.

Introduction



There are two main reasons that pressed the author to conduct this research.

First - the University of Wolverhampton is moving to a more virtual and self-learning environment through the use of WOLF. Second - the week before the first examination students are presented with an in-class formative MCQ test.

Introduction



There are a number of ways in which a VLE can be used to assess students. The most widespread one is computer-assisted-assessment (CAA).

One of the tools supplied by a VLE is self-assessment, through the use of a databank of questions, linked to automated marking and instant feedback; other frequent tool is the tracking of the students’ use of materials within the VLE. Both facilities are present in WOLF.

Literature review



The correlation between the use of formative CAA and exam results have been researched by several authors and a positive link has been found.

However, according to Clarke et al. (2004) research into the use of online formative MCQ still remains scarce and more studies are needed.

Literature review



Evaluation Methodology

Reeves (1993) suggest that using a combination of quantitative and qualitative approaches is more suitable to assess learning supported by a VLE.

In order to improve the validity of our study a number of sources – access logs, survey (with close and open end questions), automated tracking and analysis of in-class test results was used.




Access logs give us the opportunity to find out about accessing patterns of each student, in relation to date, hour and number of times a MCQ test was accessed and number of times it was completed.

Automated tracking of results allowed the researcher to find out about the marks gained for each test.



Table 4 – Tracking for first and last day of the online MCQ tests

Total of 3512 page views and 1014 interactions




Analysis of in-class test results allow for historical comparisons with the three last cohorts of students (2005-06 Semester 1 & 2 and 2006-07 Semester 1) as well as within this specific cohort - by comparing students who did take the online MCQ tests with those that did not take them and within the group that did take the tests, by comparing formative grades with summative ones.




Finally, surveys were chosen as a way to get students’ opinions regarding the usefulness of the online MCQ tasks towards their learning and final results.

Questionnaire:- 38 closed questions and 5 open questions- paper-based and completed in-class after the test- applied to 50% of the students enrolled in the module- a total of 114 questionnaires were received.



Qualitative analysis

The overall conclusion is that the online MCQ tests were a valuable learning resource for many students, helping them prepare for the summative in-class test.

“the online MCQ tests motivated me to learn and to keep trying” (86%)“the online MCQ tests forced me to study more” (81%)“the marks helped me to access how I was doing on my learning process” (80%)“as a method of learning I enjoyed doing the online MCQ tests” (76%)“the online MCQ tests were useful in revising the content of the module” (76%)




Question 3 asked the students if they have enjoyed the online MCQ tests as a method of learning. Two students who did not access the programme agreed with the questions and one student who accessed it just once and spent only 4:04 minutes strongly agreed.

One may ask why he didn’t spend more time in such an enjoyable activity!




Question 4 asked the students if the online MCQ tests were well organised and structured. One student who accessed the task only once and for just 15 seconds did agree.

We are delight to see that in just 15 seconds the student was still able to appreciate the organisation

and structure of the task.

Relying on a questionnaire alone seems, from these findings, to be highly problematic



Quantitative analysis

The last three cohorts had average grades of 16.58 (sd 3.75), 18.49 (sd 4.1), and 18.04 (sd 4.53).

The current cohort had an average of 16.94 (sd 3.83)*.

*sample of 64 students

But



Quantitative analysis

If we take into consideration only the students that fully used the online tool (12 tests ) then the

average goes up to 19.67 (sd 4.5).

AVERAGE STD Nb Students (%)

HIGH (9,10,11,12)

19.31 3.99 20.3%

MEDIUM (5,6,7,8)

17.14 4.19 21.9%

LOW

(1,2,3,4)16.25 2.95 37.5%

NIL (0) 15.62 4.05 20.3%



0.00 0.20 0.40 0.60 0.80 1.00

Average Result on formative tests

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

Fin

al R

esu

lts

Su

mm

ativ

e te

st

R Sq Linear = 0.304

236 observations



Model

Nb of page views

Nb of tests taken

Average timespent per test

Total timespent on tests

CAA results Summative result

R² = .002

R² = .103

R² = .109

R² = .097

β = .046*

β = .321

β = .330

β = .312

R² = .304β = .551

R² = .002

β = .041*

R² = .066β = .257

R² = .021β = .144*

R² = .291β = .539



Conclusion and further research



If you want to receive a copy of the final report please e-

mail me at:

[email protected]

Thank you

Work in progress

miguel martins, senior lecturer – marketing department elearning celebration 22 july 2007...

Documents

miguel martins

marketing module

assessment slide

introduction slide

progress slide

mk1004 marketing principles

literature review slide

use of formative computer