Pros and Cons of on-Pros and Cons of on-line testing using line testing using

MoodleMoodle

Pros and Cons of on-Pros and Cons of on-line testing using line testing using

MoodleMoodle

A case study of the elementary A case study of the elementary Japanese courseJapanese course

TEL@York2006April 25, 2006

by

Norio Ota, Noriko Yabuki-Soh, Gergana Ivanova (DLLL), Mike Street (IT Consultant, ATS)

PreambleTEL initiatives by the Japanese Section of DLLL • Server-based web course development• Developing interactive instructional materials for self-

study• Developed a distance education course for the

elementary level Japanese language course (tested at Glendon for 4 yrs) using videoconferencing

• Introducing Media Site Live for video-streamed lectures• Developing web-based on-line tests for the elementary

Japanese course assisted by ATS (2005-06)• Developing a distance education course for the

advanced Japanese course for St. Mary’s University in Halifax (to be implemented in September 2006)

Web-based Testing (WBT)Computer-based Testing (CBT) vs. WBTOn-line course delivery software products

– WebCT– Hot Potatoes – Sakai– FLE3 (Future Learning Environment 3)– Moodle (Modular Object-Oriented Dynamic Learning

Environment) Moodle

– Free– Open source – customizable– Non-proprietary– User-friendly– Ease of installation– Flexible– Language support– Modular & Comprehensive

Why on-line testing?– Reducing marking time– Developing distance education courses

Challenges for on-line testing for languages

Concerns by language teaching professionals• limited types of questions• lack of analytical tools (natural language

parsing)• lack of qualitative evaluation• lack of evaluation for communicative

competence• security issues

Implementation and Objectives• On-line testing is NOT comprehensive• On-line testing is to access each

learner’s knowledge and recognition of:– Vocabulary– Expressions– Conjugations– Sentence structures– Basic kana characters and basic sino-

Japanese characters (kanji) – Simple context (communicative

understanding)– Sociolinguistic and pragmatic aspects

Developing tests

• Transforming paper-based tests into web-based versions

• Modifying types of questions• Developing new types of questions• Reviewing students’ answers and

modifying possible answers• Readjusting answers to introduce

partial marking system• Developing a questionnaire for

students’ feedback

• Course Management System (CMS aka LMS)– Similar to WebCT, or Blackboard

• Our Moodle runs on Linux OS using PHP and a MySQL database backend.

• Open-source, modular and flexible• Large, growing community• Community-driven support

Overview of Moodle

Overview of the Japanese Tests

• Tests run in the Multimedia Language Lab (MLC)– 50 mins per test– 2 to 5 tests run a day, over 2 days, 5 mins between

tests

• Students self-register for accounts using Moodle– Ran an Orientation Session at beginning to create

accounts and try out sample test

• Tests were originally developed in MS Word.• Imported into Moodle via web interface

Question Types

• Aside from the normal types of questions, the tests also included:– Fill-in-the-blanks (multiple blanks within a

paragraph, using Cloze or Embedded Answers type questions)

– Wildcard characters in the answers (for partial marks)

– Custom question type:• 50 Japanese characters in a large image (png),

each given a unique number• Students answered the question by typing the

number of the character into the box, in the correct order. (able to use wildcards for partial marks here too).

Security Measures

• IP-restricted (MLC only)• Each test has unique password• Time-based release of each test• During the allowed test time, students

can review answers• Allowed a maximum of 2 attempts• Moodle ‘secure window’ test

Technical Challenges

• Accurate representation of Japanese characters– Clarity of images

• Import not functioning in Moodle, so questions were cut/pasted– Time-consuming to enter questions initially

• Cloze type question still sensitive to illegal characters

• Various minor software glitches (eg. Half marks getting rounded down)

For the Future

• Moodle has proven to be a successful way of deploying language tests online

• All of the major/minor software problems are going to be fixed over the summer 2006

• Eventually will be bringing Language Placement Tests on board (French and Italian)

What is tested?Evaluation criteria for JP1000

• Attendance & Participation (Sept.-April): 10%• Oral Presentation (4 times a year): 25% • Quizzes (dictation, every week) & Homework

(practice of Japanese characters): 15%• Tests (4 times a year, on-line testing): 50%

-grammar & structure; vocabulary; idiomatic expressions -reading of Japanese scripts

Types of questions

Pen & Paper vs. On-line• Reading words• “What would you say when…?”• Translation (E→J)

-Complete a dialogue-Short answers vs. jumbled words

• Kanji (sino-Japanese characters)• Translation (J→E)

-Short answers vs. multiple-choice

New attempts

• Short answers• Multiple-choice questions• Questions using a table• Questions using pictures• “Read a passage and answer

questions”

‘Cons’ for Students

• Less lenient marking compared to paper-based tests

• Stressful for those uncomfortable with computers and

bad eye-hand coordination; displaying time left in the

corner

• “Save without Submitting” function tedious and

demanding

• Difficult to return to an unanswered or problematic

question, slow in viewing answers

• Easier to cheat

• Blurry screens

‘Pros’ for Students

• Quiet atmosphere, easier to focus

• Easier to answer questions (multiple-choice, kanji

recognition, re-attempts)

• Easier to correct answers

• Immediate feedback

• No concern about bad handwriting; hands do not get

tired from writing; no need to use pencils

• Less stress for those familiar with technology, more fun

• Gain more knowledge about computer technology -

important for academic and professional life

On-line vs. PaperTest Average (%)

Test 1 Test 2 Test 3Test4

Paper 78.44 55.16 64.90 66.98(2004-05)

On-line 64.57 59.09 59.52 62.50(2005-06)

Students’ Self-reflection (1)

Q: How satisfied are you with the time you have allotted to prepare for the tests?

A: dissatisfied 6.67%somewhat dissatisfied 46.67%satisfied 34.67%very satisfied 12%

Students’ Self-reflection (2)

Q: How satisfied are you with the efforts you have allotted to prepare for the tests?

A: dissatisfied 8.79%somewhat dissatisfied 44.59%satisfied 38.51%very satisfied 8.11%

‘Pros’ for Instructors (1)

• Less time spent on marking, less tired,

improved quality of teaching

• A test can be automatically re-graded for the

entire class if a mistake is discovered in a

question or more possible answers are added to

the system

• Easier to deliver the correct answers to students

• Typed text easier to read than unintelligible

handwriting

‘Pros’ for Instructors (2)

• Easier coordination between instructors

regarding

reviewing students’ answers, modifying

possible answers, marking

• Faster processing of the test results

• Objective assessment (not a comprehensive

approach)

• Increased marking consistency

‘Cons’ for Instructors

• Technical problems unsolvable by instructors

• Too sensitive to spaces, hyphens, unpredictable bugs in the program

• Location and scheduling restrictions

• Limitations regarding the types of questions

• Academic honesty issues

Cheating…

Weaker classroom control Assigned seats, gaps in the seating Inability to cut off the Internet ‘Monitored activities’

Multiple attempts, higher test scores Limited attempts (two attempts only)

Easy to duplicate the test and export it to Word Question pools, randomized questions, different order

of answers; run Moodle from a CD

Outcome

• Learned more about limitations, bugs and positive features re Moodle

• Why were test scores lower this year?• Academic honesty issues• Learning process for faculty• Faculty’s willingness to learn and no quick resistance to

Moodle• Cooperation between IT consultant and faculty with

good working relationship and initiatives• Tech problems are hard to deal with by faculty alone:

requires a tech support person at the test site.• Limitations re the test site

References

• Abhijeet Chavan (2004) Open-Source Learning Management with Moodle http://www.linuxjournal.com/article/7478• Moodle (2006) Moodle for Language Teaching http://moodle.org/course/view.php?id=31• Aditya Nag (2005) Moodle: An open source learning management system http://business.newsforge.com/article.pl?sid=05/05/09/2117200• Den Pain and Judy Le Heron (2003) WebCT and Online Assessment:  The best thing

since SOAP? http://www.ifets.info/journals/6_2/7.html• Röver, C. (2000)Web-Based Language Testing: Opportunities and Challenges

http://www2.hawaii.edu/~roever/wbt.htm• _________(2001) WEB-BASED LANGUAGE TESTING http://llt.msu.edu/vol5num2/roever/default.html• Sabine Siekmann (2006) CALICO Software Report Which Web Course Management System is Right for Me?

A Comparison of WebCT 3.1 and Blackboard 5.0 http://calico.org/CALICO_Review/review/webct-bb00.htm• University of Ontario (2006) WebCT http://www.uoit.ca/EN/main/11258/12122/17767/learning_webct.html