improving pedagogical efficiency and effectiveness using an automated testing system ramon lawrence,...

Improving Pedagogical Efficiency and Effectiveness using an Automated

Testing System

Improving Pedagogical Efficiency and Effectiveness using an Automated

Testing System

Ramon Lawrence, Alyosha Pushak,Teresa Wrzesniewski

University of British Columbia Okanagan

University of British Columbia Copyright© 2011

Ramon Lawrence, Alyosha Pushak, Teresa WrzesniewskiImproving Pedagogical Efficiency and Effectiveness using an Automated Testing SystemUBCO Research Week Presentation 2011

Introduction Technology can improve teaching efficiency and delivery, but

the most repetitive and time-consuming tasks of assignment and test preparation and marking are still mostly manual.

Automated testing systems have the potential to dramatically reduce the amount of time spent on these repetitive tasks, freeing up instructor time for valuable student contact, and allowing for more uniformity on student and instructor achievement and evaluation.

We have developed the AutoEd system that is an automated question generation and marking system and evaluated it in a first year Physics course.



Background – Testing Systems There is a variety of technology for student testing:

clickers, web-based learning, course management systems, etc.

It is challenging to determine which technologies are appropriate for certain teaching styles and courses, and what are the correct products and systems that provide the appropriate payback for student learning and institutional deployment.

Most of the question systems are limited to static questions. There is no learning value in repetitive practice. The questions do not encourage or improve learning; they

simply test if previous learning has occurred.



Background – Question Templates A question template is a question that tests a specific concept

that can be re-used by varying its contents.

Example: Calculate the distance travelled by a car in 2 hours that is travelling 60 km/h.

Template: Calculate the distance travelled by a $x$ in $y$ that is travelling $z$.

Randomization: For x, y, and z.

Answer: y*z (with correct units) Incorrect answers can also be encoded with feedback.



Previous Systems Previous question template systems:

WWWAssign built by Dr. Larry Martin for physics courses. Currently a commercial product called WebAssign.

Aplia - social sciences Mastering Physics Gradiance - computer science Carnege Learning – mathematics

These products have textbook questions and provide web hosting. The cost is $15-$100 per student per course.

Course management systems such as WebCT/Blackboard have question systems and some support for randomized questions.



Previous Systems – Open Source There have been several university led efforts on building these

systems and providing access.

Quest Service at The University of Texas at Austin many science/math questions outside users can get access by requesting UT Austin accounts

LON-CAPA is an open source system built primarily at

Michigan State University. It has hundreds of thousands of resources and questions that

are shared by multiple institutions. Institutions are part of a sharing network for distributing questions and resources.



Selecting a System Selecting an appropriate system depends on one key decision:

Who pays?

1) Student pays another cost in addition to textbooks paid by students students will be reluctant to pay especially if not used

extensively commercial systems are great from an instructor point of view

2) Institution pays Highly expensive for the institution so likely limited to open

source systems. Both of which have their own restrictions. An in-house system is the only other choice.



AutoEd System Overview The AutoEd system was designed and built by Alyosha Pushak

(undergrad honours) and Dr. Lawrence in computer science. It is written completely in Java and uses open source systems

for the database and web server. The database stores all user question and answers. Users interact with the system using a web browser.

Students can: Practice questions and do assignments as given by the

instructor. Students receive immediate feedback. Instructors can:

Create practice questions and assignments. Examine student assignment progress and marks at any time.



AutoEd – Student Test



AutoEd – Question Authoring



AutoEd – Student Completion Rate



Physics Evaluation The AutoEd system was evaluated in Phys 112 in Fall 2010:

2 paper assignments, 2 electronic assignments, 1 choice Electronic assignments were converted from paper assignments

by adding randomization.

Surveys conducted at the first and last classes. First survey: 105 out of 111 surveys completed (95.5% return) Second survey: 76 out of 89 surveys completed (85.4% return)



Physics Evaluation Results Format choice: 90% of the students used AutoEd for all three

assignments and 92% used it for the last assignment. Paper or electronic preference: 70% chose electronic while an

additional 15% chose both as equal. Experience: Over 72% of the students chose satisfactory or

very satisfactory when asked about their experience using the system while only 3% rated it as very unsatisfactory.

Help learning: Almost half of the students (45%) reported that AutoEd helped their learning.

Completion: Students reported that AutoEd helped them complete their assignments (81%) and hand their assignments in on time (84%).

Other uses: Over 92% of the students thought AutoEd should be used again in Physics 112, and 87% felt that it could be used in other classes (chemistry, biology, mathematics, English).



Insights and Improvements 1) Watch technical issues with the system and questions.

Issues with incorrect marking and significant figures. Question preparation takes time and all answers must be

checked before releasing the assignment.

2) Users will use the Internet to find answers. 5% of users wanted the system to block Internet searches. Copy-and-paste electronic question versus typing it in! Effectiveness as an evaluation tool as more answers are online?

3) Students really do wait until the last minute. By removing the overhead of assignment generation and

collection/marking, more frequent, smaller assignments is better.



PHYS 122: Paper vs. Electronic

A1 A2 A3 A4

% handed in 2010 - hand in 89% 94% 89% 84%

2011 - online 93% 90% 93% 98%

Average grade [%] 2010 - hand in 60.3 82.9 67.3 62.1

2011 - online 84.4 71.3 77.5 86.3

Median grade [%] 2010 - hand in 67.1 92.2 77.9 75

2011 - online 100 100 100 100



PHYS 122: Question Writing

Q3: Using Kirchhoff’s rules for the circuit below find:

a)The current in each resistor. [8 marks]

b) The potential difference between points c and f [2 marks]

Randomize R3 from 4.00 kΩ to 10.0 kΩ, increments 1.0 kΩ.



PHYS 122: Question Writing (2)

Q2. A conductor suspended by two flexible wires (as shown below) has a mass per unit length of 0.040 kg/m. What current must exist in the conductor for the tension in the supporting wires to be zero when the magnetic field, B = 3.60 T points into the page? Give your answer with 3 sig. figs. [5 marks]

Randomize B from 2.60 T to 4.60 T, intervals 0.10 T.



Future Work and Conclusions It is possible to build a customized question answer system

with minimal cost. Although the upfront costs and time is higher, the payback in terms of student costs and instructor flexibility is significant.

We are currently evaluating the system in several other courses and will release it as open source to the community.

The system will be maintained and extended for many years. Limited time offer:

We will create the questions for you over the summer for any courses that you may be interested in using the system for!



AutoEd – Demonstation Let's have a quick tour of the AutoEd system.

improving pedagogical efficiency and effectiveness using an automated testing system ramon lawrence,...

Documents

automated testing systems

question systems

student testing

background testing systems

pedagogical efficiency

student learning

alyosha pushak

u