Labrador: A Tool for Automated GradingSupport in Multi-section Courses

Drexel University Programming Learning EXperience (DUPLEX)

Departments of Mathematics† and Computer Science‡

http://duplex.mcs.drexel.edu

Christopher D. Cera‡, Jeffrey L. Popyack‡, Bruce Char‡, Robert N. Lass‡,

Nira Herrmann†, Paul Zoski†, Aparna Nanjappa‡

May 19, 2003

Roadmap

• Introduction

• Problems and Solution Goals

• Labrador

• Discussion

Slide 2 of 56

The Duplex Project: An Overview

• Take advantage of advances in Information Technology to improveinstruction and reduce costs for computer programming courses

• Modular Structure

– Multiple Entry Points– Multiple Audiences– Multiple levels of knowledge (Bloom’s Taxonomy)

• Computer Supported Cooperative Work (CSCW) in student labs

• Online Services - Today’s Topic

Slide 3 of 56

Course Redesign

• Emphasis on online materials

• WebCT introduced for:

– Delivering online course materials– Chat/Discussion groups– Quizzes/Labs– Electronic submission, grading, and return of assignments

• Issues: WebCT Interface does not handle all course needs

• Solution: Labrador

Slide 4 of 56

Who Am I?

• First TA in MCS to experiment with WebCT in December 2000

• Migrated course content to WebCT from previous course website

• HTML assignments and labs into question database

• Gradebook maintainer

• Wrote demo’s and documentation to train other TAs

• Developer of software supplements to WebCT to support courseadministration

Slide 5 of 56

Roadmap

• Introduction

• Problems and Solution Goals

• Labrador

• Discussion

Slide 6 of 56

Before WebCT

• Hundreds of people were involvedin paper exchanges of handwrittenassignments and quizzes

• Testing programs required floppyexchanges

• No Chat

• No newsgroup-style threads

• Feedback and grades are notonline

Slide 7 of 56

WebCT

• Service from InformationResources and Technology (IRT)(since v3.1)

• MCS started Dec. 2000 (v3.5)

• Currently using version 3.8.3

Slide 8 of 56

Features

• General Course Website

• Centralized Administration

• Labs: Online Quizzes withAutomated Grading

• Homework: Online Assignments

• Joint Staff–Student Chat

• Discussion Threads

Slide 9 of 56

Large Classes and WebCT

• Even with WebCT .... still somedifficulties:

– Bulk download of assignmentfiles and quizzes for grading

– Handling select sectionsrequires searching and clicking

– Files submitted in compressed,archived, or encoded formatsare tedious to unpack manually

– Transferring data to othersystems

Slide 10 of 56

Software Design Goals

• Bulk Assignment Downloading (prior to

3.7)

• Bulk Quiz Downloading

• Section Sorting for bulk downloads, or

only one section

• Post-Processing student files

– Archive Extraction (tar, zip)

– Decompress (gz, zip)

– Decode (uue)

• Minimal staff intervention when

transferring submissions between

systems, eg. Plagiarism Detection

Systems

Slide 11 of 56

Software Design Goals [continued]

• Automatically collate source code

• Generate electronic documents tofacilitate grading and archiving

• Upload grades and marked-updocuments

• Remote Execution downloading tocomputer x (on campus) whileoperating at computer y (offcampus)

Slide 12 of 56

The Bigger Picture

• Course-Specific Tasks

• Not all processing should be doneon the server

• Select files have to be transferredto a different system for furtherprocessing and analysis by staff

• Client-side support is needed toperform this, preferablyautomated and not necessarilyusing a web browser.

Slide 13 of 56

Roadmap

• Introduction

• Problems and Solution Goals

• Labrador

• Discussion

Slide 14 of 56

Labrador: Our Solution

• Client-side WebCT Supplement

• Cross-platform

• Works for users with TA andDesigner access to WebCT

Slide 15 of 56

Labrador Execution Timeline

SubmissionDownloader

SectionSorting

Post-Processing

PDFGeneration

JPlag

Moss

SubmissionUploader

Slide 16 of 56

User Interface

• Different users / Different UIpreferences

• GUI

• Command-line

• Interactive

• Configuration File

Slide 17 of 56

Bulk Downloader

• Web-Crawler: simulates clicks ofactual staff member

• Parses HTML to find desired textor URLs to crawl to next

• Works on assignments and quizzes

• Only component in Labradorwhich interacts with WebCT

Slide 18 of 56

Required Information

http://webct.drexel.edu/SCRIPT/CS164_Fall2002/scripts/designer/dropbox_edit.pl?DROPBOX_ASSN_VIEW+_side_nav++1006285909

Username/Password st96k9ry

Server URL webct.drexel.edu

Course ID CS164 Fall2001

Submission Name or ID Recursion II or 1006285909

Optional Username List unames.txt

Slide 19 of 56

Organizing Submissions by Section

• Organizes each student’ssubmissions into a separate folderfor each section.

• How to tell Labrador the sections:

– Creating a Section column ingradebook

– “Username, Section” CSV File– Username file

Slide 20 of 56

PDF Generation for Electronic Mark-up

• Adobe Portable Document Formatis available on all major platforms

• With Adobe Acrobat, PDFs canbe annotated by graders

• Sony VAIO Slimtop PC(PCV-LX920)

• Toshiba Tablet PC: Protege 3500

• Acer Tablet PC

• Wacom Pen Tablet

Slide 21 of 56

Sample Mark-up Options

Slide 22 of 56

PDF Markup Example

Slide 23 of 56

Demonstration

Startup screen prompts for the username and password

Slide 24 of 56

Demonstration [continued]

TA enters username and password

Slide 25 of 56

Demonstration [continued]

Labrador prompts for the course name and optional student list

Slide 26 of 56

Demonstration [continued]

TA enters course name and student list

Slide 27 of 56

Demonstration [continued]

Labrador prompts for the submission type

Slide 28 of 56

Demonstration [continued]

TA selects assignments

Slide 29 of 56

Demonstration [continued]

TA selects post-processing

Slide 30 of 56

Demonstration [continued]

TA selects PDF generation

Slide 31 of 56

Demonstration [continued]

Labrador prompts for the specific assignment

Slide 32 of 56

Demonstration [continued]

TA selects “Practice Assignment” and begins downloading

Slide 33 of 56

Demonstration [continued]

Labrador notifies the TA that the job is complete

Slide 34 of 56

Demonstration [continued]

Exploded view of TA’s folder

Slide 35 of 56

Redistribution

• How can we return annotated PDF’s back to students using WebCT?

• Version 3.8 addresses this issue

• Labrador supports this upload feature

Slide 36 of 56

Labrador Applications

• Interface between WebCT documents and other software

– Decompressing files– Reformatting files for grading (PDF)– Submission to Plagiarism Detection Software (Moss/JPlag)– Other third party software programs– Returning processed/graded documents to WebCT

• Primary Issue: Compatibility with heterogeneous systems

Slide 37 of 56

Heterogeneous Systems

• Each system requires data to bepackaged in a different way

• Plagiarism Detection Systems

• Moss [1] and JPLAG [2] havebeen used extensively

• Other processes (e.g. PDFgenerator)

• Future work: Automatic programcompiling and testing

Slide 38 of 56

Automated Plagiarism Detection

• Digital formats make ”borrowing”easy

• Browsing similar works needs asimple and quick user interface.

• Careful review by faculty to assessresults and present to students

Slide 39 of 56

Moss [1]

• C, C++, Java, ML, Lisp, Scheme, Pascal, and Ada

• Common code feature reduces false positives

• http://www.cs.berkeley.edu/∼aiken/moss.html

Slide 40 of 56

Moss [1] Interface

Slide 41 of 56

MossCliques Interface

http://duplex.mcs.drexel.edu/software/MossCliques.zip

Slide 42 of 56

JPlag [2]

• C, C++, Scheme, and Java

• For plain text files, it matches a user specified number of wordsappearing in succession

• Could be used for any course grading written (text) documents

• http://wwwipd.ira.uka.de:2222/

Slide 43 of 56

JPlag [2] Interface

Slide 44 of 56

Roadmap

• Introduction

• Problems and Solution Goals

• Labrador

• Discussion

Slide 45 of 56

Recent WebCT Enhancements

• Relevant to this talk:

– 3.7 Addressed bulk downloadissue for assignments

– 3.8 Attaching documents to anassignment

Slide 46 of 56

Future WebCT Enhancements

• Power users will need functionalitynot yet supported

• Every domain will also requireadditional functionality

• Not feasible for all domain-specificfunctionality to run on theWebCT server

Slide 47 of 56

HTTP: Insufficient For Data Interchange

• Was designed for visual content

• Heavy client interaction

• An HTTP based approach is sensitive to the exact location of webpages, and format of text within them

• One possible solution is ...

Slide 48 of 56

API for non-administrators

• Stateful protocol so clients can bebuilt by 3rd parties

• Could be coupled with efforts ofthe “semantic web” (e.g. RDF)

• This would eliminate the need forLabrador to “screen-scrape” textfrom web pages

Slide 49 of 56

Labrador Availability

• Contact Us

http://duplex.mcs.drexel.edu

Slide 50 of 56

The Group

Bruce Char Professor, Computer ScienceNira Herrmann Professor and Head, MathematicsJeffrey L. Popyack Associate Professor, Computer SciencePaul Zoski Instructor, Math and Computer Science

Christopher D. Cera Computer Science Graduate StudentRobert N. Lass Computer Science Undergraduate StudentAparna Nanjappa Computer Science Graduate StudentDerek Rosenzweig Computer Science Undergraduate StudentJasper Zhang Computer Science Undergraduate Student

Slide 51 of 56

Project Support

• National Science Foundation,

Division of Undergraduate

Education, DUE − 0089009

• The Pew Learning and

Technology Program at the

Center for Academic

Transformation

• The Ramsey-McCluskey Family

Foundation, Margaret Ramsey,′84

• Drexel University

Slide 52 of 56

References

[1] Alex Aiken. MOSS: A System for Detecting Software Plagiarism(Unpublished), http://www.cs.berkeley.edu/∼aiken/moss.html.

[2] L. Prechelt, G. Malpohl, and M. Philippsen. JPlag: Finding PlagiarismsAmong a Set of Programs. Technical Report 2000-1, Fakultat furInformatik, Universitat Karlsruhe, Germany, March 2000.

Slide 53 of 56

Related Work

[1] Robert N. Lass, Christopher D. Cera, Nathaniel T. Bomberger, Bruce Char, Jeffrey L.

Popyack, Nira Herrmann, and Paul Zoski. Tools and Techniques for Large Scale Grading

using Web-based Commercial Off-The-Shelf Software (To Appear). In Proceedings of

the Eighth Annual Innovation and Technology in Computer Science Education (ITiCSE).

ACM Press, June 2003.

[2] Nira Herrmann, Jeffrey L. Popyack, Bruce Char, Paul Zoski, Christopher D. Cera,

Robert N. Lass, and Aparna Nanjappa. Redesigning Computer Programming Using

Multi-level Online Modules for a Mixed Audience. In Proceedings of the Thirty-Fourth

SIGCSE Technical Symposium on Computer Science Education. ACM Press, February

2003.

Slide 54 of 56

Related Work [continued]

[3] Jeffrey L. Popyack, Nira Herrmann, Paul Zoski, Bruce Char, Christopher D. Cera, and

Robert N. Lass. Academic Dishonesty in a High-Tech Environment (Special Session). In

Proceedings of the Thirty-Fourth SIGCSE Technical Symposium on Computer Science

Education. ACM Press, February 2003.

[4] Jeffrey L. Popyack, Bruce Char, Paul Zoski, Nira Herrmann, Christopher D. Cera,

Robert N. Lass, and Aparna Nanjappa. Course Management Systems (Birds-of-a-

Feather Session). In Proceedings of the Thirty-Forth SIGCSE Technical Symposium on

Computer Science Education. ACM Press, February 2003.

Slide 55 of 56

Related Work [continued]

[5] Jeffrey L. Popyack, Bruce Char, Nira Herrmann, Paul Zoski, Christopher D. Cera, and

Robert N. Lass. Pen-Based Electronic Grading of Online Student Submissions. In

Syllabus fall2002, Technology for Higher Education Conference, November 2002.

[6] Christopher D. Cera, Robert N. Lass, Bruce Char, Jeffrey L. Popyack, Nira Herrmann,

and Paul Zoski. Labrador: A Tool for Automated Grading Support in Multi-Section

Courses. In Proceedings of the Fourth WebCT User Conference, Integrating the

Campus: Technical Solutions for Resource Development or Wide Scale E-Learning

Deployment, July 2002.

[7] Jeffrey L. Popyack, Bruce Char, Paul Zoski, Nira Herrmann, and Christopher D. Cera.

Managing Course Management Systems (Birds-of-a-Feather Session). In Proceedings

of the Thirty-Third SIGCSE Technical Symposium on Computer Science Education,

page 423. ACM Press, February 2002.

Slide 56 of 56