The Role of Automation in Undergraduate Computer Science

Chris WilcoxColorado State University

3/5/2015

Problem: Expanding Enrollments “Bachelor’s degree production increased by a double digit percentage for the third straight year. The number of new undergraduate computing majors among U.S. computer science departments rose an astonishing 29.2 percent, 22.8 percent among those departments reporting both this year and last year. This is the fifth straight year of increased enrollment in computing majors by new students.”*1

The Role of Automation in Undergraduate Computer Science 2

1) Taulbee Survey 2013 Computer Research Association2) CSU Enrollment Numbers for Introductory Course 2010-2014

3/5/15

Problem: Decreasing Funding “Colorado has reduced its support for higher education by nearly 69.4 percent, from $10.52 in fiscal 1980 (and a peak of $13.85 in fiscal 1971) to $3.22 by fiscal 2011. At this rate of decline Colorado appropriations will reach zero in 2022, 11 years from now.”*1

The Role of Automation in Undergraduate Computer Science 3

1) American Council on Education “State Funding: A Race to the Bottom”2) Center on Budget and Policy Priorities “A Formula for Decline…”

3/5/15

Solution: Increased Automation

Automated Grading Proprietary web-based framework for the automated

submission and assessment of student programs. Peer Instruction

Partial inverted classroom with peer instruction quizzes automated using iClicker technology.

Online Tutorials – MyProgrammingLab, Zyante Bulletin Boards – Blackboard, Piazza Grading Frameworks – Blackboard, Canvas Web Portal – Proprietary

The Role of Automation in Undergraduate Computer Science 43/5/15

Key Questions

Question: How do automated tools negatively or positively affect student learning?

Question: Do the benefits of automation, such as resource savings, outweigh the disadvantages?

The Role of Automation in Undergraduate Computer Science 5

Answer: Based on my research outcome, we can conserve resources (grade less, teach more) while maintaining or improving academic performance.

3/5/15

Automated Grading

250 students * (12 assignments + 12 labs + 6 quizzes) = around 7500 programs to grade during one semester!

Let’s make it very easy, maybe 6 minutes per program, so 6 * 7500 = 45000 minutes = 750 hours.

Who has the time for that? That’s more than two teaching assistants working 20 hours per week, which is all I have.

From an economic viewpoint, that’s $10-15K based on average graduate student compensation.

From a pedagogic viewpoint, shouldn’t teaching assistants and instructors maximize time spent in front of students?

The Role of Automation in Undergraduate Computer Science 63/5/15

Software: Automated Grading

The Role of Automation in Undergraduate Computer Science 73/5/15

Software: Test Results

The Role of Automation in Undergraduate Computer Science 83/5/15

Student Feedback Cycle

The Role of Automation in Undergraduate Computer Science 9

0 4 5 6 7 8321

Workflow with Manual Grading

FinalFeedback

PreliminaryFeedback

Post Date Due Date

Workflow with Automated Grading

3/5/15

Software: Course Calendar

The Role of Automation in Undergraduate Computer Science 103/5/15

Evaluation – Exam Scores

The Role of Automation in Undergraduate Computer Science 113/5/15

Evaluation – Withdrawal Rates

The Role of Automation in Undergraduate Computer Science 123/5/15

Evaluation – Program Submissions

The Role of Automation in Undergraduate Computer Science 133/5/15

Evaluation – Online Tutorial

The Role of Automation in Undergraduate Computer Science 14

Semester CategorySampleSize

ClassPercentag

e

Average

Total

StatisticalSignificance

Fall 2013 Everyone 252 100%78.02%

Fall 2013 Control 121 48%76.60%

base population

Fall 2013 Treatment 110 44%80.09%

t = 1.73p = 0.0421

Fall 2013 SelfUnassigned

8 3%68.20%

t = 2.34p = 0.0212

3/5/15

Resource Savings

Each time we grade a student program with automation we save ~16 hours, and we give many assignments and programming quizzes.

Each time we grade a peer instruction quiz with automation, it saves us around ~6 hours of teaching resource, and we give one per week.

Equivalent to at least one TA each semester, or 300 hours at approximately $20 per hour, which is around $6000, not counting tuition, and we use automated grading in 5-6 courses, twice a year.

The Role of Automation in Undergraduate Computer Science 153/5/15

Automation Caveats

However, there are some negative aspects of automated program grading that we have observed:Automated testing can be perceived as unforgiving and overly strict by students.Clearly not as beneficial as one on one code reviews for design and coding style issues.Can create an unhealthy dependence on preliminary testing, which limits the acquisition of testing and debugging skills by the student.

The Role of Automation in Undergraduate Computer Science 163/5/15

Threats to Validity

Inherent in empirical studies performed on students in a university setting.

External Validity: Automation does not degrade academic performance, and can actually improve it.

Internal Validity: Are there extraneous factors that could account for observed differences? Semester to semester variation in students. Staff turnover among teaching assistants. Changes to course format and content.

However, we were able to tightly control the critical factors: curriculum, assignments, and exams.

The Role of Automation in Undergraduate Computer Science 173/5/15

Conclusions

The automation of key processes can save a significant amount of scarce resources without negatively impacting academic performance.

The benefits of automation are both tangible, such as higher exam scores, and intangible, such as increased student engagement and interest.

We view this as a fortunate outcome, since expanding enrollments will inevitably provide incentive for even more automation.

The Role of Automation in Undergraduate Computer Science 183/5/15