Automating Feedback & Assessment in WebLab

Eelco Visser

TU Delft | Open Education Week 2015 | Research Seminar: Data Science | March 9, 2015

2010-2011

Paper Exams on Programming

Tedious to grade!

Is this a correct solution?

Can you grade that objectively?

Is programming on paper a useful skill?

x 5 pages x 150 students

31 B A D C

32 C D A B

33 D C B A

34 A D C B

35 D A B C

36 B C D A

37 C B A D

38 A C D B

39 C A B D

40 B D C A

41 D B A C

42 A C B D

43 C A D B

44 B D A C

45 D B C A

46 C B D A

47 A D B C

48 D A C B

49 A B D C

50 B A C D

51 C D B A

52 D C A B

53 A B C D

54 B A D C

55 C D A B

56 D C B A

57 A D C B

58 D A B C

59 B C D A

60 C B A D

niet inkleuren a.u.b.

Handtekening

Gemaakt door:

TU Delft

Vakcode:

Dag Maand Jaar

/ /Naam:

Tentamen:

In BLOKLETTERS invullen a.u.b.

16 C A D B

17 B D A C

18 D B C A

19 B C A D

20 C B D A

21 A D B C

22 D A B C

23 A B D C

24 B A C D

25 C D B A

26 D C A B

27 A B C D

28 B A D C

29 C D A B

30 D C B A

61 C D B A

75 A C B D

62 D C A B

74 D B A C

63 A B C D

73 B D C A

64 B A D C

72 C A B D

65 C D A B

71 A C D B

66 D C B A

70 C B A D

67 A D C B

69 B C D A

68 D A B C

Vierkeuze antwoordformulier FB1

dagavond

Vakje volledig invullen

1 D A B C

2 B C D A

3 C B A D

4 A C D B

5 C A B D

6 B D C A

7 D B A C

8 A C B D

9 C A D B

10 B D A C

11 D B C A

12 B C A D

13 C B D A

14 A D B C

15 D A C B

StudentNummer

0123456789

Vakje volledig invullen

Opmerkingen:

4124

Constructs:

• Variable: x

• Application: MN

• Function abstraction: �x.M

Beta reduction:

• (�x.M)N ⌘ M [x := N ]

Substitution:

• x[x := N ] ⌘ N

• y[x := N ] ⌘ y if x 6⌘ y

• (PQ)[x := N ] ⌘ (P [x := N ])(Q[x := N ])

• (�x.M)[x := N ] ⌘ (�x.M)

• (�y.M)[x := N ] ⌘ (�x.M [x := N ]) if x 6⌘ y and y 62 FV (N)

Free variables:

• FV (x) ⌘ {x}

• FV (MN) ⌘ FV (M) [ FV (N)

• FV (�x.M) ⌘ FV (M)� {x}

Figure 2: Lambda Calculus

Question 8 (Bound Variables): What are the bound variables in the following statements:val contents = new Array[String](this.contents.length)for (i <- 0 until this.contents.length)contents(i) = this.contents(i) + that.contents(i)

new ArrayElement(contents)

A: {contents, i} B: {contents} C: {that} D: {that, i}

Question 9 (Recursion): Consider the following function fdef f(a: Int, b: Int): Int = if (b == 0) a else f(b, a % b)

In executing the call f(30,4), how many times is f called?

A: 1 B: 2 C: 3 D: 4

Question 10 (Interpreter): Given the definitions in Figure 3, what is the value of thefollowing expression?

eval {Let("x", Number(1),

Let("f", Fun("y", Plus(Var("x"), Var("y"))),Let("x", Number(2),

App(Var("f"), Number(3)))))}

A: NumValue(3) B: NumValue(4) C: NumValue(5) D: NumValue(6)

3

Automated grading

Easy to make (for students)

Easy to guess

Hard to design

Not test of relevant skills

Programming Assignments in the Lab

TAs download, build, test, grade manually

Installing software on lab/student machines

Long feedback cycle

Submission, grading, administration workflow

Spreadsheet administration

Uploading grades to Osiris

Christmas Break 2011

WebLabBrowser

code + test

database

success/all

code + test

success/all

securityusability scalability

An integrated learning management system with support for programming education

assignment

LabBack

execution engine

demonstration

Monitoring Student Progress

Automatic grading with unit testing

Instructor uses same environment

Program in browser, compile & run on server

WebLab

Use for programming exams

Works everywhere without installation

Scales to large numbers of students

Alignment between teaching & assessment

Objective automated assessment

Automate lab logistics

Rapid (immediate) feedback to students

Reduces tedious work of instructor

Continuous monitoring of student progress

Also: MC, essay questions, grading rubrics

Research Agenda

To Do

aka

WebLab TeamEelco Visser, Vlad Vergu, Elmer van Chastelet, Oscar Castaneda

Translating assessment into meaningful gradesRelative weight of tests

Gamification

Programming environmentgrow into online IDE

Improving Assessment & Feedback

Reduce effort of creating testsGenerate tests from more declarative specification of correctness

Property-based testing

Testing beyond functional correctnessNon-functional requirements Recognizing good design

Feedback How to help students when they are stuck?

Generating hints / solution strategies from assessment Personalized learning paths

Can we apply this to other programming courses?Algorithms & Data Structures (Java)

Algorithmics (Java) Embedded Software (C)

(and at courses at TU Darmstadt) Your course?

Generalizing Automated Assessment & Feedback

Can we apply this to non-programming courses?

Automate correctness testing

Define evaluation function that produces - a grade (correctness degree) - feedback (explanation of non-correctness)

PlagiarismHow can we make sure that students really do the work themselves? My current answer: digital exam with physical presence

Monitoring students in a courseWe need to record more data

Learning Analytics

Evaluating learning processesWhat can we learn from sequence of intermediate

solutions about solution strategies?

Evaluating the effectsDoes it work?

Do students learn more? better? How can we tell?

What do you need for your course in WebLab?

http://weblab.tudelft.nl

WebLab is open for education