Discriminative Marking of Numeric Problems

Dick BaconPhysics, Surrey.

LTSN Physical SciencesSToMP project

r.bacon@surrey.ac.uk

CAA 2005

Outline

1. Background

2. Students' opinion of previous e-coursework

3. Changes to the e-coursework system

4. Results of the changes - objective & subjective

5. Conclusions

CAA 2005

Background

The aim of this work is to produce e-coursework assignments for use in physics, that staff and students recognise as being equivalent in functionality and fairness to conventional paper based assignments.Trials have been carried out over the last few years using the SToMP testing system (QTI with numeric extensions) with first and second year students. The system and the way it is used have been changed each year, in response to students' comments on the system. The major changes this year have been with the second year e-coursework.

CAA 2005

Background

The system used in this work is the SToMP testing system. This is QTI v1.2 compliant but has extensions to enable:

randomisation of numbers for all question types numeric response checking by accuracy and

precision trivial sequencing error propagation numerical expression evaluation and analysis other prototyped facilities

CAA 2005

Student's opinions year 2002 2003 2003 2004course 1DH 1DH 2RD 1DHnos of students 53 53 37 34nos of responses 39 29 18 21

74% 55% 49% 62%negative commentsno marks for working 60% 55% 89% 58%all or nothing marking 26% 0 0 0don't like doing tests this way 0 0 22% 0cannot interrupt test 13% 17% n/a 8%question errors 13% 0 0 0technical problems 10% 31% 0 19%feedback too general 0 10% 0 15%it encourages cheating 3% 10% 6% 12%have to use laboratory 0 7% 17% 15%problems entering expressions n/a n/a 6% n/a

1DH 1DH 2RD 1DHpositive commentsflexibility of timing 51% 59% 44% 46%test can be interrupted n/a n/a 33% n/aspeed of marking/feedback 30% 0 0 19%good aid to learning 0 0 22% 0easy to use 21% 17% 11% 0less stressful 15% 7% 0 8%open book 10% 0 28% 0good quality feedback 0 7% 0 15%does not take long 0 3% 0 12%saves academic staff time 0 0 17% 19%

CAA 2005

Changes for 2004/5

The changes made, following these comments, were:

more explanation about how marks were awarded

better expression entry diagnostic messages

improved discrimination in the marking of numeric questions, using the entered expressions as well as the entered values.

CAA 2005

Changes for 2004/5

To improve discrimination in the marking of numeric questions. The 2003/4 marking scheme was (e.g.) for a numeric question involving use of this formula:

0.09 * B / A^2

correct value 12value approximate 8value correct but wrong precision 8value approx and wrong precision 6expression gives correct value 6expression gives approximate value 4

CAA 2005

Expression analysis

Expression:

0.9 * (1000*B / ( ( 100*A )^2 ) 0.09 * B / A^2

a) (0.9)*((2.3)/(3.5^2)) - no scaling

b) 0.059 * 2.6 / 4^2 - wrong constant value

c) 0.9*((2.3/1000)/((4.4^2)*10000)) - wrong scaling

20.9 BA

CAA 2005

Expression analysis

a) (0.9)*((2.3)/(3.5^2)) - no scalingb) 0.059 * 2.6 / 4^2 - wrong constant valuec) 0.9*((2.3/1000)/((4.4^2)*10000)) - wrong scalinga) and c) were addressed by a new condition that tested whether a value was equal to a variable value multiplied by a power of 10.<questvar_equal . . . expression="Yes" inpercent="5"

ordermag="Yes"/>This is of limited use!

CAA 2005

Expression analysis

b) 0.059 * 2.6 / 4^2 - wrong constant value

Need to recognise that the expression has used A and B correctly, irrespective of the final value. So . .

i) replace question values with variable names.

ii) replace other values (not exponents) with unity.

iii) use rules to reduce the expression to a common form.

e.g. 1 + 1 1

iv) check for equivalence to template(s).

CAA 2005

Expression analysis

Correct value 1.71e-3Precision too low 1.7e-3Precision too high 1.712e-3Value approximate 1.72e-3Value approx and precision too low 0.0017Value approx and precision too high 1.714e-3Order of magnitude error 0.0171Value approx and order of mag error 1.72Correct expression (1) (1.6*0.09+(0.29*100)/4.3^2)*1e-3Correct expression (2) ((0.29*100)/4.3^2+1.6*0.09)/1000approx expression (1) (1.6*0.1+(0.29*100)/4.3^2)*1e-3approx expression (2) (1.7*0.09+(0.29*100)/4.3^2)*1e-3order of magnitude error 1.6*0.09+(0.29*100)/4.3^2order of mag & approx (1) 1.3*0.09+(0.29*100)/4.3^2*1e3order of mag & approx (2) 1.6*0.1+(0.29*100)/4.3^2expression error 1 (1.6*5.0+(0.29*100)/4.3^2)*0.090expression error 2 (1.6*10+(0.29*100)/4.3)*0.085expression error 3 (1.6*10+(4.3*100)/0.29^2)*0.080expression error 4 (1.6*10+(4.3*100)/0.29)*0.075expression error 5 (1.6*10+(4.3^2*100)/0.29)*0.070expression error 6 (0.29*10+(1.6*100)/4.3)*0.065expression error 7 (0.29*100+(4.3*100)/1.6^2)*0.060expression error 8 (4.3*10+(1.6*100)/0.29^2)*0.055expression error 9 (4.3*10+(0.29*100)/1.6)*0.050expression error 10 (4.3^2*10+(0.29*100)/1.6)*0.045

CAA 2005

Expression analysis

This makes for a good range of marks, but also allows the feedback to be very specific.Feedback used to be given only when everyone had finished the tests - i.e. too late to be of much help.If students were to be allowed more than one try at questions where they did not score full marks, then high quality specific feedback could be put to immediate use.Subsequent tries can have fewer marks available.Randomised question values reduce the risk of cheating.

CAA 2005

Expression analysis and multiple tries

condition try 1 try 2 try 3 try 4correct value 20 15 12 9approximately correct value 16 12 9 6value with wrong precision 16 12 9 6val approx and wrong precision 14 10 7 5alternative value 1 12 9 6 4alternative value 2 12 9 6 4expression has correct value 10 7 5 3value order of magnitude error 9 7 5 3value approx order of mag error 8 6 4 2expression is approximate 8 6 4 2expr alternative value 1 7 5 3 1expr alternative value 2 7 5 3 1expression order of mag error 4 3 1 0expr approx order of mag error 3 2 0 0expression has right form 3 1 0 0expression has wrong form 1 2 0 0 0expression has wrong form 2 2 0 0 0both totally wrong 0 0 0 0

CAA 2005

Implementation

Each multiple try question was created by replication. Differences between tries were:a) Outcome variables:(i) Each instance had its own variable Q2a, Q2b, etc.(ii) There was one variable declared in the first instance and used in all questions, for which<decvar> was given a new attribute reuse="Yes", "No" and the <setvar> action attribute was given a new value, "Setup" (i.e. value can only be increased).

CAA 2005

Implementation

Each multiple try question was created by replication. Differences between tries were:

b) Feedback - at each attempt the feedback supplied for a particular wrong answer provided increasingly fundamental information.

c) Marks - as indicated.

d) The question values can be re-randomised if desired.

CAA 2005

Feedback

1. No. The value you entered is wrong. Your expression looks as if you are trying to calculate the final energy of the gamma ray photons rather than the energy of the Compton edge.

2 = 1 + Look up your notes (or other material) about Compton scattering and the limits to the energies of scattered photons and electrons.

You need to calculate the largest energy the electron can be given.

CAA 2005

Implementation

Each multiple try question was created by replication. To enable the scheme to work :

<item> was given a new attribute hide="Yes", "No"

A new element <exposeitem> with the attribute itemid was introduced into <respcondition>.

<item> was also given the attribute showfeedback

(<itemfeedback><solution>,<hint>

feedbackstyle was not used)

CAA 2005

Test results

2a 2b 5a 5b 5c 5d 8a 8b 8c 8d12 9 20 15 12 9 n 40 30 22 16 n3 9 16 5 32 53 9 0 15 3 24 303 9 12 15 3 20 306 9 12 10 12 0 30 90 9 12 9 7 9 0 24 20 9 9 9 12 0 0 22 2

2 0 4 2 0 21 0 160 7 12 0 0 15 130 0 0 0 0 15 0

0 0 0 0 50 0 0

Questions 2, 5 and 8. Values in blue are the final mark.

CAA 2005

Student's opinions year 2002 2003 2003 2004 2004course 1DH 1DH 2RD 1DH 2RDnos of students 53 53 37 34 32nos of responses 39 29 18 21 22

74% 55% 49% 62% 69%negative commentsno marks for working 60% 55% 89% 58% 19%all or nothing marking 26% 0 0 0 0don't like doing tests this way 0 0 22% 0 12%cannot interrupt test 13% 17% n/a 8% 12%question errors 13% 0 0 0 0technical problems 10% 31% 0 19% 8%feedback too general 0 10% 0 15% 0it encourages cheating 3% 10% 6% 12% 0have to use laboratory 0 7% 17% 15% 0problems entering expressions n/a n/a 6% n/a 12%

1DH 1DH 2RD 1DH 2RDpositive commentsflexibility of timing 51% 59% 44% 46% 35%test can be interrupted n/a n/a 33% n/a n/aspeed of marking/feedback 30% 0 0 19% 4%prompt feedback and re-try n/a n/a n/a n/a 19%good aid to learning 0 0 22% 0 27%easy to use 21% 17% 11% 0 8%less stressful 15% 7% 0 8% 0open book 10% 0 28% 0 23%entering expressions useful n/a n/a 0 n/a 16%good quality feedback 0 7% 0 15% 0does not take long 0 3% 0 12% 0saves academic staff time 0 0 17% 19% 0getting marks for method n/a n/a 0 n/a 4%

CAA 2005

Improvements

The feedback shows that even better support is needed for the entry of expressions.

The results indicate that better support is needed for students who are unable to 'get started'. This might be provided by including links to external material.

The 'track' of each student is to be analysed, to estimate the contribution made by each feedback message. This may reveal ways in which the messages can be improved.

The system must be corrected to allow all test types to be interrupted and resumed.

CAA 2005

Conclusions

The results show that with targeted help, students can improve their ability to solve these types of problem.

The results and feedback indicate that some students welcome the opportunity to improve their performance by putting in additional effort, when it is supported by information relevant to their needs.

This seems to be a good way of using CAA in coursework for simultaneous formative and summative assessment, to provide helpful learning experiences for students in the sciences.