why assessment format matters philip hedges department of economics & quantitative methods...
TRANSCRIPT
![Page 1: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/1.jpg)
Why Assessment Format Matters
Philip Hedges
Department of Economics & Quantitative Methods Westminster Business School
University of Westminster
UK
![Page 2: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/2.jpg)
Aims
• To present an analysis of coursework performance and the influence of two independent variables:
1) the learner’s mode of study 2) the learner’s nationality status
• Casual empiricism suggests that part-time and home/EU students have an advantage over full-time and overseas students – based partly on this the decision was made to alter the coursework format for the 2014/15 academic year
• To quantify the influence of the student’s study mode and nationality on performance, and examine the stability of their influence when an assessment format changes
2
![Page 3: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/3.jpg)
Format of Original Coursework Assessment
• Context: Core PG module for MA HRM students hosted by Economics dept. – original assessment pattern: 50% in-module coursework, 50% end-of-module exam
• Original coursework assessment format: 2,500 words STEEPLE analysis report of a certain market or self-selected markets for students with HR experience
• The original coursework format was designed to be student-centred, useful, engaging and differentiated according to student characteristics
• The brief was set at the beginning of the module and reports were due to be submitted in week 7 of 12 teaching weeks 3
![Page 4: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/4.jpg)
• Original coursework format was satisfactory but a number of issues became apparent:
• a) Marking & moderation of reports on this large PG module slowed down the feedback process (3-4 weeks)
• b) Coverage of the module’s content – not all students used Economics concepts & principles in the report
• c) Differences in differentiated learners’ access to info. etc. seemed to have led to different levels of achievement between Part-Time/Home/EU and Full-time/Overseas students
4
Problems with the Original Format
![Page 5: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/5.jpg)
Distribution of Learner Characteristics: 2010/11 - 2014/15, N = 610
5
![Page 6: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/6.jpg)
Coursework Mean Marks (%)by Mode of Study 2010-11 to 2013-14
6
![Page 7: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/7.jpg)
Coursework Mean Marks (%) by Nationality 2010-11 to 2013-14
7
![Page 8: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/8.jpg)
Opportunity for Change
• The module had to be reviewed internally & externally during Spring 2014 as part of a university course revalidation & accrediting body review by the Chartered Institute of Personnel and Development
• This presented an opportunity to reflect on the assessment strategy with a focus on altering the coursework format
• The new format would hopefully reduce problems a), b) and c) associated with the original format
8
![Page 9: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/9.jpg)
Solution: New Coursework Format
• The STEEPLE report was replaced by an in-class test based on the first 5 weeks’ Economics material
• The first iteration of the test (for 2014-15) was exclusively multiple-choice based and students were given 1 hour 15 minutes to complete 15 questions each worth between 1 and 6 marks
• Students were issued with 2 mock in-class tests & answers between weeks 1 & 5 , given a reading week to revise in week 6 and took the real test in week 7
9
![Page 10: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/10.jpg)
Results: 2 out of 3 Problems Solved• The in-class test was double-marked and given back to
students in week 9 (2 weeks faster turnaround!)
• For the first time in the coursework for this module most students displayed a good level of familiarity with Economics principles and concepts
• Marks were higher than with the original report format with no increase or decrease in the coursework mean differences across the different characteristics
• Therefore, for problems a) and b) the change in assessment feedback was successful – the difference in means across certain characteristics may be insurmountable?
10
![Page 11: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/11.jpg)
Coursework Means (%) by Mode of Study:
2010-11 to 2013-14 and 2014-15
11
![Page 12: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/12.jpg)
Coursework Means (%) by Nationality: 2010-11 to 2013-14 and 2014-15
12
7 percentage points
difference
![Page 13: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/13.jpg)
Unintended Outcomes: Student Feedback• Compared with the 2013-14 cohort the 2014-15
students facing the new assessment format gave much higher ratings on the module evaluation survey:
• The response rate to the survey increased from 29% to 56% - plus unsolicited positive remarks concerning the utility of the Economics discipline for HRM PGs
13
Area of FeedbackAverage % of 4 and 5 scores
(2013-14 average % in parentheses)
Teaching & Academic Support 92 (74)
Resources & Learning Environment 76 (75)
Assessment 92 (79)
Coursework Feedback 89 (76)
Overall 91 (70)
![Page 14: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/14.jpg)
Quantifying the Significance and Influence of Study Mode & Nationality
• The data permits the estimation of the sign, strength and significance of the two independent characteristics on learner performance
• For each cohort coursework marks were regressed on study mode and nationality – then repeated for all cohorts using time dummies
• “Before” and “after” regressions allow us to see if study mode and/or nationality effects are independent of assessment format 14
![Page 15: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/15.jpg)
Coursework Marks Regression Results
15
Independent Variable (1) C1 (2) C2 (3) C3 (4) C4 (5) C5 (6) C1 to C4 (7) C1 to C5
Constant 51.35*** 54.07*** 59.30*** 59.19*** 84.84*** 58.25*** 85.01***
(2.165) (1.614) (1.214) (1.229) (2.496) (-1.039) (1.181)
Study Mode (=1 if Part-Time Student) 1.06 1.62 -1.57 1.11 1.25 0.57 0.67
(3.301) (2.385) (1.781) (1.629) (3.634) (1.106) (1.090)
Nationality (=1 if Home/EU Student) 10.58*** 4.22* 9.53*** 4.85*** 6.21 6.60*** 6.55***
(3.542) (3.542) (1.749) (1.770) (3.771) (1.153) (1.135)
C1 (=1 for Cohort 1 Students) -3.87*** -30.66***
(1.481) (1.638)
C2 (=1 for Cohort 2 Students) -5.15*** -31.93***
(1.062) (1.216)
C3 (=1 for Cohort 3 Students) 1.48 -25.31***
(1.138) (1.285)
C4 (= 1 for Cohort 4 Students) -26.79***
(1.246)
Adj R-squared 0.242 0.035 0.222 0.090 0.042 0.169 0.596
n 55 167 129 146 113 497 610
Baseline for regressions (1) to (5) = Full-Time International students for each cohort from 2010-11 to 2014-15Baseline for regression (6) on 4 cohorts uses the same baseline as (1) to (5) but for 2013-14 studentsBaseline for regression (7) on 5 cohorts uses the same baseline as (1) to (5) but for 2014-15 students
Standard errors in parentheses; * p < 0.1, ** p < 0.05, *** p < 0.01
![Page 16: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/16.jpg)
Interpretation of Regression Results• “Before” regressions (1) to (4) for each annual cohort (writing reports)
suggest that mode of study is not significant in explaining coursework marks; nationality is significant and Home/EU learners outperform their Overseas peers – according to the pooled “before” regression (6) this is worth an additional 6.6 percentage points (which rises to 7 points when study mode is omitted)
• The “after” regression (5) for the 2014-15 cohort suggests that study mode and nationality are no longer significant (though nationality is only marginally insignificant at the 10% level) – the format change has had some effect in terms of problem c)
• The pooled “before & after” regression (7) suggests that nationality is a significant variable in explaining coursework performance though this sample is limited to 19% coming from after the format change (i.e. 81% of observations are before the format change)
16
![Page 17: Why Assessment Format Matters Philip Hedges Department of Economics & Quantitative Methods Westminster Business School University of Westminster UK](https://reader033.vdocuments.us/reader033/viewer/2022061518/5697bfce1a28abf838ca9921/html5/thumbnails/17.jpg)
Future Directions• The first year’s experience of the new format suggests that
the statistical influence of nationality has weakened despite the persistent difference in mean mark between Home/EU and Overseas students
• Control variables for gender, age and UG degree classification may weaken the significance of nationality
• The influence of mode of study and nationality on exam performance is yet to be modelled
• The new coursework format increased total module marks so much that the module exceeded our KPI mean mark ceiling – possible change in the test to include open questions and reducing the time allowed to complete the test 17