1

Encouraging reflective learning through formative assessment: a learning outcomes approach

Arlëne G. HunterCentre for Open Learning in Maths, Science, Computing and Technology (COLMSCT), The Open University

2

Session overview

• brief overview of the project – development of an online formative assessment tool (SOFA)

• review perceptions of the value of the student online formative assessment as a reflective learning tool– methods of evaluation

– outcomes and methods of engagement

– preliminary conclusions

3

Background to project – the challenge

• new OU level 2 course on Earth System Science – academically course is conceptually challenging

– students need to draw on broad range of skills and prior knowledge (Biology, Chemistry, Earth Sciences, Maths, Physics)

– text co-published with external partner altering standard method of in-text distance learning support

Challenge• devise mechanism to support and inform all stakeholders

of academic progress throughout course

• SOFA: Student Online Formative Assessment

4

Book 1

seven SOFAs linked to Chpts 1-7

each SOFA – 10 questions, 2 levels

Revision SOFA – random selection – specific outcomes

3-stage feedback

end results learning outcomes based– overall result– per learning outcome

mixed format (different learning styles/ skills)

SOFA framework

complexity of learning increases between SOFAs

accessed from course website

5

6

Original goals/objectives• formative

– ‘safe’ learning environment to test depth of learning and application– enhance student awareness of personal strengths, weaknesses,

achievements and progression

• adaptive – match questions to learner competence [logistically challenging]

• integrated– embed SOFAs into assessment, course and programme strategies

via learning outcomes approach– involve all stakeholders in feedback process on academic

progression

• timely – offer ‘just in time’ and self-directed study options

7

Initial challenges and problems• student-course interaction an unknown entity

– how would they engage with an online formative assessment– what academic/ time demands would affect engagement

• question setting – what represents the ‘norm’– adaptivity = 10-15 × question = question bank– dealing with non-standard characters– coping with discursive questions and answers

• stepped learning– designing questions not activities

• time and resource for development

8

Question Types

Question type Method of interaction

assign scientific term free text; drag and drop

MCQ (1 or more answers) radial buttons

complete a table free text; drag and drop

link geological processes/ interactions

free text; drag and drop

label diagrams drag and drop

simple/complex calculations (1-2 steps; sig. figs; sci. not, units…)

numeric/text entry; use or find equation, graph, info…

interpret graphs, diagrams, equations

varied

multi-format combination

9

e.g. Assign scientific term

10

e.g. MCQ

11

e.g. Complete a table

12

e.g. Link geological processes

13

e.g. Label diagram

14

e.g. Two-step calculation

15

e.g. Interpret graphs

16

e.g. Multi-format question

17

Final formative feedback

18

Learning outcomes summary

19

Evaluation of SOFAs• Metadata collected by system (all users)

– number of users (62% in total; range 15-52%; ave 27%), timing and frequency of use per student

– time spent per question/ per assessment (ave ~70 mins)– all answers (analyse common errors ; difference in scores in chapters vs

revision SOFAs)

• Question 11 at end of each SOFA (all users)– quick feedback questions – perceived usefulness: test understanding, measure

progress, revision aid– free-text comments box

• generic questionnaire on Book 1 usage (whole community)– which (if any) SOFAs used– how did they help learning

• Success Case Method targeted interviews – specific subsection of community– exemplar success and non-success cases

20

Success Case Method (SCM) evaluation

• offers quick means of gaining quantitative and qualitative insight

• develop models of absolute success and absolute failure • meaning of success will vary between stakeholders

• failure does not equal absence of absolute success

• collect extreme robust (provable) stories to identify drivers behind experiences

• focus on “what, so what, now what?”

• how to make improvements

21

SCM process

• students and tutors invited to complete generic questionnaire

• gain two perspectives on perceived value of SOFAs

• 12 respondents selected for detailed interviews• ~5% of all SOFA users interviewed* • 9 potential success cases• 3 potential non-success cases

(*based on total of 270 users who accessed at least 1 SOFA between February – May 2007; ideally, 10% is preferred representative quota)

22

SCM Interview ‘buckets’

Success interview buckets

• what was used that worked?

• what results were achieved?

• what good did it do (what was its value)?

• what helped?

• suggestions?

Non success interview buckets• barriers?• suggestions?

23

SCM – success case summaryWhat was used that worked?• access via course website ‘Check your understanding...’

• repeat function and ‘changing questions’

• instant feedback on effectiveness of their learning approach

What results were achieved?• obtained ‘impartial’ insight into their strengths, weaknesses and

misunderstandings

• demonstrated ongoing progress

• instant pacing mechanism – review learning approach/materials

What was its value?• focussed learning and helped direct areas in need of ‘extra work’

• built confidence in personal abilities and progress

• provided ‘real’ measure of personal achievement

formative, integrated, adaptive

adaptive, timely, formative

formative, timely, integrated

24

SCM – success case summary cont/.What helped?• flexible, formative structure – instantly adaptable to own needs targeted

feedback on specific errors and omission

• positive affirmation of good work

• putting all learning (content and skills) into context

Suggestions• printable version of questions and final explanation for revision

BUT recognition that targeted feedback needs SOFA to be online

General benefits• consolidated and clarified learning

• identified gaps, misunderstandings, “unknown unknowns” before it was too late and offered advice on how to improve

• shift from seeing reflective feedback to using focussed feedback

formative, integrated

formative, adaptive, integrated, timely

formative, adaptable

25

SCM – non success case summaryBarriers – general issues• working online – prefer paper, more flexible, more familiar

• academic level – too high, so demoralising when wrong

• time pressures – too much to do in course; going online is time consuming

• accessing answers – prefer immediate access to read alongside questions

Barriers to value• expected to get all questions right first time – SAQ experience

• did not like being told ‘wrong’ or to ‘try again’

• too many clicks to get to the final answer

Suggestions• focus on prior background skills and knowledge – gap filling

• downloadable/ print-based – question and answer immediately accessible

• colourful, fun and tactile formative, adaptive

26

Preliminary outcomes• majority of students who tried them, liked the SOFAs and found

them helpful– but only reached on average 27% (total usage 62%); 15% did all chapter SOFAs– how could ‘remainder’ be encouraged to engage and continue to use

• individual students who did the SOFAs were repeating them– quantitative data indicated general improvement in application of learning (content and

skills) on repeats

• qualitative feedback demonstrates more confident learners – greater personal awareness of efficiency of learning approach– growing confidence in personal skills and sense of academic progress– awareness of personal strengths, weaknesses and learning needs – changing perception from reviewing feedback (i.e. what was wrong) to focussing on

feeding forward (i.e. what needs to be developed)

• majority (success and non-success cases) still wanted ‘hard copy’ version for revision– outcome of formative versus summative usage?– familiarity with paper over PC?

formative, timely

formative, adaptive, integrated, timely

? formative, adaptive

formative, adaptive

Dr Arlëne G. HunterTeaching FellowCentre for Open Learning in Maths, Science, Computing and Technology (COLMSCT)The Open UniversityWalton HallMilton KeynesEngland. MK7 6AA

A.G.Hunter@open.ac.uk

28

Links – open access to SOFAsChapter specific SOFAs – Book 1

• https://students.open.ac.uk/openmark/s279-07.book1chapter1world/

• https://students.open.ac.uk/openmark/s279-07.book1chapter2world/

• https://students.open.ac.uk/openmark/s279-07.book1chapter3world/

• https://students.open.ac.uk/openmark/s279-07.book1chapter4world/

• https://students.open.ac.uk/openmark/s279-07.book1chapter5world/

• https://students.open.ac.uk/openmark/s279-07.book1chapter6world/

• https://students.open.ac.uk/openmark/s279-07.book1chapter7world/

Revision SOFAs – random selection of questions• https://students.open.ac.uk/openmark/s279-07.book1chapter8ran1world/• https://students.open.ac.uk/openmark/s279-07.book1chapter8ran2world/• https://students.open.ac.uk/openmark/s279-07.book1chapter8ran3world/• https://students.open.ac.uk/openmark/s279-07.book1chapter8ran4world/

29

Working with diagrams• https://students.open.ac.uk/openmark/s279-07.book1chapter8diagmodworld/ • https://students.open.ac.uk/openmark/s279-07.book1chapter8diagtaxingworld/

Understanding geological processes• https://students.open.ac.uk/openmark/s279-07.book1chapter8geoprocessmodworld/ • https://students.open.ac.uk/openmark/s279-07.book1chapter8geoprocesstaxingworld/ • https://students.open.ac.uk/openmark/s279-07.book1chapter8geotermsworld/

Using geological terms• https://students.open.ac.uk/openmark/s279-07.book1chapter8mathmodworld/ • https://students.open.ac.uk/openmark/s279-07.book1chapter8mathtaxingworld/

Summarising information with tables• https://students.open.ac.uk/openmark/s279-07.book1chapter8tablesworld/

Links – open access to SOFAs