Electronic Management of

Assessment – Critical Success Factors in

Institutional ChangeDr Barbara Newland, University of Brighton

Dr Neil Ringan, Manchester Metropolitan UniversityLindsey Martin, Edgehill University

Electronic Management of Assessment (EMA) includes a wide range of activities so the following definitions are being used:

Definitions

eSubmission electronic submission of an assignment

eMarking electronic marking (including offline marking eg in Word)

eFeedback electronic feedback (ie text, audio, video but not hard copy)

eReturn electronic return of marks

A network of senior staff in institutions engaged in promoting, supporting and developing technology enhanced learning

Over 130 nominated Heads from UK Higher Education institutions

A regular programme of well attended events

Represents the interests of its members to various national bodies and agencies including the Higher Education Academy and JISCwww.helf.ac.uk

Heads of eLearning Forum (HeLF)

AimTo identify critical success factors relating to

institutional change with EMA from longitudinal research over the past three years which has: Identified current practice with regard to EMA in

UK HEProvided a snapshot of the strategic overview

identifying key issues relating to strategic change, policies and practices

HeLF members were surveyed each March for 3 years from 2011 and asked to respond with regard to their knowledge of their own institution

The questions were a mixture of closed multiple-choice/multiple-answer and multiple selection as well as open response type

Participants were assured that all data collected in the survey would be held anonymously and securely

No personal data was asked for or retained unless the participant indicated a willingness to be contacted in the future

The results were analysed using quantitative and qualitative methods

Methodology

ResultsThe results emphasised the importance of assessment

It is mission critical as it determines whether students achieve, progress and gain awards (and at which level)

Consequently, all stakeholders in the assessment process are wary of making errors and need to be convinced of the merits in changing long-established practices

The change in practice supported by technology is only worthwhile if it effectively enhances stakeholders’ experiences and/or improves the overall process workflow

ResultsThe following comments illustrate the scale of

the change being undertaken:

‘e-Submission is quite straightforward. However, we could have managed the process of e-marking and e-feedback better. Academic staff need a lot of time to come round to the idea if they are changing years of established practice.’

‘Never underestimate the effort involved with winning hearts and minds of colleagues.’ (HeLF, 2014)

Critical Success FactorsInstitutional

culture Regulations Policy Guidelines

Support Implementation Plan

Communication

Responsibilities

Monitoring Risks Technology Health and safety

Resourcing – support staff

Attitudes of academics,

administrators, students

Quality assurance

Institutional culture An institutional culture encompasses all the critical success factors. The

change process needs to align with this culture. In some universities there has have been mandatory policies and in others a more flexible approach or a mixture of both:

‘It depends very much on the culture of your organisation. Here a partnership model between central services, and School / faculty admin and academics tends to work. Top down imposition of systems is less successful. In other institutional cultures this may be different.’

‘Policy led the mandatory eSubmission. eFeedback, while voluntary, is being adopted by staff as it is available in the same interface.’

‘One of the emerging ideas is to see submission, marking and feedback as separate entities, i.e. enabling students to submit and receive feedback online but staff to mark by whatever method is comfortable. Also not to insist on a particular feedback tool as long as the virtual point students pick up the feedback is consistent.’ (HeLF, 2014)

Institutional cultureThe EMA process involves everyone in a

university so identifying a leader of change can bring all the people together.

‘Having a central 'head of elearning' role to bring everything together across the institution: to scour the Regulations and relevant policies to produce a set of key questions for consideration by each of the key stakeholder groups.’ (HeLF, 2014)

Institutional cultureConsultation of all stakeholders and support from

senior management as well as the efforts of departmental champions and students is key:

‘Consultation was important to choose a system that would fulfil the criteria of all stakeholders. The stakeholders themselves shaped that system. The drive from students was critical in making sure people engaged with this’.

‘Senior Management support. Departmental champions. Peer and student pressure on academic staff concerning provision of eFeedback and eMarking.’ (HeLF, 2014)

RegulationsAssessment regulations will need to include EMA processes.

Often this requires just a small amendment to existing regulations. These will need to go through the committee process, which may take some time.

It may be helpful to consider how the current policy handles situations in which a student is unable to hand in a hard copy. For example, the outcome for a student who hands in work late

because his own computer fails could be the same as if the reason late submission had been missing a bus

Similarly, the outcome for a student who cannot hand in work as the university is shut due to snow could be the same if the student cannot hand in work because a university system has failed.

Policy and guidelinesPolicies

clear, up-to-date have to be agreed and approved by committees. This

may take some time

Guidelines: need to be made easily available to students,

academics, administrators and departments give clear instructions on effective ways of using the

systems and explain the responsibilities in alignment with the regulations and policies

describe contingency plans for particular scenarios occurring.

Technology Both the end user hardware and software and the central systems

need to be robust, reliable and secure

User software should be user-friendly and have the features and functionality to improve the assessment process for students, academics and administrators

There are benefits if the software integrates with other central systems such as the student record system and the Virtual Learning Environment.

‘Reliability / availability of both the VLE and Turnitin essential.’

‘Achieving a smooth system that supports the process well (technological perspective’ (HeLF, 2014)

Some departments provide second monitors and/or laptops to academics for eMarking and eFeedback.

Support Support and training both face-to-face and online is required for

academic and administrative staff as well as Helpdesk support.

Learning technologists have a key role and the move to eSubmission has generally created more work for them (Newland, 2011)

Students may require support in accessing and understanding the method of eFeedback

Support will be required both in advance and then to handle urgent queries from students and staff as the deadline approaches and stress levels increase with the urgency of submitting on time and overcoming any problems in this process

Pedagogical support to encourage effective feedback practices can enhance student learning. The quality of eFeedback is dependent on the quality of the feedback

Implementation PlanIdentifies what, when, how and who is

responsible

It needs to include the detail of the EMA processes e.g.How to handle extensions, second marking, blind

marking, re-sits. Is it e-only Submission or e and paper? If eFeedback is optional then who prints and who

pays for it.

CommunicationAlways important during a change process

Not only about the process of the change but also highlighting the benefits of engaging in the change.

Needs to be two way so that the implementation can be modified as the result of feedback and people participating in the change process are aware of the impact of their feedback

This helps to develop a sense of ownership/partnership with the change rather than it being perceived as something enforced.

ResponsibilitiesEMA will change the roles of academics,

administrators and learning technologists

This change may vary within courses and departments within a university

The consideration of responsibilities needs to be within a university’s EMA guidelines.

Monitoring and risksMonitoring

The implementation of EMA needs to be monitored and exceptions, for example, to e-only Submission to be identified and reported

The monitoring can also highlight effective practice.

RisksAssessment is mission critical and high risk in every

aspect.Appropriate planning for, and mitigation against,

potential identified risks occurring should be embedded within EMA policies.

Health and safetyConcerns expressed by certain staff regarding

the time spent looking at a screen. Usually, these concerns are already addressed through a university’s health and safety procedures

Many academics undertake eFeedback at home on their own computers, and appropriate consideration should be given to this aspect.

ResourcingImplementing change requires resourcing not

only in hardware and software but also for support and engagement in reviewing the whole EMA process.

AttitudesThe move to eSubmission is generally supported by

students, academics and administrators. However, many academics are initially reluctant to engage with eMarking and eFeedback. Positive experiences from students and staff encourage the change.

‘Positive student feedback through module evaluations and staff, student consultative committees. Exams & Assessments dept are positive about new processes.’ (HeLF, 2014)

Academic staff attitudes to eMarking are steadily changing as there is some recognition of the perceived value, benefits and efficiency and acknowledgment that it will happen in future.

Quality AssuranceeMarking and eFeedback can provide a more

consistent student experience. For example, by using a course or departmental rubric

However, greater standardisation may deter innovation in assessment

ConclusionThe critical success factors in the use of EMA as it

has moved from individual early adopters to more formalised institution–wide usage have been identified through a series of annual HeLF surveys

Consideration of these factors within an institutional context can enable the effective implementation of EMA

These factors involve everyone in a university and need to be considered simultaneously for effective implementation of, and change to, EMA

References Gibbs, G., Simpson, C., & Macdonald, R. (2003). Improving student

learning through changing assessment https://www.open.ac.uk/fast/pdfs/Gibbs%20and%20Simpson%202003.pdf

HeLF, (2014), www.helf.ac.uk

Newland, B., Martin, L., Ramsden, A. ( 2011). eSubmission – UK Policies, Practice and Support. Proceedings of the European Conference on e-Learning: 74-82.

Newland, B., Martin, L., Bird, A., & Masika, R. (2013). HeLF Electronic Management of Assessment Report, www.helf.ac.uk

Nicol, D., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Studies in Higher Education, 31: 2, 199 — 218.

Sadler, D.R. (1989). Formative assessment and the design of instructional systems, Instructional Science, 18, 119-144.

Dr Barbara Newland b.a.newland@brighton.ac.uk

Dr Neil Ringann.ringan@mmu.ac.uk

Lindsey MartinLindsey.Martin@edgehill.ac.uk

Contact details