dmw 2006 the open university's institute for educational technology forward planning: findings...
Post on 20-Dec-2015
212 views
TRANSCRIPT
![Page 1: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/1.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Forward Planning: Findings from a Roadmap for e-Assessment
Denise Whitelock
Institute of Educational Technology
The Open University,e-mail: [email protected]
![Page 2: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/2.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Background
• e-Assessment is of strategic importance to the UK
• Major global growth industry
• Recognised by JISC’s e-Learning Programme
• Funded by JISC to produce a Roadmap
![Page 3: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/3.jpg)
DMW 2006 The Open University's Institute for Educational Technology
What is a roadmap?
• Obtain a consensus view or vision of the future landscape of e-assessment to decision makers
• Kostoff & Schaller (2001)
• roadmapping provides a way to identify and evaluate and select strategic alternatives for the desired objective
![Page 4: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/4.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Road mapping
• Technology push
• Requirements pull
• Layers
• Know why – envisioning activities
• Know what – policy and theoretical developments
• Know how - technological
• Time Dimension
• Adapted from Phaal, Farrukh et al, 2004
![Page 5: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/5.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Adapting the Delphi Method
• Delphi questionnaire often used http://www.is.njit.edu/pubs/delphibook/index.html#toc
• Consensus
• Reduced to lowest common denominator
• No priorities emerge
• This consultation will identify core and outlying issues – avoid polarities
![Page 6: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/6.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Application of the Delphi Method
Select panel members based on the expertise required
Develop Questionnaire
Distribute questionnaire
Analyse responses
Determine whether a consensus has been reached
If no consensus seek further information
![Page 7: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/7.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Building the e-Assessment Roadmap
• Map current terrain
• Review current policy and initiatives
• Identify current practice
• Recognise practitioners’ concerns
• Check whether rhetoric matches reality
• Gauge stakeholders’ visions
• Recommendations for keeping on track
![Page 8: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/8.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Roadmap framework
![Page 9: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/9.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Sourcing the information for Stage 1
• Superinstitutional
• Policymakers : HEFCE, SQA etc.
• Institutional
• HE, FE etc.
• Personal
• Tutors, students, developers, commercial companies
![Page 10: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/10.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Superinstitutional: Policies and Practices (1)
• All student centred
• Personalised learning/e-portfolios
• Instant feedback
• Learner control
• More choice
![Page 11: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/11.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Superinstitutional: Policies and Practices (2)
• e-Assessment will be rolled out in post-16 education by 2009
• e-Assessment will make a significant contribution to reducing the assessment burden and improving the quality of assessment
• e-Assessment field trials should be taking place in at least two subjects per awarding body during 2005
• 75% of key and basic skills tests will be delivered on screen by 2005
• N.B. 2009 deadline made explicit by QCA but not SQA
![Page 12: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/12.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Institutional Drivers
• Increase in student retention
• Enhanced quality of feedback
• Flexibility for distance learning
• Coping with large student numbers
• Objectivity in marks
• More effective use of VLE
![Page 13: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/13.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Institutional Vision
• More learner control
• Feedback
• Flexibility
• Embedding the e-learning within the institutional tools
![Page 14: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/14.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Consulting Experts with a Survey
• Survey designed to probe the following:
• Timings of policy implementation
• How e-Assessment can cut the burden of assessment
• Ways in which the quality of e-Assessment will be improved
• Implications of the vision of the policy documents
• Visions for the future
• Barriers to the visions
![Page 15: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/15.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Experts
• Superinstitutional
• Representatives from SQA, HEFCE, BECTA etc.
• Institutional
• Directors of e-Learning Units and policy makers in HE and FE
• Personal
• Champions of e-Assessment
![Page 16: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/16.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Survey Findings (1)
• Predicted timings:
• Agree with 2010 deadline
• ICT accepted in all aspects of student experience within 2/4 years
• Tutors will have tools to assist with course design and providing electronic feedback to students within 2/4 years
![Page 17: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/17.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Survey Findings (2)
• Cutting the burden
• Technological change means revamp of old practice
• Questions need to be more interactive
• Efficiencies is in results processing, transparency in grading, develop criterion referenced exams
• Introduce formative assessment
• Faster feedback
• Better plagiarism detection
• Production of essay marking tools
• Accessibility issues require higher profile
![Page 18: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/18.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Survey Findings (3)
• Improving the quality of e-Assessment
• Offer more realistic assessments
• Cover wide range of curriculum
• Forces assessors to produce clearly defined objectives that enhance student learning
• Better regular feedback will assist students who usually under-perform
• More improvement seen in vocational sphere
![Page 19: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/19.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Survey Findings (4)
• Implications for the vision set by the policy documents
• e-Assessment sites will be open 24/7
• More students will study University courses while at school
• Parents will over-pressurise students to take exams too early
• Over-use of on-demand testing does not always increase grades and produces lack of confidence in standards
• e-Assessment to play key role in personalised learning agenda
![Page 20: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/20.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Policy implications
• 50% agreed students studying at own pace will result in higher grades and a lack of confidence in exam style
• Agreed resits should be regulated
• Agreed more breadth and depth of learning will be achieved through e-assessment
• 16 – 18 year olds will be studying some university courses while at school
![Page 21: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/21.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Paperless examinations as recommended by CCEA and Edexcel
• Authors need training
• Authors do not need same technical skills as technology partner
• Research into different question types sorely needed
• Good piloting essential
![Page 22: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/22.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Further Expert Opinions
• Adaptation of existing standards regarding security (BS7799), quality (ISO9000), test delivery (BS7988) and item production (QTI). Commercial developers will need to subscribe to these standards
• Strong encouragement of open-source software licensing models
• An agreed level of interoperability based on national standards.
![Page 23: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/23.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Visions: Superorganisational
• Large-scale testing sites
• Test sites will deliver some paper and pencil components
• Prevalent from primary through to university and vocational qualifications
• High quality question banks
• No longer referred to as e-assessment
![Page 24: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/24.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Visions: Organisational
• e-Portfolios increase
• Formative, self assessment and e-portfolios main tools
• Random internal testing
• Confidence declared in systems by teacher, awarding bodies, students and general public
• More summative assessment in HE
• Flexible entry and examination
![Page 25: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/25.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Visions: Personal
• e-Portfolios exchanged maybe as microchips in business cards
• All e-assessment except for practicals
• e-Assessment seamlessly integrated into day to day practice
• Peer e-assessment
• Assessment for Learning
• More use of diagnostics
![Page 26: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/26.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Barriers: Superorganisational
• Customer attitudes
• Lack of public confidence
• Negative stories in the media
• Lack of integration of policies
• Key antenna of quality, accessibility, reliability and security are evaluated with pilot activities
![Page 27: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/27.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Barriers: Organisational
• Lack funding to encourage pilot and R & D
• Lack of recognition in RAE
• Not engaging with in proven technology
• Lack infrastructure to pilot
• Lack of item banks
• VLE can push out champions
![Page 28: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/28.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Barriers: Personal
• Staff attitude
• Availability of resources
• Lack of IT skills in a pilot project’s target student population
![Page 29: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/29.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Barriers to adoption
• Reformation of methods
• Training and development
• Resources
• Technical
• Customer attitude
• Staff attitude
• Learner attitude
• VLE
![Page 30: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/30.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Recommendations: Superorganisational
• Clear policy directives
• Government agencies to support directions
• Standard system not acceptable to awarding bodies
• Standards for systems spelt out by QCA
• Initiatives to share expertise
• Recognition of training
• Encourage open source licensing models
• Co-ordinate dissemination at BECTA/JISC levels
![Page 31: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/31.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Recommendations: Organisational
• Tool development
• Examine role of VLEs on progress
• Follow SQA’s lead COLA and SOLAR
• Find self and peer assessments
• Item banks
• Plagiarism
• Research into automatic text recognition
• Guidelines for best practice
• Accessibility
![Page 32: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/32.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Recommendations: Personal
• Training
• Recognition of effort/time
• Assist champions
• Fund studies of changes to job allocation and staff culture
• Appoint mentors
• Free text entry needed for student assessments
![Page 33: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/33.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Research and Development
• VLE
• Evidence needed:
• Learning objectives match e-assessments
• e-Assessment promoting learning control and reflective learning
• Pilot on demand testing
• Gauging emotional pressures as well as cognitive gains
![Page 34: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/34.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Priorities
• Sort out authentication and plagiarism issues
• Train staff to devise e-assessments
• Build good test item banks following the lead of SQA
• Pass on good practice tips from current cohort of students to new learners using e-assessment for the first time
• Make more testing sites for summative assessment available and pass on good practice
• Support more research into automatic marking of e-assessment
• Address accessibility issues directly through superorganisational directives
• Public acceptance of high stake assessments needs to be achieved through dissemination of key findings using press releases, TV and press editorials and PR techniques
![Page 35: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/35.jpg)
DMW 2006 The Open University's Institute for Educational Technology
![Page 36: DMW 2006 The Open University's Institute for Educational Technology Forward Planning: Findings from a Roadmap for e-Assessment Denise Whitelock Institute](https://reader030.vdocuments.us/reader030/viewer/2022032801/56649d535503460f94a2fea6/html5/thumbnails/36.jpg)
DMW 2006 The Open University's Institute for Educational Technology
Summary
• Policy pressure main driver in England and Wales• Affects FE more than HE• More reliance than maybe anticipated on the
commercial sector • More use of formative e-Assessment• More research funding into text recognition becoming
acknowledged• Andragogy rather than pedagogy?
Roadmap final report http://www.jisc.ac.uk/elp_assessment.html