Learning Technologies EvaluationMark Jasper

IATEFL, AberdeenFriday 20-April 2007

Mark Jasper

Bio-data• British Council since 1996

• Global Product Development Consultant

• ICT Consultant & E-Learning Developer,in Manchester & Milan

• EFL Teacher, in Ho Chi Minh City, Istanbul & Tokyo

Internet Research

• Evaluation v Research

Googlefight

• Evaluation & Reflection

http://www.askaninja.com/index.php

Contents

• Internet Research

• Evaluation Definitions

• Need for Evaluation

• British Council Research

• Macmillan English Campus Evaluation

• Lessons Learned

• Towards a Generic Tool

Evaluation Definitions

• Gubba & Shufflebeam 1970

“Evaluation is the process of obtaining and providing useful information for making educational decisions”

• Jasper 2007

“Is it any good?”

Need for Evaluation

• Spend, spend, spend

• Product failures

• Impact Assessment 2005 - 2006

• Evaluation Evolution

Learning Technology Research

• BC global network of approx 100 centres

• Supported in-house research

• Ipsos-Mori

• What to believe

Macmillan English Campus

MEC Project

• Supplementary ematerials resource• January 2006 – present• Hong Kong, Bratislava, Bangkok,

Prague, Warsaw• 1000 licences• Pre-intermediate to Upper Intermediate• Varied course lengths• Teacher & learner turnover

MEC Evaluation

• Evaluation Brief

• Evaluation Plan

• Continuous Data Collection

• Qualitative & Quantitative data

Evaluation Brief & Plan• Define problem

• State why evaluation is necessary

• Data type

• Data collection (who, how & when)

• How data will be used

• What we already know

• What we want to know

• KPIs or action statements

Qualitative Data

• Focus groups

• Weekly reports

• Weekly meetings on VOIP

• Knowledge sharing forums

• Workshops

• Telephone interviews

• Consultancy reports

Quantitative Data

• Online self-completion questionnaire

• System statistics

• Reregistration figures

What Went Well

• Early planning

• Avoided evaluation as autopsy

• Evaluation as part of project

• Reasonable samples

• Mixing qualitative & quantitative

• 1 evaluation coordinator + support

What Has Nae

• Getting equal commitment

• Technology

• Meshing in anecdotes

• Behavioural evidence

Towards an Evaluation Toolkit

• Lessons Learned

• Keep a Central Resource

• Ask General Questions

Suggested Resources

• Evaluating Learning Technology Resources:Workbook

• LTSS Guide to Evaluating Learning Technology ResourcesCookbook

URLs from TalkEvaluating Learning Technologies Workbook• http://www.ltss.bris.ac.uk/publications/

guides/evalwkbk/evalwkbk.htm

Evaluation Cookbook• http://www.icbl.hw.ac.uk/ltdi/cookbook/recipes.

html

• http://www.googlefight.com• http://www.askaninja.com/