Evaluation meets costing:Evaluation meets costing:Fight or Flight?Fight or Flight?

Charlotte AshCharlotte AshSheffield Hallam UniversitySheffield Hallam University

http://www.shu.ac.uk/virtual_campus/cnl/team/cea.htmhttp://www.shu.ac.uk/virtual_campus/cnl/team/cea.htm

2121stst HE Cost-effectiveness framework HE Cost-effectiveness frameworkC

The Costs of Networked LearningThe Costs of Networked Learning

Six month study funded by the JISCSix month study funded by the JISC Additional support provided by Sheffield Additional support provided by Sheffield

Hallam UniversityHallam University Project team - director, manager and Project team - director, manager and

additional researchersadditional researchers For further information, visit the Costs For further information, visit the Costs

of Networked Learning Web page: of Networked Learning Web page: http://www.shu.ac.uk/virtual_campus/cnl/http://www.shu.ac.uk/virtual_campus/cnl/

2121stst HE Cost-effectiveness framework HE Cost-effectiveness framework

CostCost

problemsproblems

solutionssolutions

What has been done alreadyWhat has been done already

EffectivenessEffectiveness

problemsproblems

solutionssolutions

C

The Future of Higher EducationThe Future of Higher Education

Universities must:Universities must:

Change infrastructureChange infrastructure

Increase accessIncrease access

Reduce expenditureReduce expenditure

Become Become transparenttransparent

Assessing Cost-EffectivenessAssessing Cost-Effectiveness

Stage 1: Identify problems with costsStage 1: Identify problems with costs

Stage 2: Find solutionsStage 2: Find solutions

Stage 3: Testing and refinementStage 3: Testing and refinement

Stage 4: Identify effectiveness problemsStage 4: Identify effectiveness problems

Stage 5: Find solutionsStage 5: Find solutions

Stage 6: Testing and refinementStage 6: Testing and refinement

Stage 7: Joint solution toStage 7: Joint solution toCost-effectivenessCost-effectiveness

Assessing CostsAssessing Costs

ProblemsProblems

1 Lack of literature1 Lack of literature

2 What to cost?2 What to cost?

3 How to cost?3 How to cost?

4 Whom to cost?4 Whom to cost?

Assessing Costs: Problem 1Assessing Costs: Problem 1

ProblemProblem

Lack of literatureLack of literature

Lack of studiesLack of studies

Lack of frameworkLack of framework

SolutionSolution

Eclectic and Holistic Eclectic and Holistic

approachapproach

Assessing Costs: Problem 2Assessing Costs: Problem 2

ProblemProblem

WhatWhat to cost? to cost?

SolutionSolution

Identify Identify hidden hidden costscosts

Formulate list for Formulate list for

each stakeholder: each stakeholder:

staff, student and staff, student and

institutioninstitution

Structure list into Structure list into

working modelworking model

Lifecycle modelsLifecycle models

Planning the learning experience/ environment

Maintaining the learning experience/ environment

Developingthe learning experience/ environment

Managing the learning experience/ environment

Providing the learning experience/ environment

Academic Staff

Support Staff

Students

Planning and Development

Productionand

Delivery

Maintenanceand

Evaluation

Final three stage model

Previous five stage model

Expendituredimension

Stakeholder dimension Total

Institution Student Staff

Staff costs

Depreciation

Expenses

Overhead

Total

Assessing Costs: Problem 3Assessing Costs: Problem 3

ProblemProblem

HowHow to cost? to cost?

SolutionSolution

Planning frameworkPlanning framework

• Planning documentPlanning document

• Financial schemaFinancial schemaFinancial SchemaFinancial Schema

Assessing Costs: Problem 4Assessing Costs: Problem 4

ProblemProblem

WhomWhom to cost? to cost?

• Whom to include?Whom to include?

• Who to do?Who to do?

SolutionSolution

• StakeholdersStakeholders

• Translator/NegotiatorTranslator/Negotiator

SolutionsSolutions

1 Eclectic/Holistic approach1 Eclectic/Holistic approach

2 Hidden, List2 Hidden, List ModelModel

3 Planning framework3 Planning framework

• planning documentplanning document

• financial schemafinancial schema

4 Whom to cost:4 Whom to cost:

• StakeholdersStakeholders

• Translator/NegotiatorTranslator/Negotiator

Assessing CostsAssessing Costs

ProblemsProblems

1 Lack of literature1 Lack of literature

2 2 WhatWhat to cost? to cost?

3 3 HowHow to cost? to cost?

4 4 WhomWhom to cost? to cost?

• Whom to include?Whom to include?

• Who to do it?Who to do it?

Assessing EffectivenessAssessing Effectiveness

Reflections on a Model forReflections on a Model forEvaluating Learning TechnologiesEvaluating Learning Technologies

by Jones, Scanlon and Blakeby Jones, Scanlon and Blake

CIAO! evaluation frameworkCIAO! evaluation framework developed by the Computers and Learning developed by the Computers and Learning

Research Group at the Open UniversityResearch Group at the Open University

““Do my students learn better, fasterDo my students learn better, fasteror more using this technology?”or more using this technology?”

Assessing EffectivenessAssessing Effectiveness

EvaluatingEvaluatingTechnology-Based LearningTechnology-Based Learning

bybyDr Robin MasonDr Robin Mason

Holistic evaluationHolistic evaluation Good evaluations demandGood evaluations demand

skill, perception and resourcesskill, perception and resources

Flashlight 1994 - 1999Flashlight 1994 - 1999

Current Student Inventory Current Student Inventory ((19941994))

• effectivenesseffectiveness

Costs Analysis Handbook Costs Analysis Handbook ((19991999))

• costscosts

Problems:Problems:1 Costs and effectiveness kept separate1 Costs and effectiveness kept separate

2 Individualistic approach prohibits comparisons2 Individualistic approach prohibits comparisons

Assessing Cost-EffectivenessAssessing Cost-Effectiveness

Stage 1: Identify problems with costsStage 1: Identify problems with costs

Stage 2: Find solutionsStage 2: Find solutions

Stage 3: Testing and refinementStage 3: Testing and refinement

Stage 4: Identify effectiveness problemsStage 4: Identify effectiveness problems

Stage 5: Find solutionsStage 5: Find solutions

Stage 6: Testing and refinementStage 6: Testing and refinement

Stage 7: Stage 7: Joint solution Joint solution to problem ofto problem of

evaluating the cost-effectiveness of HEevaluating the cost-effectiveness of HE

Evaluation meets costing:Evaluation meets costing:Fight or Flight?Fight or Flight?

Charlotte AshCharlotte AshSheffield Hallam UniversitySheffield Hallam University

http://www.shu.ac.uk/virtual_campus/cnl/http://www.shu.ac.uk/virtual_campus/cnl/

Thank you for listeningThank you for listening