to prove, to improve or to learn? lessons for evaluating technical, practical and vocational...
TRANSCRIPT
To prove, to improve or to learn? Lessons for evaluating technical, practical and vocational educational initiatives from an analysis of STEM evaluations
Edge Research Conference 2012
Friday November 16th
Aims:
Drawing on an analysis of evaluations in STEM education we will:
• identify some of the problems with such evaluations;
• examine some potential solutions for the future;
• and examine the applicability to wider TPVL.
Introduction
CSE and CEIR
Science and Innovation Observatory
Priorities
•Research, intelligence, evaluation, polemics•Informing and influencing•Independent body
Reports, think-pieces, associates
Background - what is STEM and how does it relate to TVPL?
• In UK - developed from SET for Success (Roberts, 2002)
• STEM Framework launched 2007-8• Development of a set of 11
'ActionProgrammes'• Each of these contains more
projects, many of which were evaluated - these from the bulk of our analysis
• Technical vocational routes part of some Action Programmes, but main focus was academic
What do we know about STEM evaluations?
• Analysis of 20 STEM evaluations:– 13 Projects/activities or programmes– 4 Event evaluations– 2 Evaluations of organisations– 1 CPD evaluation
Examined:
• Aims• Timings• Methods• Evaluation models • Use of prior evidence
• Results and outcomes• Impact on policy and
practice• Limitations • Contribution to knowledge
Key points from the review
• Evaluation aims were not always explicitly stated.
• Timings do not always appear to match the purposes of the initiative being evaluated.
• Robust counterfactuals were rarely used.• Explicit evaluation models were used in only a
small number of cases.• Reviews of literature, policy or similar initiatives
were not usually presented.
Key points from the review continued• Negative results and were not usually presented
in the same depth as positive results.• Few evaluations looked to make
recommendations beyond the project at hand.• Evaluations tended not to make explicit their
limitations.• Contributing to a developing STEM knowledge
base is very rare in the evaluations we looked at.• Conclusion: The potential for learning from
these evaluations is severely limited.
Linked to key point: the purposes of Evaluation
ControllingTo understand whether the project is going to plan
ProvingTo understand if the project is achieving what was intended
ImprovingTo understand how to modify the initiative to make it work better
LearningTo provide transferable insights to help build a body of knowledge beyond the project at hand
Responses :1A single evaluation framework?• E.g. Stake’s (1996) Stufflebeam (2002):
Cronbach's (1982)• Each of these organises the focuses of
evaluation into three broad areas:• context [antecedent, context, unit of
focus/setting];• process [transaction, input/process,
treatment]; and• outcome [outcome, product,
observations/outcomes].
Responses :1 Guskey?reactions
learning
organisational support and change
use of new knowledge and skills
student outcomes
• No support for this idea:
• One approach could not be designed that would be appropriate to the aims of every STEM project or evaluation.
• A multiplicity of approaches allows greater fit, flexibility and creativity: and hence is more likely to lead to transferable learning.
Responses 2: Theory-based approaches
There are a number of well established 'theory-based' approaches e.g. Realist evaluation; Theory of Change.These develop hypotheses about the social world, and test them out using a variety of means. Close to the scientific method.
EXAMPLE - Interventions aimed at directly improving students’ attitudes to STEM subjects
• EXAMPLE THEORY - using interesting, innovative opportunities to learn improves attitudes to STEM hence improved learning outcomes and interest in STEM careers (e.g. After school Science and Engineering Clubs; Engineering Education Scheme)
Next steps for STEM:1. Development of effective use of theory-based
approaches to evaluation.2. Systematic mining of current evaluation and
research to develop a bedrock of evidence of the theoretical bases for initiatives, and their effectiveness in various contexts.
3. A commitment to using and building the evidence base through evaluation and research.
Next steps for technical, practical and vocational learning (TVPL)? Questions:
• Is there evidence that there is a similar lack of impact of evaluations in relation to TVPL?
• What analysis needs to be done to help answer this question?
• What needs to be done in TVPL to improve evaluation - and to what extent do the prescriptions in this paper for STEM evaluation apply to TVPL?
