DURHAM

•a highly-rated and long-established university

•a world heritage site•a beautiful, small city•Less than 3hrs from

London by train (Kings Cross station)

•Flights to Newcastle

Durham

1. Mentoring under-aspiring teens2. Five years’ help to ‘at risk’ teens3. Summer camps for ‘at risk’ teens - - 4. Reducing Class size from 35 to 20 +5. Computer Assisted Instruction ++6. Cross-age Tutoring ++++7. Increased time on basics +8. Teaching assistants 09. Integrated Learning Systems ILS -10. Driver training programmes --11. Counselling following trauma --12. Scared straight

What works? + = it works 0 = no impact - = it harms

GUESS …….

One task: to One task: to make a BETTER WORLDmake a BETTER WORLDWho else … if not

educators… ??……who deliver to each childwho deliver to each child

15,000 hours15,000 hours of compulsory treatmentof compulsory treatment

teenage drunkeness and teenage drunkeness and violence…...violence…...

England is top in Europe on oneLeague Table:

?

Desirable characteristics of indicators• RELEVANT RELEVANT - - to the unit of management

» In schools - - - the classroom

• INFORMATIVEINFORMATIVE about input, process, output and outcomes

• FAIRFAIR to all concerned, unbiased, reliable, not subject to demand characteristics

• BENEFICIABENEFICIAL in their impact…. This may require confidentiality - - - see CH. 6 of our report to the govt. (QCA/SCAA/DfES):

» www.cem.dur.ac.uk/downloads

Mnemonic:

BUT what matters? ..There are many lists of indicators ……

but they all generally share one feature - - -

NON-MEMORABLE THINGS NEED MNEMONICS ..… so...

*cf Bloom's Taxonomy of Educational Objectives, 1956

DOMAINS TO BE MONITORED

heart

body

GOALS

/OUTCOMES

mind

GROUPS

£ $

POLICIES

Affectivee.g. attitudes, aspirations, quality of life

Behavioural

e.g. skills, cooperation, health behaviour

Cognitive

e.g. achievements, beliefs

Demographic descriptors e.g. sex, age, SES

Expenditures e.g. resources, time and money

FLOW e.g. WHO is taught WHAT, HOW ,

and for HOW LONG?:curriculum balance, retention, methods,time

© CTF-G 2000

A

B

C

D

E

F

Value added

Two kinds of data

Monitoring Experiments

Passive

observations

Active

interventions

CLINICAL TRIALSCLINICAL TRIALSEPIDEMIOLOGYEPIDEMIOLOGY

Initiatives

Indicators, SURVEYS EXPERIMENTS,

What’s happening?

What works?

LARGE CLASSES OR SMALL CLASSES - - -

which give the best results… for achievement?

Surveys show: LARGE CLASSES

Experiments show

SMALL CLASSES ! ?

AFFECT - - - feelings

attitudes…etc

AA

©2001 C CTF-G

Psychological ‘debriefing’Psychological ‘debriefing’ aimed to prevent aimed to prevent PTSD PTSD (Post Traumatic Stress Disorder)(Post Traumatic Stress Disorder)

A Cochrane Review from BMJ (British Medical Journal)A Cochrane Review from BMJ (British Medical Journal)

11 trials of this TREATMENT after a trauma:

Emotional processing/ventilation

by encouraging recollection/reworking of the traumatic event

accompanied by normalisation of emotional reaction to the event’

©2001 C CTF-G

Impact of Psychological Debriefing followinga HIGH IMPACT trauma

0

5

10

15

20

25

30

35

40

baseline 4 months 3 years

HIGH with interventionHIGH with nointerventionLOW withinterventionLOW with nointervention

©2001 C CTF-G

A Cochrane Review : highest level of evidence……

11 randomised contol trials (RCTs)

“There was... no evidence that debriefing reduced general psychological morbidity, depression or

anxiety.”

“Compulsory debriefing of victims of trauma should cease”

“Least said, soonest mended” ????

BEHAVIOURBB

HMCI number 3

Sunday Times in Parent Power, 21 Oct 2001

‘captureEthos?’Not so!

BB

0

10

20

30

40

50

60

70

80

90

0 20 40 60 80 100 120

Percent non-white in school

% of pupils hearing racist insults used around school at least once a week

PERCENT NON-WHITE IN THE SCHOOL

Worst school(6 of the worst 10 were in

OLDHAM & BRADFORD)

20% 100%

80%

50%

60%

20%

What did the inspection report say about the worst school on racism?

“Relationships in the school are very good.”

COGNITIVE

e.g. achievement

CC

If ‘driving up standards’or ‘pushing kids’ worked….

then standards would have risen

in primary school numeracy……

the Sputnik effect?

modest improvement

They have fallen…….

…with one exception:

STANDARDS have NOT risen - except in primary school

numeracy…..

CC

Report of the Engineering Council * june 2000 re A-levels

* 10 Maltravers St. LONDON WC2R 3ER

‘At least 60 departments of Maths. Physics and Engineering give diagnostic

tests in maths. to new undergraduates’ ‘…strong evidence…of a steady

decline over the past decade of fluency in basic skills…and ..level of

preparation’

The YELLIS* Underaspirers ExperimentCC

*aka YELSIS

No of times counselled

3020100-10

Ave

rage

Res

idua

l

2

1

0

-1

-2

-3

Named?

Y

N

NAMED

THE FATES OF UNDER-ASPIRING PUPILS

Average

Progress Not named

Average ResidualN Mean SD

NOT named 59 -.55 .79 named 61 -.86 .83

YELLIS UNDER-ASPIRERS’ EXPERIMENT 1999 - results

Effect Size = ES = -0.38

ie value added was better if under-aspiring student was

not reported to the school

Repeated experiment 2002n=175 students, ES=0.12

The effect for being named was not statistically significant

On-going…… professional research……ROBUST DISTRIBUTED RESEARCH

COLLABORATIVE RESEARCHREFORMS AS EXPERIMENTS

‘design experiments’Needs qualitative and quantitative data

Reporting on a well funded project - - -

designed to prove its effectiveness………...

…and working with FAMILIES

McCord J. A thirty year follow-up of treatment effects. American Psychologist 1978;33:284-9.

McCordCounselling and support were given over a period of five years to families of ‘at risk’ young males in their teens

Thirty years later they remembered their social workers with affection andreported that the help had led themaway from crime…..

When objective measures were usedbased on records such as

the number of arreststhe seriousness of crimesand recidivism….

…some results showed NO differences( between those helped or not helped)

We may do harm when we intend to do good.

and some showed differences in favour of those who hadNOT been provided with help.

!

Good book!THE NURTURE ASSUMPTION:

parents matter less than parents matter less than you think and peers matter you think and peers matter

moremoreby J.Rich Harris

Dishion et al….1999

Summer camps for at-risk teenagers

Dishion, T.J., McCord, J. & Poulin, F. (1999) When Interventions Harm, American

Psychologist, 54(9), pp. 755-764.

A WARNING TO US ALL….

‘Some mentoring programmes had a negative

impact’

Some mentoring programmes were worse than useless:

they did harm and wasted peoples’ time and money

THE POLITE SUMMARY:

THE BLUNT SUMMARY:

The moral is……..

We must stop guessing and start experimenting

FINDING OUT WHAT WORKS WITH:

Randomised Control Trials

Demographic….E.g. gender

Ethnicity social class

DD For the school….

these are un-alterable variables

- - - WHY so popular with academics?

EFFECT SIZE = ~0.40 for PROOF READING

Scale e.g. aptitude

100100 115 130 14555 8570

GENDER DIFFERENCES - - - very popular with academics

BUT

Scared straightconclusion of Cochrane

summary

Programmes like ‘Scared Straight’

are likely to have a harmful effect

and increase delinquency relative to

doing nothing at all to the same

youths.

Donald T. Campbell

It is one of the most characteristic aspects of the

present situation that specific reforms are advocated as

though they were certain to be successful

1969!

Expenditures• Cost-benefit analysisCost-benefit analysis, in which both inputs and

outputs are measured in monetary units;• Cost-effectiveness analysisCost-effectiveness analysis, in which

comparison re made among alternatives whose inputs and outputs are not solely monetary

• Cost-utility-analysis,Cost-utility-analysis, in which alternative programs are compared based on the costs of inputs and the estimated utility or value of their outputs

RER 2002 72(1)

E

The essential point is to measure the impact and the cost

REFORMS-AS-EXPERIMENTS

Expenditures

PERSPECTIVPERSPECTIVEE

Cost analysisCost analysis

ComparatorsComparators

Estimate Estimate program program EffectsEffects

E

• Goals of the evaluation • Ingredients approach• Existing practice or

reasonable alternatives

• Rigorous experimental… designs with attention to identifying hidden and/or qualitative outcomes……

Protocol for cost-effectiveness studies…

Expenditures continued

Outcome Outcome measuresmeasures

Distributional Distributional consequenceconsequencess

Analysis of Analysis of time Effectstime Effects

E

• Standardized achievement measures or Effect Size if different achievement tests used

• Assign all types of costs and effects to appropriate parties

• Annualize costs, take into account inflation, and discount costs over time

Etc……

1. Mentoring under-aspiring teens2. Five years’ help to ‘at risk’ teens3. Summer camps for ‘at risk’ teens 4. Reducing Class size from 35 to 20 5. Computer Assisted Instruction6. Cross-age Tutoring7. Increased time on basics8. Teaching assistants9. Integrated Learning Systems ILS10. Driver training programmes 11. Counselling following trauma 12. Scared straight

EFFECTS ? +++ = very positive 0 no effect - negative

1. -, 0 2. -3. - - -4. +5. ++6. ++++7. +8. 09. -10. --11. --- 12. --

Many hope that market mechanismsmarket mechanisms of choice and competition can change the

ways that schooling is practiced. Lubienski, AERJ 2003 40(2)

• E.g. Charter Schools in the USE.g. Charter Schools in the US• Grade retentionGrade retention - - -holding back weak

students, by a year…..• THE (weak) EVIDENCE IS THAT:

IT’S THE TEACHER THAT COUNTS!

‘The lure of the structural panacea’ Tyack, 1974, p 160

pre-emptive worryingpre-emptive worrying

is important

LET US JOIN TOGETHER IN WORRYINGLET US JOIN TOGETHER IN WORRYING

in applying science toin applying science to

creating a safe and civil society for all.creating a safe and civil society for all.

Locations of mass graves… Kosovo

Where will it end?

Let’s be WISE

One task…. To make a better world

One method…. science

Dedicated to applying the best standards of evidence to any and all reforms… e.g.

initiativesinterventionspolicies

great ideas

fashionsinspirations

punishments & rewards

so that we can find out if they work….

Inaccurate

Anachronistic

Amateurish

Damaging

Feb. 2002 Feb. 2002 Select Cttee. for Select Cttee. for Education and SkillsEducation and Skills

called forcalled for

‘‘rigorous external rigorous external evaluationevaluation

of the soundness of of the soundness of Ofsted’s methods’Ofsted’s methods’

HUGE PROGRESS!

reforms-as-experiments.org.uk a new charity

Committed to professional development of evidence-based educational practice

 in order to make a better world - - -a civil, safe and fair society - - - .

WHY EDUCATION?15,000 hours of compulsory treatment should be a great

opportunity....IFF we get the evidence right....

WHY EXPERIMENTS?Because guessing isn't good enough

.... good intentions are no guarantee of good outcomes

Home of ALIS, YELLIS, MidYIS,PIPS and ASPECTS

UNIVERSITY OF DURHAM

Curriculum, Evaluation and Management Centre

Prof. Peter Tymms : Director, + PIPS

Dr.Robert J.Coe : Director for Dr.Robert J.Coe : Director for Secondary Schools and collegesSecondary Schools and collegesCarol Taylor Fitz-GibbonEmeritus Professor, former Director

Web site:www.cem.dur.ac.uk

Reforms-as-experiments.org.ukHANDOUTS……

C.T.Fitz-Gibbon@cem.dur.ac.uk0191 33 44 181

cem@cem.dur.ac.uk

R.J.Coe@cem.dur.ac.uk0191 33 44 184