conference for eef evaluators: building evidence in education
DESCRIPTION
Session 3: Implementation. Conference for EEF evaluators: Building evidence in education. Hannah Ainsworth , York Trials Unit, University of York Professor David Torgerson , York Trials Unit, University of York Professor Carole Torgerson , School of Education, Durham University. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/1.jpg)
Conference for EEF evaluators: Building evidence in education
Hannah Ainsworth, York Trials Unit, University of YorkProfessor David Torgerson, York Trials Unit, University of YorkProfessor Carole Torgerson, School of Education, Durham University
Session 3: Implementation
![Page 2: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/2.jpg)
Session 3: Implementation
• Trial registration and CONSORT (HA – 10 mins.)
• Trial management (HA - 30mins)• Model/approach (HA – 5mins)
– Discussion (5mins)
• Protocol and other tools (HA – 5mins)– Discussion (5mins)
• Relationship with delivery partner (HA – 5mins)– Discussion (5mins)
• Recruitment and retention (CT – 20 mins.)
![Page 3: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/3.jpg)
Trial registration
• Register trial with Current Controlled Trials at outset before beginning recruitment.
• You will be allocated an ISRCTN
http://www.controlled-trials.com/isrctn/submission/
![Page 4: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/4.jpg)
Why is it important to register trials?
»Public knowledge»Reduce duplication»Increase opportunities for
collaboration»Reduce selective reporting and over
reporting»Reduce publication bias
![Page 5: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/5.jpg)
CONSORT
• Conduct and report trial to the CONSORT standards
• http://www.consort-statement.org/consort-statement/
• What is CONSORT?• Why is it important?• How can it help?
![Page 6: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/6.jpg)
CONSORT checklist
![Page 7: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/7.jpg)
CONSORT checklist
![Page 8: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/8.jpg)
CONSORT flow diagram
![Page 9: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/9.jpg)
CONSORT flow diagram cluster trials
From Campbell MK, Piaggio G, Elbourne DR, Altman DG; for the CONSORT Group. Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012 Sep 4;345:e5661.
![Page 10: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/10.jpg)
Trial management
• Trial management approach/model
• Developing a trial protocol and other trial management tools
• Relationship with delivery partner
![Page 11: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/11.jpg)
Trial management model/approach• Current EEF model: Light touch approach to trial
management as delivery partner is often taking responsibility for many of the ‘usual’ trial management responsibilities
• Think about everything you would normally do/take responsibility for as trial manager. document this and share with delivery partner.
• Offer advice and guide the process
![Page 12: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/12.jpg)
Discussion
• What are YOUR experiences of trial management within EFF evaluations?
• What are the advantages and disadvantages of the current model/approach?
• Possible solutions?
![Page 13: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/13.jpg)
Trial Protocol and other tools
• Develop a trial protocol as evaluation team
• Discuss and develop trial protocol with delivery partner
• Produce clear timeframes/deadlines which both evaluation team and delivery partner can work to
![Page 14: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/14.jpg)
Produce evaluation diagrams
Control Group
Primary Schools N = 12
No intervention
Primary School Randomisation
Intervention Group
Primary Schools N = 12
Intervention in Year 6 continued intervention in Year
7 in Secondary Schools.
Children in target group n = 288
(based on average 12 children per school)
Primary Schools n = 24 Secondary Schools n = 3
Long term follow up
Routine test results and pupil characteristics recorded in National Pupil Database
Baseline data collection
Information on all Year 6 pupils including Key stage 2 English Teacher Assessments from Dec 2012
Follow up data collection Dec 2013
•Progress in English 11 (long form) (Conducted in Secondary School)
![Page 15: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/15.jpg)
Clear timeframes
![Page 16: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/16.jpg)
Provide clear information• Help delivery partner develop information for
schools, parents and children
• Work with delivery partner to ensure clear instructions are given to schools
• Help delivery partner develop school agreement documents
• Help develop parent consent forms/opt out forms
![Page 17: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/17.jpg)
Discussion
• Has the trial protocol been a useful shared document?
• What other tools have YOU used to aid the process?
![Page 18: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/18.jpg)
Relationship with delivery partner
• Intervention developer has a lot invested in the intervention
• Evaluator must remain in equipoise• Can be a challenging relationship• Try to explain that it is important you remain
impartial• Refrain from voicing your own opinions about
the intervention – let the research speak for itself
![Page 19: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/19.jpg)
Relationship with delivery partner
• Manage expectations
• Be clear from outset who is responsible for what
• Be clear from outset on the data you will require, when and the format you will require it in
• Provide clear instructions for secure data transfer
![Page 20: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/20.jpg)
Discussion
• What are YOUR experiences of the relationship with the delivery partner?
• How can challenges be overcome?
![Page 21: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/21.jpg)
Recruitment and retention (CT)
• Randomisation ensures absence of selection bias
• Selection bias can still be introduced during recruitment or because of attrition
![Page 22: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/22.jpg)
Recruitment bias
• Potential sources:» Developer-led recruitment» Timing» Randomisation of clusters before recruitment of individuals» Teacher not linked to class before randomisation
• Possible solutions:» Evaluators fully involved in recruitment process» Randomise after recruitment of clusters and within clusters» Ensure teachers are linked to classes before randomisation
![Page 23: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/23.jpg)
Attrition bias
• Attrition after randomisation can introduce bias» Those who leave a trial tend to be different
from those who remain in the trial» If there is unequal attrition this is worrying» All efforts must be made to retain participants
after randomisation for post-tests even if they don’t receive the intervention
![Page 24: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/24.jpg)
Example of attrition bias
![Page 25: Conference for EEF evaluators: Building evidence in education](https://reader036.vdocuments.us/reader036/viewer/2022062301/56815d8a550346895dcb966c/html5/thumbnails/25.jpg)
Discussion
• Discuss any issues you have experienced with the developer-led recruitment.