bmj open is committed to open peer review. as part of this

BMJ Open is committed to open peer review. As part of this commitment we make the peer review history of every article we publish publicly available. When an article is published we post the peer reviewers’ comments and the authors’ responses online. We also post the versions of the paper that were used during peer review. These are the versions that the peer review comments apply to. The versions of the paper that follow are the versions that were submitted during the peer review process. They are not the versions of record or the final published versions. They should not be cited or distributed as the published version of this manuscript. BMJ Open is an open access journal and the full, final, typeset and author-corrected version of record of the manuscript is available on our site with no access controls, subscription charges or pay-per-view fees (http://bmjopen.bmj.com). If you have any questions on BMJ Open’s open peer review process please email

[email protected]

on Novem

ber 15, 2021 by guest. Protected by copyright.

http://bmjopen.bm

j.com/

BM

J Open: first published as 10.1136/bm

jopen-2020-042133 on 12 January 2021. Dow

nloaded from

http://bmjopen.bmj.com/

[email protected]


For peer review onlyProtocol for the Process Evaluation of a Complex

Intervention Delivered in Schools to Prevent Adolescent Depression: The Future Proofing Study

Journal: BMJ Open

Manuscript ID bmjopen-2020-042133

Article Type: Protocol

Date Submitted by the Author: 01-Jul-2020

Complete List of Authors: Beames, Joanne; University of New South Wales, Black Dog InstituteLingam, Raghu; University of New South Wales, School of Women’s and Children’s HealthBoydell, Katherine; University of New South Wales, Black Dog InstituteCalear, Alison L.; Australian National University, Centre for Mental Health ResearchTorok, Michelle; University of New South Wales, Black Dog InstituteMaston, Kate; University of New South Wales, Black Dog InstituteZbukvic, Isabel; Orygen The National Centre of Excellence in Youth Mental HealthHuckvale, Kit; University of New South Wales, Black Dog InstituteBatterham, Philip; Australian National University, Centre for Mental Health ResearchChristensen, Helen; University of New South Wales, Black Dog InstituteWerner-Seidler, Aliza ; University of New South Wales, Black Dog Institute

Keywords: Child & adolescent psychiatry < PSYCHIATRY, Depression & mood disorders < PSYCHIATRY, PREVENTIVE MEDICINE

For peer review only - http://bmjopen.bmj.com/site/about/guidelines.xhtml

BMJ Open on N

ovember 15, 2021 by guest. P

rotected by copyright.http://bm

jopen.bmj.com

/B

MJ O

pen: first published as 10.1136/bmjopen-2020-042133 on 12 January 2021. D

ownloaded from


For peer review onlyI, the Submitting Author has the right to grant and does grant on behalf of all authors of the Work (as defined in the below author licence), an exclusive licence and/or a non-exclusive licence for contributions from authors who are: i) UK Crown employees; ii) where BMJ has agreed a CC-BY licence shall apply, and/or iii) in accordance with the terms applicable for US Federal Government officers or employees acting as part of their official duties; on a worldwide, perpetual, irrevocable, royalty-free basis to BMJ Publishing Group Ltd (“BMJ”) its licensees and where the relevant Journal is co-owned by BMJ to the co-owners of the Journal, to publish the Work in this journal and any other BMJ products and to exploit all rights, as set out in our licence.

The Submitting Author accepts and understands that any supply made under these terms is made by BMJ to the Submitting Author unless you are acting as an employee on behalf of your employer or a postgraduate student of an affiliated institution which is paying any applicable article publishing charge (“APC”) for Open Access articles. Where the Submitting Author wishes to make the Work available on an Open Access basis (and intends to pay the relevant APC), the terms of reuse of such Open Access shall be governed by a Creative Commons licence – details of these licences and which Creative Commons licence will apply to this Work are set out in our licence referred to above.

Other than as permitted in any relevant BMJ Author’s Self Archiving Policies, I confirm this Work has not been accepted for publication elsewhere, is not being considered for publication elsewhere and does not duplicate material already published. I confirm all authors consent to publication of this Work and authorise the granting of this licence.

Page 1 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from

https://authors.bmj.com/wp-content/uploads/2018/11/BMJ_Journals_Combined_Author_Licence_2018.pdf

http://creativecommons.org/


For peer review only

1 Protocol for the Process Evaluation of a Complex Intervention Delivered in Schools to

2 Prevent Adolescent Depression: The Future Proofing Study

3

4 Joanne R. Beames1 [email protected]

5 Raghu Lingam2 [email protected]

6 Katherine Boydell1 [email protected]

7 Alison L. Calear3 [email protected]

8 Michelle Torok1 [email protected]

9 Kate Maston1 [email protected]

10 Isabel Zbukvic4 [email protected]

11 Kit Huckvale1 [email protected]

12 Philip J. Batterham3 [email protected]

13 Helen Christensen1 [email protected]

14 Aliza Werner-Seidler1* [email protected]

15 Affiliations:

16 1 Black Dog Institute, University of New South Wales, Sydney, NSW, Australia.

17 2 Population Child Health Clinical Research Group, School of Women’s and Children’s

18 Health, University of New South Wales, NSW, Australia.

19 3 Centre for Mental Health Research, The Australian National University, Canberra, ACT,

20 Australia.

21 4 Orygen, Parkville, Victoria, Australia; Centre for Youth Mental Health, The University of

22 Melbourne, Parkville, Victoria, Australia

23 *Corresponding author:

24 Dr Aliza Werner-Seidler, Black Dog Institute, University of New South Wales, Sydney,

25 NSW, Australia. [email protected]; +61 2 9382 3808

Page 2 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from

mailto:[email protected]














Embedded process evaluation protocol for trial to prevent depression in school students 2

1 Abstract

2 Introduction: Process evaluations provide insight into how interventions are delivered across

3 varying contexts and why interventions work in some contexts and not in others. This

4 manuscript outlines the protocol for a process evaluation embedded in a cluster randomised

5 trial of a digital depression prevention intervention delivered to secondary school students

6 (The Future Proofing Study). The purpose is to describe the methods that will be used to

7 capture process evaluation data within this trial.

8 Methods and Analysis: A mixed methods design will be used with data collected in the

9 intervention arm of the Future Proofing Study. Data collection methods will include semi-

10 structured interviews with school staff and study facilitators, automatically collected

11 intervention usage data, and participant questionnaires (completed by school staff, school

12 counsellors, study facilitators, and students). Information will be collected about: i) how the

13 intervention was implemented in schools, including fidelity; ii) school contextual factors and

14 their association with intervention reach, uptake and acceptability; iii) how school staff, study

15 facilitators, and students and responded to delivering or completing the intervention. How

16 these factors relate to trial effectiveness outcomes will also be assessed. Overall synthesis of

17 the data will provide school cluster-level and individual-level process outcomes.

18 Ethics and Dissemination: Ethics approval obtained from the University of New South

19 Wales. Results will be submitted for publication in peer reviewed journals and discussed at

20 conferences. Our process evaluation will contextualise the trial findings with respect to how

21 the intervention may have worked in some schools but not in others. This evaluation will

22 inform the development of a model for rolling out digital interventions for the prevention of

23 mental illness in schools.

24 Trial Registration: The randomised controlled trial has been prospectively registered:

25 ACTRN12619000855123. Registered 31 May 2019,

Page 3 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=377664&isReview=true

2 Keywords: Process evaluation, complex intervention, mixed methods, adolescent mental

3 health, depression, prevention, randomised controlled trial, implementation.

Page 4 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from

https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=377664&isReview=true




1 Strengths and Limitations of this Study

2 This is a process evaluation embedded in a randomised clinical trial, which is critical

3 for understanding how interventions are delivered in real world contexts.

4 This protocol is the first to illustrate how to embed a process evaluation into a school-

5 based trial of a digital depression prevention intervention, including the use of flexible

6 and pragmatic quantitative and qualitative data collection methods.

7 School context factors will be rigorously assessed and analysed for contribution to

8 intervention reach and completion, uptake of study processes, and clinical outcomes.

9 Study outcomes will guide sustainability efforts in the delivery of digital depression

10 prevention interventions in schools.

Page 5 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Introduction

2 Embedded in randomised controlled trials, process evaluations provide insight into

3 reasons why interventions work in some contexts and not in others. These process evaluations

4 can help to demystify the ‘black box’ of complex intervention trials by taking into account

5 contextual factors, differences in ways the intervention is delivered, and adaptations made for

6 intervention delivery into a particular system (1, 2). Contextual factors typically include

7 features of the organisation or broader environment that influence the delivery of the

8 intervention (e.g., leadership, engagement, culture, political landscape). Considering the

9 contribution of contextual factors is necessary to aid interpretation of trial outcomes,

10 maximise the knowledge gained from trials, identify optimal delivery processes across

11 different settings, and inform broader dissemination efforts. In recognition of this need for

12 process evaluations, the UK’s Medical Research Council (MRC) set out a framework that

13 emphasises the value of these evaluations in order to capture both contextual and

14 implementation factors associated with complex interventions (3).

15 In line with best-practice recommendations that study protocols which pre-specify

16 methods and approaches should be published to maintain research integrity (4), we describe a

17 protocol for a mixed methods process evaluation embedded within a cluster randomised

18 controlled trial (cRCT) known as the Future Proofing Study (FPS). The FPS is a large school-

19 based trial examining whether depression can be prevented using cognitive behaviour therapy

20 (CBT) delivered by smartphone application (5).

21 There has been an increase in the availability of digital mental health programs which

22 show promise in addressing the significant disease burden associated with depression (6).

23 Depression often first emerges during adolescence (7) and treatment alone cannot adequately

24 reduce this burden (8). Accumulating evidence indicates that early adolescence is the ideal

25 developmental window during which to intervene because it captures young people before

Page 6 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 the incidence of depression increases exponentially around the age of 16-17 years (9, 10).

2 Therefore, increased efforts are being directed toward prevention approaches that are

3 developed specifically for delivery among adolescents.

4 While there is evidence supporting the value and effectiveness of prevention

5 programs in schools (11, 12), their uptake has been significantly limited by low levels of

6 help-seeking (13) as well as practical constraints in cost and scalability. A lack of

7 understanding about how to deliver these interventions sustainably at scale, and engage those

8 who stand to benefit, have hampered the translation of prevention approaches into the

9 community.

10 Two ways to overcome the barriers of cost and scalability of implementing adolescent

11 depression interventions are:

12 Delivering universal prevention programs in schools, and

13 Using technology to deliver programs, which tends to be lower cost than traditional

14 face-to-face methods (14).

15 Working with schools to deliver universal prevention programs, to every student,

16 regardless of their risk, circumstance, or symptom profile, not only dramatically increases the

17 potential reach of interventions, it also means that these programs can eventually be

18 integrated into the curriculum and delivered to every student, making this approach a

19 sustainable implementation strategy for widescale delivery and dissemination. Additionally,

20 working with schools to deliver such programs reduces the need for young people to actively

21 seek professional help. This is critically important - despite having a significant need, fewer

22 than 80% of young people with mental illness seek help and receive the services they need

23 (15, 16). The scaffolding provided by schools can be leveraged to deliver prevention

24 programs to all students, which fits with a trend for schools to be designated the first point of

25 contact to support youth mental health problems when they first emerge (9). Process

Page 7 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 evaluations such as the one described in the current paper are relatively rare, particularly

2 those examining digital (as compared to face-to-face) interventions in school settings. In fact,

3 we were unable to find any process evaluation descriptions in the literature that evaluate

4 digital mental health programs in school settings. This process evaluation is an initial step

5 towards addressing that gap.

6 Technology offers a promising way to deliver mental health interventions to the

7 community. Digital mental health interventions offer two key advantages over face-to-face

8 approaches – first, they are cheaper to access and more cost effective (17, 18), and second,

9 they can be used to reach people across vast geographic areas. The latter is particularly

10 important given Australia’s geography, where remoteness and low population density meant

11 that people in regional and rural areas often do not have the same level of access to mental

12 health services as people in metropolitan areas. With more young people than ever using

13 smartphones, mental health intervention delivered online or through applications represents

14 an exciting avenue for reaching adolescents.

15 The Future Proofing Study (FPS)

16 The trial that is the subject of this process evaluation is the FPS. The FPS addresses

17 barriers of reach, cost, and scalability by delivering a depression prevention program via a

18 smartphone app to Year 8 secondary school students aged 12-13 years. This study is being

19 conducted with approximately 200 Australian schools (up to 10,000 participants), of which

20 half will be allocated to the intervention condition. The primary outcome is symptoms of

21 depression. Secondary outcomes include anxiety, psychological distress and insomnia, among

22 others. Symptom outcomes will be assessed at baseline, post-intervention, six months

23 (primary outcome only), and then annually for five years. The primary endpoint is 12 months

24 following baseline. More details are available from the Australian New Zealand Clinical

25 Trials Registry (ANZCTRN12619000855123) and protocol paper (Werner-Seidler et al.,

Page 8 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 2020).

2 The intervention being delivered is known as SPARX, a CBT-based program

3 incorporating gamification principles. The development and initial evaluation of SPARX has

4 been documented in detail previously (19). SPARX has been tested in an Australian sample

5 of secondary students (delivered via computer) and shown to prevent depression in the lead

6 up to final school exams (20). The gamified intervention teaches young people about the

7 relationship between thoughts, feelings, and behaviour. Skills learnt through SPARX include

8 emotion identification, emotion regulation, behavioural activation (being active), recognising

9 and challenging unhelpful thoughts, and practical problem solving. SPARX consists of seven

10 20-minute modules. The intervention is fully automated, and the therapeutic components are

11 standardised. See the trial protocol for full intervention details (Werner-Seidler et al., 2020).

12 School Engagement and Recruitment

13 The FPS requires multiple levels of approval, beginning with state department and

14 independent school body approval (NSW Department of Education; Catholic school

15 dioceses), followed by individual school engagement. An engagement strategy for this study

16 involves sending electronic communication material to all schools across New South Wales

17 and in other Australian capital cities, targeting school principals and wellbeing staff to invite

18 them to participate in this study opportunity. Schools are invited to submit expressions of

19 interest and are subsequently followed up by the research team over the phone to explain

20 more about the study. The schools for which the FPS is a good fit (as determined by the

21 school) are then signed onto the study, with support from the principal, the school counsellor,

22 and at least one other staff member (typically a teacher). In line with best practice in

23 implementation science (21), this group of 2-3 staff members (typically not including the

24 principal) will form the school-specific ‘study implementation team’. After signing on to the

25 study, several webinars are scheduled throughout the lead up to the study start date, so that

Page 9 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 school staff and parents can listen to a 15-minute study overview from the trial manager and

2 have their questions answered.

3 Preparation

4 In preparation for in-class assessment sessions, study facilitators (volunteer research

5 assistants) are recruited to support the study. The purpose of these facilitators is to attend

6 schools to introduce the study to students, and ensure the technology is functioning so

7 students can download the SPARX app and complete the baseline (and post-assessment)

8 questionnaires. All facilitators go through an interview and screening process prior to

9 selection, then attend a half day face-to-face training session before supporting the study in

10 schools. This training provides an overview of the study and detailed information about their

11 roles within schools, including a step-by-step guide to running the sessions. Discussion and

12 practice are core components of the training.

13 Aims and Objectives

14 1. To evaluate the reach (including completion), uptake, and acceptability of the

15 intervention (school- and student-level).

16 2. To understand the contribution of contextual factors (e.g., characteristics of the

17 outer/inner setting, intervention, individuals) on:

18 o School-level fidelity to the implementation strategy

19 o Implementation outcomes (intervention reach, uptake, acceptability), as

20 assessed from the perspectives of school staff, teachers and students.

21 3. To examine the impact of school-level variation (in implementation fidelity and

22 outcomes) on clinical effectiveness outcomes at the school- and student-level.

23 The aim of this process evaluation is to understand how SPARX is implemented and

24 delivered in schools, and to identify systematic differences and variation in delivery between

25 clusters (individual schools). The differing ways in which schools support and deliver the

Page 10 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 intervention will inevitably impact its effectiveness, and this evaluation will assess these

2 differences (3, 22). For example, degree of support from senior school leaders who would be

3 expected to understand that participating in study activities is a priority for the school will

4 likely impact effectiveness at the cluster level. A secondary aim is to evaluate individual-

5 level factors likely to impact outcome. For instance, young people’s openness to receiving

6 mental health material via an app will likely impact intervention acceptability and

7 completion. This process evaluation has been designed to capture important information from

8 teachers, school staff and students at both the school and individual level, which may

9 ultimately impact the effectiveness of the intervention on clinical mental health outcomes.

10 Findings will provide insight into factors which support and/or hinder the

11 implementation of digital universal mental health programs in school settings. Knowledge

12 gained from this process evaluation will help to inform the development of a model and guide

13 for how to best deliver digital mental health programs to young people in schools.

14

15 Methods and Analysis

16 Design

17 This study uses a hybrid type 1 approach (23), with a focus on implementation

18 process factors and outcomes in the context of an effectiveness trial. The evaluation is guided

19 by the CFIR and RE-AIM framework (24, 25). In keeping with Nilsen’s categorisation of

20 implementation theories, models, and frameworks, these frameworks help us to understand

21 different parts of the implementation process (26). The CFIR will be used to identify

22 barriers and facilitators to intervention implementation and effectiveness (26). The CFIR is

23 a widely used deterministic theoretical framework which has been applied in multiple settings

24 (27), including schools (e.g., 28). The CFIR identifies five major domains, including (i) the

25 outer setting, which includes the social, political and economic context that the organisation

Page 11 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 in which implementation is occurring exists; (ii) the inner setting, which includes features and

2 characteristics of the organisation such as leadership and relative priority; (iii) the

3 characteristics of individuals, which includes organisational staff knowledge and attitudes

4 about the intervention and their role and identification within the wider organisation; (iv) the

5 characteristics of the intervention itself and; (v) implementation processes, which includes the

6 ways that the intervention will be delivered in a given context. Normalisation Process Theory

7 will be used to provide additional insights into implementation processes (NPT; 29). NPT

8 aims to identify and explain the implementation of new interventions and how they become

9 integrated into routine care.

10 The RE-AIM framework will be used to evaluate the implementation outcomes,

11 including fidelity to the implementation strategy, intervention reach, uptake, and

12 acceptability/appropriateness (24, 30). These outcomes also map onto the framework of

13 implementation outcomes proposed by Proctor and colleagues (31, 32). Fidelity refers to the

14 extent to which the intervention was delivered as intended. Reach refers to the proportion of

15 eligible participants who opened, used, and completed the intervention, as well as the

16 proportion of students from the entire cohort of eligible students in intervention schools who

17 consented to participate. Uptake refers to the proportion of schools that were onboarded to

18 the study (intervention and control arms), and of school staff who were willing to support the

19 delivery of the intervention (intervention arm only). Reach and uptake also incorporate the

20 representativeness of the sample (school-level and individual-level). Acceptability and

21 appropriateness refer to the perceived agreeableness or fit of the intervention. This evaluation

22 will incorporate how the barriers and facilitators identified through the CFIR impacted the

23 implementation outcomes, and, in turn, how these implementation outcomes impacted

24 effectiveness outcomes.

25

Page 12 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Logic Model

2 Following MRC guidance (3), the research team developed a logic model for the FPS

3 process evaluation in a series of participatory workshops. The logic model was prospectively

4 informed by key CFIR constructs identified in previous literature as being important in

5 school-based studies, but also feasible and appropriate to measure within the school context

6 in the FPS. The key constructs included the outer setting, inner setting, individual

7 characteristics, and intervention characteristics. The logic model (Figure 1) was developed to

8 consider the key factors (mapped to the CFIR) that would potentially impact implementation

9 of the intervention (mapped to RE-AIM), as well as its clinical effectiveness. The process

10 evaluation method, including the selection of dependent variables and design of surveys and

11 semi-structured interview guides, were derived from this logic model. See Table 1 for

12 CFIR/RE-AIM domains, key research questions, process data, and data that will be collected

13 within each domain.

14 A pilot study of 8 schools conducted in 2019 was used to assess the suitability of the

15 planned implementation strategy and process evaluation methods. We retrospectively applied

16 our logic model to collected data to evaluate whether our methodology captured relevant

17 constructs, and sufficient variation in these constructs. We integrated learnings from this pilot

18 study using a dynamic feedback process to strengthen our methodology for the full-scale FPS

19 phase. Some minor changes were made following this pilot which relate to the CFIR,

20 including a greater emphasis on sub-constructs such as school climate (inner setting).

21

22 <INSERT FIGURE 1 HERE>

23

24 Figure 1. Logic model. The model shows that CFIR constructs including school context

25 characteristics, school organisational characteristics, and individual characteristics, will

Page 13 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 influence how staff engage with the implementation strategy. The intervention itself, which

2 includes the core cognitive-behavioural therapeutic components, is conceptualised as

3 standardised across individuals because it is delivered digitally, follows a fixed schedule and

4 does not incorporate tailored content. The yellow input factors are expected to vary across

5 schools and individuals, thus influencing engagement and flexibility of the implementation

6 strategy and in turn, implementation outcomes and student-level outcomes. The logic model

7 and implementation plan were externally peer reviewed by an experienced and internationally

8 recognised implementation scientist outside the team within an implementation workshop.

9 For details on assessment of these factors, see Table 1.

Page 14 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



1 Table 1 – Process evaluation details including process data, outcome data, data type and source

CFIR and RE-AIM Constructs Research Aim Process or Outcome Data Data Type and Source

Outer SettingSchool contextual characteristics

What was the broad context of the schools in which the SPARX intervention was delivered? (Aim 2)

School socio-economic indexSchool location (metropolitan/regional)

Publicly available information (ICSEA, GPS)

Inner SettingSchool organisational characteristics

What were the characteristics of the delivery environments (schools)? (Aim 2)

What were the barriers and facilitators that affected buy-in, delivery, and student uptake? (Aim 2)

School size, type, composition, fundingSchool culture

Implementation climate, relative priority, competing demands, leadership, school counsellor availability and level of support, networks and communication, school culture and climate, school readiness for implementation

Publicly available information (school size, type, funding)SSPESH (assesses school culture) Interviews with school staff

Implementation Climate measures (staff and students)Relative Priority measure Competing Demands measureInterviews with school staffChecklists completed by trial managerOther administrative data including number of staff allocated to assist with delivery and consent process, level of communication with the research team

School leadership How supportive of delivering SPARX were school principals, deputy principals and executives? (Aim 2)

Level of support and buy in from school leaders

Interviews with school staff

Individual CharacteristicsSchool staff What were the characteristics (including attitudes,

beliefs, traits) of school staff supporting the delivery of the intervention? (Aim 2)

How supportive of delivery were school staff who were involved on-the-ground? (Aim 2)

Age, gender, current employment, role etc

Leadership, skills, motivations, expectations, self-efficacy, expectations, time available, knowledge and beliefs about the intervention, study buy in

Demographics questionnaire

Implementation Leadership ScaleInterviews with school staffQualitative feedback based on informal observation

Study facilitators How well did study facilitators attending schools support the delivery of the intervention? (Aim 2)

Age, employment Skills, self-efficacy or confidence, motivations and expectations

Demographics questionnaireSelf-Confidence measure Interviews with study facilitators

Page 15 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




Students What were the characteristics of young people that affected intervention uptake and effectiveness? (Aim 2)

History of mental illness Reported by Year 8 students as part of the online FPS survey

Intervention CharacteristicsSPARX Were there any barriers to intervention use? (Aim 2

& 3)

What do staff think about the efficacy and advantage of using the intervention? (Aim 2)

Technical issues

Evidence strength and quality, relative advantage

Logs of technical issues sent through schools, parents and participantsIT data pertaining to technical problemsInformal feedback provided by schools and research staff attending schools

Relative Advantages measureAnticipated Benefits measureInterviews with school staff

Implementation Processes

Normalisation and integration How did school staff perceive the implementation processes? (Aim 2)

Coherence, cognitive participation, and collective action

NoMAD

Implementation Outcomes Fidelity to the implementation strategy

To what extent was the intervention implemented as planned? (Aim 2)

School delivery of the FP program, including changes to the plan

Completed implementation checklists, emails and feedback formsInterviews with school staff

Reach What was the extent to which those who were eligible to receive SPARX used it? (Aim 1)

Proportion of eligible participants who consented to participate; proportion who opened, used and completed the SPARX interventionRepresentativeness of the student sample

Administrative data about consent. Digital analytic data including usage (app downloads, installs, opens), completion rate (modules completed) and time spent using SPARXReported by Year 8 students as part of the online FPS survey

Uptake How many eligible schools participated in the study? Within those schools, how many staff supported the delivery of SPARX? (Aim 1)

Proportion of eligible schools that were onboarded to the study; proportion of school staff (in intervention schools) who supported SPARX Representativeness of the sample

Administrative data

Publicly available information about schools and self-report demographic data from school staff

Acceptability/appropriateness How satisfied were participants with the intervention? (Aim 1)

Acceptability/appropriateness of the intervention, expectations

Acceptability/appropriateness of the Future Proofing program (including SPARX)

Reported by Year 8 students as part of the online FPS surveyInformal conversations and feedback provided by Year 8 studentsImplementation Appropriateness measureInterviews with school study staff

Page 16 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




How satisfied were school staff with supporting the intervention the FP program (including SPARX)? (Aim 1)

Informal conversations and feedback provided by school staff

Across DomainsHow might the relationship between the intervention, the staff supporting the program, and context of each school shape variation in outcomes (implementation strength metric)? (Aim 2 and 3)What key lessons emerge from this study that can be generalised to the implementation of digital mental health programs in schools more broadly?

1 Note. The process data and outcomes are mapped onto Figure 2. CFIR = Consolidated Framework for Intervention Research; RE-AIM = Reach, Effectiveness, Adoption, Implementation, 2 Maintenance; Aim = Aims outlined in the Aims and Objectives section; SSPESH = Survey of School Promotion of Emotional and Social Health; NoMAD = Normalisation Measure 3 Development questionnaire.

Page 17 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



1 School Implementation Strategy

2 The school implementation strategy was developed by the study authors. The authors

3 drew on their experience in school-based intervention delivery and integrated feedback from

4 teachers and school staff from several completed school-based trials that delivered digital

5 interventions to school students (20, 33). Stakeholder consultation specifically for this study

6 involved discussions with the Department of Education, consultation with several school

7 parent committees and consultation with both youth and parent Lived Experience Advisory

8 Panels. This strategy was also refined following the first pilot wave involving eight

9 intervention schools.

10 The school ‘study implementation teams’ are principally responsible for the

11 implementation of the intervention and liaising with the research team. As described earlier,

12 these teams typically incorporate at least one classroom teacher and one school counsellor to

13 assist with the intervention delivery. Study facilitators also support the delivery of the

14 SPARX intervention by attending schools for the first school session.

15 Implementation Strategy. During the active intervention phase, schools allocate a

16 minimum of 4 x 20-minute school class sessions during which students complete the SPARX

17 intervention. The additional three sessions may be completed either in class if permitted by

18 the schools, or in the students’ own time. The implementation strategy developed by the

19 project team and stakeholders to support the completion of the SPARX intervention

20 comprises:

21 Standardised facilitator training delivered face-to-face over a half-day

22 Study facilitators are present at schools to support students in downloading the

23 SPARX app and completing the baseline assessments

24 School implementation team provided with a SPARX user-guide and information

25 booklet they can refer to during the sessions

Page 18 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Schools provide students with weekly verbal reminders in home classrooms to use the

2 app regularly

3 Schools publish brief information about the study and mental health tips in the school

4 weekly newsletter

5 Schools liaise with research team weekly to troubleshoot problems

6 This is the strategy being outlined to schools by the research team and adherence to this

7 strategy will be assessed. Whether or not schools schedule more than the four mandated in-

8 class sessions for intervention completion is flexible and can be adapted to suit the

9 preferences of the school. Student participants receive a $20AUD voucher after the

10 intervention period to cover any phone or data related costs incurred. See Figure 2 for details

11 of training and delivery structure.

12


14

15 Data Collection Methods and Participant Groups for Process Evaluation

16 There are three participant groups taking part in the process evaluation: Year 8

17 students, school staff members (e.g., teaching and counselling staff), and study facilitators

18 (see Table 2). School staff members will be members who are responsible for leading the

19 delivery of the study in their school or will be teaching staff who have a supporting role (e.g.,

20 home room teachers who provide reminders to students to complete the intervention).

21 Four types of data will be collected and triangulated: self-report questionnaire data,

22 digital analytic data, administrative data, and qualitative interview data. Self-report

23 questionnaire data specific to the process evaluation will be collected from staff and study

24 facilitators using online survey software (Qualtrics) programmed by one of the authors (JB).

25 These questionnaires assess demographic information, school organisational characteristics,

Page 19 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 individual characteristics, intervention characteristics, implementation processes and

2 implementation outcomes. Where no available published questionnaires were identified as

3 being suitable for this process evaluation, we adapted existing standardised measures or

4 developed our own items (details below). Self-report questionnaire data from Year 8 students

5 about intervention use and feedback will be collected from an online survey (details

6 described in 5) at the post-assessment time point (after the six-week intervention stage).

7 Digital analytic data about intervention use will be captured by the purpose-built Black Dog

8 Institute research platform, which is being used for the broader FPS. Administrative data will

9 be collected through a range of sources, including staff observations, and communications

10 with the research team. Qualitative data will be collected using semi-structure interviews. All

11 data collected will be from intervention schools only as no intervention is implemented in

12 control schools. Student and school staff data will be collected immediately after the

13 intervention period has been completed; facilitator data will be collected both before and after

14 the intervention period.

15

16 Table 2. Summary of data forms provided by each of the participant groups

Questionnaire Individual interview

Digital analytics

Year 8 students ✓ ✓

School staff ✓ ✓

Facilitators ✓ ✓

17

18 Implementation Predictors

19 School Organisational Characteristics (Inner Setting).

20 Publicly Available Information. Publicly available information will be collected

21 about school contextual characteristics, including socio-economic level, size, location, type,

Page 20 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 and funding.

2 General School Culture. School staff will complete the Survey of School Promotion

3 of Emotional and Social Health (SSPESH; 34) and a measure of organisational culture (35).

4 The SSPESH assesses a school’s capacity to promote social and emotional wellbeing, and

5 contains four subscales: Positive School Community, Student Social and Emotional Learning,

6 Engaging Families, and Supporting Students Experiencing Mental Health Difficulties. Items

7 are rated on a 4-point Likert scale from 0 (not yet in place) to 3 (completely in place) and

8 preliminary investigations support the scale structure and criterion-related validity (34).

9 We adapted a 9-item questionnaire about general culture within a health care setting

10 to use within the school setting (35). Items are rated on a 5-point Likert scale from 1 (strongly

11 agree) to 5 (strongly disagree). Lower scores indicate a more positive working culture, which

12 includes transparency, productive working relationships, and receptivity to feedback.

13 Previous investigation has shown that this measure has good internal consistency, although

14 overlaps somewhat with other sub-constructs within the inner setting (e.g., learning climate;

15 35).

16 Implementation Climate. School staff will complete a measure of implementation

17 climate that we adapted for use in the school context (35). Items are rated on a 5-point Likert

18 scale from 1 (strongly agree) to 5 (strongly disagree). Lower scores indicate increased staff

19 receptivity to the program and support within the school, including rewards and recognition.

20 This measure has acceptable internal consistency and good discriminant validity (35).

21 Relative Priority. School staff will complete one adapted item from the School

22 Contextual Barriers subscale of the Perceived Attributes of the Healthy Schools Approach

23 Scale (36). The item is rated on a 5-point scale Likert scale from 1 (strongly agree) to 5

24 (strongly disagree). Higher scores indicate that other activities did not interfere with

25 implementation of the FP program.

Page 21 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Competing Demands. Staff will complete 3-items, developed specifically for this

2 study, which assess how much time they allocated, and desired to allocate, to the FP program

3 relative to other competing workload demands on visual analogue scales (anchor by 0 = no

4 work time, 100% = all of my work time). Higher scores indicate increased allocation of time

5 and prioritisation of the FP program.

6 Individual Characteristics.

7 Leadership. School staff will complete the Implementation Leadership Scale (ILS;

8 37). The ILS is a 12-item scale that assesses leadership behaviours that support

9 implementation of evidence-based practices. The ILS contains four subscales, and two will be

10 included in the current study (Knowledge Leadership and Supportive Leadership). These

11 scales assess the degree to which the staff member was knowledgeable about and offered

12 support to the program. Items are rated on a 5-point Likert scale from 0 (very much so) to 4

13 (not at all) and will be re-coded such that higher scores indicate more effective

14 implementation leadership behaviours. This measure has excellent internal validity,

15 convergent validity, and discriminant validity (37).

16 Self-Efficacy and Confidence. Study facilitators will rate their level of confidence in

17 performing 23 different tasks during school-visits on visual analogue scales, which are based

18 on the training program they completed (anchor by 0 = not at all confident, 100 = very

19 confident). Higher scores indicate higher levels of confidence. A similar questionnaire will be

20 repeated following their school-visits, along with four short answer questions that assess

21 experiences with supporting staff and students during school-visits.

22 Intervention Characteristics.

23 Relative Advantage and Evidence Strength and Quality. School staff will complete

24 the 2-item Relative Advantages subscale and one item from the Anticipated Benefits subscale

25 of the Perceived Attributes of the Healthy Schools Approach Scale (36). Adapted for the

Page 22 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 current study, items are rated on a 5-point scale Likert scale from 1 (strongly agree) to 5

2 (strongly disagree). Scores will be re-coded such that higher scores indicate greater perceived

3 advantage of the FP program compared to others and greater perceived impacts on mental

4 health, respectively.

5 Implementation Processes.

6 Normalisation and Integration into Routine Practice. School staff will complete

7 NPT’s accompanying tool, the Normalisation MeAsure Development questionnaire

8 (NoMAD; 38). The NoMAD is a 23-item measure that assesses how professionals involved

9 in the implementation of a complex intervention perceive implementation processes. It is a

10 flexible measure that can be altered to more accurately describe the adoption of new

11 interventions at the provider level. Ten items of the NoMAD will be used in this study to

12 assess intervention buy-in from school staff, and how staff members incorporated the

13 initiative into their standard work responsibilities. These items are grouped into three

14 categories: coherence (i.e., making sense of an intervention), cognitive participation (i.e.,

15 working with others to support an intervention), and collective action (i.e., the type of work

16 that people do to support an intervention). Items are rated on a 5-point Likert scale from 1

17 (strongly agree) to 5 (strongly disagree). Initial validation demonstrated that the NoMAD has

18 good face validity, construct validity, and internal consistency (39).

19 Implementation Outcomes

20 Fidelity to the Implementation Strategy. Fidelity will be captured through

21 implementation checklists, informal observations provided by staff attending schools, and

22 communications between schools and research staff.

23 Reach. SPARX app usage data from students will allow for the assessment of app use

24 (downloads, installs, and opens), completion (modules completed), and time spent using

25 SPARX. Administrative data about the number of students in intervention schools with

Page 23 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 consent to participate will also be used as an indicator of reach. Self-report data about

2 individual characteristics (e.g., gender, mental health history) will be used to gauge the

3 representativeness of the students in the intervention schools.

4 Uptake. Administrative data about the proportion of schools that were onboarded to

5 the study and the proportion of teachers who supported the intervention will be collected to

6 provide an index of uptake by schools. Representativeness of the sample will be informed by

7 publicly available information about school characteristics (e.g., socio-economic status,

8 location) and self-report data about school staff characteristics (e.g., role, gender).

9 Appropriateness. School staff will complete the Intervention Appropriateness

10 Measure (IAM; 40). The IAM is a pragmatic 4-item measure of the perceived fit, relevance

11 or compatibility of an evidence-based practice for a context, person, or problem (41). Items

12 have been adapted for this study and are rated on a 5-point Likert scale from 1 (strongly

13 agree) to 5 (strongly disagree). Scores will be re-coded such that higher scores indicate higher

14 levels of perceived appropriateness. The IAM demonstrated strong psychometric properties

15 in previous research (40). School staff will also complete one adapted item from the Agency

16 Leadership Support subscale of the Barriers and Facilitators to Implementing Survey (42).

17 The item is rated on a 5-point Likert scale from 1 (strongly agree) to 5 (strongly disagree) and

18 will be re-coded such that higher scores indicate greater compatibility of the FP program

19 within a particular school.

20 Acceptability. Year 8 students will complete an 11-item feedback questionnaire

21 about SPARX. The questionnaire assesses three domains, including: (i) reasons for non-

22 adherence; (ii) intervention acceptability; and (iii) skills learnt from the intervention. Items

23 will be quantified individually. This questionnaire has previously been used to assess the

24 acceptability of the SPARX program in the school context (43).

25 Individual Interviews

Page 24 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Interview guides for school staff and study facilitators were derived from the logic

2 model, CFIR online resources (e.g., https://cfirguide.org/evaluation-design/qualitative-data/),

3 and the broader literature investigating the delivery of interventions in school settings (44, 45;

4 see Additional File 1 for interview guides). Interviews will provide information about both

5 implementation predictors and outcomes. For school staff, questions focus on: motivations

6 and expectations about the intervention and study; knowledge and beliefs about the

7 intervention; relative advantages of the intervention; self-efficacy; barriers and facilitators

8 affecting the delivery of the intervention, including fidelity; appropriateness and acceptability

9 of the intervention; and recommendations for future implementation. School counsellors will

10 also be asked questions about their experiences of managing high risk participants within the

11 study and compatibility with existing workload. For study facilitators, questions focus on:

12 motivations and expectations about their role; confidence and competence in supporting the

13 delivery of SPARX in schools; the quality of their training; and perceptions about their ability

14 to support the study as required.

15 The interview guides allow for flexibility in questioning and diversion in responses.

16 Questions will primarily be open-ended, with specific prompts and follow-up questions being

17 used as necessary to encourage respondents to elaborate on their ideas and provide examples.

18 Patient and Public Involvement

19 The Future Proofing Study was developed with key stakeholders including school

20 personnel, school counsellors, parents, adolescents, and individuals with a lived experience of

21 mental illness. All aspects of the study design, appropriateness of outcomes measures and

22 consent procedures were developed in consultation with these stakeholder groups. The

23 current process evaluation involves consultation with school staff members to understand

24 their experience. The SPARX intervention itself was codesigned with young people. All

25 study results will be shared directly with participants and their schools through lay summaries

Page 25 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 and infographics.

2 Procedure

3 Ethical approval was provided by the University of New South Wales Human

4 Research Ethics Committee (HC180836) and NSW Government State Education Research

5 Applications Process Approval (SERAP 2019201).

6 The FPS has two waves of delivery (April-June 2020; July-Sept 2020). Process

7 evaluation data will be collected at each wave, immediately following the intervention period.

8 Purposive sampling will be used to capture a range of diverse school settings.

9 School students will complete process measures at the same time as completing

10 primary measures for the FPS. Study facilitators will provide informed consent following the

11 compulsory face-to-face training session they attend with the research team at the Black Dog

12 Institute. Questionnaires will be completed online.

13 School staff directly involved in the study will be invited to complete the

14 questionnaire by the trial manager, following the six-week SPARX intervention period. Once

15 school staff members open the letter of invitation, they will be able to consent to a 60-minute

16 individual interview (completed either in person or by phone). The semi-structured interviews

17 will be conducted with research staff who have had no previous contact with the schools

18 during the trial, thereby minimising the risk of bias and demand effects. Face-to-face

19 interviews will be held in a quiet room on school grounds. Phone and face-to-face interviews

20 will be recorded and transcribed verbatim.

21

22 Data analysis

23 Quantitative

24 Questionnaire data and open-ended questions from surveys will be exported into data

25 analytic software for analysis. Descriptive statistics will be calculated for all participant

Page 26 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 groups and will provide information about intervention use and its acceptability

2 (questionnaire data from Year 8 participants), differential implementation, fidelity to the

3 implementation strategy, school context factors within each school (questionnaire including

4 several short answer questions from school staff), and competence and experience in schools

5 from those facilitating the study (facilitator questionnaires). Differences between school

6 clusters will be assessed using Analysis of Variance methods (ANOVA). Regression models

7 will assess the on the effects of contextual factors on implementation outcomes (e.g., reach,

8 uptake, acceptability).

9 Qualitative

10 Interviews will be digitally audio-recorded and transcribed verbatim. The transcripts

11 will be checked for accuracy against the sound files as per best practice in transcription (46,

12 47). Qualitative data will then be imported into NVivo to aid in data management and

13 analysis. Thematic analysis will be undertaken to identify, interpret and report on the

14 repeated patterns of meaning within the data, drawing from Braun and Clark’s classic six

15 phase model (48, 49).

16 Initially, one coder will code a subset of the transcripts using an inductive approach,

17 incorporating themes from the data together with topics covered in the interview guide. This

18 subset will then be double coded by an independent coder who will develop their own coding

19 scheme. Codes will be compared and discussed by the two coders, with refinement occurring

20 via an interactive process until a final coding scheme is determined. A senior qualitative

21 analysist will review the coding framework and scheme before all transcripts are coded by the

22 first coder. Research rigor will be enhanced by a team approach to analysis, reflexive field

23 notes, and prolonged engagement with the subject matter (50).

24 Triangulation of Qualitative and Quantitative Data

25 Triangulation involves the use of multiple approaches to address a research question.

Page 27 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 The combination of several approaches increases confidence in the findings and provides a

2 more comprehensive account of the results than individual approaches would do alone (51).

3 In this study, reliability, validity and confidence will be maximised through cross verification

4 and exploration of differences between the outcomes of the various methods. This takes place

5 in several ways:

6 Maximising validity in analysis of qualitative data within the research team using

7 techniques such as discussing coding, constant comparison, accounting for deviant

8 cases, and systematic coding;

9 Triangulation of school staff and research assistant interviews with results from the

10 questionnaires, exploring and accounting for differences;

11 Triangulation of self-report and interview data with publicly available information

12 relating to school contextual characteristics (e.g., school socio-economic level and

13 size) and school delivery of the program, including deviations to the implementation

14 strategy.

15 Mapping the perspectives of different stakeholders across the study (school staff,

16 study facilitators).

17 Additional Analyses

18 We will generate an ‘implementation strength’ metric for each school and its

19 relationship to primary and secondary trial outcomes. This is an emerging evaluation

20 approach, used mainly in low and middle-income countries. The approach aims to understand

21 the degree of implementation effort needed during intervention delivery to achieve desired

22 benefits (52, 53).

23 The implementation strength metric will provide funders and policy makers with an

24 objective measure to monitor effectiveness of implementation if it goes beyond this trial and

25 becomes a sustainable approach. The metric can be used to assess whether the approach to

Page 28 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 implementation meets the minimum level required to prevent the onset of mental health

2 problems in adolescents. The metric will be based on implementation inputs and contextual

3 factors informed by the intervention logic model and process evaluation frameworks (CFIR

4 and RE-AIM) (e.g., adoption from teachers and other staff in the school and fidelity to the

5 intervention strategy), and will be developed with the Future Proofing research team using

6 principal component analysis.

7

8 Discussion

9 This paper describes the design of a mixed methods process evaluation of a cluster

10 randomised controlled trial, the FPS. The FPS investigates the impact of a digital cognitive-

11 behavioural therapy intervention when delivered at scale in school settings. Digital mental

12 health programs have tremendous potential to prevent up to 22% of depression cases and,

13 when delivered at scale, could have population-level impacts (54). However, these programs

14 have not been translated into practice and policy because optimal ways to scale and deliver

15 these interventions are not yet well understood.

16 As an initial step to address this issue, the current process evaluation will attend to

17 contextual and implementation factors that vary across schools and provide a lens through

18 which to interpret trial efficacy outcomes. We expect that results will provide a richly

19 detailed and nuanced understanding of the key factors involved in the effective delivery of

20 digital mental health programs across different schools. We expect that results will not only

21 contextualise our trial findings but will also be used as a model to guide the delivery of

22 school-based interventions that focus on preventing mental illness more broadly. Findings

23 from this process evaluation will indicate whether the approach used in the FPS trial is likely

24 to be sustainable in the school environment going forward and, if so, the threshold level of

25 support required in order to prevent depression and benefit student mental health.

Page 29 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 The prospective publication of this protocol outlines our planned methodological

2 approach. It also serves as a road map for other researchers on a practical way of carrying out

3 process evaluations of complex interventions in the school setting. As is the case with the

4 delivery of interventions across different contexts, we acknowledge that our approach has

5 inbuilt flexibility to explore the data and make provisions for unexpected implementation

6 factors that arise.

7 Limitations and Strengths

8 There are several limitations to our process evaluation that warrant mention. First, we

9 are not including direct observation of teachers in their role supporting the delivery of

10 SPARX. While this would provide objective fidelity data, it requires resources beyond the

11 scope of this project and is not representative of how the program will be sustained following

12 the conclusion of the trial. Second, the qualitative interviewer is a member of the research

13 team (but not the evaluation team). Care will be taken to ensure that this staff member has no

14 contact with schools prior to the interview visit to minimise bias. However, there remains a

15 risk that demand effects may impact the information that is shared. Third, the process

16 evaluation process (questionnaires and interview) will undoubtedly add to the burden placed

17 on school staff. Given that the FPS is already placing a significant burden on the time of busy

18 school staff, this additional component might contribute to low levels of participation.

19 To the point of burden on schools, one of the strengths of the design is that we have

20 undertaken a pilot phase involving eight intervention schools and have been able to refine our

21 processes (e.g., introduce an incentive) for school staff members to participate. This process

22 evaluation also involves the combination of qualitative and quantitative methods which will

23 be triangulated to provide a coherent and comprehensive picture of the data. The use of the

24 ‘implementation strength’ metric represents a novel approach in this field, borrowed from the

25 low-middle income country implementation science sector. The inclusion of this approach

Page 30 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 will provide important information to funders and policy makers following on from this trial,

2 indicating the level of implementation support required to prevent mental illness and improve

3 wellbeing of adolescent school students.

4 Notwithstanding the limitations raised above, this process evaluation will contribute

5 to the broader knowledge base and indicate how best to deliver digital mental health

6 prevention programs in school settings.

7

8 Ethics and Dissemination

9 As noted, all procedures have received ethical approval from relevant committees. Results

10 from this study will be published in an open access peer-reviewed journal.

11

12 Additional Files

13 Additional File 1 (Semi-Structured Interview Guides).docx

14

15 List of Abbreviations

16 FPS = Future Proofing Study

17 cRCT = Cluster Randomised Controlled Trial

18 CFIR = Consolidated Framework for Implementation Research

19 RE-AIM = Reach, Effectiveness, Adoption, Implementation, Maintenance framework

20

21

22

23

24

25

Page 31 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Declarations

2 Trial status

3 Recruitment for the trial is underway. Data collection commences in September 2020

4 (delayed from April 2020 due to COVID-19).

5

6 Ethical approval and consent to participate

7 This study has ethical approval from the University of New South Wales Human Research

8 Ethics Committee (HC180836; 21st January 2019) and NSW Government State Education

9 Research Applications Process Approval (SERAP 2019201; 19th August 2019). All schools,

10 staff, parents and participants must provide active consent to participate.

11

12 Consent for publication

13 Not applicable.

14

15 Availability of data and materials

16 Not applicable.

17

18 Competing interests

19 The authors declare that they have no competing interests.

20

21 Funding

22 Funding for this project came from an NSW Health Early-Mid Career Fellowship awarded to

23 Aliza Werner-Seidler, and a Black Dog Institute Post-Doctoral Fellowship awarded to Joanne

24 Beames, secured by Helen Christensen. ALC is supported by NHMRC fellowships 1122544

25 and 1173146. PJB is supported by NHMRC Fellowship 1158707. Funding for the

Page 32 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 randomised controlled trial within which this process evaluation is embedded came from an

2 NHMRC Project Grant Awarded to Helen Christensen GNT1120646. The funding bodies had

3 no role in any aspect of the study design or this manuscript.

4

5 Author contributions

6 HC and AWS conceived of the study and secured the funding. AWS and JB led the design of

7 the process evaluation, with input from all authors, and expert guidance from RL and KB.

8 AWS drafted the manuscript, with assistance from JB. ALC, MT, KM, RL, KL and HC have

9 a continuing role in monitoring the conduct and outcomes of the process evaluation. All

10 named authors contributed substantially to the approved final manuscript.

11

12 Acknowledgments

13 We would like to thank Professor Melanie Barwick for her advice on this manuscript,

14 together with all the facilitators from the Training Institute for Dissemination and

15 Implementation Research in Health (TIDIRH; Australia 2020) for their comments and

16 feedback on this project.

17

18 Author information

19 Not applicable.

Page 33 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 References

2 1. Oakley A, Strange V, Bonell C, Allen E, Stephenson J, Team RS. Process evaluation

3 in randomised controlled trials of complex interventions. BMJ. 2006;332:413.

4 2. Greenhalgh T, Papoutsi C. Studying complexity in health services research:

5 Desperately seeking an overdue paradigm shift. BMC Med. 2018;16:95.

6 3. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process

7 evaluation of complex interventions: Medical research council guidance. BMJ.

8 2015;350:h1258.

9 4. Grant A, Treweek S, Dreischulte T, Foy R, Guthrie B. Process evaluations for cluster-

10 randomised trials of complex interventions: A proposed framework for design and reporting.

11 Trials. 2013;14:15.

12 5. Werner-Seidler A, Huckvale K, Larsen ME, Calear AL, Maston K, Johnston L, et al.

13 A trial protocol for the effectiveness of digital interventions for preventing depression in

14 adolescents: The future proofing study. Trials. 2020;21:2.

15 6. Tal A, Torous J. The digital mental health revolution: Opportunities and risks.

16 Psychiatric Rehabilitation Journal. 2017;40:263.

17 7. Sawyer SM, Azzopardi PS, Wickremarathne D, Patton GC. The age of adolescence.

18 Lancet Child Adolesc Health. 2018;2:223.

19 8. Andrews G, Issakidis C, Sanderson K, Corry J, Lapsley H. Utilising survey data to

20 inform public policy: Comparison of the cost-effectiveness of treatment of ten mental

21 disorders. British Journal of Psychiatry. 2004;184:526.

22 9. Lawrence D, Johnson S, Hafekost J, de Hann KB, Sawyer M, Ainley J, et al. The

23 mental health of children and adolescents: Report on the second australian child and

24 adolescent survey of mental health and wellbeing. 2015.

Page 34 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 10. Lee FS, Heimer H, Giedd JN, Lein ES, Šestan N, Weinberger DR, et al. Adolescent

2 mental health—opportunity and obligation. Science. 2014;346:547.

3 11. Werner-Seidler A, Perry Y, Calear AL, Newby JM, Christensen H. School-based

4 depression and anxiety prevention programs for young people: A systematic review and

5 meta-analysis. Clin Psychol Rev. 2017;51:30.

6 12. van Zoonen K, Buntrock C, Ebert DD, Smit F, Reynolds CF, Beekman AT, et al.

7 Preventing the onset of major depressive disorder: A meta-analytic review of psychological

8 interventions. Int J Epidemiol. 2014;43:318.

9 13. Gulliver A, Griffiths KM, Christensen H. Perceived barriers and facilitators to mental

10 health help-seeking in young people: A systematic review. BMC Psychiatry. 2010;10:113.

11 14. Donker T, Blankers M, Hedman E, Ljotsson B, Petrie K, Christensen H. Economic

12 evaluations of internet interventions for mental health: A systematic review. Psychological

13 Medicine. 2015;45:3357.

14 15. Essau CA. Frequency and patterns of mental health services utilization among

15 adolescents with anxiety and depressive disorders. Depress Anxiety. 2005;22:130.

16 16. Rickwood DJ, Deane FP, Wilson CJ. When and how do young people seek

17 professional help for mental health problems. Med J Aust. 2007;187:S35.

18 17. Donker T, Blankers M, Hedman E, Ljótsson B, Petrie K, Christensen H. Economic


20 Medicine. 2015;45:3357.

21 18. Lee YY, Barendregt JJ, Stockings EA, Ferrari AJ, Whiteford HA, Patton GA, et al.

22 The population cost-effectiveness of delivering universal and indicated school-based

23 interventions to prevent the onset of major depression among youth in australia.

24 Epidemiology and Psychiatric Sciences. 2017;26:545.

Page 35 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 19. Merry SN, Stasiak K, Shepherd M, Frampton C, Fleming T, Lucassen MF. The

2 effectiveness of sparx, a computerised self help intervention for adolescents seeking help for

3 depression: Randomised controlled non-inferiority trial. BMJ. 2012;344:e2598.

4 20. Perry Y, Werner-Seidler A, Calear A, Mackinnon A, King C, Scott J, et al. Preventing

5 depression in final year secondary students: School-based randomized controlled trial.

6 Journal of Medical Internet Research. 2017;19:e369.

7 21. Higgins MC, Weiner J, Young L. Implementation teams: A new lever for

8 organizational change. Journal of Organizational Behavior. 2012;33:366.

9 22. Clarke DJ, Godfrey M, Hawkins R, Sadler E, Harding G, Forster A, et al.

10 Implementing a training intervention to support caregivers after stroke: A process evaluation

11 examining the initiation and embedding of programme change. Implement Sci. 2013;8:96.

12 23. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation

13 hybrid designs: Combining elements of clinical effectiveness and implementation research to

14 enhance public health impact. Med Care. 2012;50:217.

15 24. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health

16 promotion interventions: The re-aim framework. American Journal of Public Health.

17 1999;89:1322.

18 25. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering

19 implementation of health services research findings into practice: A consolidated framework

20 for advancing implementation science. Implementation Science. 2009;4:50.

21 26. Nilsen P. Making sense of implementation theories, models and frameworks.

22 Implementation Science. 2015;10:53.

23 27. Kirk MA, Kelley C, Yankey N, Birken SA, Abadie B, Damschroder L. A systematic

24 review of the use of the consolidated framework for implementation research. Implement Sci.

25 2016;11:72.

Page 36 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 28. Leeman J, Wiecha JL, Vu M, Blitstein JL, Allgood S, Lee S, et al. School health

2 implementation tools: A mixed methods evaluation of factors influencing their use.

3 Implement Sci. 2018;13:48.

4 29. Murray E, Treweek S, Pope C, MacFarlane A, Ballini L, Dowrick C, et al.

5 Normalisation process theory: A framework for developing, evaluating and implementing

6 complex interventions. BMC Medicine. 2010;8:63.

7 30. Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. Re-aim

8 planning and evaluation framework: Adapting to new science and practice with a 20-year

9 review. Frontiers in Public Health. 2019;7.

10 31. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B.

11 Implementation research in mental health services: An emerging science with conceptual,

12 methodological, and training challenges. Administration and Policy in Mental Health.

13 2009;36:24.

14 32. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes

15 for implementation research: Conceptual distinctions, measurement challenges, and research

16 agenda. Administration and Policy in Mental Health and Mental Health Services Research.

17 2011;38:65.

18 33. Calear AL, Christensen H, Mackinnon A, Griffiths KM, O'Kearney R. The

19 youthmood project: A cluster randomized controlled trial of an online cognitive behavioral

20 program with adolescents. Journal of Consulting and Clinical Psychology. 2009;77:1021.

21 34. Dix KL, Green MJ, Tzoumakis S, Dean K, Harris F, Carr VJ, et al. The survey of

22 school promotion of emotional and social health (sspesh): A brief measure of the

23 implementation of whole-school mental health promotion. School Mental Health.

24 2019;11:294.

Page 37 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 35. Fernandez ME, Walker TJ, Weiner BJ, Calo WA, Liang S, Risendal B, et al.

2 Developing measures to assess constructs from the inner setting domain of the consolidated

3 framework for implementation research. Implement Sci. 2018;13:52.

4 36. Deschesnes M, Trudeau F, Kébé M. Psychometric properties of a scale focusing on

5 perceived attributes of a health promoting school approach. Can J Public Health.

6 2009;100:389.

7 37. Aarons GA, Ehrhart MG, Farahnak LR. The implementation leadership scale (ils):

8 Development of a brief measure of unit level implementation leadership. Implement Sci.

9 2014;9:45.

10 38. Finch TL, Girling M, May CR, Mair FS, Murray E, Treweek S, et al. Nomad:

11 Implementation measure based on normalization process theory 2015 [Available from:

12 http://www.normalizationprocess.org.

13 39. Finch TL, Girling M, May CR, Mair FS, Murray E, Treweek S, et al. Improving the

14 normalization of complex interventions: Part 2 - validation of the nomad instrument for

15 assessing implementation work based on normalization process theory (npt). BMC Med Res

16 Methodol. 2018;18:135.

17 40. Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, et al.

18 Psychometric assessment of three newly developed implementation outcome measures.




22 agenda. Adm Policy Ment Health. 2011;38:65.

23 42. Salyers MP, Rollins AL, McGuire AB, Gearhart T. Barriers and facilitators in

24 implementing illness management and recovery for consumers with severe mental illness:

25 Trainee perspectives. Adm Policy Ment Health. 2009;36:102.

Page 38 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 43. Perry Y, Calear AL, Mackinnon A, Batterham PJ, Licinio J, King C, et al. Trial for

2 the prevention of depression (tripod) in final-year secondary students: Study protocol for a

3 cluster randomised controlled trial. Trials. 2015;16:451.

4 44. Punukollu M, Burns C, Marques M. Effectiveness of a pilot school-based intervention

5 on improving scottish students’ mental health: A mixed methods evaluation. J Youth

6 Adolesc. 2019:1.

7 45. Outhwaite LA, Gulliford A, Pitchford NJ. A new methodological approach for

8 evaluating the impact of educational intervention implementation on learning outcomes. Int J

9 Res Meth Educ. 2019:1.

10 46. Poland BD. Transcription quality. Handbook of interview research: Context and

11 method. 2002:629.

12 47. Poland BD. Transcription quality as an aspect of rigor in qualitative research.

13 Qualitative inquiry. 1995;1:290.

14 48. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol.

15 2006;3:77.

16 49. Braun V, Clarke V. Reflecting on reflexive thematic analysis. Qual Res Sport Exerc

17 Health. 2019;11:589.

18 50. Noble H, Smith J. Issues of validity and reliability in qualitative research. Evid-Based

19 Nurs. 2015;18:34.

20 51. Tashakkori A, Teddlie C. Handbook of mixed methods in social & behavioral

21 research. Thousand Oaks, CA: Sage; 2003.

22 52. Hargreaves JR, Goodman C, Davey C, Willey BA, Avan BI, Schellenberg JR.

23 Measuring implementation strength: Lessons from the evaluation of public health strategies

24 in low- and middle-income settings. Health Policy Plan. 2016;31:860.

Page 39 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 53. Victora CG, Black RE, Boerma JT, Bryce J. Measuring impact in the millennium

2 development goal era and beyond: A new approach to large-scale effectiveness evaluations.

3 Lancet. 2011;377:85.

4 54. Ebert DD, Cuijpers P. It is time to invest in the prevention of depression. JAMA Netw

5 Open. 2018;1:e180335.

Page 40 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



School Contextual CharacteristicsSociodemographic factors (school size,

socio economic status, area)

Intervention Characteristics (SPARX)

Standardised: Delivered by app over 6 weeks

Technical issues, evidence strength and quality, relative advantage

Student-Level Outcomes

Primary outcome: Lower levels of depression

Secondary outcomes: Lower levels of anxiety, distress,

insomnia

Additional outcomes:Improved quality of life, wellbeing

Lower levels of suicidality, self-harm

Improved academic performance on standardised tests

Lower levels of health service use

Implementation Outcomes

School-level fidelity to the implementation strategy

Reach (e.g., intervention received by students, including number of modules completed and time spent using SPARX)

School-level uptake of the FPS

Staff and student acceptability/appropriateness

Implementation Strategy

App download in school time, with support from

research staff

Intervention completion, with support from school

staff

Reminders and support provided by school staff to

complete intervention

Therapeutic Components

PsychoeducationCognitive restructuringBehavioural activationRelaxation strategies

Practical problem solvingEmotion regulation and management

Individual CharacteristicsSchool staff (e.g., skills, motivation, self-

efficacy, buy in, knowledge about the intervention, leadership)

Study facilitators (e.g., skills, confidence)

School Organisational CharacteristicsSchool culture, buy in and support for the

intervention and study, support from leadership, ability of school to support delivery (including

technological infrastructure, counsellor availability)

Individual CharacteristicsStudents (e.g., mental health

history, intervention expectations)

Page 41 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



Research Team

Facilitators

Teachers Counsellors

Year 8 Students

Training

Other staff

Delivery Support

Direct trainingSupport to access interventionDelivery support

School Implementation Team

Page 42 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



Semi-Structured Interview Guides (Version 2, 21/02/2020) HREC number: HC180836 1

Additional File 1

Semi-Structured Interview Guides

School Staff INTRODUCTION: Thank you for agreeing to take part in this interview. I appreciate your time. I am interested in hearing your thoughts and experiences of the Future Proofing program in your school. For this interview, the FP program includes the intervention app (SPARX) and everything else that was done to support the program in your school. I hope to use this information to work out what works in different schools and how to best provide support. I will ask you a few questions about your experiences. This is chance for you to reflect and offer feedback on how we could improve. The interview will last up to one hour. It will be recorded, and your responses will be transcribed to ensure we accurately reflect your words. All personal information will be removed so that your answers are non-identifiable and confidential. Nothing you personally say will be shared with anyone on the FP team that you have been liaising with or with other staff at your school. You can choose to not answer any questions. Do you have any questions before we get started? General Opening Questions:

1. How did you first get involved in teaching/counselling/psychology?

2. Could you tell me about the core values and ethos of your school?

3. To the best of your knowledge, what mental health programs or initiatives has your

school implemented to care for students besides the FP program?

Specific Questions:

4. Overall, how would you describe your experience with the FP program in your school?

5. What was your role in delivering or supporting the FP program in your school?

a. Leader/driver, decision-maker, consent/recruitment, peripheral support

CFIR DOMAIN: INDIVIDUAL CHARACTERISTICS Personal attributes (e.g., motivations, expectations, capacity, values)

6. Why were you motivated to take on this role?

a. How did you come on board with the program?

7. Why do you think your school decided to support this program?

a. To what extent do you think there was a need for a mental health program like FP in

you school (Tension for change)?

8. What were your expectations about the FP program prior to coming on board?

a. How were your expectations met (or not met)? What went differently to what

you expected? What surprised you?

b. What were your expectations about the apps?

Knowledge and beliefs about the intervention 9. How much do you know about SPARX, the intervention? E.g., principles, rationale,

content?

Page 43 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




Self-efficacy 10. How confident were you in your ability to lead/support/make decisions about the FP

program? a. What personal qualities do you think helped you to do that?

CFIR DOMAIN: OUTER SETTING Relative Advantage

11. How did SPARX compare to other mental health programs that your school has supported or delivered?

CFIR DOMAIN: INNER SETTING School contextual factors – Barriers and Facilitators

12. What factors influenced your ability to deliver/support the FP program in your school? a. What else helped or hindered your ability? b. Enquire into school-specific factors and program-specific factors e.g., What

parts of the program were easy for you to do or support (e.g., the consent process, organising rooms, weekly team meetings)? OR What aspects of your school made it difficult for you to support the program?

c. What factors got in the way of students completing the mental health app/s? d. Were you (or others) able to overcome any barriers? How did you manage to do

this? School contextual factors – Leadership

13. Tell me about the involvement of your school executive in the FP program. a. What role did this person have (e.g., principal/vice principal)? Did you meet

regularly with this person or provide updates to them? 14. How do you think the school executive saw this initiative?

Readiness for implementation

15. How would you describe the readiness of your school to take on the FP program? a. What was the preparation phase like for you/other key staff?

IMPLEMENTATION METRICS Appropriateness and Acceptability

16. How does the FP program align with your school’s approach toward student mental health?

17. How suited is the FP program to address depression prevention in your school? 18. How well does an app delivering therapy work to prevent depression in your students? 19. How do you feel about using technology to deliver psychological interventions in your

school? a. What worked well about this and what did not? b. Do you think SPARX met the student’s mental health needs? In what way?

20. What were the by-products of the intervention that you didn’t expect? a. What parts of the program do/do not suit your school?

**ADDITIONAL QUESTIONS FOR SCHOOL COUNSELLORS ONLY 21. What was your experience with responding to students identified as needing follow-up

during the FP program? a. Was there anything that could have been done to make this experience better for

you and the student? 22. What do you think of the FP risk protocol?

Page 44 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




KEY LEARNING AND FUTURE IMPLEMENTATION 23. What parts of the FP program, if any, could be made more attractive to your school to

increase uptake or support? a. In what ways could we make these parts more attractive?

24. To what extent do you think that the FP program could be integrated into your school’s existing mental health strategy?

25. What advice would you give another school contemplating taking on the FP program? 26. What’s been your major learning or insight from your school’s involvement in the FP

program? Closing

27. Is there anything else you would like to add?

Page 45 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




Volunteer Research Assistants (Study Facilitators) INTRODUCTION: Thank you for offering to take part in this interview. I appreciate your time. I am interested in your thoughts about and experiences with supporting the FP program as a Research Assistant Volunteer. I hope to use this information to work out what works in different schools and how we can help volunteers to best provide support. I will ask you a few questions about your experiences. This is chance for you to reflect and offer feedback on how we could improve. The interview will last up to one hour. It will be recorded and your responses will be transcribed to ensure we accurately reflect your words. All personal information will be removed so that your answers are non-identifiable and confidential. Nothing you personally say will be shared with the volunteer team or the Black Dog Institute more generally, so it will not affect your relationship with them. You can choose to not answer any questions. Do you have any questions before we get started? General Opening Questions:

1. What schools have you visited so far? 2. Roughly how many school visits have you attended?

Specific Questions: CFIR DOMAIN: INDIVIDUAL CHARACTERISTICS Personal attributes (e.g., motivation and expectations)

3. Why were you motivated you to volunteer? 4. What were your expectations coming on board with the FP program?

a. How were your expectations met (or not met)? b. What went differently to what you expected? What surprised you?

Perceptions of Training

5. What are your overall impressions of the training you received? a. How helpful and appropriate was the training? b. What parts were the most useful? c. What was missing from the training?

CFIR DOMAIN: INNER SETTING School visits – Barriers/Facilitators

6. What was your experience with completing the school visits? 7. Did anything unexpected happen during the school visits?

a. If so, what? b. How did you manage this? c. How well do you think you managed this?

8. In general, how was the FP program received by school staff during visits? 9. To what extent was it useful having a volunteer partner/s at the visits?

a. Were there any other benefits/disadvantages? School contextual factors - Barriers/Facilitators

10. What factors influenced your ability to assist students and teachers at the school-visits? a. Were these common across all the visits or specific to a particular visit? b. What aspects of the school made it difficult to support the program?

Page 46 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




c. What else helped or hindered your ability? (e.g., research platform/app specific factors)

d. How did you overcome these barriers? e. Were there any noticeable differences between different schools? If so, what were

they? KEY LEARNING AND FUTURE IMPLEMENTATION

11. What advice would you give a school contemplating taking on the FP program? 12. What advice would you give to others who are thinking about volunteering? 13. What parts of the FP program could be made more attractive to volunteers to increase

uptake or support? 14. What’s been your major learning or insight from taking part in the volunteer program?

Closing


Page 47 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



1

Standards for Reporting Implementation Studies: the StaRI checklist for completion

Checklist itemReported on page # Implementation Strategy

Reported on page # Intervention

“Implementation strategy” refers to how the intervention was implemented

“Intervention” refers to the healthcare or public health intervention that is being implemented.

Title and abstractTitle 1 1-2 Identification as an implementation study, and description of the methodology in the title and/or keywords

Abstract 2 2-3 Identification as an implementation study, including a description of the implementation strategy to be tested, the evidence-based intervention being implemented, and defining the key implementation and health outcomes.

IntroductionIntroduction 3 5 Description of the problem, challenge or deficiency in healthcare or public health that the intervention being implemented aims

to address.Rationale 4 9-11 The scientific background and rationale for the

implementation strategy (including any underpinning theory/framework/model, how it is expected to achieve

its effects and any pilot work).

8 The scientific background and rationale for the intervention being implemented (including evidence

about its effectiveness and how it is expected to achieve its effects).

Aims and objectives

5 9 The aims of the study, differentiating between implementation objectives and any intervention objectives.

Methods: descriptionDesign 6 10 The design and key features of the evaluation, (cross referencing to any appropriate methodology reporting standards) and any

changes to study protocol, with reasonsContext 7 11-13 The context in which the intervention was implemented. (Consider social, economic, policy, healthcare, organisational barriers

and facilitators that might influence implementation elsewhere).Targeted

‘sites’8 13 The characteristics of the targeted ‘site(s)’ (e.g

locations/personnel/resources etc.) for implementation and any eligibility criteria.

8 The population targeted by the intervention and any eligibility criteria.

Description 9 14 A description of the implementation strategy 8 A description of the intervention

Page 48 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



2

Sub-groups 10 N/A Any sub-groups recruited for additional research tasks, and/or nested studies are described

Methods: evaluationOutcomes 11 19 Defined pre-specified primary and other outcome(s) of

the implementation strategy, and how they were assessed. Document any pre-determined targets

N/A Defined pre-specified primary and other outcome(s) of the intervention (if assessed), and how they were assessed. Document any pre-determined targets

Process evaluation

12 15 Process evaluation objectives and outcomes related to the mechanism by which the strategy is expected to work

Economic evaluation

13 N/A Methods for resource use, costs, economic outcomes and analysis for the implementation strategy

N/A Methods for resource use, costs, economic outcomes and analysis for the intervention

Sample size 14 7 Rationale for sample sizes (including sample size calculations, budgetary constraints, practical considerations, data saturation, as appropriate)

Analysis 15 21-22 Methods of analysis (with reasons for that choice)

Sub-group analyses

16 23 Any a priori sub-group analyses (e.g. between different sites in a multicentre study, different clinical or demographic populations), and sub-groups recruited to specific nested research tasks

ResultsCharacteristics 17 N/A Proportion recruited and characteristics of the recipient

population for the implementation strategyN/A Proportion recruited and characteristics (if appropriate)

of the recipient population for the interventionOutcomes 18 N/A Primary and other outcome(s) of the implementation

strategyN/A Primary and other outcome(s) of the Intervention (if

assessed)Process

outcomes19 N/A Process data related to the implementation strategy mapped to the mechanism by which the strategy is expected to work

Economic evaluation

20 N/A Resource use, costs, economic outcomes and analysis for the implementation strategy

N/A Resource use, costs, economic outcomes and analysis for the intervention

Sub-group analyses

21 N/A Representativeness and outcomes of subgroups including those recruited to specific research tasks

Page 49 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



3

Fidelity/ adaptation

22 N/A Fidelity to implementation strategy as planned and adaptation to suit context and preferences

N/A Fidelity to delivering the core components of intervention (where measured)

Contextual changes

23 N/A Contextual changes (if any) which may have affected outcomes

Harms 24 N/A All important harms or unintended effects in each group

DiscussionStructured discussion

25 24-25 Summary of findings, strengths and limitations, comparisons with other studies, conclusions and implications

Implications 26 N/A Discussion of policy, practice and/or research implications of the implementation strategy (specifically

including scalability)

N/A Discussion of policy, practice and/or research implications of the intervention (specifically including

sustainability)General

Statements 27 27 Include statement(s) on regulatory approvals (including, as appropriate, ethical approval, confidential use of routine data, governance approval), trial/study registration (availability of protocol), funding and conflicts of interest

Page 50 of 49


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




Journal: BMJ Open

Manuscript ID bmjopen-2020-042133.R1


Date Submitted by the Author: 25-Oct-2020


<b>Primary Subject Heading</b>: Public health

Secondary Subject Heading: Mental health



BMJ Open on N



jopen.bmj.com

/B

MJ O


ownloaded from





Page 1 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from








3












15 Affiliations:





20 Australia.






Page 2 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from
















1 Abstract







8 Methods and Analysis: A mixed methods design will be used with data collected in the

9 intervention arm of the Future Proofing Study. Data collection methods will include semi-

10 structured interviews with school staff and study facilitators, automatically collected

11 intervention usage data, and participant questionnaires (completed by school staff, school

12 counsellors, study facilitators, and students). Information will be collected about: i) how the

13 intervention was implemented in schools, including fidelity; ii) school contextual factors and

14 their association with intervention reach, uptake and acceptability; iii) how school staff, study

15 facilitators, and students and responded to delivering or completing the intervention. How

16 these factors relate to trial effectiveness outcomes will also be assessed. Overall synthesis of

17 the data will provide school cluster-level and individual-level process outcomes.

18 Ethics and Dissemination: Ethics approval obtained from the University of New South

19 Wales. Results will be submitted for publication in peer reviewed journals and discussed at







Page 3 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from







Page 4 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from






2 This protocol is the first to illustrate how to embed a process evaluation into a school-

3 based randomised controlled trial of a digital depression prevention intervention.

4 Flexible and pragmatic quantitative and qualitative data collection methods will be

5 used to balance research rigor with feasibility within the school delivery context.

6 Process data from a range of key stakeholders will be collected, including school

7 staff, study facilitators, and students.

8 To minimise burden on schools, fidelity data from teachers and in-depth qualitative

9 data from students will not be collected.

10 The methodology and study processes were pilot tested to ensure appropriateness

11 within the school context.

Page 5 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Introduction














15 In line with best-practice recommendations that study protocols that pre-specify











Page 6 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from












9 community.

















Page 7 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 evaluations of face-to-face mental health programs have been documented, some of which

2 have been delivered in school settings (e.g., 17). However, we were unable to find any

3 process evaluation descriptions in the literature that evaluate digital mental health programs

4 in school settings. Conducting process evaluations specific to digital methods of delivery are

5 important because the contextual barriers and facilitators, as well as concerns around fidelity

6 of the intervention, are likely to have unique characteristics. For example, facilitator or

7 training in intervention delivery will be different in supporting the use of an automated

8 program relative to a face-to-face program. This process evaluation is an initial step towards

9 addressing that gap.

















Page 8 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from







4 Trials Registry (ANZCTRN12619000855123) and protocol paper (5).










14 standardised. See the trial protocol for full intervention details (5).












Page 9 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from









6 Preparation











17 We adapted the MRC framework for complex interventions to focus on effectiveness of

18 an evidence-based intervention within a specific context of delivery (i.e., schools). Overall,

19 the objective of this process evaluation is to understand how SPARX is implemented and

20 delivered in schools, and to identify systematic differences and variation in delivery.

21 Specifically, the aims are:

22 1. To evaluate the reach (including completion), uptake and acceptability of the




Page 10 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 o School-level fidelity to the implementation strategy. For example, different

2 schools will likely provide different levels of study support based on available

3 resourcing.


5 assessed from the perspectives of school staff, teachers, and students. For

6 example, young people’s openness to receiving mental health material via an

7 app will likely impact intervention acceptability and completion.


9 outcomes) on clinical effectiveness outcomes at the school- and student-level. School-

10 level clinical effectiveness is defined as changes in clinical outcomes (e.g., self-

11 reported depression) for different schools. The differing ways in which schools

12 support and deliver the intervention will inevitably impact its effectiveness, and this

13 evaluation will assess these differences (3, 23). For example, degree of support from

14 senior school leaders who would be expected to understand that participating in study

15 activities is a priority for the school will likely impact effectiveness at the cluster

16 level.

17 This process evaluation has been designed to capture important information from


19 ultimately impact the effectiveness of the intervention on clinical mental health outcomes for

20 students. Findings will provide insight into factors which support and/or hinder the




24


Page 11 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Design















16 ways that the intervention will be delivered in a given context (including fidelity to the

17 implementation strategy). Normalisation Process Theory will be used to provide additional

18 insights into implementation processes (NPT; 30). NPT aims to identify and explain the

19 implementation of new interventions and how they become integrated into routine care.


21 including intervention reach, uptake, and acceptability/appropriateness (25, 31). These

22 outcomes also map onto the framework of implementation outcomes proposed by Proctor and

23 colleagues (32, 33). Reach refers to the proportion of eligible participants who opened, used,

24 and completed the intervention, as well as the proportion of students from the entire cohort of

25 eligible students in intervention schools who consented to participate. Uptake refers to the

Page 12 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 proportion of schools that were onboarded to the study (intervention and control arms), and

2 of school staff who were willing to support the delivery of the intervention (intervention arm

3 only). Reach and uptake also incorporate the representativeness of the sample (school-level

4 and individual-level). Acceptability and appropriateness refer to the perceived agreeableness

5 or fit of the intervention. This evaluation will incorporate how the barriers and facilitators

6 identified through the CFIR impacted the implementation outcomes, and, in turn, how these

7 implementation outcomes impacted effectiveness outcomes.

8

9 Logic Model

















Page 13 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from







4


6












Page 14 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from


























Implementation Leadership ScaleInterviews with school staff and study facilitators




Page 15 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from







& 3)


Technical issues







NoMAD

Fidelity to the implementation strategy




Implementation Outcomes Reach What was the extent to which those who were eligible

to receive SPARX used it? (Aim 1)Proportion of eligible participants who consented to participate; proportion who opened, used and completed the SPARX interventionRepresentativeness of the student sample

Administrative data about consent. Digital analytic data including usage (app downloads, installs, opens), completion rate (number of modules completed) and time spent using SPARXReported by Year 8 students as part of the online FPS survey



Administrative data






Page 16 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from






Across DomainsHow might the relationship between the intervention, the staff supporting the program, and context of each school shape variation in outcomes (implementation strength metric)? (Aim 2 and 3)What key lessons emerge from this study that can be generalised to the implementation of digital mental health programs in schools more broadly?


Page 17 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from























20 comprises:






Page 18 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from





2 app regularly


4 weekly newsletter








12


14












Page 19 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from









6 described in 5). Digital analytic data about intervention use will be captured by the purpose-

7 built Black Dog Institute research platform, which is being used for the broader FPS.

8 Administrative data will be collected through a range of sources, including communications




12 intervention period has been completed (i.e., after the six-week intervention stage); facilitator

13 data will be collected both before and after the intervention period.

14

15 Table 2. Summary of data forms (and collection point) provided by each of the participant

16 groups

Questionnaire Individual interview Digital analytics

Year 8 students ✓ (post-intervention) ✓ (ongoing)

School staff ✓ (post-intervention) ✓ (post-intervention)

Facilitators ✓ (before first school visit and after final post-intervention visit)

✓ (after final school post-intervention visit)

17




Page 20 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from





2 and funding.














16 35).










Page 21 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from





























Page 22 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from
























21 implementation checklists, communications between schools and research staff, and

22 interviews with school staff.



25 (downloads, installs, and opens), completion (number of modules completed, out of a total of

Page 23 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 seven), and time spent using SPARX. Administrative data about the number of students in

2 intervention schools with consent to participate will also be used as an indicator of reach.

3 Self-report data about individual characteristics (e.g., gender, mental health history) will be

4 used to gauge the representativeness of the students in the intervention schools.







11 Measure (IAM; 41). The IAM is a pragmatic 4-item measure of the perceived fit, relevance,















Page 24 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from










7 and expectations about the intervention and study processes more broadly; knowledge and

8 beliefs about the intervention; relative advantages of the intervention; self-efficacy; barriers

9 and facilitators affecting the delivery of the intervention, including fidelity; appropriateness

10 and acceptability of the intervention; and recommendations for future implementation.

11 School counsellors will also be asked questions about their experiences of managing high risk

12 participants within the study and compatibility with existing workload. For study facilitators,

13 questions focus on: motivations and expectations about their role; confidence and

14 competence in supporting the delivery of SPARX in schools; the quality of their training; and

15 perceptions about their ability to support the study as required.











Page 25 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from





2 and infographics.

3 Procedure

4 Ethical approval was provided by the University of New South Wales Human

5 Research Ethics Committee (HC180836) and NSW Government State Education Research

6 Applications Process Approval (SERAP 2019201).

7 The FPS has two waves of delivery (October-December 2020; April-July 2021).

8 Process evaluation data will be collected at each wave. All school students will complete

9 relevant survey questions at the same time as completing primary measures for the FPS. All

10 study facilitators will be asked to provide informed consent for their data to be used for

11 research purposes following the compulsory face-to-face training session they attend with the

12 research team at the Black Dog Institute. Data provided by facilitators will be from two

13 surveys, one completed immediately following the training and another completed after their

14 final school visit. All school staff from intervention schools (on average, three from each

15 school, estimated number of intervention schools = 100) who were directly involved in the

16 study will be invited to complete one survey following the six-week SPARX intervention

17 period. All surveys will be completed online.

18 After completing their respective online questionnaires, all school staff and study

19 facilitators will be given the option to participate in a 60-minute individual interview

20 (completed either in person or remotely). Purposive sampling will be used to capture a range

21 of diverse school settings and experiences. Face-to-face interviews will be held in a quiet

22 room on school grounds or at the Black Dog Institute. Virtual interviews, which may be

23 required due to COVID-19 restrictions, will also be conducted. All interviews will be audio-

24 recorded and transcribed verbatim. All contact with study facilitators and school staff,

25 including the semi-structured interviews, will be made by research staff who have had no

Page 26 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 previous contact with them during the trial. This independence minimises the risk of bias and

2 demand effects.

3

4 Data analysis

5 Quantitative

6 Questionnaire data and open-ended questions from surveys will be exported into data

7 analytic software for analysis. Descriptive statistics will be calculated for all participant

8 groups and will provide information about intervention use and its acceptability

9 (questionnaire data from Year 8 participants), differential implementation, fidelity to the

10 implementation strategy, school context factors within each school (questionnaire including

11 several short answer questions from school staff), and competence and experience in schools

12 from those facilitating the study (facilitator questionnaires). Differences between school

13 clusters will be assessed using Analysis of Variance methods (ANOVA). Regression models

14 will assess the on the effects of contextual factors on implementation outcomes (e.g., reach,

15 uptake, acceptability).

16 Qualitative

17 Interviews will be digitally audio-recorded and transcribed verbatim. The transcripts

18 will be checked for accuracy against the sound files as per best practice in transcription (47,

19 48). Qualitative data will then be imported into NVivo to aid in data management and

20 analysis. Thematic analysis will be undertaken to identify, interpret and report on the

21 repeated patterns of meaning within the data, drawing from Braun and Clark’s classic six

22 phase model (49, 50). An iterative and reflexive approach will be used to analyse the data,

23 incorporating themes from the data together with topics covered in the interview guide. Two

24 coders will independently engage in a familiarisation phase before generating codes and

25 initial themes for a subset of the data. These codes and themes will be reviewed and

Page 27 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 discussed by the two coders, with refinement occurring via an interactive process. A senior

2 qualitative analysist will also review the first-stage coding framework and scheme before all

3 transcripts are coded by the first coder. Refinement will continue to occur via an interactive

4 process until final codes and themes are realised and defined across the whole data set.

5 Research rigor will be enhanced by a team approach to analysis, reflexive field notes, and

6 prolonged engagement with the subject matter (51).

7 Triangulation of Qualitative and Quantitative Data

8 Triangulation involves the use of multiple approaches to address a research question.

9 The combination of several approaches increases confidence in the findings and provides a

10 more comprehensive account of the results than individual approaches would do alone (52).

11 In this study, reliability, validity and confidence will be maximised through cross verification

12 and exploration of differences between the outcomes of the various methods. This takes place

13 in several ways:









22 strategy.



25 Additional Analyses

Page 28 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 We will generate an ‘implementation strength’ metric for each school and its

2 relationship to primary and secondary trial outcomes. This is an emerging evaluation

3 approach, used mainly in low and middle-income countries. The approach aims to understand

4 the degree of implementation effort needed during intervention delivery to achieve desired

5 benefits (53, 54).










15

16 Discussion










Page 29 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from























20 the conclusion of the trial. Second, given the complexity of the study and high demand placed

21 on students (e.g., engaging with the study apps and completing the online surveys at multiple

22 time points over five years), we are not collecting in-depth qualitative data by way of

23 interviews. Instead, we collect information about students’ perceptions of the intervention

24 (e.g., acceptability) through short self-report questions in the online survey. Second, the

25 qualitative interviewer is a member of the research team (but not the evaluation team). Care

Page 30 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 will be taken to ensure that this staff member has no contact with schools prior to the

2 interview visit to minimise bias. However, there remains a risk that demand effects may

3 impact the information that is shared. Third, the process evaluation process (questionnaires

4 and interview) will undoubtedly add to the burden placed on school staff. Given that the FPS

5 is already placing a significant burden on the time of busy school staff, this additional

6 component might contribute to low levels of participation.














20

21

22

23

24

25

Page 31 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from





2 As noted, all procedures have received ethical approval from relevant committees. Results

3 from this study will be published in an open access peer-reviewed journal.

4

5 Additional Files







12

13

14

15

16

17

18

19

20

21

22

23

24

25

Page 32 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Declarations

2 Trial status

3 Recruitment for the trial is underway. Data collection commences in October 2020 (delayed

4 from April 2020 due to COVID-19).

5

6 Ethical approval and consent to participate

7 This study has ethical approval from the University of New South Wales Human Research

8 Ethics Committee (HC180836; 21st January 2019) and NSW Government State Education

9 Research Applications Process Approval (SERAP 2019201; 19th August 2019). All schools,

10 staff, parents and participants must provide active consent to participate.

11

12 Consent for publication

13 Not applicable.

14

15 Availability of data and materials

16 Not applicable.

17

18 Competing interests


20

21 Funding

22 Funding for this project came from an NSW Health Early-Mid Career Fellowship awarded to

23 Aliza Werner-Seidler, and a Black Dog Institute Post-Doctoral Fellowship awarded to Joanne

24 Beames, secured by Helen Christensen. ALC is supported by NHMRC fellowships 1122544

25 and 1173146. PJB is supported by NHMRC Fellowship 1158707. Funding for the

Page 33 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 randomised controlled trial within which this process evaluation is embedded came from an

2 NHMRC Project Grant Awarded to Helen Christensen GNT1120646. The funding bodies had

3 no role in any aspect of the study design or this manuscript.

4

5 Author contributions

6 HC and AWS conceived of the study and secured the funding. AWS and JRB led the design

7 of the process evaluation, with input from all authors (including KH, IZ, PJB), and expert

8 guidance from RL and KB. AWS drafted the manuscript, with assistance from JRB. ALC,

9 MT, KM, RL, KB and HC have a continuing role in monitoring the conduct and outcomes of

10 the process evaluation. All named authors contributed substantially to the approved final

11 manuscript.

12

13 Acknowledgments





18


20 Not applicable.

Page 34 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 References







8 2015;350:h1258.



11 Trials. 2013;14:15.














Page 35 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from
















13 Medicine. 2015;45:3357.





18 17. Kösters MP, Chinapaw MJM, Zwaanswijk M, van der Wal MF, Utens EMWJ,

19 Koot HM. FRIENDS for Life: Implementation of an indicated prevention program

20 targeting childhood anxiety and depression in a naturalistic setting. Mental Health &

21 Prevention. 2017:6,44.



24 Medicine. 2015;45:3357.

Page 36 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from
























21 1999;89:1322.




Page 37 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from








5 2016;11:72.













18 2009;36:24.




22 2011;38:65.




Page 38 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from







4 2019;11:294.






10 2009;100:389.



13 2014;9:45.







20 Methodol. 2018;18:135.




Page 39 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from















12 Adolesc. 2019:1.





17 method. 2002:629.




21 2006;3:77.


23 Health. 2019;11:589.


25 Nurs. 2015;18:34.

Page 40 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from











8 Lancet. 2011;377:85.


10 Open. 2018;1:e180335.

Page 41 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from











insomnia












research staff


staff














Page 42 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



Research Team

Facilitators


Year 8 Students

Training

Other staff

Delivery Support



Page 43 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




Additional File 1







Specific Questions:















content?

Page 44 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from

























Page 45 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from








program? Closing


Page 46 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from

















Page 47 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from









Closing


Page 48 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



1














Aims and objectives









Page 49 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



2





Process evaluation


Economic evaluation





Sub-group analyses

16 26-28 Any a priori sub-group analyses (e.g. between different sites in a multicentre study, different clinical or demographic populations), and sub-groups recruited to specific nested research tasks





assessed)Process


Economic evaluation



Sub-group analyses


Page 50 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



3




Contextual changes









Statements 27 31-33 Include statement(s) on regulatory approvals (including, as appropriate, ethical approval, confidential use of routine data, governance approval), trial/study registration (availability of protocol), funding and conflicts of interest

Page 51 of 50


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




Journal: BMJ Open

Manuscript ID bmjopen-2020-042133.R2


Date Submitted by the Author: 26-Nov-2020


<b>Primary Subject Heading</b>: Public health

Secondary Subject Heading: Mental health



BMJ Open on N



jopen.bmj.com

/B

MJ O


ownloaded from





Page 1 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from








3












15 Affiliations:





20 Australia.






Page 2 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from
















1 Abstract







8 Methods and Analysis: Using a hybrid type 1 design, a mixed methods approach will be

9 used with data collected in the intervention arm of the Future Proofing Study. Data collection

10 methods will include semi-structured interviews with school staff and study facilitators,

11 automatically collected intervention usage data, and participant questionnaires (completed by

12 school staff, school counsellors, study facilitators, and students). Information will be

13 collected about: i) how the intervention was implemented in schools, including fidelity; ii)

14 school contextual factors and their association with intervention reach, uptake and

15 acceptability; iii) how school staff, study facilitators, and students and responded to

16 delivering or completing the intervention. How these factors relate to trial effectiveness

17 outcomes will also be assessed. Overall synthesis of the data will provide school cluster-level

18 and individual-level process outcomes.

19 Ethics and Dissemination: Ethics approval was obtained from the University of New South

20 Wales Human Research Ethics Committee (HC180836; 21st January 2019) and the NSW

21 Government State Education Research Applications Process (SERAP 2019201; 19th August

22 2019). Results will be submitted for publication in peer reviewed journals and discussed at




Page 3 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from










Page 4 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from






2 The methodology of this embedded process evaluation is underpinned by

3 implementation frameworks and logic modelling.

4 Flexible and pragmatic quantitative and qualitative data collection methods will be

5 used to balance research rigor with feasibility within the school delivery context.

6 Process data from a range of key stakeholders will be collected, including school

7 staff, study facilitators, and students.

8 To minimise burden on schools, fidelity data from teachers and in-depth qualitative

9 data from students will not be collected.

10 The methodology and study processes were pilot tested to ensure appropriateness

11 within the school context.

Page 5 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Introduction














15 In line with best-practice recommendations that study protocols that pre-specify











Page 6 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from












9 community.

















Page 7 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 evaluations of face-to-face mental health programs have been documented, some of which

2 have been delivered in school settings (e.g., 17). However, we were unable to find any

3 process evaluation descriptions in the literature that evaluate digital mental health programs

4 in school settings. Conducting process evaluations specific to digital methods of delivery are

5 important because the contextual barriers and facilitators, as well as concerns around fidelity

6 of the intervention, are likely to have unique characteristics. For example, facilitator or

7 training in intervention delivery will be different in supporting the use of an automated

8 program relative to a face-to-face program. This process evaluation is an initial step towards

9 addressing that gap.

















Page 8 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from







4 Trials Registry (ANZCTRN12619000855123) and protocol paper (5).










14 standardised. See the trial protocol for full intervention details (5).












Page 9 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from









6 Preparation











17 We adapted the MRC framework for complex interventions to focus on effectiveness of

18 an evidence-based intervention within a specific context of delivery (i.e., schools). Overall,

19 the objective of this process evaluation is to understand how SPARX is implemented and

20 delivered in schools, and to identify systematic differences and variation in delivery.

21 Specifically, the aims are:

22 1. To evaluate the reach (including completion), uptake and acceptability of the




Page 10 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 o School-level fidelity to the implementation strategy. For example, different

2 schools will likely provide different levels of study support based on available

3 resourcing.


5 assessed from the perspectives of school staff, teachers, and students. For

6 example, young people’s openness to receiving mental health material via an

7 app will likely impact intervention acceptability and completion.


9 outcomes) on clinical effectiveness outcomes at the school- and student-level. School-

10 level clinical effectiveness is defined as changes in clinical outcomes (e.g., self-

11 reported depression) for different schools. The differing ways in which schools

12 support and deliver the intervention will inevitably impact its effectiveness, and this

13 evaluation will assess these differences (3, 23). For example, degree of support from

14 senior school leaders who would be expected to understand that participating in study

15 activities is a priority for the school will likely impact effectiveness at the cluster

16 level.

17 This process evaluation has been designed to capture important information from


19 ultimately impact the effectiveness of the intervention on clinical mental health outcomes for

20 students. Findings will provide insight into factors which support and/or hinder the




24

25

Page 11 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from





2 Design















17 ways that the intervention will be delivered in a given context (including fidelity to the

18 implementation strategy). Normalisation Process Theory will be used to provide additional

19 insights into implementation processes (NPT; 30). NPT aims to identify and explain the

20 implementation of new interventions and how they become integrated into routine care.


22 including intervention reach, uptake, and acceptability/appropriateness (25, 31). These

23 outcomes also map onto the framework of implementation outcomes proposed by Proctor and

24 colleagues (32, 33). Reach refers to the proportion of eligible participants who opened, used,

25 and completed the intervention, as well as the proportion of students from the entire cohort of

Page 12 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 eligible students in intervention schools who consented to participate. Uptake refers to the

2 proportion of schools that were onboarded to the study (intervention and control arms), and

3 of school staff who were willing to support the delivery of the intervention (intervention arm

4 only). Reach and uptake also incorporate the representativeness of the sample (school-level

5 and individual-level). Acceptability and appropriateness refer to the perceived agreeableness

6 or fit of the intervention. This evaluation will incorporate how the barriers and facilitators

7 identified through the CFIR impacted the implementation outcomes, and, in turn, how these

8 implementation outcomes impacted effectiveness outcomes.

9

10 Logic Model
















Page 13 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from








5


7

Page 14 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from


























Implementation Leadership ScaleInterviews with school staff and study facilitators




Page 15 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from







and 3)


Technical issues







NoMAD

Fidelity to the implementation strategy




Implementation Outcomes Reach What was the extent to which those who were eligible

to receive SPARX used it? (Aim 1)Proportion of eligible participants who consented to participate; proportion who opened, used and completed the SPARX interventionRepresentativeness of the student sample

Administrative data about consent. Digital analytic data including usage (app downloads, installs, opens), completion rate (number of modules completed) and time spent using SPARXReported by Year 8 students as part of the online FPS survey



Administrative data






Page 16 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from






Across DomainsHow might the relationship between the intervention, the staff supporting the program, and context of each school shape variation in outcomes (implementation strength metric)? (Aim 2 and 3)How might the school-level variation (in implementation fidelity and outcomes) affect clinical effectiveness outcomes (e.g., self-reported depression)? (Aim 3)What key lessons emerge from this study that can be generalised to the implementation of digital mental health programs in schools more broadly?


Page 17 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from























20 comprises:






Page 18 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from





2 app regularly


4 weekly newsletter








12


14












Page 19 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from









6 described in 5). Digital analytic data about intervention use will be captured by the purpose-

7 built Black Dog Institute research platform, which is being used for the broader FPS.

8 Administrative data will be collected through a range of sources, including communications




12 intervention period has been completed (i.e., after the six-week intervention stage); facilitator

13 data will be collected both before and after the intervention period.

14

15 Table 2. Summary of data forms (and collection point) provided by each of the participant

16 groups

Participant group Questionnaire Individual interview Digital analyticsYear 8 students ✓ (post-intervention) ✓ (ongoing)School staff ✓ (post-intervention) ✓ (post-intervention)Facilitators ✓ (before first school

visit and after final post-intervention visit)

✓ (after final school post-intervention visit)

17





22 and funding.

Page 20 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from

















14 35).












Page 21 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from





























Page 22 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from






















19 implementation checklists, communications between schools and research staff, and

20 interviews with school staff.



23 (downloads, installs, and opens), completion (number of modules completed, out of a total of

24 seven), and time spent using SPARX. Administrative data about the number of students in

25 intervention schools with consent to participate will also be used as an indicator of reach.

Page 23 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Self-report data about individual characteristics (e.g., gender, mental health history) will be

2 used to gauge the representativeness of the students in the intervention schools.







9 Measure (IAM; 41). The IAM is a pragmatic 4-item measure of the perceived fit, relevance,

















Page 24 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from








5 and expectations about the intervention and study processes more broadly; knowledge and

6 beliefs about the intervention; relative advantages of the intervention; self-efficacy; barriers

7 and facilitators affecting the delivery of the intervention, including fidelity; appropriateness

8 and acceptability of the intervention; and recommendations for future implementation.

9 School counsellors will also be asked questions about their experiences of managing high risk

10 participants within the study and compatibility with existing workload. For study facilitators,

11 questions focus on: motivations and expectations about their role; confidence and

12 competence in supporting the delivery of SPARX in schools; the quality of their training; and

13 perceptions about their ability to support the study as required.












25 and infographics.

Page 25 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Procedure

2 The FPS has three waves of delivery (October-December 2020; April-July 2021; July-

3 September 2021). Process evaluation data will be collected at each wave. All school students

4 will complete relevant survey questions at the same time as completing primary measures for

5 the FPS. All study facilitators will be asked to provide informed consent for their data to be

6 used for research purposes following the compulsory face-to-face training session they attend

7 with the research team at the Black Dog Institute. Data provided by facilitators will be from

8 two surveys, one completed immediately following the training and another completed after

9 their final school visit. All school staff from intervention schools (on average, three from

10 each school, estimated number of intervention schools = 100) who were directly involved in

11 the study will be invited to complete one survey following the six-week SPARX intervention

12 period. All surveys will be completed online.

13 After completing their respective online questionnaires, all school staff and study

14 facilitators will be given the option to participate in a 60-minute individual interview

15 (completed either in person or remotely). Purposive sampling will be used to capture a range

16 of diverse school settings and experiences. Face-to-face interviews will be held in a quiet

17 room on school grounds or at the Black Dog Institute. Virtual interviews, which may be

18 required due to COVID-19 restrictions, will also be conducted. All interviews will be audio-

19 recorded and transcribed verbatim. All contact with study facilitators and school staff,

20 including the semi-structured interviews, will be made by research staff who have had no

21 previous contact with them during the trial. This independence minimises the risk of bias and

22 demand effects.

23

24 Data analysis Plan

25 Quantitative. Survey questionnaire data (from approximately 100 schools) will be

Page 26 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 exported into data analytic software for analysis. Descriptive statistics will be calculated for

2 all participant groups and will provide information about intervention use and its

3 acceptability (questionnaire data from Year 8 participants), differential implementation,

4 fidelity to the implementation strategy, school context factors within each school

5 (questionnaire including several short answer questions from school staff), and competence

6 and experience in schools from those facilitating the study (facilitator questionnaires).

7 Differences between school clusters will be assessed using Analysis of Variance methods

8 (ANOVA). Regression models will assess the on the effects of contextual factors on

9 implementation outcomes (e.g., reach, uptake, acceptability).

10 Qualitative. Interviews will be digitally audio-recorded and transcribed verbatim.

11 The transcripts will be checked for accuracy against the sound files as per best practice in

12 transcription (47, 48). Qualitative data will then be imported into NVivo to aid in data

13 management and analysis. Thematic analysis will be undertaken to identify, interpret and

14 report on the repeated patterns of meaning within the data, drawing from Braun and Clark’s

15 classic six phase model (49, 50). An iterative and reflexive approach will be used to analyse

16 the data, incorporating themes from the data together with topics covered in the interview

17 guide. Two coders will independently engage in a familiarisation phase before generating

18 codes and initial themes for a subset of the data. These codes and themes will be reviewed

19 and discussed by the two coders, with refinement occurring via an interactive process. A

20 senior qualitative analysist will also review the first-stage coding framework and scheme

21 before all transcripts are coded by the first coder. Refinement will continue to occur via an

22 interactive process until final codes and themes are realised and defined across the whole data

23 set. Research rigor will be enhanced by a team approach to analysis, reflexive field notes, and

24 prolonged engagement with the subject matter (51).

25 Triangulation of Qualitative and Quantitative Data. Triangulation involves the use

Page 27 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 of multiple approaches to address a research question. The combination of several approaches

2 increases confidence in the findings and provides a more comprehensive account of the

3 results than individual approaches would do alone (52). In this study, reliability, validity and

4 confidence will be maximised through cross verification and exploration of differences

5 between the outcomes of the various methods. This takes place in several ways:









14 strategy.



17 Additional Analyses. We will generate an ‘implementation strength’ metric for each

18 school and its relationship to primary and secondary trial outcomes. This is an emerging

19 evaluation approach, used mainly in low and middle-income countries. The approach aims to

20 understand the degree of implementation effort needed during intervention delivery to

21 achieve desired benefits (53, 54).





Page 28 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from









6


8 Ethics approval was obtained from the University of New South Wales Human

9 Research Ethics Committee (HC180836; 21st January 2019) and the NSW Government State

10 Education Research Applications Process (SERAP 2019201; 19th August 2019). Results will

11 be submitted for publication in peer reviewed journals and discussed at conferences. Our

12 process evaluation will contextualise the trial findings with respect to how the intervention

13 may have worked in some schools but not in others. This evaluation will inform the

14 development of a model for rolling out digital interventions for the prevention of mental

15 illness in schools.

16

17 Discussion









Page 29 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from
























21 the conclusion of the trial. Second, given the complexity of the study and high demand placed

22 on students (e.g., engaging with the study apps and completing the online surveys at multiple

23 time points over five years), we are not collecting in-depth qualitative data by way of

24 interviews. Instead, we collect information about students’ perceptions of the intervention

25 (e.g., acceptability) through short self-report questions in the online survey. Second, the

Page 30 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 qualitative interviewer is a member of the research team (but not the evaluation team). Care

2 will be taken to ensure that this staff member has no contact with schools prior to the

3 interview visit to minimise bias. However, there remains a risk that demand effects may

4 impact the information that is shared. Third, the process evaluation process (questionnaires

5 and interview) will undoubtedly add to the burden placed on school staff. Given that the FPS

6 is already placing a significant burden on the time of busy school staff, this additional

7 component might contribute to low levels of participation.














21

22

23

24

25

Page 31 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Additional Files


3






9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

Page 32 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 References







8 2015;350:h1258.



11 Trials. 2013;14:15.















Page 33 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from















12 Medicine. 2015;45:3357.





17 17. Kösters MP, Chinapaw MJM, Zwaanswijk M, van der Wal MF, Utens EMWJ, Koot

18 HM. FRIENDS for Life: Implementation of an indicated prevention program targeting

19 childhood anxiety and depression in a naturalistic setting. Mental Health & Prevention.

20 2017:6,44.



23 Medicine. 2015;45:3357.



Page 34 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from






















19 1999;89:1322.







Page 35 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from





2 2016;11:72.













15 2009;36:24.




19 2011;38:65.







Page 36 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 2019;11:294.






7 2009;100:389.



10 2014;9:45.







17 Methodol. 2018;18:135.









Page 37 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from










7 Adolesc. 2019:1.





12 method. 2002:629.




16 2006;3:77.


18 Health. 2019;11:589.


20 Nurs. 2015;18:34.






Page 38 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from






3 Lancet. 2011;377:85.


5 Open. 2018;1:e180335.

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

Page 39 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Declarations

2 Authors’ Contributions

3 HC and AW-S conceived of the study and secured the funding. AW-S and JRB led the design

4 of the process evaluation, with input from all authors (including KH, IZ, PJB), and expert

5 guidance from RL and KB. AW-S drafted the manuscript, with assistance from JRB. ALC,

6 MT, KM, RL, KB and HC have a continuing role in monitoring the conduct and outcomes of

7 the process evaluation. All named authors contributed substantially to the approved final

8 manuscript.

9

10 Funding Statement

11 Funding for this project came from an NSW Ministry of Health Early-Mid Career Fellowship

12 awarded to AW-S, and a Black Dog Institute Post-Doctoral Fellowship awarded to JRB,

13 secured by HC. ALC is supported by NHMRC fellowships 1122544 and 1173146. PJB is

14 supported by NHMRC Fellowship 1158707. Funding for the randomised controlled trial

15 within which this process evaluation is embedded came from an NHMRC Project Grant

16 Awarded to HC GNT1120646. The funding bodies had no role in any aspect of the study

17 design or this manuscript.

18

19 Competing Interests Statement


21

22 Acknowledgments




Page 40 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from





2

3 Trial Status

4 Recruitment for the trial is underway. Data collection commences in October 2020 (delayed

5 from April 2020 due to COVID-19).

6

7 Consent to Participate

8 All schools, staff, parents, and participants must provide active consent to participate.

9

10 Consent for Publication

11 Not applicable.

12

13 Availability of Data and Materials

14 Not applicable.

15


17 Not applicable.

Page 41 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




1 Figure Legends












13

14 Figure 2. Details of implementation strategy training and delivery structure.

15

16

Page 42 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from











insomnia












research staff


staff














Page 43 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



Research Team

Facilitators


Year 8 Students

Training

Other staff

Delivery Support



Page 44 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from




Additional File 1







Specific Questions:















content?

Page 45 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from

























Page 46 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from








program? Closing


Page 47 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from

















Page 48 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from









Closing


Page 49 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



1














Aims and objectives









Page 50 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



2





Process evaluation


Economic evaluation





Sub-group analyses

16 26-28 Any a priori sub-group analyses (e.g. between different sites in a multicentre study, different clinical or demographic populations), and sub-groups recruited to specific nested research tasks





assessed)Process


Economic evaluation



Sub-group analyses


Page 51 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from



3




Contextual changes









Statements 27 31-33 Include statement(s) on regulatory approvals (including, as appropriate, ethical approval, confidential use of routine data, governance approval), trial/study registration (availability of protocol), funding and conflicts of interest

Page 52 of 51


BMJ Open

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

on Novem


http://bmjopen.bm

j.com/

BM



nloaded from


bmj open is committed to open peer review. as part of this

Documents