A Realist Evaluation of Family Navigation in Youth Mental Health and Addictions
by
Nadine Reid
A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy
Institute of Health Policy, Management and Evaluation University of Toronto
© Copyright by Nadine Reid 2017
ii
A Realist Evaluation of Family Navigation in Youth Mental Health
and Addictions
Nadine Reid
Doctor of Philosophy
Institute of Health Policy, Management and Evaluation
University of Toronto
2017
Abstract
In Canada today, many families of youth with mental health and/or addiction concerns are
struggling to access the care they need. The Family Navigation Project is a service affiliated with
Sunnybrook Health Sciences Centre in Toronto, Ontario, which aims to provide needs-based,
family-centred system navigation to families of youth aged 13 to 26 with mental health and/or
addiction concerns. The current study is a Realist Evaluation of the Family Navigation Project.
The objectives of this study were a) to describe the population being served by the Family
Navigation Project; b) to develop a conceptual framework for family navigation and a program
theory for the Family Navigation Project; and c) to test the program theory, and refine it based on
the results. This multi-phase, mixed methods study applied a Realist Evaluation framework and a
cross-sectional methodological design in which both quantitative and qualitative data were
collected through an online survey package in order to evaluate the sample characteristics;
perception of navigation; the impact of perceived experience on family empowerment, family
quality of life, and service satisfaction; and the influence of context. Data was collected from a
convenience sample of 134 families seeking care on behalf of youth, who were registered with
the Family Navigation Project at the time of the study. Descriptive, inferential and qualitative
iii
analyses were performed. Results indicated that the Family Navigation Project reached its target
population in this sample; that families in this sample were highly satisfied with the services they
received; that most families in this sample perceived care to be accessible, continuous, and
family-inclusive; that this perceived experience significantly influenced family empowerment,
family quality of life, and service satisfaction; and that both individual and systemic-level
contexts influence experience and outcomes to varying extents. The conceptual framework and
program theory were subsequently refined. Applications, contributions and limitations are noted.
iv
Acknowledgments
I would like to first express my sincere gratitude to the families involved in this study for sharing
their stories. I would also like to thank the staff at the Family Navigation Project for their
ongoing enthusiastic participation and feedback. Specifically, I would like to thank Dr. Anthony
Levitt for introducing me to the topic of navigation and including me in the development of the
Family Navigation Project many years ago; and for his ongoing support since then.
To my supervisors, Dr. Rhonda Cockerill and Dr. Janet Durbin, thank you for your consistent
(re)direction over the last four years, and for patiently and reliably keeping my feet (and ideas)
on the ground.
Finally, thank you to my husband and family, who always help to keep things in perspective; and
to Ruby, for cuddle breaks, bottomless laughs, and daily walks in the park.
v
Table of Contents
Acknowledgments.......................................................................................................................... iv
Table of Contents .............................................................................................................................v
List of Tables ...................................................................................................................................x
List of Figures ............................................................................................................................... xii
List of Appendices ....................................................................................................................... xiii
Chapter 1 Introduction .....................................................................................................................1
Description of the current state ...................................................................................................1
1.1 Navigation: A potential solution ..........................................................................................4
1.1.1 Navigation in mental health and addictions .............................................................5
1.2 The Family Navigation Project ............................................................................................9
1.2.1 Background and project development .....................................................................9
1.2.2 Project description .................................................................................................10
Chapter 2 Theoretical and Conceptual Frameworks ......................................................................14
Theoretical evaluation framework: Realist evaluation .............................................................14
2.1 Theory-building process ....................................................................................................16
2.1.1 Program theory.......................................................................................................18
2.2 Conceptual framework .......................................................................................................19
2.2.1 Context ...................................................................................................................19
2.2.2 Mechanisms ...........................................................................................................20
2.2.2.1 Accessibility ............................................................................................20
2.2.2.2 Continuity of care ....................................................................................21
2.2.2.3 Family involvement .................................................................................21
2.2.3 Outcomes of interest ..............................................................................................22
vi
2.2.3.1 Family empowerment ..............................................................................23
2.2.3.2 Family quality of life ...............................................................................24
2.2.3.3 Service satisfaction ..................................................................................25
Chapter 3 Research Questions and Rationale ................................................................................27
Research questions ....................................................................................................................27
3.1 Hypotheses .........................................................................................................................27
3.2 Objectives ..........................................................................................................................28
3.3 Rationale for the current study ...........................................................................................29
Chapter 4 Measurement .................................................................................................................30
Specification of variables ..........................................................................................................30
4.1 Context variables ...............................................................................................................30
4.2 Mechanism variables .........................................................................................................30
4.3 Outcome variables .............................................................................................................31
4.3.1 Family empowerment ............................................................................................31
4.3.2 Family quality of life .............................................................................................32
4.3.3 Service satisfaction ................................................................................................33
4.4 Qualitative measures ..........................................................................................................34
4.5 Measurement validity.........................................................................................................35
Chapter 5 Methods .........................................................................................................................37
Design overview .......................................................................................................................37
5.1 Study setting.......................................................................................................................37
5.2 Ethics..................................................................................................................................37
5.3 Sample................................................................................................................................37
5.4 Data collection ...................................................................................................................38
5.4.1 Survey methods ......................................................................................................38
vii
5.4.1.1 Pilot phase overview................................................................................39
5.4.1.2 Response rates .........................................................................................39
5.4.2 Chart review ...........................................................................................................41
5.5 Sample size and power.......................................................................................................41
5.6 Data storage .......................................................................................................................42
5.7 Overview of analytical approach .......................................................................................42
Chapter 6 Quantitative Analysis ....................................................................................................44
Overview of quantitative approach ...........................................................................................44
6.1 Importing and coding of data .............................................................................................44
6.2 Data quality ........................................................................................................................45
6.3 Descriptive analyses...........................................................................................................45
6.3.1 Context variables ...................................................................................................45
6.3.1.1 Demographics ..........................................................................................46
6.3.1.2 Mental health and addiction characteristics ............................................48
6.3.1.3 Previous service use ................................................................................51
6.3.1.4 Reasons for contact..................................................................................52
6.3.2 Mechanism variables .............................................................................................53
6.3.2.1 Accessibility ............................................................................................53
6.3.2.2 Continuity of care ....................................................................................54
6.3.2.3 Family involvement .................................................................................56
6.3.3 Outcome variables .................................................................................................56
6.3.3.1 Family empowerment ..............................................................................57
6.3.3.2 Family quality of life ...............................................................................60
6.3.3.3 Service satisfaction ..................................................................................62
6.4 Correlational analysis.........................................................................................................64
viii
6.4.1 Overview of approach ............................................................................................64
6.4.2 C-M and C-O dyads: Selection of covariates for inferential modelling ................65
6.4.3 M-M, M-O and O-O dyads: Satisfaction of modeling assumptions ......................71
6.4.3.1 M-M dyads ..............................................................................................71
6.4.3.2 M-O dyads ...............................................................................................72
6.4.4 Implications for modeling ......................................................................................72
6.5 Factor analysis ...................................................................................................................73
6.6 Inferential modelling ..........................................................................................................75
6.6.1 Overview of statistical approach ............................................................................76
6.6.2 Testable hypotheses ...............................................................................................78
6.6.3 Models of family empowerment ............................................................................80
6.6.3.1 FES Family subscale ...............................................................................81
6.6.3.2 FES Service-seeking subscale .................................................................83
6.6.4 Model of family quality of life...............................................................................85
6.6.5 Models of service satisfaction ................................................................................87
6.6.5.1 Inverse NAVSAT total score...................................................................87
6.6.5.2 Inverse satisfaction with referred services (SRS) score ..........................89
Chapter 7 Qualitative Analysis ......................................................................................................93
Overview of qualitative approach .............................................................................................93
7.1 Descriptive and thematic analyses .....................................................................................95
7.1.1 Context ...................................................................................................................95
7.1.1.1 Theme 1: You can’t force someone to get well .......................................95
7.1.1.2 Theme 2: You can’t navigate to services that don’t exist........................97
7.1.1.3 Theme 3: Existing services lack accessibility and continuity of care .....99
7.1.1.4 Theme 4: Privacy, consent and capacity legislation ..............................101
ix
7.1.2 Mechanisms .........................................................................................................103
7.1.2.1 M(resources) ..........................................................................................103
7.1.2.2 M(reasoning) .........................................................................................107
7.1.2.3 The link between resources and reasoning ............................................110
7.1.3 Outcomes .............................................................................................................111
7.1.3.1 Positive outcomes ..................................................................................112
7.1.3.2 Negative outcomes ................................................................................114
Chapter 8 Discussion ...................................................................................................................117
Results as per the research questions, conceptual framework and program theory ................117
8.1 Research question 1 .........................................................................................................117
8.2 Research question 2 .........................................................................................................120
8.3 Research question 3 .........................................................................................................122
8.4 Results in relation to the conceptual framework and program theory .............................125
Chapter 9 Conclusions .................................................................................................................129
Limitations, mitigations and contributions .............................................................................129
9.1 Study limitations and mitigations ....................................................................................129
9.2 Contributions....................................................................................................................132
References ....................................................................................................................................134
Appendix A Themes from the Family Navigation Project’s Focus Groups ................................142
Appendix B Logic Model and Conceptual Framework ..............................................................147
Appendix C Measurement Summary ...........................................................................................150
x
List of Tables
Table 1. Components of family navigation
Table 2. Hypotheses by research question and dependent variable
Table 3. Descriptive statistics for continuous current program use variables
Table 4. Frequency statistics for demographic variables
Table 5. Descriptive statistics for continuous mental health and addiction characteristic variables
Table 6. Frequency statistics for mental health characteristic variables
Table 7. Frequency statistics for addiction characteristic variables
Table 8. Frequency statistics for service use characteristic variables
Table 9. Frequency statistics for reasons for contact variable
Table 10. Descriptive statistics by item for the “Accessibility scale”
Table 11. Descriptive statistics by item for the “Continuity of care scale”
Table 12. Descriptive statistics for the single item “Family involvement”
Table 13. Descriptive statistics by item for the “FES Family subscale”
Table 14. Descriptive statistics by item for the “FES Service-seeking subscale”
Table 15. Descriptive statistics for the “m-BCFQoLS scale”
Table 16. Descriptive statistics for the “NAVSAT total score scale”
Table 17. Results of a correlational analysis between select C-M and C-O dyads
Table 18. Summary of covariates and predictors for modeling by outcome
Table 19. Spearman’s rho correlation coefficients for mechanism and outcome variables
Table 20. Results of a Principal Components analysis for mechanism variables
Table 21. Descriptive statistics for the Bartlett Factor Score for “navigation mechanism”
Table 22. Correlation coefficients for the Bartlett Factor Score with outcomes
Table 23. Dependent variables and corresponding hypotheses
Table 24. Covariates selected for inclusion in modeling by outcome
Table 25. Parameter estimates for the dependent variable “FES Family score”
Table 26. Parameter estimates for the dependent variable “FES Service-seeking score”
Table 27. Parameter estimates for the dependent variable “m-BCFQoLS total score”
Table 28. Parameter estimates for the dependent variable “Inverse NAVSAT total score”
Table 29. Parameter estimates for the dependent variable “Inverse satisfaction with referred
service (SRS) score”
Table 30. Omnibus Test of Model Fit results by dependent variable
Table 31. Generalized linear model equations by dependent variable
Table 32. M-M dyad combination by theme
xii
List of Figures
Figure 1. Continuum of service delivery models with examples of existing programs
Figure 2. Word cloud depicting word frequency in all phrases coded as an outcome
xiii
List of Appendices
Appendix A: Themes from the Family Navigation Project’s Focus Groups
Appendix B: Logic Model and Conceptual Framework
Appendix C: Measurement Summary
1
Chapter 1 Introduction
Description of the current state
In Canada today, three out of four youth struggling with mental health and/or addiction (MHA)
concerns do not receive the care they need (Mental Health Commission of Canada [MHCC],
2015b). Mood and anxiety disorders rates among youth are rising, and suicide remains the
second leading cause of death at a rate that has not decreased significantly in the last decade
(MHCC, 2015a). Of all age groups, youth aged 15-24 are the most likely to experience these
types of health concerns and yet least likely to access services (Patton, 2007). Recently released
data from the Canadian Institute for Health Information [CIHI] indicates that families of youth
aged five to 24 with mental health disorders are increasingly seeking care in hospital settings due
to the unavailability or inaccessibility of community-based services; youth emergency
department (ED) visits and inpatient admissions for mental health reasons have increased by
56% and 47%, respectively, in a 10-year period (CIHI, 2017).
There are a variety of explanations behind these statistics, many of which are associated with
systemic challenges in the amount of awareness, knowledge, capacity, integration, coordination,
accessibility and accountability for youth MHA treatment (Institute for Clinical Evaluative
Sciences [ICES], 2017; MHCC, 2012, 2015b, 2017; Ministry of Health and Long Term Care
[MOHLTC], 2011). There is a particular lack of capacity for the specialized treatment of
complex conditions involving one or more MHA concerns (Canadian Mental Health Association
[CMHA], 2013; CIHI, 2013). The impact of this gap is significant as MHA concerns frequently
co-occur (Kessler, et al., 2005, 2012; Pearson, Janz, & Ali, 2013). The fragmented system that
results from insufficient capacity means that it is often extremely difficult to identify and access
the right doors and subsequent pathways to appropriate care (MHCC, 2012, 2015a; MOHLTC,
2011). In the field of youth MHA concerns, it is often the family (i.e. parent or guardian) who
facilitates the interaction with the health care system and negotiates access to resources on the
youth’s behalf, yet family-centred services are uncommon (Curtis & Singh, 1996). Families
seeking care for youth in crisis are often forced to try a wide range of resources before finding an
appropriate match, and frequently encounter multiple long wait lists along the way that can lead
to a closed window of opportunity before families can find the help they need (MHCC, 2012,
2
2015a; MOHLTC, 2009, 2011). A concerning byproduct of this process is reduced capacity in
the service system over the long-term (Children’s Mental Health Ontario [CMHO], 2016a;
MHCC, 2017).
Delays in access to care can have serious, long-term health, social, and economical
consequences. Nearly 70% of MHA concerns have their onset in adolescence, and without
timely and appropriate care, these concerns become lifelong struggles that impact individual and
family health, quality of life, and productivity, beyond placing a significant burden on the health
care system (MHCC, 2017; Statistics Canada, 2006). Timing is critical when seeking care and
intervening for this vulnerable population as the developmental period from adolescence to
young adulthood is a cornerstone in which individuals establish long-term patterns of behaviour
and make life decisions that can have significant impacts throughout the lifespan (Kessler, et al.,
2005, 2012). A recent report by the Mental Health Commission of Canada indicates that the
economic burden of mental health problems exceeds $50 billion per year; and lifetime economic
costs of childhood mental health problems are estimated at approximately $200 billion in Canada
(MHCC, 2017; Smith & Smith, 2010).
The provincial health care system is difficult to navigate for patients, families, caregivers,
providers and policymakers alike (MOHLTC, 2011). MHA services include both public and
private services; community agencies and hospital services; services with different eligibility
criteria, age restrictions and residency requirements; and services provided by many different
designated professions that may or may not be regulated and/or funded by up to four different
provincial ministries. As a consequence, service providers and professionals working in one
service sector often lack knowledge of other sectors offering services and resources that are still
relevant to a family’s multidimensional needs, making it difficult for a family to obtain the
comprehensive basket of services they require (MHCC, 2012).
For families of youth with suspected MHA concerns, equally challenging is the question of
knowing what to do and where to go for help in the face of a suspected problem that may be
apparent and yet difficult to articulate. Research suggests that families are continuing to look to
their family doctors or pediatricians for a first source of help (CIHI, 2017; MOHLTC, 2011).
However, this is typically only the first stop of many; in most cases, families make contact with
multiple agencies or providers before they find what they need, and they learn about treatments
3
and professionals informally as they go along. Moreover, while some families feel fully engaged
in their youth’s treatment, others may feel excluded. This may reflect the particular treatment
philosophy or bias of the provider, the youth’s choice, involvement of other service sectors, or
limiting interpretations of current privacy and consent legislation. Because there is no age of
consent in Canadian health care law, youth may be deemed capable of refusing treatments in
spite of certain need, and parents may not be informed or recruited to be involved in the youth’s
treatment pathway (Ontario Hospital Association [OHA], 2016). Generally, health care providers
will work with the youth and their family to reach a consensus on an agreeable course of action
for all parties involved. However, in some cases, consensus is not reached, and this is a situation
that occurs disproportionately in the management of MHA concerns.
Even for physicians, navigating such a complex health care system is challenging. American
families have reported that, although they were initially able to connect with a mental health
provider, often there was no clear pathway from that point onwards (Lazear, Worthington, &
Detres, 2004). Although the Canadian system is largely incomparable, here too family physicians
are most often the first and preferred point of contact with the health care system, and are also
expected to provide access to primary comprehensive mental health care and serve as
gatekeepers to specialty services (MOHLTC, 2011; Pautler, 2005). In fact, Ontario Family
Health Teams (FHT) were designed to serve as “patient navigators,” and so “care navigators”
were an explicitly recommended staffing option in the FHT collaborative practice guidelines
(although no studies evaluating how many FHTs have chosen to employ this position to date and
whether or not it is effective are known to exist) (Conference Board of Canada, 2014; MOHLTC,
2005).
The burden of suffering for families can be immense; parents often speak of feeling adrift and by
themselves. A 2006 report from the Canadian Mental Health Association voiced families’ desire
for better information and education, peer support to share experiences and coping skills, and
opportunities for caregiver relief, such as respite (CMHA, 2006). Today, for most families,
resources still remain extraordinarily difficult to navigate, despite repeated calls for navigation
services since then on both the provincial and national stage (MOHLTC, 2009, 2011).
4
1.1 Navigation: A potential solution
Navigation and related terms such as patient navigation, care navigation, and in the current
scenario, family navigation, is a concept that has gained traction over the last two decades and is
now increasingly emerging in a range of health care settings (Freeman & Rodriguez, 2011).
Navigation is often categorized as a care coordination intervention in that navigators are thought
to facilitate standardized care coordination activities that support physicians’ essential care tasks
typical of the general patient population (McDonald, et al., 2007).
Although navigation tends to get grouped with care coordination (and particularly case
management, which is a type of care coordination intervention), navigation-specific literature,
albeit limited, does suggest that navigation can and should be distinguished from care
coordination and case management. Whereas care coordination is defined by a standardized set
of activities, proponents of navigation emphasize that navigation is distinguished from care
coordination by its unique patient-focused approach. The central focus is helping patients to
overcome perceived barriers, similar to a case management model of care (Longest & Young,
2000; McDonald, et al., 2007). By its champions, professional navigator roles are envisioned as
extending beyond even the role of case managers, who are considerably more task-oriented and
typically manage appointment coordination and adherence. Instead, navigators favour a
comprehensive social model of patient management that values humanization of the care
trajectory and empowerment of the patient and family to overcome their perceived barriers
(Fillion, et al., 2006). However, because of this, navigation role descriptions are often highly
context-specific, which contributes to the lack of consensus and consistency in the literature
(Pedersen & Hack, 2010).
Navigation services have their roots in cancer care (breast and cervical cancer, in particular),
where programs were predominantly focused on overcoming barriers among disadvantaged
patients in order to ensure equitable access to adequate care for all, regardless of gender, ethnic
or socioeconomic status (Robinson-White, Conroy, Slavish, & Rosenzweig, 2010). Dr. Harold
Freeman is widely known to have first coined the term “patient navigation” when thinking about
“a metaphor for what patients have to do to negotiate the medical system; that of being on a
small boat in the south seas when you can see an island in the distance you want to get to but
there are rocks in between, and if you hit them, you’ll sink and possibly die. But let’s say there’s
5
a navigator on board who can chart the course and get you there safely.” He goes on to
emphasize the value navigators hold for patients and systems alike because “it’s very easy to get
lost in the impersonal health care system, but navigators are like problem solvers. They
coordinate all the disjointed elements and move patients through fast and more efficiently”
(Freeman, 2011).
Within Canada, in 2004, a formal evaluation of pilot projects in breast cancer navigation by
Cancer Care Nova Scotia found consistent benefits of navigation for patients and families that
included providing emotional support, preparing patients for their cancer journey, referrals to
appropriate health professionals, increasing patient knowledge about cancer, helping to
coordinate appointments, referral to community supports, assisting with the logistics of getting to
cancer centres and finding sources of funding for medications and supplies (Corporate Research
Associates Inc., 2004).
1.1.1 Navigation in mental health and addictions
Care coordination and related interventions have had a place in the field of MHA concerns for
decades, but since they operate within a fragmented system where silos and barriers span the
wide range of sectors and providers involved, any one provider faces inherent impediments to
meeting the extensive and diverse service needs of the population. Incompetent navigation of
these barriers is associated with poor patient outcomes and inappropriate health care utilization,
but a catch-22 exists when the results of disparate coordination efforts further perpetuate
fragmentation (McDonald, et al., 2007). Here, it is also important to reiterate that coordination is
not navigation; coordination interventions are driven by a pre-defined set of services, whereas
navigation helps patients overcome their perceived barriers. This approach is likely to be both
more efficient and effective when it comes to finding resources for MHA problems, which are
known to manifest via highly unique and individualized pathways and require similarly
individualized responses.
Undoubtedly, there is an intuitive place for navigation services in addressing MHA concerns,
and even more so for youth and families. The gap has not gone unrecognized. Ontario’s 2011
mental health and addiction strategy, Open Minds, Healthy Minds, states that “mental health and
addictions services are fragmented, spread across several ministries and offered in a variety of
care settings;” and that “families struggle to navigate services and get the right support for their
6
children and youth.” With over $250 million in new funding over three years, one of the key
youth priorities moving forward was the provision of “fast access to high quality service,”
including specific initiatives meant to “improve public access and help children, youth and
families find the right kind of services” and “provide supports in select communities for families
to navigate the system” (MOHLTC, 2011).
Since then, a handful of examples of MHA navigation initiatives emerged across Canada, such as
British Columbia’s Sooke Navigator Project and F.O.R.C.E. Society for Kids’ Mental Health,
Halton (Ontario) Healthcare Services’ Navigator Program, and COAST Navigation in Niagara,
Ontario (McPhee, Syed, Nunes, & Mobilizing Minds Research Group, 2012; Roberts & Schmidt,
2012). More recently, St. Joseph’s Centre in Toronto, Ontario launched a Family Navigation
Program. While this list is not exhaustive, the following examples are self-defined navigation
programs that share an emphasis on overcoming patient-perceived barriers to accessing
appropriate MHA care. However, it is important to note the extensive variation and lack of
standard practices in the way navigation programs are implemented, organized and delivered.
Because of this, while a fixed list of components cannot be neatly attributed to each category,
one can more generally describe these models to aid in understanding the service continuum.
Numerous jurisdictional reviews and conversations with experts in the field have led to the
conclusion that currently in Canada, navigation programs exist along a continuum that varies in
terms of which services are offered, to what extent, and to whom. In general, as you move along
the continuum toward programs with more to offer to more people, one expects increased
accessibility, increased client engagement, and increased continuity of care (Figure 1).
Online or tele-help
• ConnexOntario
• Kids Help Phone
• Mental Health Helpline
Interorganizational referral networks
• C.O.A.S.T. Niagara
• Halton Mental Wellnes Navigator Program
• Contact Hamilton
Navigation representative
• FHT Navigators
• St. Joseph's Family Navigation Program
• Hamilton's Child and Youth Aboriginal Navigator
• CMHA BC Community Navigator
• SickKids Mental Health Patient Navigator
Navigation team
• Sooke Navigator Project
ContinuityEngagementAccessibility
7
Figure 1. Continuum of service delivery models with examples of existing programs
The most basic delivery of navigation services exists by one-dimensional online or tele-help
resources. ConnexOntario is an excellent example of this model. It can be accessed online or
over the phone, which is convenient, but may or may not allow you access to a live agent, which
is not ideal due to the highly complicated nature of the MHA service systems. Accessibility of
information is thus limited as it only provides help-seekers with access to some information
about some of the services available in their area, along with a contact number. There is no
follow up or long-term engagement, and particularly for youth with MHA concerns, the
limitations of this model quickly becomes clear. For example, the platform can only search for
resources for one presenting problem; if a youth has an eating disorder and depression, or an
anxiety disorder and substance use problems, there are no results because the MHA service
systems are separate and the platform does not account for this. Yet, the co-occurrence of MHA
problems is widely known. Moreover, since ConnexOntario is a publicly-funded initiative, it
only returns search results for publicly-funded resources. This could be problematic due to the
number of private service providers, particularly for substance use treatment programs.
The second type of navigation exists within a predefined interorganizational network; structured
referral processes allow programs to facilitate inter-organizational client transfers when certain
needs cannot be accommodated by a partner program. Halton Healthcare’s Mental Wellness
Navigator Program is an example of this in that navigation services are provided only to youth
up to the age of 18 who live in the Halton catchment area and were referred because they are
already clients of Halton Healthcare. Here, they are engaged in care planning and some degree of
continuity is experienced, but once the youth turns 19, they become the responsibility of another
Ministry and are no longer eligible for the program. Contact Hamilton is another example of a
program which provides youth and families with central intake and access to a number of partner
programs in the Hamilton area. The immediate improvement over online or tele-help programs is
access to a live agent and engagement in personalized assessment. However, in both programs,
there may be some initial follow-up while connecting families with a service, but once
connected, the relationship is terminated.
The third modality is a designated navigation official or representative. This is an individual who
functions as a designated navigator within a team setting and generally aims to work with
patients throughout their help-seeking journey. For example, as mentioned earlier, Ontario FHTs
8
were encouraged to include a patient navigator in their collaborative team practices. Similarly,
many cancer care programs, including the Odette Cancer Centre at Sunnybrook Health Sciences
Centre, have a dedicated navigator on staff. These individuals may be trained professionals or
peers with lived experience; either way, they tend to have a nuanced understanding of the service
system and as such, are often able to offer clients personalized care planning. British Columbia’s
branch of the Canadian Mental Health Association employs both a Community Navigator,
dedicated to finding and connecting clients with housing and social supports, and a Peer
Navigator, who focuses on providing and connecting clients to peer supports. In 2016, St.
Joseph’s Health Centre launched their Family Navigation Program, which pairs family members
with social workers who guide them through their time at the hospital and provide information,
education, counselling and support groups, and help to facilitate connections to community-
based resources in order to ensure continued support post-hospital.
At the far end of the spectrum are navigation teams, where clients should theoretically
experience the greatest degree of accessibility, engagement and continuity of care. These are
collaborative teams of designated navigation personnel dedicated to helping clients understand,
identify and access a wide range of appropriate resources to meet the range of their perceived
needs. They have the mandate and capacity to stay involved with their clients long-term,
providing follow-up and responsiveness to change as needed. The Sooke Navigator Project
(2007) is perhaps Canada’s most well-known example of the potential of navigation teams.
Developed in British Columbia and embedded within a local family service organization, the
primary intent was to improve inter-provider efficiency and capacity within the primary care
service system by providing low-barrier access, assessments, collaborative service planning,
respectful communication, referrals, tracking and follow-up for anyone who asked, including but
not targeted to youth and families. The Sooke Navigator Project is one of the few identifiable
examples of independent navigation team initiatives, and evaluative research suggests the model
effectively improved understanding of, access to and connectivity of community services
(Anderson & Larke, 2009). Indicative of its face validity, the model is being implemented at
three additional sites in British Columbia.
Overall, the range of service delivery models appears to differ significantly in several important
ways, including but not limited to the engagement method, provider type, location and
organization of services, and intensity and length of contact. These are defining organizational
9
characteristics that strongly depend upon the needs of a community and the program inputs,
which will trigger different mechanisms and outcomes for families seeking care for their youth
with MHA concerns.
1.2 The Family Navigation Project
The Family Navigation Project (FNP) is a highly unique and innovative family navigation team
initiative that launched in Toronto, Ontario in 2014 (Roberts & Schmidt, 2012). The Primary
Investigator (PI) for this study was employed as the FNP’s Project Manager from 2011 to 2013,
and as such, participated in much of the FNP’s design, development and implementation. No
existing programs are known to be adequately comparable to the FNP. As discussed in the
previous section, extensive jurisdictional and literature reviews confirm that what potential
comparators do exist vary very significantly in context, target population, goals, and actual
services offered. Some of the key features that distinguish the FNP from its comparators are its
target population (families of youth and transitional-aged youth with mental health and/or
addiction needs); its defining family-centred approach; its dedicated capacity (with seven full-
time Navigators), having navigated for over 1850 families to date; its service organization (e.g.
phone and email based, flexible hours, not for profit status).
1.2.1 Background and project development
In 2009, three families of youth treated at Sunnybrook Health Sciences Centre’s (SHSC)
inpatient youth psychiatry unit independently approached the department to express frustration
with how their children’s care plans were managed; each family had been discharged from
inpatient care with unclear follow-up or connection to community resources or supports. As a
site-based quality improvement project, three focus groups were conducted by an independent
consultant with parents of youth cared for on the unit to better understand the families’
perspectives on what was needed to improve their help-seeking and care experiences.
Strong themes emerged from the focus group data, which were then used to inform the FNP’s
business case (see Appendix A for a summary of focus group themes, which were included in the
business case; and for the full business case, refer to Roberts & Schmidt, 2012). Families
reported that having comprehensive, objective information about treatment options helps,
especially understanding the different treatment approaches, availability and a sense of the way
10
in which a particular provider works and the likelihood of their being a good match for the child
and family. Parents also reported that it is helpful to hear other families’ advice and experience.
Most commonly however, parents spoke of the immeasurable value of having someone who is
knowledgeable, compassionate and committed as a guide who is prepared to “get in the boat”
with the family and stay with them throughout their journey. Indeed, this is reminiscent of Dr.
Freeman’s original metaphor.
As a whole, the data suggested that in the field of youth MHA services, pockets of knowledge
and awareness of available services are isolated from each other, a result of which is that families
experience many gaps in care. Interviews were then conducted with a wide range of families,
public and private service providers for both mental health and addictions (i.e. general
practitioners who are often first points of contact; psychiatrists; hospital and- community-based
programs and agencies; residential programs in Canada and the United States) in order to collect
as many perspectives as possible as to what the problem is, and what would be the most efficient
and effective solutions.
1.2.2 Project description
With an indication from families of what would be most helpful to them, a team of clinicians,
consultants, parents and research personnel in Toronto, Ontario developed an original model for
a lived-experience-driven, privately funded, non-profit, relationship-based navigation service for
families of youth aged 13 to 26 with MHA concerns. The program is affiliated with Sunnybrook
Health Sciences Centre, and officially launched in June 2014.
The FNP purports to address identified service gaps by partnering families with navigators who
are skilled, knowledgeable and experienced mental health professionals able to provide families
with needs-specific and family-centred information in order to facilitate identification of and
access to the appropriate services and resources for the identified youth client, as well as the
entire family, as needed. On an individual (i.e. family)-level, navigators offer low-barrier access
to highly individualized, family-centred, comprehensive perceived needs assessments;
informational support; collaborative resource and service planning; referrals; care coordination;
and follow up and support throughout to ensure that the process is working. If not, navigators re-
evaluate perceived needs, barriers and the service plan in order to provide the family with a more
currently appropriate solution. The program is driven by the underlying assumption that MHA in
11
youth are experienced by the entire family, and consequently, that a family-centred coordinated
and continuous care plan in which families are engaged, empowered, and supported, will lead to
improved satisfaction with the service and with help-seeking in general, and improved family
quality of life.
At the systems-level, the FNP liaises with other resource providers and system stakeholders to
help match resources to need and to build relationships over time, which may help to facilitate
system coordination, and eventually capacity. A full-time Director of Strategy and Partnerships
cultivates relationships throughout the health care community to improve integration and
awareness. Educational events and presentation to a wide range of stakeholders help raise
awareness about the resources available, and empower families with the tools to recognize
problems and potential solutions. The FNP is also continuously developing an in-house live
database of resources, programs and services for youth MHA treatment, which includes nuanced
service details and data on treatment outcomes so families can be directed toward the most
innovative, successful approaches and programs for a particular situation; and so evidence of
unmet needs can be identified and used to promote informed, efficient distribution of system
resources.
Logistically, the “program team” consists of an intake worker, seven full-time navigators, several
administrators and a growing research staff, all of whom are led by a Medical Director. Access to
the program is primarily through an intake email and phone line. Similarly, the program is
primarily phone- and email-based, although onsite meeting facilities are available should this be
a family’s preference. For families who phone or email central intake, an assessment process is
quickly initiated. Initial demographic information, reported MHA concerns, and reasons for
contacting the program (among other variables) are collected, and clients are assessed by an
intake worker so they can be paired with a Navigator particularly suited to that family’s needs.
The assigned Navigator then completes a more comprehensive, family-based needs-assessment
and commits to navigating for that family from that point onward, as long as needed. Although
client status will automatically update to discharged if there has not been contact within a certain
period of time, families are encouraged to return to the program, as needed, at any point in time.
12
Following extensive discussions with the program team, family navigation was determined to be
defined by the key components outlined and defined in Table 1. The specific process by which
these components were determined is discussed in Chapter 2.
Table 1. Components of family navigation
Component Definition
Accessible expertise
Services are organized to respond to families' needs; phone and
email-based, extended and flexible hours, meeting space; medical
consultation and supervision available; Navigator experience and
expertise
Family education
Navigators provide families with information and resources aimed
at improving their understanding of the problem itself, and the
roles of the family, Navigators, and health care system in the
recovery process
Resource assessment and
information sharing
Navigators provide appropriate and sufficient information on
which resources are available
Resource matching Navigators identify the resources that best meet the family's
identified needs
Referrals Navigators directly connect families and facilitate relationships
with identified resources or services
Connections to peer support Navigators provide families with connections to support networks
of peers with lived experience
Family-based perceived needs
assessment
Navigators discuss and determine the youth's and family's
perceived needs and perceived barriers to care
Family-based collaborative care
planning
Navigators engage and work closely with families to develop care
plans with clear steps and supports that meet their perceived needs
and overcome perceived carriers
Consistent family engagement Navigators consistently engage with families in all steps
throughout the process
Information dissemination and
exchange
Navigators ensure continuous and current information exchange
across care settings so families perceive transitions as seamless
Ongoing follow up and response to
changing needs
Navigators communicate with families regularly to continuously
monitor progress and adjust course as necessary throughout the
process
Relationship and trust-building Navigators strive to develop a relationship and build rapport with
families by providing care and communicating in a professional,
13
respectful and responsive manner
Facilitate service-system
relationships*
Navigators visit and build relationships with service providers in
the system
Promote evaluation and
accountability*
Navigators evaluate their own service and other resources in order
to promote efficiency and effectiveness in the system through
evidence-based care
Promote education and awareness
for youth MHA*
The Family Navigation Project conducts research and shares
findings at educational events targeted at a wide range of
stakeholders and settings
Advocate for youth MHA and
navigation services*
The Family Navigation Project advocates for its cause on a variety
of platforms using primary data
Competency training* Navigators must be highly skilled and experienced mental health
and/or addiction professionals.
*Indicates service-system interaction; this level is beyond the scope of the current study, which is limited to family-
service interaction.
14
Chapter 2 Theoretical and Conceptual Frameworks
Theoretical evaluation framework: Realist evaluation
The FNP’s design and development aligned with a traditional realist synthesis process in that it
began with highly contextualized data from a wide range of sources to identify and understand
the gaps in care before soliciting feedback regarding the most efficient and effective way to
mitigate these gaps (Pawson & Tilley, 1997). Research was collected over two years and
incorporated into a constantly evolving model of care. Similar to the FNP’s development
process, an evaluation framework for the FNP that would be highly adaptive, realistic, context-
specific, and stakeholder-driven was determined by the PI (in consultation with the program
team and supervisory committee) to be the most appropriate approach to this particular
program’s evaluation. For this reason, the current study proposed a realist evaluation perspective.
Realist evaluation (RE) is a theory-driven measurement and evaluation framework that was
developed in the late 1990s, most notably by Ray Pawson and Nick Tilley (1997). It is praised
for its pragmatism, and most importantly, because of its context-specific approach, RE is
particularly well suited to complex interventions, which are defined by their context (Rogers,
2008). The FNP meets all criteria for a “complex intervention,” including the presence of
multiple, interacting components; non-linear trajectories; multiple feedback mechanisms; and
multiple alternative and simultaneous causal strands, as what works for one family may not work
or may unfold differently for another (Rogers, 2008). Thus, RE was a logical choice that aligned
with the FNP’s own program philosophy, development, daily operation and program goals.
RE is grounded in scientific realism, a school of thought necessitating the inclusion of both
observable and latent aspects of the world in our theories and models because both have real
world effects (Bhaskar, 1979; Leplin, 1981; Miller, 1987; Putnam, 1982). RE and scientific
realism are distinguished from critical realism, which assumes that the inability in social sciences
to create the ‘closed system’ study available to the natural sciences necessitates an approach
centred on abstract a priori reasoning and acknowledging that a moral lens is applied to all
evaluation (Boyd, 1989).
15
In the context of program evaluation, RE presumes that social programs, such as the FNP, are
intended to address a social problem, such as the struggle for families to quickly find appropriate
MHA care for their youth where and when they need it most (Pawson & Tilley, 1997). RE
surmises that programs generate outcomes by providing resources (e.g. information, skills,
support) and/or influencing their participants’ reasoning (such as their values, beliefs or
attitudes), and thus decision-making. Because of this, the way in which these social programs
‘work’ in any given interaction strongly depends on the participant’s response to the resources
provided by the program (Pawson & Tilley, 1997). This is why RE encourages open-endedness
in knowledge gathering efforts - due to the theoretically infinite number of contingencies that
could continue to shape understanding of the phenomenon or program of study.
The contexts in which a program operates, in terms of multi-level social, economic and political
structures (such as participant subgroups, stakeholders, program staff characteristics, or larger
social, cultural, economic or political conditions) are also extremely important to understanding
how a program achieves change because as little as one factor can influence whether or not a
particular mechanism is triggered for a participant, either by influencing reasoning directly or via
how resources are provided (Pawson & Tilley, 1997).
Again, RE is theory-driven; its hallmark is the distinctive “generation” of causality. As such, a
program theory or a series of conjectures about how an intervention is proposed to work is
needed to guide the evaluation design (Pawson & Tilley, 1997). Realist program evaluations are
intended to answer the question, which goes on to form the program theory, “what works, for
whom, in what respects, to what extent, in what contexts, and how?” The question is addressed
by gathering information on observable and latent mechanisms that explain how any given
outcome was caused - what resources were provided, and what was the participant’s reasoning in
response - and what the influence of the context was:
C + m(resources) + m(reasoning) = O.
In RE, analyses result in a series of conjectures relating contexts (C) and mechanisms (M) to
outcomes (O), known as “C-M-O configurations,” each of which is associated with certain risks
and assumptions. C-M-O configurations are arrived at by hypothesizing and testing individual
dyads of data (i.e. C-C, C-M, C-O, et c.), which are then amalgamated into C-M-O triads. C-M-
O configurations, most notably individual dyads, serve several purposes. The first is to add to the
16
description of the dataset in terms of the relationships between variables; the second is to
confirm that hypothesized covariates are related to the outcomes of interest; and the third is to
satisfy inferential modeling assumptions. The most robust C-M-O configurations are then often
tested in inferential models and compared to the initial program theory for iterative refinement
and retesting (Pawson & Tilley, 1997).
Because of the importance of context, realist evaluation is method-neutral and encourages a wide
range of approaches to data collection (Dalkin, Greenhalgh, Jones, Cunningham, & Lhussier,
2015). However, most often, quantitative methods are used to evaluate context and outcome,
while qualitative methods are used to further describe these factors and gather in-depth
information on mechanisms, which are typically social or psychological processes that can be
difficult to adequately capture using quantitative methods.
2.1 Theory-building process
Another tenet of RE is the importance of involving the program team in the evaluation design
process (Pawson & Tilley, 1997). In the context of the current study, this included collaborating
to develop a conceptual framework, program theory and corresponding logic model underlying
family navigation, all of which then informed the measurement framework. This approach
aligned well with the FNP’s stakeholder-driven philosophy as families, providers and
policymakers were intentionally involved in project design and development from the early
conceptualization stages; and evaluation is an explicit part of the program model.
The conceptual framework, program theory, logic model and outcomes of interest in the current
study were developed in collaboration with the program team over a series of meetings
facilitated by the PI, beginning in August 2015.
The process began with a discussion of the concept of family navigation and the construction of
an appropriate definition for the family navigation services by using their own program
information as a starting point. This included promotional materials such as educational
pamphlets, website information, and the FNP’s business case (Roberts & Schmidt, 2012). This
information was then compared with existing definitions for navigation and examples of similar
constructs, such as care coordination and case management, from available peer-reviewed and
grey literature. After several iterations, the group came to a consensus on the following baseline
17
definition of family navigation: “Family Navigation is the provision of continuous needs-specific
and family-centred information and support in order to facilitate timely identification of and
access to appropriate resources for the youth and their family.”
It is worth noting that this definition is consistent with the findings of the AHRQ review on the
topic discussed earlier, which determined that while no consensus on the definition of navigation
exists, a few authors do distinguish navigation by its focus on helping to overcome patient-
perceived barriers to care (McDonald, et al., 2007). The definition used here reiterated the idea
that navigation should be client-based and focused on overcoming perceived barriers, rather than
a predefined set of services, which is how care coordination is defined and measured.
The second step was to collaboratively develop a logic model that represents how the FNP is
thought to work (see Figure 1, Appendix B). The process began with a discussion with the
program team, again facilitated by the PI, of program goals and the needs of the target
population. Then, the inputs and range of activities that were required to meet those needs were
determined, in addition to the results and impacts of those activities (tangible outputs). A series
of tools and techniques were employed by the PI throughout the process. These included
articulation of mental models, which involves questioning key informants about how they
understand an intervention to operate and contribute to addressing the needs of their target
population; group model building, in which a models of system dynamics are built using causal
loops based on perceptions of the problem; Strength Weakness Opportunity Threat (SWOT)
analyses, and hierarchical results chains, which identify the chain of effects from inputs through
activities, outputs and outcomes [Bryson, 2011; Richardson & Andersen, 1995; Vennix, 1996).
This extensive list of activities and outputs was then aligned and condensed into a list of key
defining components detailed in Table 1 (Section 1.2.2).
Next, the logic model (Figure 1, Appendix B) was used to clarify the program theory. The PI
facilitated a discussion with the program team focused on the identification of themes within and
across the range of inputs, activities and outputs associated with the outcomes in the logic model.
The program team was also encouraged by the PI to recall past example narratives that helped
clarify how one component linked to another; and prompted to include how any contextual
factors, and any risks and/or assumptions might enable or hinder any step along the way. The
results of this discussion were recorded by the PI and then integrated with literature that offered
18
evidence for the stated problem (e.g. lack of resources for youth, particularly transition-aged;
lack of coordination; calls for “navigation”); and evidence for why the possible solutions
generated (e.g. the provision of continuous needs-specific and patient-centred information and
support in order to access appropriate resources) would address the problem.
An overall theory about how the program is believed to work was constructed and presented by
the PI to the program team for further refinement. Based on the theory-building process
described above, the program theory hypothesizes that three key mechanisms – accessibility,
continuity of care, and family involvement - directly enable the three outcomes of interest -
family empowerment, improved family quality of life and family service satisfaction.
2.1.1 Program theory
The preliminary program theory was as follows:
➢ IF we supply the inputs that allow us to employ Navigators who are continuously
accessible, interpersonally skilled and adept at assessing and meeting needs from a
family-oriented perspective;
➢ THEN families will have ongoing access to family-based, needs-specific and supportive
care that best responds and adapts to the family’s changing needs;
➢ SO THAT families will have a better understanding of their needs, the range of options
available to them, and how to quickly and efficiently access sufficient and appropriate
resources to meet their needs;
➢ SO THAT families feel empowered to better manage their situation; and
➢ SO THAT families are able to improve their quality of life; and
➢ SO THAT families will be satisfied with the services they receive.
The last step was for the PI to synthesize findings from the literature and conclusions from group
discussions in order to produce a succinct conceptual framework for family navigation (Table 1,
Appendix B). Together, the logic model for the program and conceptual framework for family
navigation proposed that in the presence of certain contextual factors, three key “mechanisms”
characterize family navigation – family involvement, accessibility, and continuity of care –
which enable the project goals and outcomes of interest of empowering families, improving their
family quality of life, and ensuring service satisfaction. Included concepts are described below.
19
2.2 Conceptual framework
2.2.1 Context
There are several categories of contextual characteristics with bases in the literature and program
theory that were a) included for descriptive purposes; and b) theorized to influence outcomes and
client-reported experience with the FNP to varying extents. It should be noted that because the
FNP employs a “family model of care” in which the family is the client (based on the premise
that it is families (i.e. parents or other primary caregivers) who facilitate the interaction between
a youth and the service systems), in this study, individual-level characteristics of the youth were
conceptualized as attributes of the family (Chovil, 2009).
Categories of contextual characteristics first included demographics, namely age and gender of
the youth, since MHA concerns are known to manifest differently across both variables. Because
navigation was originally designed to help disadvantaged populations overcome perceived
barriers (and because cultural and financial barriers to MHA care are a frequent occurrence) to
evaluate equity, other demographics of interest included geographical location, ethnicity, and
socioeconomic status. Type of MHA concern(s) was the second main category and included
whether care was being sought for mental health concerns, addiction concerns, or both, since
mental health and addiction services are often located in separate service silos and as such, care
for concurrent disorders can be especially challenging to navigate. It also included the specific
MHA concerns, since there are some concerns for which the FNP may be particularly adept at
navigating – relatively common concerns like depression and anxiety - or which are acute and
particularly responsive to attention – suicidality, for example. Other concerns are theorized to be
inherently more difficult to address and/or have fewer designated resources available; this would
be typical of chronic complex conditions like OCD, personality and eating disorders, for
example. Relatedly, youth’s level of acuity, in terms of whether or not families perceived
improvement since enrolling with the FNP, was expected to influence mechanisms and
outcomes, particularly service satisfaction. It is important to reiterate that reported concerns are
parent-perceived; and parent-perceived MHA concerns cannot be equated with formally
diagnosed mental disorders. However, because this study is centred on family experiences and
outcomes, family-reported MHA concerns were determined to be the most relevant variable.
20
Also theorized to influence mechanisms and outcomes were current program use characteristics,
including the length of time and intensity with which they were engaged with the FNP and
whether they were actively receiving services or had been discharged at the time of evaluation.
Extent and nature of past service utilization was also a category of interest that was intended to
further describe the sample and included variables such as whether they had a formal diagnosis,
previous emergency department (ED) visits, or inpatient stays. Other past service use variables
were expected to influence mechanisms and/or outcomes, such involvement with the justice
system or Children’s Aid Society were, which can add additional layers of complexity to
navigation cases. Reason for contact was the last contextual category of interest and was
included to contribute toward understanding who is using navigation and for what purposes; and
because it was expected that certain reasons would be more difficult to accommodate than
others. For example, families contacting the FNP in search of general recommendations,
information and/or family support may be more likely to have their needs met than families
contacting the FNP in need of a relatively inaccessible resource like residential treatment.
Altogether, a range of contextual factors were included in the conceptual framework, some of
which were included primarily to add to the description of the dataset, whereas others were
specifically hypothesized to influence mechanisms and/or outcomes for families receiving
services from the FNP.
2.2.2 Mechanisms
2.2.2.1 Accessibility
Accessibility was the first proposed mechanism. In this case, it referred to the accessibility of
expertise (i.e. the Navigators), information, and supports. Data from the program team and
families with lived experience were both firm that a program responsive to families’ needs would
be highly accessible, but there were many different examples of what this entailed. For some,
this meant having extended hours to cover weekends and evenings or having the opportunity to
call, email, Skype or attend in person; for others, it meant having someone make the direct
referral for them; and for others, it meant having a Foundation so families who could not afford a
private treatment option, despite high need, would have that opportunity available to them.
21
To address the range of perspectives, the current framework employed Roy Penchansky and J.
William Thomas’ (1981) classic model of accessibility, which provides a comprehensive
taxonomy of five distinct components of “access” - availability, accessibility, accommodation,
affordability, and acceptability. Each of these components were determined to be relevant to the
current study. Availability refers to the adequacy of supply of FNP services and resources in
relation to families’ needs. Accessibility refers to the relationship between the location of supply
and the location of families, taking account of factors such as transportation resources and travel
time, distance, physical accessibility and cost. Accommodation refers to the relationships
between how FNP services and resources are organized, such as the appointment and client
record systems, hours of operation, meeting facilities, telephone and web-based services; a
family’s relative ability to accommodate to these factors; and their perception of appropriateness.
Affordability refers to the relationship of cost of services (both the FNP itself and the services to
which it refers) to the clients’ ability to pay; this includes “client perception of worth relative to
total cost.” Finally, acceptability refers to the relationship between a family’s attitudes about
what the personal and practice characteristics should be and what the actual characteristics
actually are.
2.2.2.2 Continuity of care
The second theorized mechanism is continuity of care. The framework adapted Canadian
researcher Jeannie Haggerty and colleagues’ conceptualization, which suggests three types of
continuity should be offered to families: 1) informational continuity, the use and accumulation of
personalized information over time on which providers can draw to ensure current care is
appropriate; 2) management continuity, a consistent, coherent and responsive management of a
care plan that can provide a sense of predictability and security, especially to families who are
facing MHA crises; and 3) relational continuity, arguably the most essential need, which refers to
the ongoing therapeutic relationship between patients/families and provider(s) (Haggerty, et al.,
2003). Recall that in focus groups, as per Dr. Freeman’s analogy, families most strongly
emphasized the need for someone to “get in the boat.”
2.2.2.3 Family involvement
“Family involvement” was the final proposed mechanism. This was agreed to be one of the
defining features of the program and its measurement was intended to capture the client-centred
22
nature of navigation as it applies to families who facilitate the interaction between youth and
service system, and as such, are the central dimension of the care planning system (Tannenbaum,
2001). Family involvement is increasingly referred to in the youth mental health literature,
although no single, empirical definition is known to exist. Inter-related terms include family
empowerment, family-centredness, family-focused, and family-driven (Chovil, 2009; Curtis &
Singh, 1996). In the current framework, a decision was made by the PI in consultation with the
program team to employ Wood’s definition of family involvement as “respecting families as
experts on their children, enlisting them as partners in the care of their children, supporting them
in their caregiver role, and involving them as partners in decision-making at all levels of the
system” (Wood, 2004, p. 6). This conceptualization was selected because it best emphasized the
need to understand family members as part of the context, mechanism and outcome; and as
experts on their children, making them invaluable partners in care planning (McCammon,
Spencer, & Friesen, 2001).
2.2.3 Outcomes of interest
Three main outcomes of interest were derived in collaboration with the program team following
the facilitated discussion regarding program goals. As a unique grassroots project in its early
stages of operation, the program team’s primary interest was in the subjective experience of their
clients. They acknowledged the difficult-to-treat nature of MHA concerns, and the many
unavoidable barriers to help-seeking and positive outcomes. Furthermore, they understood
themselves to broker treatment services rather than provide them directly. As such, they were
most interested in knowing that the program and the time, money and extensive efforts invested
to be responsive to families’ needs had made some positive difference(s) in the daily lives of the
families they set out to help; and whether families were satisfied with their services or not (and if
not, why, and how they could improve).
In the facilitated discussions, members of the program team mentioned wanting to offer families
“a sense of relief,” to “encourage confidence” and “hope for the future,” and “help families help
themselves.” They specifically wanted families to feel more in control of and better able to
manage their youth and family’s situation because of the accessible, needs-responsive
informational, management and relational support they offered. Accordingly, the three outcomes
chosen for the current study were as follows.
23
2.2.3.1 Family empowerment
Empowerment is not a particularly well-defined construct in the literature, and yet the term is
widely used across health care settings. Definitions, measures and underlying conceptual
frameworks vary significantly and particularly across conditions but tend to revolve around
patients building the beliefs, knowledge and skills to actively participate in their care and
improve their situation. In mental health settings, efforts to facilitate empowerment are generally
associated with positive perceptions and outcomes and have been encouraged for use with
families of youth with extreme behavioural disorders (Battaglino, 1987; Dunst, Trivette, Davis,
& Cornwell, 1998; Freund, 1993; Kopp, 1989; Singh, 1995; Staples, 1990).
Family empowerment has been previously defined as “a process by which the families access
knowledge, skills and resources that enable them to gain control of their own lives as well as
improve the quality of their lifestyles” (Singh, 1995, p. 13). It requires both changes in conscious
beliefs and attitudes, as well as “practical knowledge, solid information, real competencies,
concrete skills, material resources, genuine opportunities, and tangible results” (Staples, 1990, p.
30). Many of these definitions position empowerment as an interactive process – a mechanism
involving both resources and the family’s reasoning – but empowerment can also be
conceptualized as a state in which “a family perceives itself as being able to successfully
negotiate the […] system,” and “efficiently utilize it to meet their needs” (Curtis & Singh, 1996,
p. 504). However, this state is likely dynamic, changing over time in response to evolving
experiences, conditions and circumstances. This idea is reflected in another often-cited definition
of empowerment as “the ongoing capacity of individuals or groups to act on their own behalf to
achieve a greater measure of control over their lives and destinies (Staples, 1990, p. 30). For this
reason, empowerment can theoretically be conceptualized as both a process and a state, an
individual or collective characteristic, and one that can be facilitated in a wide range of settings
and circumstance (Koren, DeChillo, & Friesen, 1992).
In this study, because family empowerment was an explicitly stated program goal in and of itself,
family empowerment was conceptualized as an outcome of successful navigation. That is, family
empowerment was theorized to be a state produced, at least in part, by the process of navigation.
This approach aligned with emerging literature suggesting empowerment is a valuable patient-
24
reported outcome independent of health status or service utilization, and particularly relevant in
chronic conditions like MHA concerns (McAllister, Dunn, Payne, Davies, & Todd, 2012).
2.2.3.2 Family quality of life
When a youth has an MHA concern, the situation affects the entire family. Youth exist within an
ecological framework in which family is central; family systems theory defines families as goal-
directed, self-correcting, dynamic, interconnected systems that influence and are influenced by
their environment and inherent qualities (Klein & White, 1996). That is, families have different
strengths capabilities, resiliencies and skills that can significantly impact a youth’s health and
family’s functioning; and each family member is linked to the other such that what impacts one
impacts all. Disabilities are thus challenging for everyone involved in the interconnected system,
and family functioning often declines as a result. Thus, in family models of care, outcomes
reflecting enhanced family management and quality of life are increasingly desired (Chovil,
2009).
Family quality of life (FQOL) is an evolving construct in the literature that is increasingly being
used to offer a more nuanced and robust understanding of families’ daily lives when interacting
with the health care system (Gill & Renwick, 2007; Park, et al., 2003). Impact on FQOL is an
important outcome of quality of service delivery in health services research because it may
reflect the nature, content, accessibility and delivery of services and consequently inform quality
improvement (Turnbull, et al., 2000). However, due to its inherently complex nature, there have
been few and disjointed attempts to measure the concept comprehensively and holistically. The
rationale for the application of the FQOL framework in family support service evaluations is
located in the idea that “families that function well support societies, and families with effective
quality of life are seen as a social resource;” and the best way for disability service systems to
improve FQOL is therefore through family-centred intervention models which assess families’
perceived needs and support and build upon families’ strengths to help them function better as a
system (Isaacs, et al., 2007; Samuel, Rillotta, & Brown, 2012). Families are typically the primary
participant in their youth’s help-seeking and care planning, and are increasingly being
understood as experts on their children who are able to provide meaningful and relevant
information from a unique perspective that can significantly improve the appropriateness of care.
25
Moreover, families who participate in their youth’s care are more likely to feel in control and as
if their needs were met (Curtis & Singh, 1996; Koren, et al., 1997).
Optimal FQOL is conceptualized as having a) perceived needs met, and b) appropriate
opportunities to make active choices according to need (Renwick, Brown, & Raphael, 1998).
There are several domains of FQOL the FNP expected to impact, including family interaction,
parenting, wellbeing, and available support. Navigators provide information and cultivate
knowledge and awareness, skills and strategies, resiliency and capacity to cope, which was
theorized to result in a better understanding of their role and ability to manage daily activities
and interactions with both the youth and the service system. The supportive environment of the
FNP itself, long-term involvement by the navigators, and connections to additional support
networks can increase resilience, support coping skills, decrease perceived burden and help to
reduce the stigma, shame and embarrassment that often accompany help-seeking for MHA
concerns. This, alongside increased access to appropriate and relevant disability-related supports
that meet the perceived needs of both the youth and other family members, are hypothesized to
reduce caregiver burden and stress and improve both physical and emotional well-being.
For the FNP, the family is the client and the key resource for change in the youth’s health and
family’s functioning. As such, family-focused outcomes were preferred by the program team,
who again wanted to know whether families were “better off in some way,” in their day-to-day
life because of the services offered.
2.2.3.3 Service satisfaction
Satisfaction is one of the most widely used patient-reported health care service outcomes that has
value in and of itself; as a key indicator of quality of care; and by examining its predictors, in
informing quality improvement initiatives (Al-Abri & Al-Balushi, 2014; Barr, et al., 2006;
Mpinga & Chastonay, 2011). Measures can reflect both client expectations and the actuality of
services provided (Ware, Synder, Wright, & Davies, 1983). Although cancer-specific, what few
evaluations exist do suggest that patient navigation is associated with improved self-reported
service satisfaction over treatment as usual (Campbell, et al., 2010; Hook, Ware, Siler, &
Packard, 2012; Seek & Hogle, 2007).
26
Families’ satisfaction with the navigation services received was an essential indicator of success
for the FNP, a program specifically designed to respond to families’ stated needs. In 2014, the
FNP methodically developed an in-house satisfaction measure to evaluate both satisfaction with
the navigation services, and satisfaction with the services to which Navigators referred families
were of interest in the current study. It was suggested that this satisfaction measure be employed
in the current study both because it is already validated and because data resulting from the
current study could then be compared to data from the pilot study of that measure.
27
Chapter 3 Research Questions and Rationale
Research questions
With the conceptual framework and program theory fleshed out, three broad research questions
were determined to guide this study:
1. a) Who is the Family Navigation Project serving?
b) Is the Family Navigation Project reaching its target population?
c) Overall, are families satisfied with the services they received?
2. a) Do families perceive the Family Navigation Project to be providing accessible,
continuous, family-inclusive care?
b) How does context influence perceived experience of the program?
3. a) Do families who perceive the Family Navigation Project as accessible,
continuous and family-inclusive experience better outcomes in terms of family
empowerment, family quality of life, and service satisfaction?
b) How does context influence these outcomes?
3.1 Hypotheses
With regard to the third research questions, which asks whether a family’s context and perceived
experience of navigation (defined by accessibility, continuity and family involvement) influence
the outcomes of interest, several broad hypotheses following from the program theory were
proposed and are outlined in Table 2 below.
Table 2. Hypotheses by research question and dependent variable
Research question Dependent
variable Hypothesis
Do families who perceive
the Family Navigation
Project as accessible,
Family
empowerment
H1: Context influences outcomes to varying
extents and family navigation positively
predicts family empowerment scores
28
continuous and family-
inclusive experience
better outcomes in terms
of family empowerment,
family quality of life, and
service satisfaction?
How does context
influence these
outcomes?
Family quality
of life
H1: Context influences outcomes to varying
extents and family navigation positively
predicts family quality of life
Navigation
satisfaction
H1: Context influences outcomes to varying
extents and family navigation positively
predicts navigation satisfaction
Satisfaction
with referred
services (SRS)
H1: Context influences outcomes to varying
extents and family navigation positively
predicts satisfaction with referred services
3.2 Objectives
This study was further guided by several broad objectives. The first was to develop a conceptual
framework and testable program theory for family navigation. The groundwork for this objective
was laid through extensive collaboration by the PI with the program team to design the current
study and the underlying conceptual and measurement framework. The hope was that the
resulting data supported this framework, which could then be iteratively retested and refined in
future evaluations.
The second objective was to evaluate perceived experience of family navigation in terms of
accessibility, continuity and family involvement; to report on the influence of perceived
experience on desired outcomes of family empowerment, family quality of life, and service
satisfaction; and to explore the influence of context on experience and outcomes.
Addressing these two objectives would contribute evidence regarding if and how the FNP is
achieving its goals (i.e. having the impact on families that it intended to have); and the various
ways in which a family’s experience may vary across contexts. Should the analyses show
positive associations between program components and the desired outcomes, it would lend
support to the value of family navigation.
The final objective was to test a theory about which mechanisms are underlying the observed
outcomes. If mechanisms were found to have a significant statistical impact on the outcomes of
interest, this would help to answer the questions of if and how family navigation helps families,
and what the relative influence of context was for families in this sample. Identifying the
underlying mechanisms is particularly important to implementation of the model elsewhere.
29
Although implementation elsewhere would need to take local context into account (different
populations and/or settings may have different needs), the underlying mechanisms should remain
the same.
3.3 Rationale for the current study
The FNP navigates for all family members. That this evaluation was limited to family-based
outcomes was a conscious choice that reflects this, as well as the critical understanding that it is
families who primarily facilitate the interaction with the health care and social services systems
when seeking care for their youth. The decision was also practical; only a very small number of
FNP clients are youth seeking care for themselves and are unlikely to engage in outcomes
research, leading to a small and inconsistent sample.
The study, as designed, was warranted for several reasons. First, ongoing evaluation and change
in response to feedback was a key tenet of this evolving program, and this study provided an
opportunity to test an evaluation framework for the concept of family navigation. Second, since
the FNP was intended to be a prototype for future navigation initiatives and was expected to
scale up, it was especially important to collect data that could be used by program developers to
inform its implementation and success in other jurisdictions. Lastly, this study has the potential
to make a significant contribution to the literature base. Navigation is increasingly being
employed across a range of health care settings despite the lack of literature, particularly around
conceptual and evaluation frameworks, and the lack of distinction from care coordination, case
management and related terms. This study yields both a conceptual framework for family
navigation and a mixed methods measurement framework, as well as a series of testable program
theories with both qualitative and quantitative data to the literature base.
30
Chapter 4 Measurement
Specification of variables
4.1 Context variables
The context variables explicitly included in the conceptual framework were collected as part of
the FNP’s standard intake procedure and as such, could be abstracted from the client charts.
Recall from Section 2.3.1 that contextual characteristics of interest fell into several categories.
The first main category of contextual variables was demographics, including age, gender, and
geographical location. There were two demographic variables that could not be abstracted from
client charts, however; the FNP currently does not explicitly ask families to self-report their
ethnicity or socioeconomic status. For the purposes of this study, a decision was made by the PI
in consultation with the program team and supervisory committee to add original questions to the
survey package in order to collect these variables of interest.
Another key category of contextual variables was program use characteristics, including the
length of time and intensity (number of contacts) with which they were engaged with the FNP.
Also of chief importance was type(s) of MHA concern(s), both in terms of whether care was
sought for mental health concerns, addiction concerns, or both (and for which specific
conditions); and youth’s level of acuity, in terms of whether or not families perceived
improvement in their youth’s functional status since enrolling with the FNP. Lastly, past service
utilization, including past ED visits, inpatient stays, and involvement in their case by other
service sectors; and reasons for contact were contextual categories of interest given they can add
additional layers of complexity to navigation.
Measurement of context variables is summarized in Table 1, Appendix C.
4.2 Mechanism variables
The three mechanisms proposed by the program theory were operationalized using original
survey questions grounded in peer-reviewed literature. Questions were written by the PI, in
consultation with the supervisory committee, and validated for face validity by the program team
and pilot sample. Accessibility was measured using five five-point ordinal scale items that
31
reflected each of Penchansky and Thomas’s (1981) accessibility framework – availability,
accessibility, accommodation, affordability, and acceptability; refer back to Chapter 3.3.2 for
further detail on each of the mechanisms. Similarly, continuity of care was measured using three
original five-point ordinal scale items that reflected the three types of continuity of care proposed
in Haggerty and colleagues’ (2003) continuity of care framework – informational continuity,
management continuity, and relational continuity. Lastly, a single five-point item was written to
measure family involvement (Wood, 2004). Five-point ordinal scales ranged from “very
dissatisfied (1)” to “very satisfied (5).”
The details of each mechanism measure, including the proposed survey questions, are
summarized in Table 2, Appendix C.
4.3 Outcome variables
Measurement of outcome variables is summarized in Table 3, Appendix C, and is discussed in-
depth by outcome below.
4.3.1 Family empowerment
Family empowerment was quantitatively operationalized using the Family Empowerment Scale
(FES), a scale with which the program team was already familiar. Developed in the 1990s by
P.E. Koren, N. DeChillo and B.J. Friesen for use with families who have children with emotional
and developmental disabilities, this measurement tool since been extensively used in a variety of
related settings (1992). The scale includes three separate subscales representing family
empowerment at different levels or spheres of influence: at home in the family (FES Family), in
seeking services (FES Service-seeking), and advocating in the community (FES Community).
The first dimension, level of empowerment, can occur at the family level, in terms of their ability
to manage their immediate circumstances at home; the service system level, in terms of families’
ability to actively command and negotiate the services necessary for their youth and family’s
needs; and at the community/political level, in terms of families’ ability to advocate for
improved services for the youth MHA population in general. The second dimension measures the
expression of empowerment at any of the above levels through families’ attitudes; knowledge
and skills; and behaviour.
32
The FES consists of 34 items, each containing a subjective statement (for example, “I believe I
can solve problems with my child when they happen”) and a five-point response scale ranging
from “not true at all (1)” to “very true (5).” The scoring strategy reflects the three levels of
empowerment by summing scores on the items from each level (family, service system and
community/political) to yield three separate subscores. The scale has previously been extensively
psychometrically evaluated and shows strong internal consistency (Chronbach’s alpha ranges
from 0.87 to 0.88 for each of the three levels) and good test-retest reliability (Pearson’s
correlations ranging from 0.77 to 0.85, and paired t tests for mean differences between subscores
were non-significant). With regard to validity, inter-rater agreement is also good, with kappa
coefficients ranging from 0.70 to 0.83 across levels and 0.77 overall. Lastly, factor analysis
suggests the items generally correspond to the levels conceptualized in the tool.
4.3.2 Family quality of life
Family quality of life (FQOL) was quantitatively operationalized using a modified version of the
Beach Center Family Quality of Life Scale (BCFQoLS) (Hoffman, Marquis, Poston, Summers,
& Turnbull, 2006). The tool was developed on the basis of two empirical studies conducted by
the Beach Center on Disability at the University of Kansas, in collaboration with families,
service providers and researchers. The goal for the researchers was to create an instrument that a)
accurately reflected families’ perception of the most important aspects of FQOL, b) was
psychometrically sound for use in research studies, and c) would be a pragmatic tool for use in
program evaluation and policy planning. Like the FES, the BCFQoLS was originally designed
for use with families who have youth up to age 21 with disabilities, but can be adapted to the
youth MHA setting.
The BCFQoLS consists of 20 items scored on a five-point response scale ranging from “very
dissatisfied (1)” to “very satisfied (5).” The items span five conceptual domains of family life:
family interaction, parenting, emotional well-being, physical/material well-being, and disability-
related support. Scores can be summed by domain and/or in total for a continuous measure.
Psychometric analyses, including confirmatory factor analyses with a sample of 488 families,
support the five-factor model and suggest that all subscales individually have good to excellent
fit for both importance and satisfaction rating (Chronbach’s alpha was 0.94 and 0.88,
respectively). Subscales correlate significantly with hypothesized similar measures, suggesting
33
convergent validity; and test-retest reliability correlations were significant at the 0.01 level
across the board.
Following consultation with the program team and supervisory committee, the PI removed
several items in this scale due to irrelevancy to the population and modified wording to increase
relevancy. In addition, feedback from the program team led the PI to add three more items
intended to capture the full range of expected impacts of family navigation of FQOL. The first
added item asks families about their sense of control over care. This was deemed highly relevant
by the PI because a) it was a stated goal to improve families’ ability to manage their daily lives,
and b) as mentioned earlier, existing evidence suggests that families who participate in their
youth’s care are more likely to feel in control and as if their needs were met (Curtis & Singh,
1996). The second added item asks families about their hopefulness and future outlook. When
reviewing potential outcome measures, this concept of hopefulness and future outlook was found
to be a unique and desirable feature of the Bakas Caregiving Outcomes Scale (Bakas, 2014). As
such, a corresponding item was added by the PI that similarly captures the FNP’s stated goal of
providing families with hope for the future of their youth. Lastly, an item subjectively assessing
overall FQOL was added. What little FQOL literature does exist has shown a degree of discord
between a total sum of the rating of individual items (which is how the BCFQoLS is scored), and
the rating of a single overall satisfaction item rating of FQOL on a five-point Likert scale; this
finding aligns with the well-known tendency for satisfaction measures to be negatively skewed
(Samuel, et al., 2012). A comparison of ratings on this single item versus the summed total score
allows for the contribution of further evidence toward this topic.
Following the removal and addition of specific items described above, the resulting modified
scale (“m-BCFQoLS scale”) remained at 20 items, which were summed to a single continuous
variable representing a total cross-sectional score.
4.3.3 Service satisfaction
Satisfaction with navigation services was operationalized using the FNP’s in-house satisfaction
survey tool mentioned earlier (Section 2.3.3). The “NAVSAT” was developed by the FNP in the
summer of 2014 in response to a gap in the literature around valid and reliable measures specific
to navigation in the field of MHA (Fishman & Levitt, 2014). No similar MHA navigation
satisfaction scales are known exist. The scale was developed by the FNP team, which included
34
staff navigators, a medical director, project manager, and parent advisory council members at
Sunnybrook Health Sciences Centre. Questions were adapted from six valid and reliable
satisfaction scales in the literature, including the Verona Satisfaction Scale and the UKU-
Consumer Satisfaction Rating Scale. This 25-item scale consists of two sections: satisfaction
with navigation, and satisfaction with the resources to which families were referred by the
navigation team. Items are rated on five- or seven-point Likert-style scales.
The first section includes 15 items that ask families to assess their satisfaction with the treatment
recommendations received, their navigator’s knowledge and fluency in the mental health and
addictions systems, respect for confidentiality, and nature/frequency of contact. Designated
outcome variables include likelihood of recommending the service, navigator helpfulness, and
overall service satisfaction. For the purposes of the current study, scores on the three outcome
variables were summed to a continuous total score representing satisfaction with navigation
services. The second section presents 10 items that evaluate families’ satisfaction with the
referred resource in terms of type, delivery method, location and effectiveness. The designated
outcome variable for this section was a seven-point item measuring overall satisfaction with
referred services. For the purposes of this study, this item was treated as a continuous variable,
an approach that is considered generally acceptable for variables with more than five categories
(Rhemtulla, Brosseau-Liard, & Savalei, 2012).
The NAVSAT was previously validated in a sample of 80 families receiving services from the
FNP in July 2014. The results suggested the scale was easy to administer, took 10-15 minutes to
complete, and had excellent psychometric properties that reflect the scale’s grounded
development process. The first 15-item section on navigation satisfaction yielded a Chronbach’s
alpha value of 0.957, and in factor analyses, all variables loaded on a single factor, with loadings
ranging from 0.776 to 0.936. The second 10-item section on referred service satisfaction
similarly yielded a reliability coefficient of 0.948, with a factor analysis and scree plot
suggesting a one-factor solution was again most appropriate (factor loadings in this section
ranged from 0.702 to 0.913) (Fishman & Levitt, 2014).
4.4 Qualitative measures
To enhance the robustness of the data, participants were prompted to qualitatively expand on
their context, experiences and outcomes at multiple points throughout the survey. The survey
35
was organized such that quantitative context, mechanism, and outcome measures were presented
on their own pages. As a means of efficiently collecting concurrent qualitative data that would
help to enhance understanding of the quantitative responses, following the completion of each
page or measure, participants were asked: “Do you have anything you would like to add?”
(Driscoll, Appiah-Yeboah, Salib, & Rupert, 2007). Responses could then be entered in unlimited,
open-text fields. As with all questions in this survey, qualitative responses were entirely
voluntary.
4.5 Measurement validity
All measures were extensively refined and validated for face validity, clarity, comprehension,
applicability, and relevancy in collaboration with the program team, who had an in-depth
understanding of the nature of the study, having extensively participated in the design of the
conceptual framework and program theory. However, it was also necessary to verify whether
families understood and responded to the questions and underlying concepts as expected. This
type of validity evidence, based on response processes, was first proposed by S. Messick and
refers to adequacy with which respondents’ actions and thought processes reflect their
understanding of the construct in the same way as intended by the researchers, such that the
rationale for use of the measure and interpretation is maintained (1990).
To further validate the measures, open-ended prompts were embedded in the survey package to
gather feedback from a randomly selected pilot sample of families from the FNP registry (n=30,
or to theoretical saturation). The preamble explained that as the first group of participants, they
were being asked to provide feedback intended to improve the survey experience for the formal
study phase. Open-ended prompts related to a) the convenience and ease of use of the survey
platform and b) the understanding of the survey items for the remaining participants. Feedback
generated from the pilot sample was thematically analyzed, amalgamated with feedback from the
program team, and proposed refinements re-presented to the program team for final consensus.
The pilot sample data were not included in the study sample for the analyses.
36
37
Chapter 5 Methods
Design overview
This multi-phase, mixed methods study employed a cross-sectional design whereby both
quantitative and qualitative data were collected via an online survey package including both
open- and closed-ended items. Data subsets were then examined individually and in combination
via statistical and thematic analyses with findings merged to produce a more comprehensive
understanding of whether and how family navigation is helping families.
5.1 Study setting
This study was conducted electronically from September through December 2016 in Toronto,
Ontario.
5.2 Ethics
The study is considered to be minimal risk so only delegated and administrative reviews were
required. Ethics approval was jointly sought from and granted by Sunnybrook Research Institute
(SRI) and University of Toronto using the Toronto Academic Health Sciences Network
(TAHSN) Human Subjects Research Ethics Application.
5.3 Sample
The intended sample for the study was any individual officially registered with the FNP at the
time of the study who was seeking care on behalf of a youth with an MHA concern. This
represented approximately 95% of the client roster at the time of the study (n=1045) and
excluded youth seeking care for themselves due to the family-focused nature of the outcomes
employed. Families who did not provide the FNP intake coordinator with verbal consent to begin
navigation services or who did not provide an email address (or who indicated they did not to
want be contacted by email) were excluded (n=303). All individuals meeting inclusion criteria
were identified and abstracted from the client roster by the program’s intake coordinator and
research fellow, resulting in an eligible convenience sample of 742 who were registered with the
FNP at the time of the study.
38
Since the FNP is a relatively new program whose reach is continuously extending throughout the
GTA, the client roster has grown considerably since the study took place. At the time of the
study, the sample of 742 individuals represented approximately 65.0% of the roster.
5.4 Data collection
5.4.1 Survey methods
Data collection occurred over two four-week phases beginning in September 2016; a pilot phase
preceded the official launch of the survey. All individuals identified by the FNP as meeting the
established inclusion criteria were surveyed using a modified Dillman’s Tailored Design Method
(Dillman, et al., 2014). Communication regarding participation in the study was initiated by an
FNP staff member personally known to the clients. The FNP team suggested that communication
by email is generally their clients’ preferred method of contact, and that web-based surveys were
most likely to yield the highest response rate, so eligible individuals were sent an introductory
email briefly describing the study, along with a request to participate. Interested individuals were
instructed to follow a live link to access a personal letter from the PI more fully explaining the
nature and goals of the study, and an option to proceed to the web-based survey platform.
At the time of administration, the survey platform, FluidSurveys, was a Canadian company
which stored data on Canadian servers only; all data was exported prior to FluidSurveys’
acquirement by American counterpart, SurveyMonkey. Upon following the live link, participants
were presented with an online consent form detailing the full protocol, all details of participation
including risks, benefits, ethical concerns and time to complete, as well as an assurance that all
data would be deidentified and confidential. Only individuals who provided electronically signed
and dated consent were permitted to continue to the survey questions; failure to provide consent
redirected individuals to the exit page. Individuals were also asked for consent to be contacted
directly by the PI should any follow up questions about the nature of their responses or
opportunities for participation in follow up research arise.
Individuals who did not respond to the introductory email received a reminder email with a
replacement survey package following one and three weeks’ non-response, respectively. Surveys
were accepted up to four weeks after the initial email, a cut-off point that was selected based on
the program team’s experience that families who did not respond within that time period were
39
unlikely to respond at all. Indeed, only a couple surveys were submitted after the four-week
mark. Lastly, as compensation for their time and effort, families who did complete surveys were
offered the option to have their name entered in a draw for a $150 Loblaws gift certificate.
5.4.1.1 Pilot phase overview
Of the 742 eligible individuals, 30 were randomly selected by the intake coordinator to receive
an invitation to participate in a pilot phase of the survey in September 2016. The purpose of the
pilot phase was to gain any client perspectives that would allow further refinement of face and
content validity of the survey items beyond what the program team was able to contribute. The
survey was administered as per the protocol and respondents were prompted in open fields for
feedback on convenience of the survey platform and understanding of the survey items for each
scale, along with any generalized suggestions for improvement. Feedback suggested the survey
was clear, asked important questions, and was not overly long but it required some modifications
including simplifying wording on one item, correcting a few formatting errors, and addressing
some technical difficulties with the website. All feedback and suggestions were reviewed by the
PI in consultation with the program team and the supervisory committee before being
implemented into the final version of the survey accordingly.
5.4.1.2 Response rates
5.4.1.2.1 Pilot survey phase
Of the 30 randomly selected individuals invited to participate in the pilot phase of the survey,
one email was found to be invalid, 11 individuals responded by following the live link to the
survey package, and five of those 11 respondents completed the informed consent and provided a
complete set of responses. The complete response rate (16.7%, or five of 30) was lower than
expected. To better understand and address this low rate, the PI presented the results to the
program team, who helped to identify several potential explanatory variables that informed the
PI’s subsequent decision not to modify the recruitment process.
The first explanatory variable discussed was logistical and related to the fact that the pilot study
was initiated on the first day of school in a population whose youth are school-aged and
struggling with mental health concerns that are often related to school attendance and experience.
This is a busy, stressful and inconvenient time for parents and youth alike where voluntary, time-
40
consuming surveys are unlikely to take priority. The second related variable discussed was that
there is extensive variability in the nature, intensity and severity of MHA concerns families that
come to the FNP are dealing with, and some families were likely too entrenched in their
difficulties to be in a position to participate, even if they had wanted to. However, this is not a
feature that was easily identified from the client roster. Lastly, it was noted that the roster
consisted of individuals who had registered with the FNP at any point since the program
launched in 2014, and it is possible that some individuals were less motivated to provide
feedback on an experience that may have taken place as long as two years ago.
With regard to the low completion rate, the survey platform allows users to identify at which
point in the survey respondents exit. It was noted that three of the six participants who failed to
complete the survey did not proceed beyond the introductory letter from the PI, and so a decision
was made to shorten the introductory letter, simplify the study description further, and highlight
the minimal time required to participate in hopes of engaging more individuals in the formal
survey phase. A contingency plan to have the FNP’s Medical Director send out the email was
arranged. The survey pages themselves were also shortened and further simplified, and a
progress bar was included at the bottom of each page.
5.4.1.2.2 Formal survey phase
The survey was formally launched following the integration of pilot phase feedback in
November 2016. The individuals invited to participate in the pilot phase were removed from the
list of potential participants, resulting in an eligible sample of 712; and of 712 email invitations
sent out, 24 addresses were found to be invalid and bounced back. The resulting valid study
sample was 688.
Of 688 individuals who were sent an introductory email, 276 (or 40.1%) followed the live link to
the survey package, which is slightly more but comparable to the pilot phase response rate. Of
these 276 individuals who accessed the survey package, 48.6% (n=134) provided electronically
signed informed consent and a complete set of responses, which again is slightly more but
comparable to the pilot phase completion rate. Of the 688 eligible individuals, 134 complete
responses yield an overall response rate of 19.5%.
41
Participants were encouraged to reach out to the PI with any questions or concerns about the
survey or their participation. Several individuals highlighted an unfortunate technical difficulty
with the website whereby individuals repeatedly lost their progress when they attempted to take a
break and return to the survey at a later point in time. It is likely that this frustrating experience,
an unexpected feature of the survey platform and beyond the control of the investigator,
discouraged completion to some extent.
5.4.2 Chart review
Client chart reviews were completed using a secure, cloud-based management software called
EMHware, which is an advanced electronic medical record platform primarily intended for use
by outpatient mental health care agencies in Ontario. As per the measurement framework, a
number of additional contextual variables were abstracted from client charts for all survey
respondents who provided both their full name and informed consent to access personal health
information (n=134). Chart reviews took place over several weeks following the survey, in
December 2016.
The purpose of the chart review was to collect additional information about the nature of a
family’s concerns and engagement with the FNP, while reducing additional burden on the client.
For example, logistical details like clients’ status (active or discharged), documented dates of
contact (to calculate time engaged with the program), address (required to identify catchment
area), and history of previous service use were abstracted from charts. Also abstracted were
reported MHA concerns, and more nuanced details like school avoidance, history of bullying,
and any legal involvement. A complete list of variables by source is provided in Appendix C.
5.5 Sample size and power
Although the resulting sample size was smaller than anticipated, the Rule of Ten for multiple
regressions with more than six predictors, along with Green’s comprehensive guide indicating
that N > 50+8m where m is the number of predictors is sufficient for multiple correlations, and N
> 104 + m is adequate for testing individual predictors, all suggest that the current study’s
sample size of 134 is sufficiently powered to accommodate as many as 10 predictors (Green,
1991; VanVoorhis & Morgan, 2007). Further, statistical analyses were supplemented with
qualitative data, the goal of which was to gain a more robust understanding of any patterns that
42
emerged versus relying entirely on standardized but decontextualized statistical indicators of
model fit and power.
5.6 Data storage
Only the PI has access to identifying information. All data was deidentified, and a separate file
linking clients to their unique identifiers is kept in a locked drawer in a locked office on a
password-protected and encrypted hard drive. Data will be stored for the recommended 10 years
as per hospital policy, and then destroyed.
5.7 Overview of analytical approach
A multi-phase, mixed methods analytical plan was developed to evaluate the data according to
the theoretical framework, program theory and research questions. The first phase involved a
quantitative descriptive analysis of the variables in order to assess the distribution of the
variables and describe the sample with regard to who was using the FNP at the time of the study,
for what reasons, what their outcome scores were, and how they were distributed.
The second phase involved a quantitative correlational analysis, the purposes of which were to
test hypothesized relationships within and between select context, mechanism and outcome
variables, generating a series of C-M-O configurations that further described the sample and
supported the program theory (refer back to Chapter 2 for an overview of C-M-O
configurations). Results of this second phase were then used to expand upon the descriptive
analysis; and to inform the selection of covariates and refine hypotheses for inferential modeling
in a third, quantitative phase. This final inferential quantitative phase was implemented to
statistically test the impact of the proposed context and mechanism variables on the outcomes of
interest using a series of generalized linear models.
A criticism of Realist Evaluation has been the lack of clarity around how to construct C-M-O
configurations, as few realist evaluations clearly delineating C-M-O construction have been
published. The analytical approach in the current study was thus primarily informed by the
methods and resulting data types, but also draws from and expands upon one of the few
examples in the literature in which a method for incrementally building C-M-O configurations is
well-described (Byng, et al. 2005).
43
Finally, a qualitative phase followed the quantitative analyses in order to enhance the robustness
of the analysis. As mentioned earlier, open-text fields prompting qualitative feedback were
included throughout the survey package. This qualitative data was then described, thematically
coded and analyzed in relation to key themes from the program theory. The analytical approach
is further detailed at the beginning of each phase in the chapters that follow.
44
Chapter 6 Quantitative Analysis
Overview of quantitative approach
RE is a theory-driven framework and as such, the quantitative analysis followed a deliberate,
theory-building process designed to accumulate as much information as possible about the
relationships between context, mechanism and outcome variables. First, descriptive statistics
were used to determine for whom the FNP is navigating and for what reasons; how clients rated
the services they received; and how they scored on the outcomes of interest. Descriptive statistics
were also used to assess the nature of the data in order to ensure that the most appropriate
statistical procedures were chosen.
Following descriptive analyses, correlational analyses were used to identify factors that tend to
co-occur. Correlational analyses in this study served several foundational purposes. Correlations
amongst context variables themselves (C-C dyads) helped to further describe the population
seeking navigation, including the complex relationships within and between demographics,
mental health and addiction concerns, and interactions with the service system. Correlations
within and between mechanisms and outcomes were also employed to ensure preliminary
modeling assumptions were met; M-M dyads reflected risk of multicolinearity (subsequently
informing and resulting in the use of a single component score), and M-O dyads indicated
whether the assumption that the mechanism is linearly related to the outcome had been met.
However, the primary goal of the correlational analyses was to identify meaningfully correlated
C-M and C-O dyads, which were used as covariates in outcome models.
6.1 Importing and coding of data
All survey responses and client chart data were exported from the FluidSurveys and EMHware
platforms, respectively, and imported into a pre-organized data collection form in Microsoft’s
Excel where data were combined, coded and scored as per the measurement framework and
variable specifications. Once all survey and chart data was amalgamated for each individual, data
was deidentified using unique client identity numbers. All variables were then imported, defined
and analyzed using IBM’s SPSS Statistics 22.0 (IBM Corporation, 2013).
45
6.2 Data quality
Overall data quality was excellent. Data for all descriptive variables were complete for all
individuals with the exception of missing data (n=12) for self-reported household income. Data
for all mechanism variables (accessibility, continuity of care, and family involvement) were also
complete for all individuals. With regard to outcome data, single item responses on the family
empowerment scales were missing entirely at random in a small handful of cases (n=5) and so
the decision was made to impute the middle value in these instances (equivalent to the response,
“Neither”). However, there was systematically missing data that resulted from cases in which
respondents completed the survey in full through the family empowerment scales, but then
completed only some or none of the remaining outcome measures (n=5). In these cases, data
were coded as missing and excluded pairwise from analyses to avoid overfitting (Tabachnick &
Fidell, 2007). In addition, several outliers were identified but based on the theoretical
framework, were retained in the analyses as they are thought to reflect the inherent variability in
families’ experiences. In this framework, every case is understood to have a unique ability to
contribute to understanding who navigation works best for, when and why.
The significance of variables was interpreted using an alpha level of 0.05 where appropriate.
6.3 Descriptive analyses
Descriptive analyses were performed on all variables. The particular statistical approach
employed was determined by the type of variable.
6.3.1 Context variables
Descriptive statistics including frequencies and measures of central tendency, where appropriate,
are presented for each category of context variables (refer back to Section 2.3.1 and 4.1):
demographics, MHA concern characteristics, current program and past service use
characteristics, and reasons for contacting the FNP. This section is intended to answer the first of
the guiding research questions, which was “Who is the Family Navigation Project serving? Is the
Family Navigation Project reaching its target population?”
46
6.3.1.1 Demographics
6.3.1.1.1 Clients
The vast majority of the sample, 93.3%, are parents who were seeking help for their own youth
and family, although other client relationships such as step-parents and grandparents were
reported. Since the roster included clients registered from 2014 onward, only 27.6% were
actively receiving services at the time of the study, while 72.4% were classified as discharged.
Tests of group differences indicated that discharge status at the time of the survey was not
meaningfully related to any variables of interest. However, due to the highly variable and often
chronic nature of MHA concerns, clients may be discharged and reactivated multiple times
according to the needs of the family at any point in time. The number of re-engagements was not
measured in the present study but may have been associated with the total number of weeks a
family spent engaged with the program. The average length of time families spent as a client of
the program varied significantly across the sample, ranging from as little as one week to 106
weeks (or just over 2 years), with clients speaking to their Navigator anywhere from three to 100
times. Clients were actively engaged for an average of 24 weeks (SD=21.7; median=17.1), or six
months, and had an average of 19.7 documented contacts with a Navigator (SD=14.9; median
16.0). As indicated by the range and standard deviation, both length of time and number of
contacts varied widely across the sample.
The FNP operates independently from but is still affiliated with SHSC. Correspondingly, clients
were located across the City of Toronto and the GTA; 29.1% of clients lived in SHSC’s official
catchment area, 41.8% live in the City of Toronto outside the SHSC catchment, and 28.4% live
in the GTA. One client in Nova Scotia was included.
As a whole, the sample consisted primarily of high income families, with 65.6% reporting
household incomes over $100,000 and 40.2% alone over $150,000. This sample is also 85.1%
Caucasian, which does not reflect demographic MHA trends in the highly diverse Toronto area
(Ontario Human Rights Commission, 2015).
6.3.1.1.2 Youth of the clients
The demographic trends among the youth in this sample reflected demographic trends in the
youth MHA literature (ICES, 2017; Kessler., et al., 2005, 2012; Statistics Canada, 2006). The
47
majority of youth in this sample were males (59.0%) who lived at home with their family
(72.4%). The FNP specifies an age range of 13 to 26 years but occasionally accepts youth under
or over the range on a case by case basis. A large majority of youth in this sample are older teens
and young adults; transitional-aged youth (18 to 24 years old) in particular account for 61.9% of
the sample. Youth ranged in age from 12 to 25 and over. Descriptive statistics are presented for
continuous and categorical contextual variables in Table 3 and Table 4, respectively, below.
Table 3. Descriptive statistics for continuous current program use variables
Variable
(n=134) Mean Median SD Min Max IQR Skew Kurtosis K-S
Weeks engaged 24.0 17.2 21.7 1 106.1 24.1 1.6* 2.5* 0.165**
No. contacts
with Navigator 19.7 16.0 14.9 3 100 14 2.3* 7.7* 0.204**
*Statistic is more than twice its standard error **p<.01
Table 4. Frequency statistics for demographic variables
Variable N % of cases*
(N=134)
Demographics
Youth age
12 to 15 years 9 6.7%
16 to 17 years 23 17.2%
18 to 19 years 28 19.9%
20 to 24 years 55 41.0%
25 years and over 19 14.2%
Youth gender
Male 79 59.0%
Female 55 41.0%
Youth ethnicity
Caucasian 114 85.1%
Asian 7 5.2%
Other 10 7.5%
Family household income $25 to 49,999 6 4.9%
48
(n=122) $50 to 74,999 14 11.5%
$75 to 99,999 20 16.4%
$100 to 149,999 31 25.4%
$150k+ 49 40.2%
Family location
Sunnybrook catchment 39 29.1%
City of Toronto 56 41.8%
GTA 38 28.4%
Family living arrangement
with youth
Home with family 97 72.4%
On their own 18 13.4%
Treatment centre 8 6.0%
Home with single parent 7 5.2%
Family member’s relation to
youth
Parent 125 93.9%
Other 9 6.7%
*Percentages and totals may not add to 100.0 due to rounding and/or because cell counts < 5 are suppressed
6.3.1.2 Mental health and addiction characteristics
Clients reported a wide range of youth MHA concerns. Nearly the entire sample (98.5%)
reported at least one mental health concern (mean = 2.3), the most common of which was
depression (57.5% of cases), followed by anxiety (48.5%). ADD/ADHD (22.4%), suicidality
(defined as suicide attempts or ideation; 17.9%), OCD (14.9%), and self-harm (12.7%) were the
next most frequently reported concerns. Less common but still significant were chronic, severe,
and difficult-to-treat concerns including personality disorders, eating disorders, and bipolar
disorder. A history of bullying was reported in 18.7% of cases; and 27.6% of cases explicitly
reported school avoidance or refusal. Descriptive statistics for mental health and addiction
characteristics are presented for continuous and categorical variables in Tables 5, 6 and 7,
respectively.
In this sample, 45.5% of clients (n=61) reported concurrent mental health and addiction
concerns. Refer to Table 7 for the full range of reported addiction concerns in this sub-sample,
49
most of which were assumed to co-occur with a mental health concern as only two cases reported
standalone addiction concerns. Clients who reported addiction concerns reported an average of
1.7 concerns. The most common concern was cannabis use, reported in 74.6% of cases with
addiction concerns (35.1% of total cases). Alcohol use was also a frequent concern, reported in
44.4% of addiction cases (20.9% of the total sample). Among addiction concerns, the next
highest reported rate was for stimulant use (17.5%), which includes cocaine, ecstasy, MDMA,
and methamphetamines. It is also important to note that these MHA concerns were parent-
reported and did not make the distinction between use and abuse or dependence. Other reported
substance use concerns in this sample included opioids, benzodiazepines and non-opioid
prescription drugs, and behavioural addictions, such as to the Internet, video games, and sex.
Nearly two thirds of clients reported that since enrolling with the FNP, their youth’s functional
status had stayed the same (64.2%), while just over a quarter had improved (26.9%) and a
handful had worsened (9.0%).
Table 5. Descriptive statistics for continuous mental health and addiction characteristic
variables
Variable N Mean Median SD Min Max Skew Kurtosis K-S
Number of mental
health concerns 134 2.3 2.0 1.4 0 7 0.7* 0.4 0.165**
Number of addiction
concerns 63 1.7 1.0 1.1 0 5 1.5* 2.0* 0.295**
*Statistic is more than twice its standard error **p<.01
Table 6. Frequency statistics for mental health characteristic variables
Variable N % of cases*
(N=134)
Youth functional status since
enrolling
Improved 36 26.9%
Stayed the same 86 64.2%
Worsened 12 9.0%
Reported mental health Depression 77 57.5%
Anxiety 65 48.5%
50
concerns** ADD/ADHD 30 22.4%
Suicidality 24 17.9%
OCD 20 14.9%
Self-harm 17 12.7%
Personality disorder 13 9.7%
Eating disorder 10 7.5%
Developmental disorder 9 6.7%
ODD/Anger 9 6.7%
Bipolar disorder 8 6.0%
Panic attacks 7 5.2%
Psychosis 6 4.5%
Trauma 5 3.7%
Bullying Yes 37 27.6%
No 97 72.4%
School avoidance Yes 37 27.6%
No 97 72.4%
Concurrent mental health
and addiction concerns
Yes 61 45.5%
No 73 54.5%
*Percentages may not add to 100.0 due to rounding and/or because cell counts < 5 were suppressed **Indicates multiple response variable
Table 7. Frequency statistics for addiction characteristic variables
Variable N % of cases
(N=134)*
% of addiction
cases (N=63)
Reported addiction
concerns**
Cannabis 47 35.1% 74.6%
Alcohol 28 20.9% 44.4%
Stimulants 11 8.2% 17.5%
Opioids 5 3.7% 7.9%
51
Internet and gaming 5 3.7% 7.9%
*Percentages may not add to 100.0 due to rounding and/or because cell counts < 5 were suppressed **Indicates multiple response variable
6.3.1.3 Previous service use
Previous service use characteristics gave insight into the nature and intensity of families’ service
needs. Nearly all families contacting the FNP (92.5%) had already received prior MHA services
for their youth. Despite the numerous and wide-ranging mental health and addiction concerns
reported by parents, only approximately 60.0% of youth had received a formal diagnosis from a
physician or allied mental health professional at some point in their help-seeking journey.
Further, 38.1% of youth had had at least one previous ED visit, and 40.3% had had at least one
inpatient stay.
A case history of legal involvement (which includes involvement by the Youth Justice system
and Children’s Aid Society) was theorized to impact help-seeking by adding another service
sector to navigate. Prior or current legal involvement was reported in over a quarter of cases
(26.9%).
Previous service use characteristics are presented in Table 8 below.
Table 8. Frequency statistics for service use characteristic variables
Variable N % of cases*
(N=134)
Service use
characteristics
Previous service use
Yes 124 92.5%
No 10 7.5%
Current formal diagnosis
Yes 80 59.7%
No 54 40.3%
Previous ED visit(s)
Yes 51 38.1%
No 83 61.9%
Previous inpatient stay(s) Yes 54 40.3%
No 80 59.7%
52
History of legal
involvement
Yes 36 26.9%
No 98 73.1%
*Percentages may not add to 100.0 due to rounding and/or because cell counts < 5 were suppressed
**Indicates multiple response variable
6.3.1.4 Reasons for contact
The FNP collects information from its clients upon intake with regard to their reasons for
contacting the program. This data was abstracted from client charts during the chart review
phase, and the results indicate that a wide range of reasons was reported (Table 9).
Most often (in 70.1% of cases), clients contacted the FNP actively seeking recommendations for
their given scenario. The next most common reason clients connected with the FNP was to seek
out family support (38.8% of cases). Many families were also simply seeking information
(34.3% of cases).
The majority of the remaining reported reasons for seeking navigation were more specific and
deliberate, reflecting the wide range of needs for families who have a youth with MHA concerns.
A quarter of clients were specifically seeking a psychiatrist, and another 19.4% were specifically
seeking a therapist. A number of clients were also seeking residential treatment (15.7% of cases).
Less commonly, clients contacted the FNP seeking addiction-specific programs.
A range of other reasons for contacting the FNP were reported, including need for crisis
supports, day treatment, aftercare, supportive housing, case management, out-of-province or
country treatment, psychoeducational assessments, and cognitive behavioural therapy (CBT).
The type and frequency of reasons for contact is presented for in Table 8 below.
Table 9. Frequency statistics for reasons for contact variable
Variable N % of cases*
(N=134)
Reason for contact**
Recommendations 94 70.1%
Family support 52 38.8%
Information 46 34.3%
53
Psychiatrist 33 24.6%
Therapist 26 19.4%
Residential treatment 21 15.7%
Addiction program 12 9.0%
Crisis supports 6 4.5%
Day treatment 5 3.7%
*Percentages may not add to 100.0 due to rounding and/or because cell counts < 5 were suppressed
*Indicates multiple response variable
6.3.2 Mechanism variables
This section addresses the first half of the second guiding research question, “Is the Family
Navigation Project providing accessible, continuous, family inclusive care? Which contextual
factors matter?” Descriptive statistics related to the frequency distribution of each of the three
mechanism variables are reported below. The program theory guiding this study suggested that
family navigation (as per the definition developed during the theory-building stage of this study)
is characterized by three key mechanisms: accessibility, continuity of care, and family
involvement. The evaluation feedback will inform understanding of whether clients perceived
the services they received as highly accessible, continuous, and family-inclusive.
6.3.2.1 Accessibility
Accessibility is a widely known and accepted literature-based concept that was assessed using
five items each rated on five-point Likert-style scale; the sum of the item ratings was calculated
to produce a continuous total score out of 25. Overall, accessibility was rated very highly across
the sample (mean = 21.1, median = 23, range = 5 to 25, SD = 4.9). There are a handful of outliers
reflecting a handful of clients with low ratings, which widened the standard deviation.
Examining the individual items, the ratings were highly comparable for the first four (mean =
4.3, median = 5.0), but on the last item, which represented appropriateness of care and asked
whether the FNP met their expectation of what family navigation entailed, ratings were slightly
lower (mean = 3.9, median = 4.0).
54
Normality was assessed using a range of descriptive statistics. A median to the right of the mean;
a non-linear Q-Q plot; a boxplot with high positive values, a long tail and several outliers;
negative skew and positive kurtosis values twice their standard error; and a significant
Kolmogorov-Smirnov (K-S) statistic (0.241, p=.000) all led to the conclusion that the
distribution of total accessibility scores were significantly skewed and leptokurtic.
Table 10 presents the statistics for total score and for each item, respectively.
Table 10. Descriptive statistics by item for the “Accessibility scale”
Variable: Accessibility (5 items; n=134)
Item Mean Median SD Skew Kurtosis K-S
To what extent do you feel you were
able to reach the Navigator whenever
you needed? (/5)
4.32 5.0 0.91 - - -
To what extent do you feel it was
convenient to communicate or meet with
the Navigator? (/5)
4.25 5.0 1.18 - - -
To what extent do you feel the Navigator
accommodated your schedule when
making arrangements with you? (/5)
4.37 5.0 1.1 - - -
To what extent do you feel the Navigator
was considerate of your resources? (/5) 4.29 5.0 1.12
- - -
To what extent do you feel the Navigator
met your service expectations for family
navigation? (/5)
3.90 4.0 1.31 - - -
Accessibility total score (/25) 21.13 23.0 4.90 -1.7* 2.3* 0.241**
Scale: 1=Very dissatisfied, 2=Dissatisfied, 3=Neither, 4=Satisfied, 5=Very satisfied
*Statistic is more than twice its standard error **p<.001
6.3.2.2 Continuity of care
Continuity of care, like accessibility, is a literature-based concept that was assessed using three
items each rated on five-point Likert-style scale; the sum of the item ratings provided a
continuous total score out of 15. Overall, continuity of care was also rated very highly (mean =
55
11.8, median = 13.0, SD = 3.8) and the distribution followed a similar pattern to accessibility. A
histogram, median to the right of the mean, non-linear Q-Q plot, boxplot with high positive
values and a long tail, negative skew value twice its standard error, and significant K-S statistic
(0.218, p=.000) indicated that the distribution of scores was also significantly negatively skewed.
However, relative to accessibility, continuity ratings are lower. This is because there was a
greater proportion of low-scoring outliers on this measure and so the mean is relatively lower
and the standard deviation relatively wider. Inter-item means were also slightly lower for one
item in particular, which asked clients whether their Navigator “continuously communicated and
coordinated” with the family and other service providers.
Table 11 presents the statistics for total score and for each item, respectively.
Table 11. Descriptive statistics by item for the “Continuity of care scale”
Variable: Continuity of care (3 items; n=134)
Item Mean Median SD Skew Kurtosis K-S
To what extent do you feel the Navigator
continuously communicated and
coordinated information with you and the
service providers to whom you were
referred? (/5)
3.88 4.0 1.35 - - -
To what extent do you feel the Navigator
continuously adequately responded to
changes in your family’s situation and
needs? (/5)
3.90 4.0 1.40 - - -
To what extent do you feel the Navigator
was continuously committed to
understanding and helping your family
until you no longer felt you require their
services? (/5)
4.01 5.0 1.35 - - -
Continuity of care total score (/15) 11.80 13.0 3.83 -1.1* 0.1 0.218**
Scale: 1=Very dissatisfied, 2=Dissatisfied, 3=Neither, 4=Satisfied, 5=Very satisfied
*Statistic is more than twice its standard error **p<.001
56
6.3.2.3 Family involvement
Unlike the accessibility and continuity of care measures, the family involvement measure in this
framework was a single five-point ordinal scale item that was based in the literature developed
by the PI in consultation with the program team and supervisory committee. Like accessibility
and continuity of care though, family involvement followed similar score patterns and was rated
very highly by the sample overall (mean = 4.0, median = 4.0, mode = 5.0, SD = 1.3). Again, all
plots and statistics suggested the distribution was similarly significantly negatively skewed (K-S
= 0.284, p=.000), but as with the accessibility and continuity of care measures, there were also
some low-scoring clients.
Table 12 present the descriptive statistics for this item.
Table 12. Descriptive statistics for the single item “Family involvement”
Variable: Family involvement (1 item; n=134)
Item Mean Median SD Skew Kurtosis K-S
To what extent do you feel the
Navigator consistently involved you
and your family in all stages of care
planning and decision making?
(/5)
4.0 5.0 1.3 -1.2* 0.2 0.284**
Scale: 1=Very dissatisfied, 2=Dissatisfied, 3=Neither, 4=Satisfied, 5=Very satisfied
*Statistic is more than twice its standard error **p<.001
6.3.3 Outcome variables
The following section reports descriptive statistics for each of the outcomes of interest. In
building the program theory, the program team was asked to identify its goals, which were
empowering families, improving their quality of life, service satisfaction (Figure 1, Appendix B).
Several representative quantitative measures were then identified and slightly modified to better
fit the population: the Family Empowerment Scale (FES), a modified Beach Center Family
Quality of Life Scale (-mBCFQoLS), and the NAVSAT tool. Descriptive statistics for NAVSAT
responses in particular answer research question 1c: “Overall, are clients satisfied with the
services they receive?”
57
6.3.3.1 Family empowerment
The FES consisted of 34 items across three separate subscales representing family empowerment
at different levels or spheres of influence: at home in the family (FES Family), in seeking
services (FES Service-seeking), and advocating in the community (FES Community). The
continuous total subscale scores could be used individually, or summed to create a total
empowerment score. Only the first two subscales (FES Family and FES Service-seeking) were
selected for use in this study as the community-level scale was later determined to be too
conceptually distal a measure to expect it would be meaningfully or statistically impacted by the
program mechanisms proposed in this study; and that including community subscale scores as a
third of a “total empowerment score” would limit ability to detect a statistical impact at the more
relevant and proximal family and service-seeking levels. A decision was thus made by the PI in
consultation with the program team and supervisory committee not to proceed with scores from
the community-level subscale, and to use scores for the two subscales (FES Family and FES
Service-seeking) as individual outcome measures rather than summing to a single family
empowerment score.
6.3.3.1.1 FES Family
The family subscale of the FES (FES Family) was measured using 12 five-point items summed
to a continuous total score out of 60. Unlike the mechanism scores, empowerment scores were
more evenly distributed. The mean for the sample was 43.9 (median = 44, SD = 6.9, IQR = 8).
All normality statistics were non-significant, and plots confirmed that scores on this variable may
be slightly leptokurtic but can safely be assumed to follow a normal distribution with long but
even tails.
Table 13 presents the statistics for item-specific and total scores.
Table 13. Descriptive statistics by item for the “FES Family subscale”
Variable: FES Family subscale (12 items; n=134)
Item Mean Median SD Skew Kurtosis K-S
When problems arise with my child, I handle
them pretty well (/5) 3.65 4.0 0.73
- - -
58
I feel confident in my ability to help my
child grow and develop (/5) 3.49 3.0 0.87
- - -
I know what to do when problems arise with
my child (/5) 3.34 3.0 0.76
- - -
I feel my family life is under control (/5) 3.37 3.0 1.0 - - -
I am able to get information to help me
better understand my child (/5) 3.49 4.0 0.96
- - -
I believe I can solve problems with my child
when they happen (/5) 3.14 3.0 0.80
- - -
When I need help with problems in my
family, I am able to ask for help from others
(/5)
3.80 4.0 1.07 - - -
I make efforts to learn new ways to help my
child grow and develop (/5) 4.25 4.0 0.77
- - -
When dealing with my child, I focus on the
good things as well as the problems (/5) 4.05 4.0 0.78
- - -
When faced with a problem involving my
child, I decide what to do and then do it (/5) 3.77 4.0 0.85
- - -
I have a good understanding of my child’s
disorder or behavioural concern (/5) 3.57 4.0 0.94
- - -
I feel I am a good parent (/5) 3.93 4.0 0.77 - - -
FES Family subscale total score (/60) 43.84 44.0 6.29 -0.6 2.4 0.074
Scale: 1=Very dissatisfied, 2=Dissatisfied, 3=Neither, 4=Satisfied, 5=Very satisfied
6.3.3.1.2 FES Service-seeking
Like the family subscale, the services subscale (FES Service-seeking) included another 12 five-
point items, summed to a total score out of 60. Scores on the child services subscale were similar
to the family subscale scores, with equivalent means and medians (43.9 and 44.0, respectively)
59
but slightly more variability across the sample (SD = 9.0, IQR = 12). This was reflected in a
negative skew statistic more than twice its standard error, a Q-Q plot revealing a slight
curvilinear pattern, and a boxplot with relatively less even tails due to a greater number of low-
scoring outliers. However, the K-S normality statistic was non-significant and so the distribution
was cautiously specified as normal.
Table 14 presents the statistics for total score and for each item.
Table 14. Descriptive statistics by item for the “FES Service-seeking subscale”
Variable: FES Service-seeking subscale (12 items; n=134)
Item Mean Median SD Skew Kurtosis K-S
I feel that I have a right to approve all
services my child receives (/5) 3.57 4.0 1.25
- - -
I know the steps to take when I am
concerned my child is receiving poor
services (/5)
3.30 3.0 1.16 - - -
I make sure that professionals understand
my opinions about what services my child
needs (/5)
3.77 4.0 1.07 - - -
I am able to make good decisions about
what my child needs (/5) 3.54 4.0 1.01
- - -
I am able to work with agencies and
professionals to decide what services my
child needs (/5)
3.72 4.0 1.10 - - -
I make sure I stay in regular contact with
professionals who are providing services to
my child (/5)
3.82 4.0 1.18 - - -
My opinion is just as important as
professionals’ opinions in deciding what
services my child needs (/5)
3.85 4.0 1.02 - - -
I tell professionals what I think about
services being provided to my child (/5) 3.55 4.0 1.08
- - -
I know what services my child needs (/5) 3.18 3.0 0.87 - - -
60
When necessary, I take the initiative in
looking for services (/5) 4.45 5.0 0.75
- - -
I have a good understanding of the service
systems my child is involved in (/5) 3.42 4.0 1.09
- - -
Professionals should ask me what services
I want for my child (/5) 3.78 4.0 1.13
- - -
FES Service-seeking subscale total score
(/60) 43.72 44.0 8.56 -0.8* 1.0 0.075
Scale: 1=Very dissatisfied, 2=Dissatisfied, 3=Neither, 4=Satisfied, 5=Very satisfied
*Statistic is more than twice its standard error
6.3.3.2 Family quality of life
FQOL was scored using a modified version of the BCFQoLS (“m-BCFQoLS scale”) that
consisted of 20 five-point ordinal items summed to a total FQOL score out of 100. Overall,
FQOL scores were similar to family empowerment subscale scores with a median of 4.0 and a
mean in the 70th percentile (mean=72.73, SD=14.3). However, three specific items stood out as
rated lower than the rest, with medians of 3.0 and means less than 3.3; these items related to
availability and accessibility of resources to support families to manage youth MHA concerns.
The first item that was rated lower referred to families’ availability of outside help to take care of
needs; the second item referred to the quality of families’ relationships with service providers;
and the third referred to families’ overall feeling of control over their youth’s MHA care.
Descriptive statistics and normality plots indicate that this variable is normally distributed.
Table 15 presents the descriptive statistics for item-specific and total scores.
Table 15. Descriptive statistics for the “m-BCFQoLS scale”
Variable: m-BCFQoLS scale (20 items; n=131)
Item Mean Median SD Skew Kurtosis K-S
Your family enjoys spending time together
(/5) 3.76 4.0 0.88
- - -
61
Your family has the supports you need to
relieve stress (/5) 3.43 4.0 1.03
- - -
Your family helps each other out with
schoolwork, tasks or activities (/5) 3.79 4.0 0.86
- - -
Your family talks openly with each other
(/5) 3.66 4.0 0.95
- - -
Your family members get along with each
other (/5) 3.69 4.0 1.04
- - -
Your family members have time to pursue
their own interests (/5) 3.98 4.0 0.92
- - -
Your family solves problems together (/5) 3.50 4.0 0.93 - - -
Your family members support each other
to accomplish goals (/5) 3.79 4.0 0.89
- - -
Your family members show that they love
and care for each other (/5) 4.15 4.0 0.91
- - -
Your family has outside help available to
take care of the special needs of all family
members (/5)
3.37 3.0 1.14 - - -
Adults in your family feel they are able to
parent the youth in your family (/5) 3.50 4.0 1.02
- - -
Your family gets medical or other care
when needed (/5) 4.10 4.0 0.94
- - -
Your family is able to handle life’s ups
and downs (/5) 3.71 4.0 0.90
- - -
Adults in your family have time to take
care of the individual needs of every youth
(/5)
3.73 4.0 0.98 - - -
Your family’s youth with a mental health
and/or addiction problem has support to
accomplish goals at school and/or in the
workplace (/5)
3.51 4.0 1.20 - - -
62
Your family’s youth with a mental health
and/or addiction problem has support to
make friends (/5)
3.87 4.0 0.95 - - -
Your family has good relationships with
those who provide services and supports
to your family members (/5)
3.25 3.0 1.20 - - -
Your family feels in control over the care
of your youth with a mental health and/or
addiction problem (/5)
2.97 3.0 1.25 - - -
Your family feels more hopeful for the
future of your youth with a mental health
and/or addiction problem (/5)
3.37 4.0 1.22 - - -
Your family feels your overall quality of
life is better (/5) 3.57 4.0 1.07
- - -
m-BCFQoLS total score (/100) 72.73 73.0 14.30 -0.3 -0.6 0.064
Scale: 1=Very dissatisfied, 2=Dissatisfied, 3=Neither, 4=Satisfied, 5=Very satisfied
6.3.3.3 Service satisfaction
As discussed earlier (Section 4.3.3), the NAVSAT is the FNP’s in-house satisfaction measure
that captures both proximal satisfaction with the navigation services received, and more distal
but still relevant satisfaction with the services to which they were navigated. One of the research
questions identified by the program team was, “Overall, are clients of the Family Navigation
Project satisfied with the services they receive?”
6.3.3.3.1 Satisfaction with navigation services
The “NAVSAT total score” was a continuous total score out of 19 that reflected satisfaction with
the navigation services received based on the sum of three ordinal outcome variables: two seven-
point items and one five-point item representing likelihood of recommendation, general
Navigator helpfulness, and overall service satisfaction. The NAVSAT is highly specific to the
program and underwent extensive validation. As such, the three items included in the total score
were simply summed to preserve the integrity of the items as they were originally written, rather
than calculating a weighted total score. What matters in the current study, particularly in terms of
63
modeling the impact of mechanisms on satisfaction, is the relative difference in total score
between individuals.
Overall, total scores were very high (mean = 15.9, median = 18.0, SD = 3.9). Normality statistics
indicated the distribution is not normal (K-S = 0.238, p=.000); like the mechanisms, it was
significantly negatively skewed and leptokurtic. This was confirmed by a boxplot with high
values and a long tail, and a curvilinear pattern on the Q-Q plot similar to those observed with
the mechanism variables. This was expected given both the sample’s high mechanism scores and
the general tendency for satisfaction measures to yield positive ratings.
Table 16 presents the statistics for item-specific and total scores.
Table 16. Descriptive statistics for the “NAVSAT total score scale”
Variable: NAVSAT total score scale (3 items; n=130)
Item Mean Median SD Skew Kurtosis K-S
In general, how helpful did you find your
Navigator? (/7) 5.76 6.0 1.49 - - -
In general, how satisfied are you with the
Family Navigation Project? (/7) 5.80 7.0 1.57 - - -
How likely are you to recommend this
service to family and friends? (/5) 4.35 5.0 1.01 - -
-
NAVSAT total score (/19) 15.90 18.0 3.9 -1.4* 1.2* 0.238**
Overall, how satisfied are you with the
services to which you were referred?
(/7)
5.0 5.0 1.7 -0.6* -0.2 0.188**
5-point scale: 1=Very unlikely, 2=Unlikely, 3=Not sure, 4=Likely, 5=Very likely
7-point scale: 1=Extremely dissatisfied or unhelpful, 2=Dissatisfied or unhelpful, 3=Fairly dissatisfied or unhelpful, 4=Neither, 5=Fairly satisfied or helpful, 6=Satisfied or helpful, 7=Extremely satisfied or helpful
*Statistic is more than twice its standard error **p<.001
6.3.3.3.2 Satisfaction with referred services
The second outcome of interest derived from the NAVSAT tool was an outcome variable for
satisfaction with referred services. This was a single seven-point item with a mean and median of
64
5.0 (SD = 1.7), which was also not normally distributed (K-S = 0.188, p=.000). Like the
NAVSAT total score and satisfaction measures in general, the distribution of scores was highly
negatively skewed. However, there was considerably more variability and a significantly greater
proportion scoring the middle value (which equates to “Neither satisfied nor dissatisfied”),
reflected in the fact that this distribution was not at all leptokurtic like the NAVSAT total score
was. This was also evident when examining the boxplot, which showed a much wider spread of
scores with a short upper tail and a very long lower tail. This was not unexpected, since the FNP
can directly manage the ways in which they provide navigation services, but cannot directly
influence the services to which they refer.
Statistics for this item are included in Table 16 above.
The descriptive analysis of the satisfaction measures in particular answered the research
questions of whether, overall, families were satisfied with the services they received. The results
suggested that families in this sample were highly satisfied with both the navigation services
received, and the services to which they were referred. Later in the analysis, the influences of
context and family experience on all outcomes of interest is evaluated.
6.4 Correlational analysis
6.4.1 Overview of approach
As discussed earlier (see Sections 2 and 5.7), RE is a theory-driven framework operationalized
through the use of C-M-O configurations, which are essentially buildable, testable hypotheses
about the relationships between context, mechanism and outcome variables (Byng, et al., 2005;
Pawson & Tilley, 1997). To arrive at C-M-O configurations, select dyads (i.e. C-M, C-O, M-M,
and M-O) were tested from the dataset according to hypotheses generated from the conceptual
framework and program theory (see Chapter 2 and Appendix B). Testing select C-M and C-O
dyads was a process primarily used to provide further rationale for the hypothesized covariates to
be included in the inferential models of each outcome variable, namely by confirming that the
proposed variables were linearly related to the mechanisms and outcomes of interest. Secondly,
M-M and M-O dyads were tested to provide evidence used to satisfy initial inferential modeling
assumptions.
65
The following section is organized according to the two functions outlined above. Spearman’s
rho was used for ordinal and skewed continuous variables, and the Phi coefficient was calculated
for correlations between two dichotomous variables. To increase statistical reliability, no
variables with cell counts less than 10 were included in correlational analyses. For multiple
response variables, including types of MHA concerns and reasons for contact, numerous
categories were excluded due to small sample size. The significance of correlations was
interpreted at an alpha level of 0.05.
The dataset consisted of many variables that are generally highly interactive. With the number
and type of interrelated variables tested and a relatively modest sample size, correlation
coefficients were expected to be modest in effect size. It is also possible that type 1 error was
inflated. However, a decision was made by the PI in consultation with a statistician to forgo
adjustment for multiple tests in order not to risk underestimating the importance of contextual
factors that are known to be significant for families’ help-seeking experiences and outcomes.
This is a decision that is validated in the literature, which suggests such adjustments (Bonferroni
and false discovery rates, for example) are often far too conservative for real world application,
particularly in social sciences (Armstrong, 2014; Cabin & Mitchel, 2000; Curtin & Schulz, 1998;
Sirotich & Durbin, 2014).
6.4.2 C-M and C-O dyads: Selection of covariates for inferential modelling
The first function of the correlational analysis was to confirm whether or not the hypothesized
covariates for inferential modeling were, in fact, linearly related to the outcomes of interest. This
process also helps to answer the research questions related to which contextual factors matter for
each of the proposed mechanism and outcomes. As discussed earlier (Sections 2.3.1 and 4.1),
select context variables that typically would be expected to influence outcomes were tested for
their relationships with both mechanisms and outcomes.
Hypothesized contextual variables of influence included age, since different service systems
serve different age groups; acuity or youth improvement (i.e. clients whose youth improved were
expected to report better outcomes); concurrent addiction, since mental health and addiction
services are provided by separate sectors; type(s) of MHA concern(s), since some conditions are
inherently more difficult to treat, and since different conditions have different resources
available; and reason for contact, specifically whether clients were seeking a psychiatrist referral
66
(expected to be positively related to outcomes, given the FNP’s ability to facilitate referrals and
individually match youth with particular treatment providers) or residential treatment, which was
expected to negatively influence outcomes due to the widely acknowledge lack of available
residential services. Other reasons for contact that reflected service gaps like day treatment and
aftercare did not have sample sizes larger enough (>10) to be included in inferential analyses.
Lastly, legal involvement in cases was expected to negatively influence both mechanisms and
outcomes due to the complexity associated with navigating justice system requirements.
The FNP was designed to provide families with needs-based help in overcoming perceived
contextual barriers to care by offering them highly accessible, continuous navigation that
specifically prioritized family involvement. If it is true that these three service mechanisms –
accessibility, continuity of care and family involvement – are what enable the FNP to help
families overcome perceived barriers, then correlational analyses should reveal largely negligible
relationships between clients’ contexts and their mechanism scores. The anticipated exception
was related to the extensiveness of current program (FNP) use. For example, clients who were
relatively more engaged with the program, as determined by the number of documented contacts
with a Navigator, were expected to inherently experience greater accessibility, continuity of care,
and family involvement throughout the process.
The results of the correlational analysis of select C-M and C-O dyads are presented in Table 17,
which follows. With regard to C-M dyads, results were generally as expected. Very few
contextual variables were significantly associated with clients’ mechanism scores; the same three
context variables were found to be significantly associated with all three mechanism variables.
First, as expected, the number of contacts with a Navigator was significantly positively
associated with accessibility (r=.24, p=.005), continuity of care (r=.24, p=.005) and family
involvement scores (r=.21, p=.016). The other two influential context variables were related to
type of MHA concern. Interestingly, suicidality concerns were significantly positively associated
with clients’ scores on all three mechanisms (accessibility: r=.27, p=.001; continuity of care:
r=.20, p=.023; and family involvement: r=.24, p=.005). In contrast, OCD concerns in particular
were negatively associated with clients’ scores on all three mechanisms (accessibility: r=-.29,
p=.001; continuity of care: r=-.21, p=.016; and family involvement: r=-.23, p=.007). Legal
involvement was negatively related with each of the mechanisms but the relationship did not
approach significance. In general, with the exception of suicidality and OCD, mechanism scores
67
were not significantly associated with context variables that typically influence delivery of
mental health and addiction care.
Compared to C-M dyads, relationships between contexts and outcomes (C-O dyads) were more
readily identified but not to the extent expected (i.e. 100.0% of hypothesized relationships).
While most variables trended in the expected direction, only approximately 30.0% of the
hypothesized C-O dyads were found to be statistically significant. With regard to the two FES
subscales, the same three contextual variables were associated with both; in this sample, older
youth age was associated with lower empowerment scores on both FES Family and FES Service-
seeking subscales (r=-.24, p=.006; and r=-.27, p=.002, respectively). Concurrent addiction
concerns were also associated with lower scores on both empowerment subscales (FES F: r=-.20,
p=.021; FES S: r=-.26, p=.00d). Again, of note, suicidality concerns were positively associated
with clients’ empowerment scores (FES F: r=.21, p=.000; FES S: r=.33, p=.000). Legal
involvement was again negatively, but not significantly, related with empowerment subscale
scores.
Similar to family empowerment, concurrent addiction concerns were also associated with lower
cross-sectional m-BCFQoLS scores (r=-.26, p=.003). Personality disorder concerns were also
negatively associated with m-BCFQoLS scores (r=-.18, p=.036). In contrast, depression concerns
were positively associated with m-BCFQoLS scores (r=.20, p=.025). Consistent with the other
suicidality-related dyads noted above, reports of suicidality concerns were positively associated
with clients’ m-BCFQoLS scores (r=.27, p=.002). In the case of FQOL, legal involvement was
significantly negatively associated with m-BCFQoLS scores (r=-.28, p=.001).
68
Accessibility total score
Continuity of care total
score
Family involvement
score
FES Family
total score
FES Service-seeking
m-BCFQoLS total score
NAVSAT total score
SRS score
Number of contacts with Navigator
.244** .241** .208** -.016 .085 .100 .229** .104
Age -.050 -.010 -.058 -.235** -.268** -.056 -.016 -.019
Perceived youth improvement
-.115 -.167 -.116 -.017 -.056 -.037 -.146 -.205*
Concurrent addiction -.030 -.026 -.106 -.199* -.255** -.260** -.157 -.127
Type of mental health concern
Depression .132 .160 .113 .076 .045 .196* .186* .195*
Anxiety -.025 -.059 -.095 -.129 -.082 -.128 -.007 .027
ADD/ADHD .087 .033 .046 -.135 -.045 -.144 -.003 -.041
Suicidality .274** .196** .241** .307** .327** .272** .287** .273**
OCD -.289** -.207** -.234** -.055 .109 -.156 -.247** -.254**
Eating disorder
-.011 .054 .059 -.082 -.115 .018 .086 .100
Personality disorder
-.108 -.166 -.113 -.045 .007 -.184* -.121 -.113
Reason for contact
Psychiatrist referral
.140 .133 .112 .035 .033 .055 .198* 199*
Residential treatment
.039 -.006 -.047 -.141 .066 -.137 -.128 -.150
Legal involvement -.040 -.145 -.142 -.138 -.029 -.283** -.227** -.180*
*p<.05 **p<.01
Table 17. Results of a correlational analysis between select C-M and C-O dyads
69
C- dyads related to satisfaction outcomes followed similar patterns. Satisfaction with navigation
services (NAVSAT total score) was, as expected, positively and significantly associated with
number of contacts with a Navigator (r=.23, p=.009). Satisfaction with navigation was also
positively associated with reported depression and suicidality concerns (r=.19, p=.034; and
r=.29, p=.001, respectively). In contrast, OCD was negatively associated with navigation
satisfaction (r=-.25, p=.005). Whereas concurrent addiction concerns were significantly
negatively associated with empowerment and FQOL outcomes, the negative association between
concurrent addiction concerns and navigation satisfaction did not reach significance (r=-.16,
p=.074). Surprisingly, family-perceived youth improvement was not significantly correlated with
navigation satisfaction. In fact, the data suggested a counter-intuitive inverse relationship
between improvement and satisfaction (r=-.15, p=.098); this should be interpreted in the context
of the fact that the majority of the sample (64.2%) indicated their youth’s status had stayed the
same, with only 9.0% (or 12 families) indicating their youth’s status had worsened.
As expected, the need for a psychiatrist referral was positively associated with navigation
satisfaction (r=.20, p=.024), whereas there was a trend for the need for residential treatment to be
negatively associated with navigation satisfaction (r=-.13, p=.148). Legal involvement was, as
anticipated, significantly negatively associated with navigation satisfaction as well (r=-.23,
p=.009).
Satisfaction with referred services were generally associated with the same contextual variables
as satisfaction with navigation services at comparable effect sizes and levels of significance.
Satisfaction with referred services was positively associated with depression (r=.20, p=.027) and
suicidality concerns (r=.27, p=.002), whereas scores were negatively associated with OCD
concerns (r=-.25, p=.004). Clients seeking a psychiatrist referral had positively associated
satisfaction with referred service scores (r=.20, p=.024). Again, counter-intuitively, perceived
youth improvement was found to counter-intuitively be negatively, and this time significantly,
associated with satisfaction with referred services scores (r=-.21, p=.020). Lastly, legal
involvement was also significantly negatively associated with satisfaction with referred services
(r=-.18, p=.041).
Based on the correlation results of C-M and C-O dyads, predictors and covariates that were
confirmed to be significantly linearly related to the mechanisms and outcomes were selected for
70
inclusion in subsequent inferential models and are summarized according to outcome of interest
in Table 18 below. The exceptions were that a decision was made by the PI to preserve
concurrent concerns in the model of navigation satisfaction both because it trended near
significance, but more importantly, because it had a strong theoretical rationale for inclusion.
Similarly, youth improvement was maintained in the model of satisfaction with referred services
despite the counter-intuitive significant inverse relationship between improvement and
satisfaction observed.
Table 18. Summary of covariates and predictors for modeling by outcome
Outcome
domain
and
variables
Family empowerment Family
quality of life Satisfaction
FES F FES S m-
BCFQoLS NAVSAT SRS
Mechanisms
Accessibility
Continuity
Family
involvement
Accessibility
Continuity
Family
involvement
Accessibility
Continuity
Family
involvement
Accessibility
Continuity
Family
involvement
Accessibility
Continuity
Family
involvement
Contextual
covariates
No. contacts
with Navigator
No. contacts
with Navigator
No. contacts
with Navigator
No. contacts
with Navigator
No. contacts
with Navigator
Suicidality Suicidality Suicidality Suicidality Suicidality
OCD OCD OCD OCD OCD
Concurrent
concerns
Concurrent
concerns
Concurrent
concerns
Concurrent
concerns
Concurrent
concerns
Age Age Depression Depression Depression
Personality
disorder Psychiatrist Psychiatrist
Legal
involvement
Legal
involvement
Legal
involvement
Youth
improvement
71
6.4.3 M-M, M-O and O-O dyads: Satisfaction of modeling assumptions
In addition to examining relationships involving context variables, associations within and
between mechanisms and outcomes were also tested. The primary purposes of this step were to
satisfy modeling assumptions by a) ensuring mechanisms and outcomes were linearly related as
expected; and b) to identify the magnitude of risk of multicolinearity by comparing the relative
strength of M-M dyads to M-O dyads, since higher correlations within mechanisms than between
mechanisms and outcomes is an indicator of a high risk of multicolinearity, if all variables were
to be used in modelling (Tabachnick & Fidell, 2007). The results of this correlational analysis
are presented in Table 19.
Table 19. Spearman’s rho correlation coefficients for mechanism and outcome variables
Acc
essi
bil
ity
Conti
nuit
y
Fam
ily
involv
emen
t
FE
S F
FE
S S
m-B
CF
QoL
S
NA
VS
AT
Accessibility 1
Continuity 0.822 1
Family
involvement 0.774 0.828 1
FES F 0.303 0.301 0.407 1
FES S 0.217 0.223 0.325 0.642 1
m-BCFQoLS 0.411 0.444 0.463 0.539 0.367 1
NAVSAT 0.735 0.772 0.762 0.267 0.225 0.418 1
SRS 0.591 0.594 0.560 0.291 0.201 0.459 0.689
*All correlations have corresponding p-values < 0.001
6.4.3.1 M-M dyads
The results suggested that client scores on all three mechanisms – accessibility, continuity of
care, and family involvement - were significantly and strongly correlated with each other,
yielding three M-M dyads each coefficients ranging from 0.77 to 0.83 and corresponding critical
72
alpha values of 0.000. Theoretically, high correlation between the mechanisms was anticipated
as these service characteristics complement each other in practice. Although these are three
distinct concepts with distinct measures in the literature, and every effort was made to ensure the
corresponding survey items had high face and content validity, it is possible that differences in
wording on the selected measures were perceived by clients as relatively minute, thus yielding
similar scores across the three measures. It is also possible that they were able to make the
distinction and were simply highly satisfied with all three mechanisms.
6.4.3.2 M-O dyads
The program theory presupposed that provision of accessible, continuous, family-inclusive
services would encourage family empowerment, improve FQOL, and ensure service satisfaction.
Since family empowerment and FQOL are theoretically more distal to the provision of certain
service characteristics than satisfaction with said service is, stronger correlations were expected
between mechanisms and satisfaction outcomes than with empowerment or FQOL outcomes.
Table 30 suggests that all three mechanisms were most strongly associated with NAVSAT score,
with coefficients somewhat consistent across the three, ranging from 0.74 to 0.78 and alpha
values of .000. The same was true for referred service satisfaction (0.56 < r < 0.59, p<0.001).
The associations between mechanisms and family empowerment and FQOL were more
moderate, with coefficients ranging from 0.22 to 0.46 (p<0.001).
6.4.4 Implications for modeling
Referring to back to Table 17, it was observed that there were no cases in which the correlations
between mechanisms and outcomes higher than the correlations within mechanisms. Higher
correlation between individual mechanisms than between mechanisms and outcomes is a strong
indication of likely multicolinearity, which violates a key assumption of inferential modeling
(Tabachnick & Fidell, 2007). Multicolinearity does not significantly impact overall model fit or
the actual parameter estimates, but is a concern because it decreases statistical power and at high
levels, can increase the variance of the model’s parameter estimates to the extent that they can
become unpredictable (so much so that signs can change) and problematic to interpret
(Studenmund, 2010).
73
The results of the M-M and M-O correlational analyses indicated that, due to high risk of
multicolinearity, all three mechanism variables could not be individually reliably included in the
modeling of the outcome variables. To mitigate this issue, factor analysis was proposed as a data
reduction technique and is discussed in the next section.
6.5 Factor analysis
As identified above, the correlational analyses revealed that the associations between
mechanisms were always stronger than those between mechanisms and outcomes. This was
interpreted as evidence of a high risk of multicolinearity, which violates an assumption of
inferential modeling and can significantly undermine the interpretation of individual effects.
Despite the fact that the mechanisms were theorized to be three distinct concepts, the dataset did
not reflect this. Instead, it revealed similarly high scores, distributions, and correlation
coefficients amongst the three measures. Although they may have been genuinely independent
ratings, based on the current measures, they remained very likely to co-occur, so it was
considered a reasonable statistical solution to use a single variable representative of the
combined effects of navigation for modeling purposes (Yong & Pearce, 2013).
For this reason, a Principal Components Analysis was performed on the three mechanism
variables. Typically, large sample sizes greater than 300 are preferred for factor analyses,
however for datasets in which factor loadings are expected to be very high (>0.80), smaller
sample sizes are sufficient (Tabachnick & Fidell, 2007). In addition, only three items were
included, which more than satisfies Nunnally’s widely cited 10:1 cases per item rule (1978).
A Principal Components Analysis was performed on the three mechanism variables, the results
of which are presented below in Table 20. Principal Components was selected as the method of
extraction with a Direct Oblimin rotation for its primary use in data reduction and subsequent
modeling. A principal component (called “navigation mechanism”) with an eigenvalue of 2.714
was extracted; it explained 90.5% of the variance in scores across the three mechanisms. Since
only one component with an eigenvalue > 1 was extracted, rotation was not performed.
Communalities for all three variables were very high (0.885 < h2 < 0.927), indicating that all are
well represented by the extracted component. Component loadings for accessibility, continuity
and family involvement were 0.941, 0.963, and 0.950, respectively, suggesting each of the three
variables were very highly correlated with the extracted principal component. A scree plot
74
confirmed the adequacy of a single component. The Kaiser-Meyer-Olkin statistic (0.763) is
relatively close to 1.0 and greater than the suggested 0.5 minimum, confirming sampling
adequacy; and the results of Bartlett’s Test of Sphericity were also significant (p=0.000),
confirming that the correlation matrix is not an identity matrix. Lastly, the determinant of the
correlation matrix was 0.052, which is significantly greater than 0.00001 and indicates that
multicolinearity is no longer a concern (Field, 2009; Tabachnik & Fidell, 2007; Yong & Pearce,
2013).
Table 20. Results of a Principal Components analysis for mechanism variables
Mechanism
variables
Communalities
(h2) C
om
pon
ents
Eig
en
valu
es
% o
f varia
nce
Com
pon
ent
load
ings
KM
O
Bartlett’s
Dete
rm
inan
t
Init
ial
Extr
acti
on
Accessibility 1.000 0.885 1 2.714 90.458 0.941
0.763 0.000 0.052
Continuity of
care 1.000 0.927 2 0.178 5.922 0.963
Family
involvement 1.000 0.902 3 0.109 3.621 0.950
The results of the Principal Components analysis suggested that a single component score in
place of the three mechanism variables was highly suitable for use in subsequent modeling. As
such, Bartlett’s factor (i.e. component) scores were calculated for each case and saved as new
variables in the dataset. Bartlett’s approach was most appropriate for the data since the manner in
which the scores were calculated (using maximum likelihood estimates) minimized the error
across the three mechanism variables and produced unbiased estimates that were most likely to
represent the true component scores (Yong & Pearce, 2013). Bartlett’s scores thus represented
the overall “navigation mechanism.” Descriptive statistics for the Bartlett’s scores are presented
in Table 21 below.
75
Table 21. Descriptive statistics for the Bartlett Factor Score for “navigation mechanism”
Variable N Mean Median SD Min Max Skew Kurtosis K-S
Bartlett Factor
Score for
“navigation
mechanism”
134 0.00 0.36 1.0 -2.74 0.84 -1.32* 0.71 0.199**
*Statistic is more than twice its standard error **p<.01
To confirm the Bartlett’s scores were appropriately correlated with the outcome variables, a
quick correlational analysis was performed (Table 22). The results mimicked the patterns and
effect sizes observed when the mechanism variables were tested individually, confirming the
appropriateness of Bartlett’s scores for use in subsequent modeling. Therefore, the three
individual mechanisms variables were replaced with a representative component score –
“navigation mechanism” – in the inferential models that follow this section.
Table 22. Correlation coefficients for the Bartlett Factor Score with outcomes
Outcome
Family
empowerment FQOL Satisfaction
Variable FES F FES S m-
BCFQoLS NAVSAT SRS
Bartlett Mechanism Score
(“Navigation mechanism”) 0.368 0.296 0.458 0.812 0.623
* All correlations have corresponding p-values < 0.001
6.6 Inferential modelling
This study was guided by three broad research questions:
1. a) Who is the Family Navigation Project serving?
b) Is the Family Navigation Project reaching its target population?
c) Overall, are families satisfied with the services they received?
76
2. a) Do families perceive the Family Navigation Project to be providing accessible,
continuous, family-inclusive care?
b) How does context influence perceived experience of the program?
3. a) Do families who perceive the Family Navigation Project as accessible, continuous and
family-inclusive experience better outcomes in terms of family empowerment, family
quality of life, and service satisfaction?
b) How does context influence these outcomes?
The first two research questions were addressed through descriptive and correlational analyses.
The third research question required an inferential approach that statistically tested the impact of
the navigation mechanism on each of the three outcomes of interest in the presence of the
associated contextual factors identified in the preceding correlational analyses.
6.6.1 Overview of statistical approach
Recall from Chapter 2 that RE is a theory-driven framework, and one of the goals of the
correlational analyses was to identify dyads and triads of variables that would help explain the
influence of context and mechanisms on outcomes. The next step in the framework was to test
these C-M-O configurations for statistical significance in a series of generalized linear models
intended to support the proposed program theory. Since only one model per dependent variable
was tested, adjustments for multiple analyses were not required and significance was confidently
interpreted at a critical alpha level of 0.05.
The study was designed to yield a set of continuous dependent variables for use in generalized
linear modelling. This was the preferred approach as generalized linear models allow for
specification of the probability distribution of the dependent variable, and at least some of the
dependent variables in this study were anticipated to be significantly skewed (McCullagh &
Nelder, 1989). The results of the descriptive analysis did indicate that while the FES Family and
FES Service-seeking subscales and m-BCFQoLS scores could be fit with a normal distribution,
the two satisfaction outcomes were very highly negatively skewed. As such, generalized linear
models were selected and run for all dependent variables as they can fit different distributions
while producing consistent statistical output across outcomes. Different distributions are fit using
a canonical link function, which relates the mean of the response variable to the linear predictors
77
in the model. Identity links are the most basic link function and can be used in any case where
transformation is not required, which is most commonly in the case of normal and gamma
distributions (McCullagh & Nelder, 1989). Other link functions are transformative; examples
include log links, for use when the response variable cannot be negative (e.g. Poisson counts),
and logit and probit links, for use when data are binary.
For the three normally distributed outcome variables (i.e. FES Family, FES Service-seeking and
m-BCFQoLS scores), generalized linear models can be used by specifying a normal distribution
and an identity link. This procedure produces identical coefficients to those produced by
traditional linear regression; the main difference is in the specific significance tests used (F-test
for linear regression, Wald Chi-Square for generalized linear models) (McCullagh & Nelder,
1989).
Descriptive analyses of the normality statistics, Q-Q and P-P plots, as well as literature on the
topic, suggested that the satisfaction outcomes were highly negatively or left-skewed and could
not be reliably fit with a normal distribution. For generalized linear modelling of non-normal
outcomes (i.e. satisfaction scores), more appropriate distributions can be specified. Literature
suggests satisfaction data can be fit with a beta distribution, which can take on many shapes
provided appropriate shape parameters are applied; however, shape parameters can be difficult to
estimate (Lindsey & Jones, 1998; McCullagh & Nelder, 1989). The second option was to apply a
gamma distribution, which is the inverse of a beta distribution; visually, it looks like a highly
positive or right-skewed distribution (Hardin & Hilbe, 2007; McCullagh & Nelder, 1989). As
such, to apply a gamma distribution to negative or left-skewed satisfaction scores, scores would
need to be inverted. Gamma distributions typically employ either an identity or log link. For the
current study, a gamma distribution with an identity link was determined to be most appropriate
by the PI, a choice that was confirmed as appropriate following consultation with a statistician at
the University of Toronto.
This latter approach (i.e. applying a gamma distribution) was considered preferable over risking
inappropriately specified shape parameters for a beta distribution; and also preferable over
categorizing the variables and using a logistic regression or gamma distribution with a log link
for several reasons. First, categorizing negates the design efforts to yield a continuous dependent
variable, which inherently has more information. Second, the heavily skewed distribution means
78
the naturally emerging categories would be highly unbalanced unless condensed into even larger
categories, losing even more information. Third, categories with smaller sample sizes lack the
statistical power needed to include the proposed predictors. Lastly, an identity link was selected
over a model using the log link because the dependent variable was positive and continuous and
did not need to be further log-transformed, which would unnecessarily complicate interpretation
of the regression coefficients for categorical variables with more than one level (Lindsey &
Jones, 1998). Therefore, following a statistical consultation, inverse scores were calculated by
subtracting the original score from the denominator plus one, and then saved as new variables.
These new variables then served as the dependent variable for satisfaction, or dissatisfaction; that
is, use of the inverse score means the applied generalized linear model models dissatisfaction.
For these models, a gamma distribution was specified along with an identity link.
There are several important differences to note in the accompanying statistics produced by
generalized linear models (Hardin & Hilbe, 2007; McCullagh & Nelder, 1989). First,
standardized beta coefficients were not available as they would have been through the linear
regression procedure so effect sizes of individual predictors are less easily compared. Second,
whereas the linear regression procedure calculates variance explained (R2), most of the goodness
of fit statistics produced by the generalized linear model procedure are less easily interpreted in
this particular study. For example, the procedure produces deviance and Pearson chi-square
statistics, which are known to be sensitive to responses equal to zero (that occur often in dummy
coded categorical covariates, such as the ones included in this study) and empty cells (that often
occur with continuous covariates, such as the ones included in this study). Instead, the selected
approach was to report the results of the Omnibus Test of model fit for each dependent variable.
This procedure tested the complete fitted model against the null model using a likelihood ratio
chi-square and corresponding critical alpha value for significance testing. Models for each
dependent variable were then compared by the magnitude of the likelihood ratio chi-square
value, where higher values indicate better model fit.
6.6.2 Testable hypotheses
At the outset of this study, several broad hypotheses were proposed in relation to each of the
outcomes of interest (Table 2, Section 3.1). Following measurement specification and the
confirmation of covariates for each outcome in Section 6.4.2, hypotheses could then be further
79
specified prior to modelling. The specific hypotheses for each outcome variable are presented in
Table 23 below.
Table 23. Dependent variables and corresponding hypotheses
Dependent variable Specific hypotheses
FES Family
H1: Clients’ “navigation mechanism” score positively predicts clients’
FES Family scores even when selected contextual covariates are
included
FES Service-seeking
H1: Clients’ “navigation mechanism” score positively predicts clients’
FES Service-seeking scores even when selected contextual covariates
are included
m-BCFQoLS
total score
H1: Clients’ “navigation mechanism” score positively predicts clients’
m-BCFQoLS total scores even when selected contextual covariates are
included
Inverse NAVSAT
total score
H1: Clients’ “navigation mechanism” score negatively predicts
clients’ NAVSAT total scores even when selected contextual covariates
are included
Inverse Satisfaction
with referred
services (SRS) score
H1: Clients’ “navigation mechanism” score negatively predicts
clients’ satisfaction with referred services (SRS) scores even when
selected contextual covariates are included
As a reminder, Table 24 presents a summary of the covariates selected for inclusion, which were
grounded in the program theory and confirmed for inclusion by the results of the correlational
analysis (refer back to Section 6.4.2). “Navigation mechanism” factor score was the primary
predictor for all outcomes. Contextual covariates were included because they were expected to
influence the mechanism and/or outcomes. However, the navigation mechanism was
hypothesized to maintain an independent effect on the outcomes of interest even when selected
contextual covariates were included.
80
Table 24. Covariates selected for inclusion in modeling by outcome
Dependent
variable
Family empowerment Family
quality of life Satisfaction
FES F FES S m-
BCFQoLS
Inverse
NAVSAT
total score
Inverse
SRS score
Mechanism
(i.e. predictor)
“Navigation
mechanism”
factor score
“Navigation
mechanism”
factor score
“Navigation
mechanism”
factor score
“Navigation
mechanism”
factor score
“Navigation
mechanism”
factor score
Contextual
covariates
No. contacts with
Navigator
No. contacts
with Navigator
No. contacts
with Navigator
No. contacts
with Navigator
No. contacts
with Navigator
Suicidality Suicidality Suicidality Suicidality Suicidality
OCD OCD OCD OCD OCD
Concurrent
concerns
Concurrent
concerns
Concurrent
concerns
Concurrent
concerns
Concurrent
concerns
Age Age Depression Depression Depression
Personality
disorder
Psychiatrist
referral
Psychiatrist
referral
Legal
involvement
Legal
involvement
Legal
involvement
Youth
improvement
Results are discussed by outcome below.
6.6.3 Models of family empowerment
Recall that the third initial guiding research questions in this study asked, “What is the impact of
the Family Navigation Project on family empowerment and quality of life? What components are
most important, for whom, in which circumstances, and why?” Descriptive and correlational
analyses led to the identification of predictors of empowerment and covariates of mechanisms,
which were used to form specific hypotheses about the context and mechanism variables that
best predict FES Family and FES Service-seeking scores. The two hypotheses related to these
two dependent variables were tested in generalized linear models. Results are reported below.
81
6.6.3.1 FES Family subscale
The specific hypothesis related to family empowerment at the family level was:
H1: Clients’ “navigation mechanism” scores positively predict FES Family scores even when
contextual covariates are included.
Navigation services are designed to help clients overcome real and perceived barriers to care,
which in this model, are represented by the contextual factors found to be most strongly
associated with the mechanism and outcome (FES Family): number of contacts with a navigator,
OCD, suicidality, concurrent concerns, and age. If navigation was provided as intended, the
coefficient for “navigation mechanism” score should be statistically significant even when the
associated context variables are included in the model. Results are presented in Table 25 below.
Table 25. Parameter estimates for the dependent variable “FES Family score”
DV – FES Family score (n=134)
Parameter B SE CI < 95%< CI Wald
Chi-square df p-value
Intercept 44.19 2.87 38.56, 49.82 236.38 1 .000
Mechanism factor score 2.06 0.56 0.96, 3.16 13.40 1 .000*
No. contacts with Navigator -0.02 0.04 -0.09, 0.05 0.39 1 .535
Suicidality 3.04 1.52 -0.05, 6.02 3.98 1 .046*
OCD -0.02 1.57 -3.10, 3.07 0.00 1 .992
Concurrent concerns -2.26 1.10 -4.41, -0.10 4.21 1 .040*
Age 12-15 5.53 2.59 0.46, 10.61 4.57 1 .033*(.178)a
16-19 1.57 1.73 -1.83, 4.98 0.82 1 .365(178)a
20-24 1.01 1.67 -2.27, 4.28 0.36 1 .546(.178)a
25+ (ref)
*p<.05 aIndicates significance level of variable main effect
A generalized linear model specifying a normal distribution and an identity link was performed
where FES Family subscale score was the dependent variable, navigation mechanism score was
82
the primary predictor, and number of contacts with a Navigator, suicidality, OCD, concurrent
concerns and age were all included as covariates. The results suggested that navigation
mechanism score significantly, positively predicted FES Family score (B=2.06, SE=0.56, 0.96 <
95% CI < 3.16; Wald 2=13.40, p=0.000) even when the aforementioned context variables were
included. That is, for every unit increase in navigation mechanism score, FES Family scores
increased by 1.97 units. A relatively small standard error and narrow confidence intervals
suggested this coefficient estimate was statistically reliable.
Several context variables were also significantly predictive of the dependent variable. Following
from the results of the correlational analysis, the results of the inferential model suggested that
suicidality significantly, positively predicted individual FES Family scores in this sample
(B=3.04, SE=1.52, -0.05 < 95% CI < 6.02, Wald 2=3.98, p=0.046). In contrast, reported
concurrent MHA concerns were found to significantly, negatively predict FES Family scores in
this sample (B=-2.26, SE=1.10, -4.41 < 95% CI < 0.10, Wald 2=4.21, p=0.040). The main
effect of age was not significantly predictive, meaning as a whole, age does not significantly
influence family empowerment scores (Wald 2=4.92, p=0.178). The results did indicate that
relative to the reference group, which was youth aged 25+, clients with youth aged 12 to 15 were
significantly more likely to score higher on family-level family empowerment. However, only
nine youth in this sample fell into the aged 12 to 15 category.
For each variable, the Wald Chi-square statistic indicated the magnitude and significance of the
main effect size (scaled using the estimated standard error). The results suggested that the
navigation mechanism score had over three times the effect relative to the contextual predictors;
and that concurrent concerns and suicidality still modest effects - negative and positive,
respectively. In comparison, the effect sizes for OCD and number of contacts with a Navigator
were minimal in both magnitude and significance.
Overall model fit was assessed using the results of the Omnibus Test of Model Fit, which uses a
likelihood ratio chi-square test (G test) to compare the fit of two models, in this case the null
model (with no predictors) and the model fitted with the theorized predictors above. The
likelihood ratio chi square test compares the log likelihoods of both models; higher log
likelihoods (distributed chi square) indicate better model fit, and better model fit indicates a
greater proportion of explained variance in response data. If the test is statistically significant
83
(i.e. p<.05), the model fitted with predictors is confirmed to fit the data significantly better than
the null model (Hardin & Hilbe, 2007).
The Omnibus Test of Model Fit for the current model indicates that the fitted model performed
significantly better than the null model (LR 2=34.03, df=8, p=0.000); that is, when the proposed
predictors were included, the model was better able to predict individual FES Family scores than
when predictors were not included.
6.6.3.2 FES Service-seeking subscale
The specific hypothesis related to family empowerment at the level of youth service seeking was:
H1: Clients’ “navigation mechanism” scores positively predict FES Service-seeking scores even
when contextual covariates are included.
Program theory and correlational analysis confirmed a number of theorized contextual predictors
to be significantly related to the outcome, in this case, FES Service-seeking scores. As such, the
following contextual variables were included as covariates in the model where FES Service-
seeking score was the dependent variable: number of contacts with a Navigator, OCD,
suicidality, age, and concurrent MHA concerns. If navigation was provided as intended, the
coefficient for “navigation mechanism” score should be statistically significant even when the
associated context variables mentioned are included in the model. Results are presented in Table
26 and interpreted below.
Table 26. Parameter estimates for the dependent variable “FES Service-seeking score”
DV – FES Service-seeking score (n=134)
Parameter B SE CI < 95%< CI Wald
Chi-square df p-value
Intercept 48.90 2.86 43.3, 54.51 292.91 1 .000
Mechanism factor score 1.52 0.74 0.68, 2.97 4.21 1 .040*
No. contacts with Navigator 0.01 0.05 -0.83, 0.11 0.08 1 .777
Suicidality 5.45 2.01 1.15, 9.38 7.38 1 .007*
OCD 2.97 2.07 -1.10, 7.04 2.05 1 .152
84
Concurrent concerns -3.19 1.44 -6.03, -0.35 4.86 1 .027*
Age 12-15 7.61 3.41 0.92, 14.29 4.97 1
.026*
(.158)a
16-19 2.88 2.29 -1.61, 7.36 1.58 1 .209
(.158)a
20-24 1.97 2.20 -2.35, 6.28 0.80 1 .371
(.158)a
25+ (ref)
*p<.05 a Indicates significance level of variable main effect
A generalized linear model specifying a normal distribution and an identity link was performed
where FES Service-seeking subscale score was the dependent variable, navigation mechanism
score was the primary predictor, and number of contacts with a Navigator, suicidality, OCD,
concurrent concerns and age were all included as covariates. The results suggested that
navigation mechanism score significantly, positively predicted FES Service-seeking score
(B=1.52, SE=0.74, 0.68 < 95% CI < 2.97; Wald 2=4.21, p=.040) even when the aforementioned
context variables were included. That is, for every unit increase in navigation mechanism score,
FES Service-seeking score increased by 1.52 units.
In this model, several context variables were also found to be significantly predictive of FES
Service-seeking scores. As with FES Family scores, concurrent MHA concerns predicted poorer
outcomes (B=-3.19, SE=1.44, -6.03 < 95% CI < -0.35; Wald 2=4.86, p=.027), while suicidality
predicted better outcomes (B=5.45, SE=2.01, 1.15 < 95% CI < 9.38; Wald 2=7.38, p=.007).
OCD and number of contacts with a Navigator were non-significant in this model as well. The
main effect of age was again not significantly predictive (Wald 2=5.19, p=0.158); however,
again relative to the reference group, youth aged 25 and older, clients with youth aged 12 to 15
were significantly more likely to score higher. Again however, only nine clients had youth aged
12 to 15.
Comparing the magnitude of the Wald Chi-square statistics suggested that the navigation
mechanism score in this model (2=4.21) had less of an effect overall and relative to context than
in the FES Family model (2=13.40). The standard error estimates were also slightly larger and
the confidence intervals were slightly wider, suggesting the estimates within the FES Service-
85
seeking model are slightly less reliable than those in the FES Family model. However, the
Omnibus Test of model fit indicates that the fitted model still performs significantly better than
the null model (LR 2=32.34, df=8, p=0.000); that is when the proposed predictors were
included, the model was better able to predict individual FES Service-seeking scores than when
predictors were not included. The magnitude of the likelihood ratio (distributed chi square)
indicates that the model of FES Service-seeking scores was similar in overall fit to the FES
Service-seeking model.
6.6.4 Model of family quality of life
The specific hypothesis related to FQOL was as follows:
H1: Clients’ “navigation mechanism” scores positively predict m-BCFQoLS total scores even
when contextual covariates are included.
The program theory proposed the following contextual factors to influence FQOL, which were
confirmed to be related to the outcome variable (m-BCFQoLS total score) in the correlational
analysis and included in the current model as covariates: number of contacts with a Navigator,
OCD, suicidality, depression, personality disorders, concurrent MHA concerns and legal
involvement. If navigation was provided as intended, the coefficient for “navigation mechanism”
score should be statistically significant even when the covariates mentioned are included in the
model. Results are presented in Table 27 and interpreted below.
Table 27. Parameter estimates for the dependent variable “m-BCFQoLS total score”
DV – m-BCFQoLS total score (n=131)
Parameter B SE CI < 95%< CI Wald
Chi-square df p-value
Intercept 73.83 2.34 69.25, 78.14 997.22 1 .000
Mechanism factor score 4.95 1.20 2.80, 7.12 20.32 1 .000*
No. contacts with Navigator 0.04 0.07 -0.10, 0.17 0.29 1 .588
Suicidality 6.59 2.70 1.30, 11.88 5.97 1 .015*
OCD -4.06 2.84 -9.62, 1.50 2.05 1 .152
86
Concurrent concerns -4.73 2.15 -8.94, -0.53 4.86 1 .027*
Depression 3.14 2.02 -0.82, 7.09 2.41 1 .121
Personality disorder -7.55 3.53 -14.46, -0.64 4.59 1 .032*
Legal involvement -4.96 2.48 -9.81, -0.10 4.01 1 .045*
*p<.05
A generalized linear model specifying a normal distribution and an identity link was performed
where m-BCFQoLS total score was the dependent variable, navigation mechanism score was the
primary predictor, and number of contacts with a Navigator, suicidality, OCD, depression,
personality disorders, concurrent MHA concerns and legal involvement were all included
covariates. The results suggested that navigation mechanism score significantly, positively
predicted m-BCFQoLS total score (B=4.95, SE=1.20, 2.80 < 95% CI < 7.12; Wald 2=20.32,
p=.000) even when the aforementioned context variables were included. That is, for every unit
increase in navigation mechanism score, FQOL total score increased by 4.95 units.
In this model, four context variables were also significantly predictive of m-BCFQoLS total
scores. As with the FES models, concurrent MHA concerns again predicted poorer outcomes
(B=-4.73, SE=2.15, -9.62 < 95% CI < 1.50; Wald 2=4.86, p=.027), while suicidality predicted
better outcomes (B=6.59, SE=2.70, 1.30 < 95% CI < 11.88; Wald 2=5.97, p=.015). In this
model, personality disorder concerns were found to be significantly predictive of poorer FQOL
total scores (B=-7.55, SE=3.53, -14.46 < 95% CI < -0.64; Wald 2=4.59, p=.032), as was legal
involvement in cases (B=-4.96, SE=2.48, -9.81 < 95% CI < -0.10; Wald 2=4.01, p=.045).
Comparing the magnitude of the Wald Chi-square statistics suggested that the navigation
mechanism score in this model had stronger effect overall and relative to the context variables
than both FES models. The Omnibus Test of model fit also indicated that the fitted model still
performed significantly better than the null model (LR 2=63.14, df=8, p=0.000); that is when
the proposed predictors were included, the model was better able to predict m-BCFQoLS total
scores than when predictors were not included. The magnitude of the likelihood ratio (distributed
chi square) indicates that the model of FQOL performed better in overall fit compared to either
of the FES models.
87
6.6.5 Models of service satisfaction
Service satisfaction was evaluated using the FNP’s in-house navigation satisfaction tool, which
captures both satisfaction with navigation services and the services to which families were
referred. It was theorized that satisfaction with navigation services would be the most proximal
outcome of interest in this study. Modelling results would lend support to this theory if the
influence of the navigation mechanism score is significantly greater in effect size than any of the
other modelled outcomes. As discussed previously, inverse satisfaction scores were modelled
using generalized linear models with specified gamma distributions and identity links. The
dependent variable was therefore dissatisfaction with navigation services and dissatisfaction with
referred services.
6.6.5.1 Inverse NAVSAT total score
The specific hypothesis related to satisfaction with navigation services was:
H1: Clients’ “navigation mechanism” scores negatively predict inverse NAVSAT total scores
even when contextual covariates are included.
The following contextual factors were theorized to influence satisfaction and confirmed to be
related to the outcome variable in the correlational analysis: number of contacts with a
Navigator, OCD, suicidality, concurrent concerns, depression, legal involvement and reason for
contact being need for a psychiatrist. These contextual factors were included as covariates in the
current model. If navigation was provided as intended, the coefficient for “navigation
mechanism” score should be statistically significant even when the covariates mentioned are
included in the model. Results are presented in Table 28 and interpreted below whereby each
coefficient represents the influence on increasing or decreasing dissatisfaction with navigation
services.
Table 28. Parameter estimates for the dependent variable “Inverse NAVSAT total score”
DV – Inverse NAVSAT total score (n=130)
88
Parameter B SE CI < 95%< CI Wald
Chi-square df p-value
Intercept 3.99 0.32 3.36, 4.61 154.99 1 .000
Mechanism factor score -3.12 0.31 -3.70, -2.51 103.18 1 .000*
No. contacts with Navigator 0.00 0.00 -0.01, 0.00 0.88 1 .348
Suicidality -0.15 0.18 -0.51, 0.21 0.67 1 .414
OCD 0.49 0.58 -0.64, 1.62 0.73 1 .394
Legal involvement 0.61 0.30 0.02, 1.19 4.12 1 .042*
Concurrent concerns 0.14 0.18 -0.21, 0.49 0.65 1 .421
Psychiatrist referral -0.28 0.18 -0.62, 0.07 2.48 1 .115
Depression -0.11 0.20 -0.51, 0.28 0.314 1 .575
*p<.05
A generalized linear model specifying a gamma distribution and an identity link was performed
where inverse NAVSAT total score was the dependent variable, navigation mechanism score
was the primary predictor, and the number of contacts with a Navigator, OCD, suicidality,
concurrent concerns, legal involvement, and reason for contact being need for a psychiatrist
referral were all included as covariates. The results indicated that navigation mechanism score
significantly, strongly, negatively predicted dissatisfaction with navigation (B=-3.12, SE=0.31, -
3.70 < 95% CI < -2.51; Wald 2=103.18, p=.000) even when the aforementioned context
variables were included. That is, for every unit increase in navigation mechanism score, inverse
NAVSAT total score decreased by 3.12 units. In other words, as navigation mechanism score
increased, dissatisfaction with navigation decreased; thus, navigation mechanism positively
predicted satisfaction with navigation.
In this model, only one context variable was significantly predictive of the dependent variable.
Legal involvement in a client’s case was positively predictive of dissatisfaction (B=0.61,
SE=0.30, 0.02 < 95% CI < 1.19; Wald 2=4.12, p=.042). That is, clients with legal involvement
in their cases were significantly less likely to be satisfied with navigation.
89
Comparing the magnitude of the Wald Chi-square statistics suggests that the navigation
mechanism score in this model had the strongest effect overall and relative to the context
variables observed to date; the statistic was nearly 300% greater than the next highest statistic
(FQOL model) and had much less distance between it and the intercept statistic, suggesting
better model. The standard error estimates were also significantly smaller relative to their
estimates, and the confidence intervals were much narrower, suggesting the estimates within the
inverse NAVSAT total score model were highly statistically reliable. Finally, the Omnibus Test
of model fit confirms that the fitted model still performed significantly better than the null model
(LR 2=170.04, df=8, p=0.000); that is when the proposed predictors were included, the model
was better able to predict individual inverse NAVSAT total scores than when predictors were not
included. The magnitude of the likelihood ratio (distributed chi square) indicates that the
NAVSAT model performed exponentially better in overall fit compared to FES and FQOL
models.
6.6.5.2 Inverse satisfaction with referred services (SRS) score
The specific hypothesis related to satisfaction with referred services was:
H1: Clients’ “navigation mechanism” scores negatively predict inverse SRS scores even when
contextual covariates are included.
In this model, the following contextual factors were included as covariates: number of contacts
with a Navigator, OCD, suicidality, concurrent concerns, depression, legal involvement, youth
improvement, and reason for contact being need for a psychiatrist referral. If navigation was
provided as intended, the coefficient for “navigation mechanism” score should be statistically
significant even when the covariates mentioned are included in the model. Results are presented
in Table 29 and interpreted below whereby each coefficient represents influence on increasing or
decreasing dissatisfaction with referred services.
Table 29. Parameter estimates for the dependent variable “Inverse satisfaction with
referred service (SRS) score”
DV – Inverse satisfaction with referred services (SRS) (n=129)
90
Parameter B SE CI < 95%< CI Wald
Chi-square df p-value
Intercept 3.59 0.51 2.60, 4.59 50.33 1 .000
Mechanism factor score -0.85 0.18 -1.20, -0.49 21.55 1 .000*
No. contacts with Navigator -0.01 0.01 -0.02, 0.00 1.67 1 .196
Suicidality -0.36 0.21 -0.93, -0.03 4.37 1 .089
OCD 0.80 0.42 -0.03, 1.62 3.53 1 .060
Concurrent concerns 0.17 0.21 -0.24, 0.57 0.65 1 .419
Depression -0.19 0.21 -0.60, 0.22 0.82 1 .366
Psychiatrist referral -0.39 0.19 -0.77, 0.02 4.17 1 .041*
Legal involvement 0.12 0.26 -0.40, 0.63 0.20 1 .656
Youth
improvement
Improved 0.42 0.55 -0.66, 1.50 0.58 1
.446
(.001)*a
Stayed the
same -0.66 0.49 -1.61, 0.29 1.85 1
.174
(.001)*a
Worsened
(ref)
*p<.05 aIndicates significance level of variable main effect
A generalized linear model specifying a gamma distribution and an identity link was performed
where inverse SRS score was the dependent variable, navigation mechanism score was the
primary predictor, and the number of contacts with a Navigator, OCD, suicidality, concurrent
concerns, depression, legal involvement, youth improvement, and reason for contact being need
for a psychiatrist referral were all included covariates. These variables were included as
covariates because they were theorized to influence satisfaction with referred services, and were
confirmed to be significantly related to the outcome variable in the correlational analysis. The
results suggested that navigation mechanism score significantly, modestly, negatively predicted
dissatisfaction with referred services (B=-0.85, SE=0.18, -1.20 < 95% CI < -0.49; Wald
2=21.55 p=.000) even when the aforementioned context variables were included. That is, for
every unit increase in navigation mechanism score, inverse SRS score decreased by 0.85 units. In
91
other words, as navigation mechanism score increased, dissatisfaction with referred services
decreased; thus, navigation mechanism positively predicted satisfaction with referred services.
In this model, only two context variables were significantly predictive of the dependent variable.
This time, the need for a psychiatrist referral negatively predicted dissatisfaction with referred
services (B=-0.39, SE=0.19, -0.77 < 95% CI < 0.02; Wald 2=4.17, p=.041). That is, clients who
contacted the FNP in need of a psychiatrist were significantly more likely to be satisfied with
referred services. The model also indicated that although perceived youth improvement had a
significant main effect on the outcome (Wald 2=13.41, df=2, p=.001). Referring to Table 29,
the positive and negative signs for each category suggested that families with no change in youth
perceived functional status were less likely to be dissatisfied than those whose youth had
worsened, while families who perceived youth improvement were counter-intuitively more likely
to be dissatisfied. These differences aligned with the counter-intuitive inverse relationship
between improvement and satisfaction revealed in the correlational analysis. However, in the
current inferential analysis, differences between categories were not found to be statistically
significant.
Comparing the magnitude of the Wald Chi-square statistics suggested that the navigation
mechanism score in this model was much closer in effect size to the FES and FQOL models than
it was to the inverse NAVSAT total score model. The standard error estimates were still small,
and the confidence intervals were still narrower, suggesting the estimates were reliable.
The Omnibus Test of model fit confirmed that the fitted model performed significantly better
than the null model (LR 2=80.84, df=10, p=0.000); that is when the proposed predictors were
included, the model was better able to predict individual inverse satisfaction with referred
services (SRS) scores than when predictors were not included. The magnitude of the likelihood
ratio (distributed chi square) indicates that the model of SRS scores fit better than the FES
models, and was similar in fit to the FQOL model. However, overall model fit for SRS was not
nearly as robust as it was for the inverse NAVSAT total score model. This aligns with the
premise that NAVSAT score was the most proximal outcome measure of those selected in this
study.
92
Table 30 presents the overall model fit statistics by outcome below and Table 31 presents the
final model equations by dependent variable. Overall, the results support the conclusion that
family navigation, as defined by the navigation mechanism score, has a significant, positive
impact on each of the outcomes of interest in this study.
Table 30. Omnibus Test of Model Fit results by dependent variable
Omnibus Test Likelihood
ratio chi-
square
df p-value
DV
FES F 34.03 8 0.000
FES S 32.34 8 0.000
m-BCFQoLS 63.14 8 0.000
NAVSAT 170.04 8 0.000
SRS 80.84 10 0.000
Table 31. Generalized linear model equations by dependent variable
Dependent
variable Equation
FES
Family
= 44.19 + 2.06(navigation mechanism) + 3.04(suicidality) -2.26(concurrent
concerns)
FES
Service-
seeking
= 48.90 + 1.52(navigation mechanism) + 5.45(suicidality) -3.19(concurrent
concerns)
m-
BCFQoLS
= 73.8 + 4.95(navigation mechanism) + 6.59(suicidality) -4.73(concurrent
concerns) -7.55(personality disorder) – 4.96(legal involvement)
Inverse
NAVSAT
total score
= 3.99 -3.12(navigation mechanism) + 0.61(legal involvement)
Inverse
SRS score = 3.59 -0.85(navigation mechanism) - 0.39(psychiatrist referral)
93
Chapter 7 Qualitative Analysis
Overview of qualitative approach
During administration of the online survey, participants were prompted at the end of each section
to add any additional qualifying information, points of interest, and/or to further expand upon
their experiences with the FNP in unlimited open text fields. As with responses to the
quantitative measures, providing additional qualitative data was entirely voluntary. Qualitative
data was collected and merged with quantitative data to generate rich, qualitative thematic
descriptions that would expand upon and enhance the program theory and quantitative findings,
particularly with regard to the influence of context. Descriptive qualitative analyses are an
effective and often undervalued means of supplementing and better understanding a dataset. It is
an approach that borrows many techniques from many other approaches, and falls somewhere
between the limits of quantitative inference and more in-depth and rigorous qualitative
approaches, such as grounded theory or phenomenology (Sandelowski, 2000). The goal of
descriptively analyzing the open-ended qualitative data was simply to further contextualize and
improve understanding of the data.
Of the 134 survey participants, 71 (or 53.0% of the study sample) provided complementary
qualitative data that corresponded to at least one context, mechanism and/or outcome of interest.
Qualitative data from these 71 unique sources were imported, coded and analyzed using QSR
International’s NVivo 11.4 for Mac, a software package supporting qualitative and mixed
methods research (2015). Once the qualitative data was imported, contextual quantitative data
from the survey and chart review phases were also imported to create a merged dataset for
thematic analysis. Logistically, this was accomplished in NVivo using case classifications and
attributes, a feature which allows the user to record categories of descriptive information (like
demographics or type of MHA concern) about each client. Users can then view themes and
coded phrases within and across classifications and themes, producing a variety of C-M-O
configurations.
These qualitative C-M-O configurations further contributed toward understanding the
experiences and outcomes of families in this study, especially in relation to contextual influences
94
that were less easily captured and/or elucidated with quantitative measures. For example,
systemic-level contexts such as long wait lists or consent and capacity legislation, were factors
theorized to influence experience and outcomes, but which were difficult to pre-define and
measure quantitatively. With regard to individual-level contexts, a range of factors, such as youth
age and whether youth had concurrent mental health and addiction concern, were theorized to
influence experience and outcomes. Of the theorized contextual influences, only approximately
30.0% were found to be statistically significantly associated with experience and outcomes in the
correlational analysis. Of those that contextual influences that were statistically significantly
associated, only approximately 33.0% were found to be statistically significantly predictive in
inferential modeling of outcomes. However, the program theory maintained that such contextual
influences were very likely to have played a role; it is possible that the degree of influence was
not to the extent that statistical significance could be detected in the current sample. Qualitative
C-M-O configurations were thus helpful in identifying if and describing how and to what extent
context may have influenced a family’s experience and outcomes.
Since the RE framework is theory-driven, it presupposes which key contexts, mechanisms and
outcomes of interest are likely to be present in the qualitative data (i.e. those identified in the
program theory and background research). However, RE is also meant to adapt to emergent
themes, and so any additional contexts, mechanisms and outcomes outside the program theory
were also recorded (Pawson & Tilley, 1997). NVivo is an ideal analytical software for this
purpose as it allows for users to code phrases of data according to pre-identified themes (or
“nodes”), as well as allow for groupings of similar phrases to accumulate under emergent
themes.
To begin, phrases were first coded by domain as related to either contexts (C), mechanisms (M),
or outcomes (O). Phrases describing a condition existing external to or prior to enrolment with
the FNP were coded as contextual (C); mechanisms were coded when phrases spoke directly to
program activities and resources (M); and outcomes were coded when phrases spoke to a direct
result of a client’s experience with the FNP (Pawson & Tilley, 1997). With the integrated
dataset, each code also had a case classification and a corresponding set of attributes from the
quantitative data (i.e. age, gender, type of concerns). As mentioned above, the purpose of the
qualitative data was to generate evidence for additional C-M-O configurations that would add to
a more robust understand of a family’s experience and outcomes, particularly in terms of further
95
describing contextual influences (i.e. if, to what extent, and how context exerted influence on
experience and/or outcomes). Codes were then grouped under proposed and emergent themes
related to how context influenced experiences and outcomes, and hierarchies were assembled
where theoretically appropriate (Byng, et al., 2005). To ensure confidentiality, clients were
identified by number only and all names have been changed. Themes are discussed in depth by
domain – context, mechanisms, or outcomes - below.
7.1 Descriptive and thematic analyses
7.1.1 Context
Based on the program theory and background research, several contextual themes known to
impact this population at different levels were proposed. At the individual level, most theorized
contextual factors such as demographics and illness characteristics were collected from closed-
ended survey questions and client charts. However, there are some individual-level contexts that
were theorized to influence experience and outcomes, but which were not found to be
statistically significant in effect, or which were not easily captured quantitatively. The qualitative
data spoke to one theme in particular that relates to individuality, the nature of MHA conditions,
and the fact that the client in this program is the family member, not the youth.
7.1.1.1 Theme 1: You can’t force someone to get well
Of the 71 clients who provided qualitative data, over a quarter (n=19, or 26.8%) spoke to the fact
that their experience and/or potential for positive outcomes was hindered by the youth in
question’s unwillingness or inability to get well. The single most commonly associated
contextual variable under this theme was age. Most often, this theme related to the youth being
of adult age and in control of their own treatment, and/or generally unwilling to engage in the
help-seeking and treatment processes. Many youth in this sample were over the age of 19 and are
technically adults responsible their own behaviour; 16 of 19 clients who were coded on this
theme had youth who were aged 19 and older at the time of seeking help. Clients explained how
this affected their experience: “My daughter is 25 years old so I have little control about what
services she will accept or decline. I feel that at this point I can only offer her suggestions and
options for her to help herself” (10059); and, “…it is difficult because our son is 19. It is up to
96
him how he decided to utilize the services that are presented to him. The drugs have got [sic]
worse and he is no longer living with us or accessing any services” (10016); and,
“The concept is excellent and initially was helpful. Because my nephew was 21 years old
and living in youth hostels, it was necessary for him to be engaged in getting help.
Despite the efforts of the navigator and myself, it’s not clear that [Tim] was very engaged
... as [Tim] showed limited commitment to seeking help, the role of the navigator
lessened” (10011).
Younger age did not guarantee engagement though, as even younger youth have the right to
refuse treatment in Canada. One client noted, “Unfortunately all the service recommendations
could not work because our teen refuses meds and refuses counselling or psychological support”
(10068); another, whose child is only 16, “My child refuses to seek treatment and doesn’t think
there is anything wrong with him” (10082); and another, “Since there is no way for parents to
force their children to go or stay in treatment in Canada, he never wanted to, or did” (10123).
Type of MHA concern mattered for families. Different MHA conditions can manifest uniquely
and vary significantly in nature, particularly in terms of disability, chronicity and responsiveness
to treatment (Kessler, et al., 2005, 2012). For example, depression is highly treatable and
associated with positive outcomes, whereas personality disorders and OCD are chronic severe
conditions that are difficult to treat and for which fewer resources are available. Clients who
spoke to this idea of being unable to force their children to get well tended to have youth with
these complex and severe conditions: three with personality disorders, two with eating disorders,
two with psychosis, one with mania. Twelve of 19 had concurrent substance use concerns as
well.
Parents in the sample were generally aware of and affected by the limitations type of MHA
concern can impose on treatment expectations. They wrote: “My daughter’s issues [personality
disorder] are proving difficult to treat…she has tried alot [sic] of treatments and therapies with
little success” (10089); “The situation with our daughter [personality disorder] will be a life
struggle” (10021); and, “It takes so long for people to get well even after they get quality care”
(10082).
There were also important contextual factors at the systemic level that could not be measured
with a survey tool but were theorized to impact both mechanisms and outcomes - conditions such
as long wait lists and a general lack of specialized services which make access to timely,
97
appropriate, quality MHA care difficult even with the help of a Navigator. Another known
important contextual condition for this population was the role of privacy, consent and capacity
legislation. These three themes were hypothesized based on the background literature and data
and were notably present in the qualitative data; 41 of 71 clients (57.7%) provided over 80
quotations that spoke to these types of systemic barriers.
7.1.1.2 Theme 2: You can’t navigate to services that don’t exist
The term navigation presumes there is a system in place to navigate, but the research underlying
the FNP suggested that the system lacks not just integration between services but absolute
capacity in terms of specialized service providers for youth MHA treatment. This presents a
known significant barrier to providing effective navigation services as one cannot navigate to
services that do not exist. The FNP recognizes this constraint and strives to piece together the
best basket of services given what is available. For the clients in this sample, the results
suggested that sometimes the FNP was able to find solutions, but sometimes families were
disappointed as the solutions they found were less than ideal.
For the most part, clients of the FNP were still appreciative of their Navigators’ efforts, even
when the results were less than satisfactory. Most clients kept their comments brief: “Our
exposure to the shortcomings of the system has been a shock” (10091); “FNP did their best.
Service providers are extremely limited” (10025); “The initial intake and assessment were very
good however the options for my son were limited” (10129); and “My main problem was not
with FNP which I support wholeheartedly, but with the lack of options/resources available for
them to recommend” (10005).
Many, however, took advantage of the opportunity to express their frustration and
disappointment:
“First and foremost, the Project wasn’t the least bit helpful except to make it clear that there
was very little expeditious or effective help for Michael in the public sector…Any
improvement that [Michael] has enjoyed has come because we moved him to the private
sector ([Michael] is now in a program in the US) which is firstly, immediately available,
secondly, is comprehensive in figuring out if there is a link between addiction and mental
illness, and thirdly, has the willingness and capacity to treat these kids in a macro way. Our
system is not effective and the Project is reflective of that ineffectiveness…In treatment
addiction and mental health issues for young adults, the system is woefully inadequate. It is a
system where kids get “punted down the road.” There is no leadership or infrastructure in
98
place to deal with these kids and their issues, and further, the system doesn’t seem to have
been designed contemplating that addiction and mental health issues may be related.
Addiction facilities aim to get kids “clean” and then send them back out into the world
without dealing with the possibility that there may be underlying causes of that addiction.
Mental health resources are extremely limited, very difficult to navigate and often all but
impossible to understand” (10003);
“Sadly for us, this program was no help at all. My son is now 18. I have very little hope for
his future. Our mental health system tells me, “this is normal teenaged behaviour” or “he is
overprivilaged [sic] and this with [sic] what we get.” My son grew up in a very normal house
with 2 biological parents that loved him dearly, there is no reason for his life to look like it
does. Now the criminal justice system will be his [sic] mental health advisor” (10012);
“We struggled for years trying to negotiate the system. There are huge deficiencies in the
mental health system. Sick Kids was awful, Sunnybrook was a disaster, CAMH had no
programs for youth, the school was useless, the police response was traumatic…The system
is entirely broken” (10068);
“It did not really help me or him to deal with the system. I want to be clear that this was not
[the Navigator’s] fault in any way. She did what she could, but it seems there is not much
help available in Canada (US treatment programs were recommended). The inpatient
psychiatrists at Sunnybrook seemed overwhelmed and too busy, the nurses were not
psychiatric nurses for the most part, and there is no therapy offered to the inpatients, just
drugs” (10015); and,
“Please note again I do not feel this is really the fault of Family Navigation who I believe did
try to an extent…there is just NOTHING in the Canadian Medical system to help teens with
Mental Health issues other than expensive individual therapists who may or may not be
equipped to assist. The other option we were told was to go to Sunnybrook emergency if our
child was suicidal. We have done that now twice and ended up leaving both times after a few
hours of waiting as we were triaged and not seen” (10018).
The results suggested that families in this sample were significantly struggling but that the
current system could not meet their needs. The data indicated that age, type of MHA concern,
and required service types were the most commonly associated context variables for this theme.
Clients who spoke about this overall lack of services tended to have transitional-aged youth (18
to 24 years old) for whom there are fewer targeted resources available; were often seeking care
for concurrent addiction (n=14) or complex, severe conditions like personality disorders (n=7),
bipolar disorder (n=3) that are relatively less common; and were often in need of service types
that are particularly lacking in the current system, such as residential treatment (n=6), day
treatment (n=2), and aftercare programs (n=2).
99
7.1.1.3 Theme 3: Existing services lack accessibility and continuity of care
The second systemic-level contextual theme that was present in the qualitative data was related
to the inadequate supply of services, which was that the services that did exist were inaccessible,
untimely, and discontinuous. Lack of capacity has created service fragmentation and gaps in care
pathways, which made it inordinately difficult for some families in this sample to identify and
access the appropriate services in a timely manner. Twenty-four unique clients (33.8%) spoke to
the impact of this context on their experience and outcomes:
“We had to wait 6 months for her first psychiatric appt [sic] and employed therapists and
psychologist and DBT to keep her alive in the interim. That year cost us thousands and we
remortgaged but most can’t. Nothing was covered except annual max of $200 with health
insurance provider. Even now there is no coordination between psychiatrist, doctor,
endocrinologist, and therapist. The whole system is shocking and disgraceful. All the
advertising and sponsorships and walks are really just crap. The system is disjointed and
impossible to navigate. Agencies competing for funds and NOT comprehensive care. Psych
wards were basically homeless shelters. I work in education and the chances of arranging
help for kids is just as elusive” (10091).
This was a systemic barrier that also impacted the FNP’s ability to perform:
“There was certainly some frustration that is not coming across with the questions above.
There were times when practitioners were recommended/suggested, but then appointments
with them never happened because of paperwork issues between the recommended
practitioner and the regular physician’s office. This was frustrating as we often felt
encouraged by the people who were being recommended, only to go 1-2 months and nothing
would happen. Also initially I was hopeful in terms of the contact I received at the beginning,
but then it took a few months (3-4) before the first real connection to a recommended service
was made. Sometimes, at the beginning, it felt like I was back where I had been over the
previous few years, and that was frustrating” (10107).
Most clients were particularly frustrated by how long it takes to receive services across the
system. One client explained, “I spent days and weeks trying to find out what was available to
her as things got progressively worst [sic] only to find a huge waiting list;” (10049); and another,
“There’s a lot of “try this” or “maybe this” and attention is available 6 to 8 months from now”
(10003); and another, “The problem has been that we are still on wait lists for both case
management and CAMH treatment. We completed the intake process for these about 9 months
ago but we are still waiting for services to begin” (10100). Some families encountered waitlists
as long as 18 months: “We were on wait lists for any appointments or assessments and were
looking at 12-18 months. We essentially had to wait until an emerg[ency] admission bumped
100
him up the list. We had to wait for criminality to access school-based interventions” (10068);
and, “The wait lists for things are also extremely long – some years…so there is no ability in
some cases to get resources (we were looking for “residential mental health” assistance)”
(10125). This was a challenging situation for some parents to manage: “I wish it didn’t take so
long to see a specialist. It’s difficult to see your child go through such depression while you’re
waiting for your appointment” (10035).
Another important issue was continuity in service. Pathways to care for MHA concerns are often
complex, involving multiple providers, and long-term, in addition to this being a particularly
difficult population to engage. Continuity of care is essential, and the FNP strives to “get in the
boat” with families for the duration of the help-seeking process because, as one parent described:
“The most challenging thing as a parent was when my son was an inpatient at Sunnybrook
and I felt helpless and afraid about what he was going through, not being able to speak
directly to the doctors regularly, and what the next steps would be after he got out” (10115).
In this sample, Navigators often stayed engaged with families over the long-term, but could not
always guarantee the services to which they referred would be as responsive: “I was more than
happy with the services of the FNP. Unfortunately, the services that were recommended to us
took months (CAMH) to get and lasted 2 hours when we got them and my son lost interest in
getting help” (10002); “He has been taken to hospital twice threatening suicide and released with
barely a question” (10012); and,
“The questions about treatment services were hard to answer, because our child has not yet
received clinical services or treatment yet. We do pay for private counselling support as a
way of bridging until we can access treatment. It is going on one year now. A great deal of
delay originated with our own G.P., whose office, twice, omitted to include [sic] critical
information in the referral to Sunnybrook” (10047).
Clients also spoke about inaccessibility of services due to such barriers as prohibitive eligibility
criteria, geographic constraints and costs: “Hard to say it in multiple choice, but we feel helpless
at times that “the system” is full of “gatekeepers” who hold the balance of power to let your child
enter, or else just let them flounder in a sea of despair” (10047). Catchment areas were a
common barrier to care: “I wanted someone from Sunnybrook to help my son as I work here, but
they wouldn’t as I am not in the cachment [sic] area” (10035); and, “It was incredibly stressful to
have to go downtown to SickKids for every appointment (and there were many). We didn’t know
where to start to look for resources in the community” (10062).
101
Financial resources meant some families had the option to go elsewhere rather than wait: “We
were forced to take our child to the USA due to wait times and the lack of cohesive services
available” (10063); “I am deeply sad and frustrated about how difficult it is to obtain paid for
services through OHIP. We luckily had the financial means to pay for services, otherwise I truly
believe our daughter would not be alive today” (10049); and “I can’t imagine what other families
with less are doing” (10082). Unfortunately, other families did not have this option, as “The only
options provided were all in the US and extremely expensive,” (10018) and yet “There is no
funding available to seek quality care” (10131).
No particular C-C dyads stood out, which may reflect the widespread impact of a systemic lack
of capacity for youth MHA services. The merged dataset suggested that clients in this sample
who commented on this theme had a wide range of associated types of MHA concerns; had
youth of all ages; and were seeking a variety of service types.
7.1.1.4 Theme 4: Privacy, consent and capacity legislation
Another important theme present in the qualitative data was the unique role of privacy, consent
and capacity legislation as a barrier to care in the field of youth MHA concerns.
Sixteen clients (22.5%) referred specifically to the negative impact privacy, consent and capacity
legislation has had on their navigation and general help-seeking experiences. Most were
extremely frustrated by their exclusion from care because they felt it impeded their ability to
parent their child and that they could be an important resource in recovery:
“You are not taking into consideration the 18 year olds…Our daughter had gotten very
immature in some aspects and certainly was during her mania. Even when she was sedated
and in four point restraints in a psych ward they wouldn’t tell us anything!!!!!!” (10091);
“My experience vis-à-vis the professionals involved in my daughter’s care has been
haphazard despite the ongoing involvement with my daughter. This has been due to various
reasons including the fact that I was originally told that because she was considered an adult,
she could make her own decisions. This information I received when she had been admitted
to the hospital the week after her 16th birthday. Apparently her decisions that lead to her
hospitalization were considered sound enough to exclude me” (10105);
“Privacy laws are important for the child but severely impact what a parent can do. The
parent is usually the one who most wants to help their own child. The road blocks to assisting
one’s own child are now too much. VERY frustrating as a parent” (10008);
102
“Due to the youth’s’ age, above 18, no one can obtain any information regarding any of the
treatments, diagnosis etc. unless the patient (youth) approves it. Due to this factor, I could get
no information whatsoever as she was 21 at the time. When it comes to mental health,
especially if the parent is the one seeking, finding and getting the patient to admit themselves
or participate in it, the parent should be able to gain pertinent information to assist both
parent and child to work towards the same goal” (10020).
“When a youth of 17 or 18 is in severe depression and the primary caregiver is kept outside
the doctor’s room, it upset me as a parent as I knew more about the situation than my son. I
should have been more involved in the plan to get him help” (10035);
“It is frustrating that if a child is of legal age, then the medical professionals do not share
information with parents. My son’s initial diagnosis, which I arranged, was not shared by the
assessing psychiatrist or his GP, and if it was, then I could have addressed his treatment
needs before it turned into a crisis. Now I feel that he will never get better – too little, too
late” (10061); and,
“I also think it is frustrating that when a troubled youth turns 16 he/she is suddenly allowed
to keep things confidential with the health care provider. How can we help them as parents if
we are not aware of their needs and concerns?” (10066).
The findings suggested that there may be a degree of misinformation communicated to parents in
this sample about the criteria for privacy, consent and capacity being age-based. Other clients
highlighted the fact that the same applied to parents of youth under the age of 16 as well: “It was
appalling that at 13 patient confidentiality excluded us as parents from having any information
about his diagnosis and symptoms” (10068). Regardless of age, restrictive interpretations of
current legislation were a significant barrier to care for these families, despite the fact they were
often in the primary role of interacting with the system and negotiating access to resources on the
youth’s behalf.
Under this theme, age was the most commonly associated contextual variable. While quantitative
analyses did not always find age to be a statistically significant influence on experience and/or
outcomes, qualitative data indicated that age did play a role for some families; thirteen of the 16
clients who spoke about the impact of privacy, consent and capacity legislation on their
experience and/or outcomes had youth who were aged 19 years or older. Altogether, it was
evident from clients’ feedback that the contexts in which families were seeking help for their
youth with MHA concerns were highly complex and impactful; and that understanding these
contexts is essential to appropriate service navigation and provision.
103
7.1.2 Mechanisms
Qualitative data were also coded for mechanisms. The three mechanisms or service
characteristics identified by the program theory and measured quantitatively were accessibility,
continuity of care, and family involvement; these three themes were pre-set, and others were
again allowed to emerge from the qualitative data in order to add to the program theory about
which mechanisms may be operating and how; and if and how the experience of these
mechanisms was influenced by context.
Pawson and Tilley state that mechanisms naturally involve both resources and reasoning, but do
not provide a method for distinguishing between these two processes. Recent contributions to RE
suggest that the importance of distinguishing these two mechanistic aspects have been
understated to date, which has contributed to inconsistency and confusion in identifying how
mechanisms operate and how C-M-O configurations are generated (Dalkin, et al., 2015). Instead,
it has been suggested that mechanisms (M) be further categorized as related directly to the
resources or activities provided by the program (m(resource)), or to the human reasoning in
response to receiving those resources (m(reasoning)). This allows for clearer specification of
how resources and reasoning are interacting with each other and with context to generate
outcomes.
For this reason, qualitative data referring to one of the original three proposed mechanisms were
coded as m(resources) because it is the provision of these three service characteristics that were
thought to define navigation. In addition, when qualitative data referred directly to clients’
attitudes, values, feelings, or beliefs that resulted from the provision of those resources, phrases
were coded as m(reasoning) (Dalkin, et al., 2015).
7.1.2.1 M(resources)
7.1.2.1.1 Theme 1: Accessibility
Although the FNP cannot directly influence the accessibility of the services it refers to, it can
guarantee that their own Navigators are highly accessible in the meantime. The FNP provides
accessible navigation by organizing its services to respond to families’ needs, whether this means
offering clients extended and/or flexible hours of operation; having an intake line with a live
voice on the end; the ability to communicate by phone, email, Skype, or in person; free services;
104
and/or guaranteed response within 48 hours. To families, these are resources that should enable
them to manage their youth’s care. Qualitative data supported the quantitative findings that the
FNP is excelling at providing highly accessible care to families of youth with MHA concerns
within and across the GTA.
Nearly 20% of clients (n=13) specifically spoke to the FNP’s level of accessibility. In general,
clients were impressed and grateful to be able to access the FNP itself whenever they required.
Clients reported, “Thanks for FNP – it was there when we needed it” (10035); “I was very
impressed with the fast response” (10029); “They were very responsive, providing resources
very quickly” (10110); “In my view, one of the best aspects of Family Navigation was how
quickly the intake procedures and involvement with the Family Navigator happened. In our case
this occurred within two weeks – a miracle!” (10037); and “Thank you Family Navigation for no
wait list and always being available to provide prompt information for my request” (10049).
Immediate access to care is particularly important in crisis situations as it can prevent both
worsening of the MHA conditions as well as inefficient and ineffective use of acute care (i.e.
emergency department) services. The results suggested that the FNP was adept at responding to
clients’ crises in this sample: “We met our Navigator when our child was in crisis. We received
quick in-take and he was seeing a psychiatrist very quickly” (10088); “My daughter was
receiving the care needed [for suicidality and self-harm] shortly afterwards” (10134); and “We
immediately had one visit and about three phone calls with FNP including [Jane] meeting
[Medical Director]” (10097).
7.1.2.1.2 Theme 2: Continuity of care
As previously mentioned, the FNP aims to “get in the boat” with families and remain engaged
throughout their help-seeking journey. In this sample, it was a common complaint that the
existing service system is extremely discontinuous, and clients were highly aware of the FNP’s
efforts to remain available to the family as long as required. Twelve clients (16.9%) explicitly
commented on the ongoing involvement of their Navigators and the frequent follow ups they
received. Overall, the twelve clients who spoke to continuity of care had been engaged with the
program for longer than the sample mean (28.2 weeks vs. 24.0 weeks). Clients commented:
“Very good at keeping in touch and trying to find solutions to our problems” (10103); “They did
not abandon us and stayed connect [sic] when initial steps did not work and we were able to try a
105
different option which did help” (10113); “The Family Navigation [sic] continuously checked in
via email to see how things were going” (10001); “I’m very impressed with…the frequent follow
up” (10029); “She [Navigator] remained in contact with follow up phone calls after the therapy
sessions started” (10087); “Support staff kept emailing me on and off just to see if we had found
him and also to make sure I was ok” (10124); and,
“In the beginning, we were in contact with our navigation [sic] often. Then my son left to
live in another country for 6 mos. When we had problems again, when he came back, we
contacted our navigator again. We are satisfied with their follow up, including the parent
advocate. Thank you for the help and follow up” (10092).
The FNP also strives to facilitate continuity of care between the providers to which it refers. This
is more difficult as they cannot control the service system, but the data suggested they were
successful in facilitating some cases: “I was also impressed with how the Family Navigator was
able to quickly and directly liaise with our son’s former therapist at CAMH as well as a CAMH
psychiatrist who had conducted an assessment” (10037); “They are always available to help me
find services for my daughter and have even gone further to help when she was planning to
attend post-secondary schooling outside the area” (10049); and “Once [John]’s condition was
stabilized, they also made the referral to an experienced psychiatrist to monitor [John]’s ongoing
mental health” (10009).
7.1.2.1.3 Theme 3: Knowledge, insight and expertise in youth mental health and addictions
The results of the thematic analysis suggested that the specific knowledge, insight and/or
expertise in the youth MHA system offered to families was a standalone mechanism in and of
itself; of all program resources and activities, clients most often referred to the value of
Navigators’ knowledge, insight and expertise (n=23, or 32.3% of clients providing qualitative
feedback).
Some clients spoke specifically to the Navigators’ knowledge, insight and expertise on the nature
of youth MHA concerns themselves, and how families could better manage as a result. One
client explained, “The Family Navigation Project allowed my situation to be finally “heard” by a
team of professionals who collectively provided suggestions as to where I might get help and
explained how they might help” (10113). Others commented, “Their insight into [David]’s
condition and suggestions as to treatment were invaluable” (10009); “Your service was
106
invaluable and worked as a mediator between my child (who was 21 then) and I as I seemed to
be where her finger was pointing and I could not understand how or why” (10020); and “I did
and still do appreciate all of the ideas, information and support from the navigator” (10081).
Other clients spoke to the Navigators’ knowledge, insight and expertise related to the service
system and providers available to this population: “The Family Navigation Project was great at
referring to services that I either did not know about or was having trouble accessing” (10100);
“Given the availability of so much information online, I wasn’t sure I would get much value but
the Navigator definitely provided new options” (10041); “The lists that FNP has are constantly
being updated which is really really helpful” (10021); and “We would never have found this
service on our own” (10037).
A few more clients spoke specifically to the Navigators’ ability to use their knowledge, insight
and expertise to make highly appropriate matches between families’ needs and service providers:
“She found the perfect therapist for my daughter who in turn recommended a treatment facility
that was instrumental in helping” (10094); “…guided us to exactly the help we needed at the
time” (10053); “She has provided so many resources that are excellent” (10127); and, “The
challenges I faced and still face shook me. [Navigator]’s support, problem solving discussions
and access to professionals with special background in areas I needed to access made
considerable difference” (10097).
7.1.2.1.4 Theme 4: Family support
Family support was theme that was assumed to fall under family involvement based on the
definition employed. Instead, as with the previous theme of knowledge, insight and expertise, the
qualitative results suggested that family support should be considered a standalone mechanism.
Fourteen clients (nearly 20.0%) specifically noted the support provided by the FNP. Clients said,
“[Medical Director] and his team supported our family as we looked for appropriate services for
[James]” (10009); “The family navigate [sic] that I spoke to was kind and caring and very
supportive at a time when I felt very alone” (10062); “Thanks as well for such a calm and
sympathetic ear” (10082); “[Navigator] took so much time to comfort, reassure, and encourage
me in the early days of the diagnosis” (10099); and “I really appreciate their support and
professionalism and caring” (10134).
107
7.1.2.2 M(reasoning)
From the background research, program theory and primary qualitative data results on
m(resources) above, it was clear that having continuous access to expertise and support helped
families. As mentioned previously, qualitative data was also unique in its ability to explore how
these resources impacted families. In general, RE theory proposes that programs primarily
achieve their goals by providing resources that directly influence clients’ reasoning in terms of
their attitudes, values, feelings, and/or beliefs. This reasoning, in turn, influences decisions they
make with regard to care planning for their family going forward (Dalkin, et al., 2015; Pawson &
Tilley, 1997).
Qualitative data were therefore coded for phrases expressing clients’ attitudes, values, feelings
and/or beliefs as a result of resources received from the FNP. Since reasoning is a highly
individual process, themes in reasoning were not presupposed and instead allowed to emerge
organically from the data. Several reasoning processes were evident. The first was that as a result
of the services provided by the FNP, clients felt a sense of relief; second, clients felt reassured;
and third, clients felt hopeful for the future.
7.1.2.2.1 Theme 1: Relief
Many clients spoke about the sense of relief they felt as a direct result of engaging with the FNP
(n=18, or 25.4%). Relief occurred on two levels. The first revolved around the physical relief
from the burden of seeking help for youth MHA concerns. It is understood that the system is
extremely complex and difficult to navigate, requiring an inordinate amount of time and effort to
find, understand and access the range of available resources. This was an incredibly stressful
situation for families who are already dealing with stressful youth MHA concerns and having a
Navigator available provided a huge sense of relief by helping to reduce the overall time and
effort clients had to invest themselves. One client explained:
“When the navigator told me that she would help me make calls and find services because
she appreciated that it was a challenge and she acknowledged that I was working and caring
for my daughter and family, I felt incredible relief. I cannot tell you how much I appreciated
having a partner to navigate the system” (10062).
Six other clients expressed similar sentiments, such as: “I know I can work with FN [sic] to find
new options and this has taken some of the pressure off me to constantly come up with new
108
treatment programs” (10021); “I am exhausted by the whole process, but so appreciate
everything the Navigation project has done to help” (10061); “Such timely access greatly
reduced stress and anxiety” (10037); “Navigator definitely…cut down on the time and
uncertainty of searching” (10041); and “Having a service such as yours helped me to look at
other possibilities that I could consider for my child during a very stressful time for all
concerned” (10058).
Clients also spoke of the emotional relief associated with finding a service provider who cared,
listened, and most importantly, understood. Twice the number of clients referred to this
emotional sense of relief over being heard (n=14). One client expressed this succinctly saying,
“You have no idea how important having someone who understands what its [sic] like at the
other end of the line” (10082). Another explained, “It gave me, the sole parent who was alone
and who had no one who really understood our situation, support from a group that really “got”
our case” (10113).
Other clients echoed: “I appreciate that I have been heard by a Family Navigation staff person
and she was kind” (10018); “We were grateful to have another ear to hear our concerns”
(10102); “Your service provide [sic] guidance and as well an outlet for me to let off steam due to
my frustration in not being able to get the help my daughter needed in a timely manner” (10058);
“I thought it was the sweetest thing when after my son ran away, your support staff kept emailing
me on and off just to see if we had found him and also to make sure I was ok. Such a caring
group of people. Thank you” (10124); “Family navigation offered me a piece of mind [sic] when
I had nowhere else to connect” (10090); and “She [Navigator] was very concerned about me and
how I was handling the stress, and the impact on family dynamics and marriage. I will be forever
grateful for her kind and gentle voice on the phone” (10099).
7.1.2.2.2 Theme 2: Reassurance
Families of youth with MHA concerns face many challenges to their ability to effectively parent
and seek help for their children: stigma, scarce resources, unclear legislation, and symptoms that
affect the entire family. The FNP outwardly acknowledges the complexity and stress that
families can face when trying to effectively and efficiently manage their child’s care, and
purports to help by offering continuous access to support and comprehensive, objective
information about treatment options. Thematic analysis of the qualitative data suggested that one
109
of the reasons this information is, in fact, helpful is because it helps to reassure clients that they
are doing right by their child; parents (and other caregivers) in this sample wanted to know they
were doing the best they could for their youth with what they had available to them. This
involved reassurance that parents themselves were doing all they could as parents to help their
children, and/or that the professionals or resources they enlisted to help their children were the
most appropriate available.
Ten clients (14.1%) spoke to this topic. They explained how the FNP reassured them they were
doing their child justice, despite the complex system:
“The service we received was very useful because we wanted to know if there were any other
options we may have missed. It turned out that there weren’t but that gave us the confidence
to know we are on the right track and doing all we can for him” (10038); and,
“I think your programme is an important bridge to mitigate some of the confusion, fear and
helplessness that a parent can feel when knowing where to turn and how to access
appropriate services for their child, and also helping to discern if a parent’s concerns are
valid or not and in a non-judgmental way” (10067).
Other clients reiterated: “FNP was unable to add much to what I already know, but provided
reassurance that I was pursuing the best avenues” (10025); and “It was useful to talk to our
Navigator to understand the landscape of services available. As a parent, I wanted to make sure
that our family reasonably explored all services available” (10010).
Clients also mentioned how the FNP created confidence in the treatment providers selected:
“They solidified my decision” (10134); “We feel that the worker was instrumental in leading us
to the right resources and giving us great advice” (10016); “It has given us more choice which
results in being able to confidently tailor a selection to something that fits our daughter the best it
can” (10021); and “[Navigator] further enhanced our confidence in the supports that were being
recommended to us” (10037).
7.1.2.2.3 Theme 3: Hopefulness
The results suggested that as clients experienced relief and reassurance through their engagement
with the FNP, they also gained hopefulness for the future. Although they recognized that many
MHA concerns would be chronic and/or recurrent, clients of the FNP felt hopeful that they
110
would be better able to manage their youth and family as the situation evolves as a direct result
of their engagement with the program. Nine clients spoke specifically to this topic.
Clients explained: “It looks at the entire problem and the people involved and helps on all fronts.
One is not left at a dead end when initial steps may not work. It gives hope” (10113); “I believe I
am better prepared as a parent to deal with the problems as they arise. Knowing that I can contact
my navigator for advice definitely helps” (10087); “It has improved our ability to cope with our
son’s illness so much better” (10127); “I do feel now that I have somewhere to turn if things go
off the rails and I’m very thankful for that” (10081); “It does not always work, but at least I
know we can continue to try” (10021); “This gave me a real boost and improved my outlook and
hope for the situation” (10100); and “We remain optimistic and are hopeful that the navigation
project will continue to act as a resource as our situation evolves and changes” (10102).
It was interesting to reflect back on the theory-building discussions with the program team
around what they hoped to achieve (refer back to Chapter 2 and/or Figure 1, Appendix B).
During this process, team members specifically mentioned wanting to offer families “a sense of
relief,” to “encourage confidence” (i.e. by providing reassurance) and “hope for the future.” For
most of the 71 families who provided qualitative data, the results suggested the FNP was
operating as intended.
7.1.2.3 The link between resources and reasoning
According to the program theory, mechanisms occurred in unique combinations for unique
clients; generally, clients were expected to experience all mechanisms to varying degrees
according to their needs. Moreover, earlier thematic analyses suggested that several different
reasoning responses occurred as a direct result of those mechanism resources. This, in addition to
the fact that each client was coded at a number of nodes and allowed to make a number of
references, yielded a wide range of M(resource)-M(reasoning) dyads that contributed to
understanding how navigation helped families. The commonly recorded dyads most often related
to a sense of relief as a result of the various resources offered; the single most common was that
of expertise and relief (n=10), followed closely by links between support and relief (n=9), and
continuity and relief (n=7). Combinations are presented in Table 32 below where “N”
corresponds to the number of unique clients coded on both mechanisms in the specified dyad.
111
Table 32. M-M dyad combination by theme
M-M dyads
M(reasoning)
Relief Reassurance Hope
M(resource)
Accessibility 5 3 2
Continuity 7 1 4
Expertise 10 3 6
Support 9 2 4
7.1.3 Outcomes
These theory-building discussions around what the FNP hoped to achieve led to the selection of
a set of quantifiable, standardized outcome measures that could theoretically be expected to
result from the services provided. For example, the FNP team wanted to “encourage
confidence,” and so empowerment and a corresponding standardized outcome measure – the
FES – was selected. In place of “a sense of relief” and/or “hope for the future,” the FNP team
settled on an improvement – any improvement – in family quality of life, quantified using a
modified version of the BCFQoLS. Standardized measures such as these were essential because
they allowed for quantitative analyses and comparison. However, other than service satisfaction,
the selected outcome measures were distal, highly complex and subject to a wide range of
influences. Particularly with small sample sizes and cross-sectional designs, statistical
significance is difficult to detect and effect sizes are likely to be quite modest (Tabachnick &
Fidell, 2007). Moreover, in a program providing unique combinations of services to unique client
contexts, unique outcomes should also be expected.
Clients’ qualitative feedback was therefore coded for phrases that spoke directly to a result of
being engaged with the FNP, and analyzed in terms of how these outcomes were influenced by
context and/or mechanisms. Outcomes were then simply categorized as positive or not positive.
112
Once all outcomes were coded, an initial word frequency query was run to generate an
exploratory word cloud, which is a simple visual depiction of the most common words in the
coding category (Figure 2). The most frequently occurring stem word by a count of 50 was
“help” (and its derivatives: helps, helped, helping, helpful.
Figure 2. Word cloud depicting word frequency in all phrases coded as an outcome
7.1.3.1 Positive outcomes
Thirty-eight unique clients (53.5% of those providing qualitative feedback) spoke to a general
outcome of the FNP. Of those references, 31 (or 81.6%) were positive, and no contextual factors
were identified as more often associated with positive feedback than others, which lent support
to the quantitative findings that most clients in this sample were satisfied with their navigation
experience. As mentioned above, positive feedback generally fell under two themes. The first
was when services were successfully accessed and youth and family are now better off as a
result. As one client explained:
“The Family Navigation Project was a life saver. I truly believe that it was the first step that
helped to get a young man who had withdrawn from the world back out and making his way
into a productive happy individual. I believe that many of our other healthcare services
should follow this format” (10113).
Others echoed: “I feel that through involvement in the Family Navigation Project, we were
directed to a program that is working well with our son as well as the whole family” (10037); “I
did get great service which helped my son” (10035); “Not only were we directed to get help for
my son but for me as well and within a few months my son started to come back to the world”
(10120); “It DID result in their help in finding a psychiatrist for our daughter… who has been a
113
godsend” (10091); “I have had such an amazing experience with family navigation services. My
son currently is thriving” (10108); “I was extremely satisfied with the family navigation project
for my nephew. He received the immediate care he needed. Family navigation connected him
with a doctor who he has been in touch with regularly” (10090); and,
“We have been thrilled with the care and with the referral to the addiction counselor. We
firmly believe that our child has the best care – just need him to continue on his path to
wellness before we can check all those boxes as ‘extremely satisfied’” (10088).
Qualitative data effectively complemented the quantitative satisfaction measures, with feedback
speaking to satisfaction with the FNP and satisfaction with the services to which they were
referred. Clients spoke to FNP-specific satisfaction outcomes above and beyond overall
satisfaction, including likelihood to recommend and support for scaling the model, a long-term
stated goal of the FNP. Results aligned with the findings from the quantitative analysis, which
suggested that overall, most clients were highly satisfied.
Clients volunteered: “I recommend this service on a regular basis to friends, patients, colleagues”
(10072); “This program needs to be located in every region. Families need support now more
than ever to get help” (10046); “I have already recommended this service to a number of people”
(10039); “We believe that every family with a mentally ill family member could use the
resources and guidance offered by this team. Something like it should be offered everywhere and
to all ages” (10009); “I would highly recommend this service!” (10094); “I have really nothing
to say to improve your services. I would like to say just keep doing what you are doing” (10124);
and “The services provided through the project were phenomenal…other cities should have such
navigation resources” (10097)
The last theme under which positive feedback fell was a more subjective sense of appreciation
and gratitude for the FNP team. The parents and youth in this sample represented a unique and
vulnerable population who had faced repeated barriers to accessing the care in the past. As a
result, when prompted for concluding feedback on their experiences, many clients used the
opportunity to simply express their appreciation and gratitude: “I am exceedingly grateful”
(10097); “I am extremely grateful that someone I knew had put me in touch with the Family
Navigation Project” (10120); “I want to thank Family Navigation for providing names of
services and people to help us through this difficult time” (10049); “I went on sheer faith in your
recommendations and grateful for the services. I have donated to your service and would be
114
pleased to help on your Board of Directors or with other parents” (10065); “I will be forever
grateful for her kind and gentle voice on the phone…we are very thankful to our navigator”
(10099); “I can’t say enough good things about the program. [Navigator] has been so helpful.
The services are a godsend” (10127); “Thanks for FNP – it was there when we needed it”
(10035); “Your service was wonderful and our contact was amazing. Thank you” (10020); “I am
grateful that I found this program” (10039); “Thank you for all your help and hard work. You are
the ONLY organization that made a connection for my nephew to obtain help…Great Work!”
(10071); and, “This is the best resource and support for families with children with mental health
issues…such a caring group of people…Thank you for helping us” (10124).
7.1.3.2 Negative outcomes
It is important to note that not all outcomes were positive; and that all mechanisms can have
unintended consequences. There were also cases in which, despite Navigators’ efforts, families’
needs and expectations were unmet. For most clients, this was not the case: only seven clients
reported negative outcomes, which was 18.4% of clients providing qualitative outcome data and
9.9% of the entire qualitative population. Recalling earlier discussions about the many
challenging contexts this population must navigate, it is not unforeseen that some families will
fail to find the full range of supports they require. As one client expressed,
“I don’t mean to slam your project. I was very excited when I learned of it. I thought we
could find help here. We have…seen absolutely no improvement in our son. I am personally
falling apart over the loss of my son” (10012).
Others reported similar outcomes: “We are still weighing our options for treatment and our child
continues to be very ill and unhappy” (10018); “My son still has reoccurring mental issues my
family and I have to deal with now” (10087); “Lots of good recommendations… unfortunately,
none have really helped (10098); “When I think about the lack of services being implemented
thus far I feel discouraged” (10100); “The provider did not align with my son and so he ended up
with no care until I changed his family doctor” (10129); and succinctly linking a non-positive
outcome with low service satisfaction, “Navigation services didn’t help her, so I can’t say I’m
very satisfied with the services” (10089).
Qualitative outcomes were subcategorized simply as positive and negative, and since 31 of 38
unique clients with unique contexts provided positive feedback, it was instead the link between
115
context and negative outcomes that stood out. In most cases, number and type of MHA concerns
were again the most common contextual factors associated with negative outcomes in this
qualitative sample. The merged dataset indicated that the seven clients who reported negative
outcomes had a higher number of mental health concerns overall (2.7 vs. 2.3 in the total client
population). In addition, four of the seven had concurrent addiction concerns, which the
quantitative results suggested was related to poorer outcomes; and the three with concurrent
addiction concerns had complex, chronic-severe conditions including personality disorders,
OCD, and autism. While the particular type of mental health concern was not found to be
significantly predictive of negative outcomes in inferential modelling, the qualitative data
suggested that type of mental health concern did influence, to some extent, the experience and
outcomes of navigation for several families in the current sample.
116
117
Chapter 8 Discussion
This concluding chapter first discusses the merged results as they relate to the research questions
of this study. Then, key findings are discussed in relation to and used to refine the original
program theory and conceptual framework for family navigation. The chapter and thesis closes
with a discussion of study limitations, strengths, implications for and contributions to the field.
Results as per the research questions, conceptual framework and program theory
This study was guided by a conceptual framework and three broad research questions, the
answers to which would describe, in depth, the clients and youth of the FNP; the impact of
family navigation on families in terms of their level of family empowerment, family quality of
life, and satisfaction with the services they received; and the influential role of context. The
results were then used to refine the original conceptual framework and program theory. It is
noted that the results reported in this study cannot be generalized beyond this sample of
convenience as both the clientele and the program have evolved since the current study was
initiated.
8.1 Research question 1
The first research question asked, “Who is the Family Navigation project serving? Is the Family
Navigation Project reaching its target population? Overall, are families satisfied with the
services they received?”
The FNP has several service mandates that help to define its target population. Broadly, the FNP
is intended to serve families of youth aged 13 to 26 with mental health and/or addiction
concerns. First, the FNP was an initiative that was started by parents, for parents. The results
indicated that over 90.0% of the sample were parents seeking help for their youth. In this respect,
the study results suggest FNP reached its clientele in this sample. When the FNP was first
designed, lived experience and extensive research by the program team highlighted two priority
sub-populations most in need of navigation: youth with concurrent mental health and addiction
concerns, and transitional-aged youth. The addition of an addiction concern(s) to mental health
118
concern(s) further complicates an already challenging mental health service context. Mental
health services and addiction services are generally provided by separate entities and sectors;
often, qualifying criteria for a service in one category precludes access to services in the other
category (CIHI, 2013; MHCC, 2012; MOHLTC, 2011; Pearson, et al., 2013). Transitional-aged
youth have to navigate an additional barrier: the transition between child and adult mental health
service systems, which are also provided by different sectors (MHCC, 2015).
The sample described in this study is consistent with the populations that the FNP is intending to
reach. Families in this sample reported a very wide range of mental health concerns, from
relatively more common and manageable conditions like anxiety and depression to complex,
chronic, severe conditions like personality and eating disorders, OCD, and bipolar disorder.
Addiction concerns ranged from cannabis and alcohol concerns (most commonly), to stimulant
and opioid use, to behavioural addictions like video games. Moreover, 45.5% reported
concurrent mental health and addiction concerns. This is consistent with the rates of concurrent
mental health and addiction concerns reported in the literature; rates of concurrent addiction
among adults with a mental disorder (and youth with major depression) typically vary from
20.0% to 50.0%, which suggests the FNP has effectively reached this population (CMHA, 2013;
Centre for Behavioral Health Statistics and Quality, 2015).
Correlational results suggest extensive interaction within and between conditions, with clients
reporting an average of 2.3 and 1.7 mental health and addiction concerns, respectively. The
associations found in this study sample tended to follow established patterns in the literature
(Kessler, et al., 2005, 2012; Nguyen, Fournier, Bergeron, Roberge, & Barrette, 2005; Pearson, et
al., 2013). For example, mental health and addiction concerns were related to both age and
gender in this sample, with older males tending toward substance use and complex mental
illnesses, whereas younger females tended toward anxiety, self-harm, eating and personality
disorders. Some concerns themselves were also highly associated and reflect the literature base,
such as the links between depression and anxiety, suicidality and self-harm, ADD/ADHD and
addiction concerns, and eating and personality disorders.
Consistent with a wide range of interactive mental health and addiction concerns, clients in this
sample had complex case histories. Nearly all had previously received some form of youth
mental health or addiction service; over a third had previous ED visits or inpatient stays; and
119
over a quarter had some history of legal involvement in their care. The high levels of ED and
inpatient use among this sample reflect the increasing trend data for ED and inpatient use for
youth mental health reasons that was recently released by CIHI (2017).
A wide range of reasons were reported for seeking navigation, from general guidance to specific
service requests. Most often, families were seeking recommendations. Since nearly all families
had already connected with the service system, it is possible that they continued to have unmet
needs and/or were dissatisfied with the recommendations given. They were also frequently
simply seeking information. This aligns with the focus group data from which the FNP was
conceived; families in the focus group reported that simply having access to comprehensive,
objective information about their treatment options was immeasurably helpful (Appendix A).
Although sample size was relatively small and this may threaten generalizability of study results,
the specific reasons clients had for contacting the FNP may reflect known gaps in preventative
community, outpatient and step-down services (CMHO, 2016a). For example, clients were
seeking services such as crisis supports, access to day treatment, and aftercare. Identifying and
addressing such gaps could have positive system-wide effects.
A number of clients were specifically seeking information on or a referral to residential
treatment. The FNP purposely developed an aptitude for navigation to residential treatment due
to the fact that this is a known, significant gap in Canadian mental health and addiction treatment
(CMHO, 2016b). There are very limited choices of residential treatment centres for youth with
mental health and/or addiction concerns in Canada; there are only a handful of centres across the
country that are typically have extremely long wait lists and restrictive out-of-pocket costs.
Moreover, mental health and addiction treatments are generally not provided concurrently, and it
can be difficult to find a residential program that will treat both concerns despite the high rate of
co-occurrence. The results of this study suggest that residential treatment services in Canada are
still inaccessible for families of youth with MHA concerns.
The FNP is an organization positioned to overcome several common barriers to accessing care.
The first barrier for many help-seeking families is geographical in that public services are
typically restricted by catchment area. However, because the FNP is privately funded, it is not
subject to the same catchment and eligibility restrictions as public services, and as such intended
to provide services to the whole of the GTA. The results of this study suggest the FNP has
120
extensive reach, with clients distributed fairly evenly across the SHSC catchment area, the City
of Toronto, and the GTA. The second mandate is to provide equitable navigation services at no
cost to families; and to recommend services that are considerate of families’ financial resources.
Clients in this sample were predominantly Caucasian and of high socioeconomic status, which is
certainly not reflective of general demographic trends among those with MHA concerns. These
results were presented to and discussed with the program team, who were able to further
contextualize the finding. The program team explained that the statistics are likely reflective of
the fact that they do not advertise their services, and that in the early stages of operation, clients
arrived at the FNP primarily through word of mouth and social networks that included members
of the FNP’s Parent Advisory Committee, who were predominantly Caucasian and of high
socioeconomic status. The program team also noted that these findings are not reflective of their
current clientele, which has grown considerably in size and variability as program reach
continuously expands. In addition, this study employed a convenience sample and it may have
been the case that families with higher socioeconomic status had contexts (such as time and/or
willingness) that better enabled them to respond to the survey.
Although the results of this study were taken from a small and non-representative convenience
sample which may not reflect all families with youth MHA concerns, they are consistent with the
determination that the FNP is reaching its target population by providing navigation services to
families of youth and young adults with mental health and/or addiction concerns.
With regard to whether or not families are satisfied with the services they received, study data
supports the conclusion that overall, clients in this particular sample were highly satisfied with
the services they received. This included both satisfaction with the FNP itself, as well as
satisfaction with the services to which the FNP referred families. Scores for satisfaction were
significantly higher than for any other outcome, which is thought to reflect both the considerably
more proximal nature of the outcome and the appropriateness of the theorized mechanisms.
8.2 Research question 2
The second research question asked, “Do families perceive the Family Navigation Project to be
providing accessible, continuous, family-inclusive care? How does context influence perceived
experience of the program?”
121
Study results suggested that the FNP provided care that most families in this sample perceived as
accessible, continuous, and family-inclusive. The results of the quantitative analyses indicated
that item-specific and total scores across all three mechanism measures were very high and the
majority of the sample was very satisfied. Several low scoring outliers on continuity and family
involvement were explained by context in the qualitative analysis. Qualitative results indicated
that most clients who rated continuity of care and family involvement items poorly tended to
either not require continuity of care or family involvement, which was typical of clients who
only wanted information; or they did not objectively have the opportunity to experience or
receive continuity of care and family involvement due to the duration of time they had been
registered with the program. This was confirmed by the quantitative results, which suggested that
a client’s number of contacts with a navigator was understandably and significantly associated
with their rating of accessibility, continuity and family involvement.
The FNP was designed to provide families with needs-based help despite varying contextual
conditions by offering them highly accessible, continuous navigation that specifically prioritizes
their involvement. If it is true that these three service mechanisms – accessibility, continuity of
care, and family involvement – are what enable them to overcome contextual barriers, then very
few contextual factors should have mattered to the FNP’s ability to navigate for families. With
regard to which factors influenced experience, the results of this study suggested the answer is
that there are a few individual-level contextual factors matter, and that there remain significant
systemic-level contextual barriers.
At the individual-level, few contextual factors were associated with perceived experience of
family navigation. Accessibility, continuity and family involvement did not appear to be
associated with typically influential factors such as demographics and type(s) of MHA
concern(s). This suggested the FNP was effectively adapting its service mechanisms to the
contexts of families in this study sample. Only two contextual factors stood out as significantly
related to the mechanisms. Suicidality concerns positively associated with all three, whereas
OCD concerns were negatively associated. The positive association with suicidality is
particularly interesting and is thought to reflect the FNP’s expertise in responding to this
particular population, in which the reported rate of suicidality was significant (17.9%). The FNP
has become particularly adept at educating families around how to handle acute youth mental
health and addiction concerns and crises, and in navigating families to youth crisis supports;
122
these were some of the most critical needs and difficult situations to manage that families
identified when the program was first conceived. On the other hand, OCD is a medically
complex, chronic and relatively less common condition for which highly specialized services are
required and while some do exist (two adult-only centres are available in Toronto), access is very
limited and hindered by the need for a physician referral and long wait times (Richter, 2017). It
could be the case that there are systemic barriers to OCD treatment in particular that the FNP is
simply unable to overcome.
The results of the qualitative analysis align with research underlying the program which suggests
that there are significant systemic-level contexts that influence families’ experience of
accessibility, continuity and family involvement. These were discussed at length in the
qualitative results chapter but to review, they include an overall inadequate supply versus the
demand for youth mental health and addiction services, especially community-based step-up and
step-down services; a particular dearth in specialized services for concurrent conditions and
transitional-aged youth; poor accessibility and continuity of care within existing services across
the system; and restrictive interpretations of privacy and capacity legislation that can impede a
family’s ability to participate in their youth’s care. Overall however, the results of this study
suggest that families perceived the care they received from the FNP as accessible, continuous,
and family-inclusive.
8.3 Research question 3
The third research question asked, “Do families who perceive the Family Navigation Project as
accessible, continuous and family-inclusive experience better outcomes in terms of family
empowerment, family quality of life, and service satisfaction? How does context influence these
outcomes?”
Study results indicated that families in this sample who perceived the FNP to have provided
accessible, continuous and family-inclusive care reported significantly better outcomes in terms
of family empowerment, FQOL, and service satisfaction.
With regard to family empowerment, results suggested the impact of perceived experience was
significant but modest in effect; this was particularly true for family empowerment at the service-
seeking level. Conceptually, family empowerment is a broad concept that is likely influenced by
123
a wide variety of interactive factors, and a distal outcome of navigation when compared to
navigation satisfaction, for example. This was reflected in the quantitative results by much more
modest correlations between mechanisms and family empowerment and quality of life outcomes
than between mechanisms and satisfaction outcomes. Nonetheless, the results of the inferential
analyses indicated that more positive perceptions of navigation, as defined by accessibility,
continuity of care and family involvement, predicted higher levels of family empowerment in the
home and in service-seeking for the families in this sample.
In terms of how context influences family empowerment, correlational analyses suggested that
age mattered, as did type(s) of MHA concern(s). Families with younger children were more
likely to be empowered in both domains, possibly because they are more in control of their
youth’s care at this age. Families with concurrent mental health and addiction concerns had
lower associated family empowerment scores in both domains. In contrast, families with
concerns about suicidality were more likely to be empowered. This may relate to several factors,
some of which have been mentioned previously. The first is that the FNP is thought to be
particularly skilled in managing cases with suicidality concerns; and suicidality concerns are also
extremely acute, families with this type of concern are likely to have received immediate
attention and consequently experienced a relatively immediate result or change in state. It is
possible that families were significantly empowered by this immediate response, as well as the
provision of specifically appropriate information and supports.
With regard to FQOL, study results suggested the impact of perceived experience was again
significant but similarly modest in effect size to the influence on family empowerment. The
results suggested that families who had more positive experiences of navigation were likely to
score significantly higher on FQOL. Similar to empowerment, suicidality predicted better
outcomes, likely for the same reasons detailed above. In contrast, again as with empowerment,
concurrent concerns significantly predicted poorer FQOL outcomes; this was true of families
who reported personality disorder concerns as well. This could be related to the inaccessibility of
resources for both these conditions.
In addition to the aforementioned individual-level contexts, the systemic-level contexts
determined to influence mechanisms were also expected to influence outcomes through their
interaction with mechanisms. That is, if these systemic-level barriers to care resulted in families
124
experiencing poor accessibility, continuity and family involvement in youth MHA treatment
planning and management, it is reasonable to expect that as a consequence of not receiving the
high quality care they needed, they would feel considerably less empowered, particularly
because their youth’s health concerns had the potential to escalate without proper care and
support. For families with less experience of negative systemic influences and/or more
experience of positive influences, FQOL could reasonably be expected to be lower, as a result.
Qualitative data from the current sample lent support to the idea that these contexts have
significant impacts on families’ empowerment and quality of life. As a result of systemic barriers
to care, families in this sample spoke of how “difficult” and “elusive” it was to find the help they
needed, and that as a result, they felt “discouraged,” “deeply sad,” “frustrated,” and “helpless.”
Overall though, the findings from this study support the idea that navigation has a significant, if
only modest, positive impact on family empowerment and family quality of life. Qualitative data
supported the impact of family navigation on family empowerment and family quality of life as
well. Families in the qualitative sample spoke extensively about feeling relieved, reassured, and
better able to manage in their daily lives as a direct result of the FNP’s involvement. They also
felt more hopeful for the future. Certainly, this can be considered a relative improvement in
FQOL for the families in this sample.
With regard to service satisfaction, inferential modelling results suggested that families who
perceived the FNP as providing accessible, continuous, family-inclusive care were more likely to
be highly satisfied with the services they received. Naturally, the impact of perceived experience
was exponentially greater on satisfaction with the FNP itself, again reflecting the proximity of
the outcome and the appropriateness of the measures. Overall model fit was substantially better
for this dependent variable (i.e. NAVSAT total score) as well. Another indication that the
mechanisms were operating as theorized is that only one of seven contextual variables identified
as significantly associated with the outcome was a statistically significant predictors of
individual differences in NAVSAT total score. While legal involvement was a barrier that was
found to be a significant negative predictor of satisfaction with navigation, more important is the
finding that satisfaction with navigation was not significantly influenced by factors like youth’s
improvement, number and type of MHA concerns, and reasons for needing navigation. In this
study, factors that were expected to be negative influencers of outcomes were not impactful. This
may suggest that the provision of navigation that is perceived as highly accessible, continuous,
125
and family-inclusive may mitigate, to some extent, the negative impact of those factors such that
families’ satisfaction with the navigation services themselves is preserved.
With regard to satisfaction with referred services, the influence of context was similarly limited.
Modelling results implied that clients were more likely to be satisfied with the referred service if
they were specifically seeking a psychiatrist. Youth psychiatrists are a limited and difficult to
access resource, and finding the right patient-doctor fit is particularly important to engage this
population (CMHO, 2016; Chovil, 2009). That the need for a psychiatrist was associated with
high satisfaction may be a reflection of the FNP’s ability to make appropriately individualized
assessments and facilitate timely access to the right psychiatrist for that youth and family’s
needs.
Again, the same systemic contexts influencing mechanisms and the other outcomes mattered for
satisfaction too. First, they mattered because they highlighted the innovation and quality of the
FNP’s services. The qualitative study results indicated that mental health and addiction services
were often inaccessible for families seeking care on behalf of their youth. In comparison, clients
valued with level of accessibility, continuity and involvement in their navigation services. They
spoke of being “thrilled with,” and “so impressed by” the fast intake and ongoing responsiveness
relative to their previous experiences; and of “finally being heard.” In contrast, clients who
expressed dissatisfaction in the qualitative sample did so not so much because of the navigation
services themselves, but because they were unable to access the services to which they were
referred, or because the service did not end up being helpful.
Overall, the study results strongly supported the conclusions that most families in this sample
were satisfied with the services they received; that perception of navigation significantly,
positively impacted satisfaction; and that context often mattered but to varying extents.
8.4 Results in relation to the conceptual framework and program theory
As a whole, the results of this study supported the original program theory that the program is
situated amid a variety of influential individual- and systemic-level contexts, but that with
appropriate inputs, Navigators are enabled to provide accessible, continuous, family-inclusive
care to families of youth with mental health and/or addiction concerns, which has a significant
126
positive impact on their level of family empowerment, family quality of life, and satisfaction
with the services they receive. This pathway was particularly true for satisfaction; a comparison
of quantitative model fit statistics suggested satisfaction with navigation was the most proximal
outcome to the proposed mechanisms; and that the proposed mechanisms had the largest
combined main effect on satisfaction with navigation compared to any other outcome.
In view of the theoretical emphasis on individual and system-level contexts presented in the
original logic model (Figure 1, Appendix B), the results of this study strongly supported the
importance of understanding the influence of context on families’ help-seeking experiences.
Quantitative results highlighted extensive associations amongst context variables themselves,
and between context variables and both mechanisms and outcomes. These reflected a range of
individual-level contextual considerations that were determined to be relevant for families,
including age, service use, number and type of mental health and/or addiction concerns, and
reasons for seeking navigation. Qualitative results further expanded on how and why contexts
impact families, both at the individual- and system-level. Age and type of mental health and/or
addiction concern, for example, may have created contexts in which, despite the efforts of the
family and service providers, the youth in question refuses to engage; and at the systemic level,
flexible interpretations of legislation may have created contexts in which families and service
providers were less able to effectively collaborate on the youth’s behalf. A larger and more
significant context for families in this sample was the systemic lack of high quality services for
youth MHA concerns, particularly if youth had concurrent concerns or was transitional-aged.
Throughout this study, the importance of understanding and accounting for context was
consistently reaffirmed.
However, qualitative results did suggest that some refinement of the program theory and
conceptual framework, particularly with regard to the range of mechanisms underlying family
navigation and how they interact, was necessary. Chapter 2 discussed the underpinnings of
realist evaluation theory, which presumes that social programs like the FNP generate outcomes
by providing resources (e.g. information, skills, or support) and/or influencing their participants’
reasoning (e.g. values, beliefs, or attitudes), which goes on to influence decision-making and
subsequent outcomes (Pawson & Tilley, 1997). More recent work in realist evaluations has
reiterated the importance of specifically identifying mechanisms as related to either resources or
127
reasoning, which allows for clearer specification of how resources and reasoning are influenced
by context, and how this is associated with the given outcomes (Dalkin, et al., 2015).
The qualitative results of this study first indicated that beyond accessible, continuous, and
family-inclusive services, several other “resources” were at play. The first was specific
knowledge, insight and expertise in youth mental health and addictions. Families benefitted
significantly from having access to a Navigator who had a strong understanding of the conditions
themselves, the range of available service options, and the nuances of the service systems. In the
original conceptual framework (Table 1, Appendix B), “accessible expertise” was listed as the
first defining component of family navigation. The underlying mechanism was proposed to be
the accessibility of expertise, but the results of this study suggested that in such a complex
service system, expertise itself is a specifically important resource for families. Clients in the
qualitative sample spoke about how their Navigator’s expertise led them to make the most
appropriate recommendations and referrals for that client’s particular needs. Qualitative data
maintained accessibility was also an important mechanism for families, but that the concept had
potentially been overextended in the original framework; clients tended to speak about
accessibility specifically in regard to their ability to tangibly connect with the program and
receive attention as needed, quickly and efficiently.
The second additional resource to which the qualitative results pointed was family support.
Family support was a theme assumed to fall under family involvement based on the definition
employed, but which instead emerged from the data, suggesting it was a unique resource that
may be more applicable to the framework than family involvement. In fact, few clients
commented specifically on their involvement in the process. Instead, families spoke more
generally about the program’s family-centredness, and specifically about support. It appears that
families may have sought, received and benefitted from a family perspective and family support,
but may not have placed as much emphasis on being involved or did not necessarily want to be
actively involved in the process. The results suggest that the broader mechanism may thus be a
family-centred perspective, with one of the specific resources being the provision of family
support.
Qualitative data also suggested that in response to these mechanistic resources, several specific
reasoning processes occurred that were then linked to client outcomes. Clients who received
128
expertise and/or support that they perceived as accessible, continuous, and/or family-centred
(according to their needs) felt significantly relieved, reassured, and/or more hopeful for the
future as a direct result. This increased sense of relief, reassurance and hopefulness represents a
change in clients’ reasoning that subsequently enabled families to better manage their daily lives
and the ongoing needs of their youth and family. The role of reasoning in elucidating how these
mechanistic resources generate changes in outcomes is compelling and important for
understanding how best to help families with youth who have mental health and/or addiction
concerns.
Altogether, the results of this study suggest several considerations for the original program
theory and conceptual framework. The refined program thus states that the Family Navigation
Project is situated amid a variety of influential individual- and systemic-level contexts, but that
with appropriate inputs, Navigators are enabled to provide accessible, continuous, family-centred
expertise and support to families of youth with mental health and/or addiction concerns, which
engenders a sense of relief, reassurance, and/or hopefulness that in turn, has a significant positive
impact on their level of family empowerment, FQOL, and service satisfaction.
129
Chapter 9 Conclusions
Limitations, mitigations and contributions
9.1 Study limitations and mitigations
This study employed a mixed methods design within a RE framework in order to better
understand the impacts of the FNP on families’ level of empowerment, quality of life, and
service satisfaction as a mutable, complex product of biopsychosocial contexts and mechanisms.
However, there were several significant design limitations in this project. First and foremost, this
evaluation relied solely on families; it did not collect data from the youth themselves. This
deliberate decision was made for several reasons: 1) in the context of youth mental health and/or
addictions, families are understood to be the primary facilitator when interacting with the service
system; 2) youth mental health and/or addiction concerns are understood by the program to be a
family disease; and 3) youth mental health and/or addiction concerns have incredibly unique and
complex pathways, and are particularly difficult to treat, meaning significant causal associations
and effects are difficult to detect. Related to the reliance on family-focused outcome measures
was the assumption that one family member’s response adequately reflects the family as a whole.
Similarly, it is important to note that in the current study, reported MHA concerns were those
that are parent-perceived, which is conceptually distinct from a formally diagnosed mental
disorder. Future research should strive to collect additional family members’ perspectives, youth
perspectives, and navigator perspectives as well.
The current study evaluated the role of context at the individual/family level, and at the systemic
level. However, organizational-level contextual factors should be considered in future research,
as there are undoubtedly unique program qualities to which desired outcomes may be attributed.
For example, the FNP is privately funded, which allows it to operate independently and outside
restrictions placed on publicly funded programs. For example, the FNP is not required to operate
within a particular catchment area, and can thus accept clients from across the GTA. Similarly,
because they are not publicly funded, they are not mandated to refer only to public services. This
allows them to refer families to the full range of available resources. This is particularly
130
important to families who are seeking care for MHA concerns for which there are few accessible
public options available (eating or personality disorders, for example).
Another main limitation was the lack of comparator group. Because this program was both new
and unique, a cross-sectional design relying on self-report data, without a comparison or control
group was necessary. However, employing an RE framework mitigated these drawbacks to some
extent by prioritizing patient-reported outcomes and a robust understanding of context, which
was believed to be more appropriate and pragmatic, given the family-focused goals of the study.
History and maturation threats to validity still exist though, as “navigation” has become a
buzzword and policy changes that affect empowerment, quality of life, and satisfaction scores
across the system may have arisen since the study was first conceptualized. Mental health and
addiction conditions are also evolving and recurrent in nature, especially among youth;
perceptions and outcomes may have been influenced by the natural progression of the illness(es)
and accumulation of missed opportunities. Again, RE attempted to mitigate this with a thorough
description of context (Pawson & Tilley, 1997). Nonetheless, it would be interesting to pursue a
longitudinal design in the future.
The results of the study indicated that the overall sample of youth was slightly older than
expected; the majority of the sample was over the age of 19. As such, it is likely that the
“child/youth” language in some outcome measures, such as the FES, may not have been as
appropriate as anticipated. Some clients specifically noted in qualitative fields that specific
questions, such as those related to school work or extracurricular activities, were inappropriate.
There are, however, limitations with the qualitative data as well, particularly in terms of the
manner in which it was collected. Open-ended prompts following each of the quantitative
measures were originally intended to gather supplemental qualitative data that would help to
contextualize and further understand respondents’ scores. However, these prompts unexpectedly
yielded spontaneous, highly extensive, and rich descriptions of families’ experiences. While the
originally proposed study design included semi-structured in-person interviews as a follow up to
survey responses, the selected theoretical framework - Realist Evaluation - strongly encourages
adaptability and responsiveness throughout the course of an evaluation. As such, following a
discussion with the program team and the PI’s supervisory committee with regard to 1) the
surprising richness and quality of the qualitative data resulting from the open prompts; and 2) the
131
ethical considerations related to the burden of asking families to attend in-person interviews after
so many had voluntarily provided rich accounts that touched on many of the proposed interview
topics, a decision was made by the PI to proceed with a descriptive analysis of the qualitative
data resulting from the open prompts.
RE as an evaluative framework also has its limitations. Because of the methodological
flexibility, there is a lack of clarity and explicit direction in the field around how to generate C-
M-O configurations from the data (Dalkin, et al., 2015). However, the current study proposed
and expanded upon a method that has previously been clearly articulated in the literature (Byng,
et al., 2005); and it is the hope of the PI that the current study will serve as another example of
how C-M-O configurations can be constructed from both quantitative and qualitative data.
This this study (and RE studies in general) resulted in data that is highly contextualized and as
such, is not intended to be generalizable to other settings. This is not to say it will not still be
useful. To enhance external validity, the Theory of Proximal Similarity was used to ensure the
context and sample itself was extensively described and analyzed in relation to both mechanisms
and outcomes such that future researchers have the information necessary to adequately interpret
the relevance and implications of the findings in light of their specific contexts (Campbell &
Stanley, 1963). This will be particularly relevant for other jurisdictions looking to implement
similar service models.
External validity was deprioritized in favour of a high degree of internal validity in this study.
Extensive efforts were undertaken to ensure that the underlying program theory, study
objectives, research question and methods were stakeholder-driven and appropriately validated.
This approach, along with the use of mixed methods to mitigate the limitations of each individual
method and enhance robustness of the data, although time-intensive, lent heavily to the internal
validity and utility of the study. Other favourable aspects included the participatory nature of the
population and the quickly growing roster of registered families, which helped to minimize
concerns about adequate survey response rates and sample sizes. It is noted, though, that the
overall response rate in this study was quite low. Although inferential models had sufficient
sample sizes to accommodate the proposed predictors for each outcome, a larger sample would
have provided increased opportunity to detect individual differences in score; and perhaps the
three individual mechanisms could have been tested separately.
132
Another limitation is the fact that this study employed a convenience sample that was likely
biased against families with more challenging contexts that would deter participation; and the
contexts that deter participation could similarly be the contexts in which less positive
experiences and outcomes may have been observed. Similarly, it is possible that the sample
opting to provide qualitative data was biased toward families whose contexts, experiences and/or
outcomes were more complex, compelling them to volunteer additional qualitative information.
In addition, as the program has grown and extended its reach in throughout the GTA, the
clientele is expected to have evolved as well. This expectation was confirmed by program team,
who indicated that while most descriptions of the study sample aptly described their current
client roster (such as the types of reported MHA concerns, proportion of concurrent cases and
average length of stay), other characteristics of the study sample (namely the high proportion of
Caucasian clients and those with high household incomes) were no longer true. As such, the
results of this study are not considered generalizable outside the study sample.
Quantitative mechanism and outcome measures were employed so that inferential modelling of
the desired outcomes could be performed. However, there are limitations in using quantitative
measures to capture complex processes and outcomes like family empowerment and quality of
life as quantitative measures may not have sufficiently accounted for the wide range of possible
internal and external influences on an individual response. As a result, in modelling the
quantitative responses, identifiable relationships were generally quite modest in effect size. It is
likely the case that these concepts were better captured and elucidated qualitatively in present
study. Also, qualitative feedback could inform quantitative tools that may better capture family
experience. In this study, modelling results are believed to be sufficiently supplemented by the
qualitative data.
9.2 Contributions
The current study has important implications and makes several contributions to the field.
Internally, the results of this study offer an evaluation of the FNP’s efforts by highlighting
several service strengths, as well as opportunities for quality improvement. It also offers the FNP
conceptual and measurement frameworks populated with baseline data, which will be used to
inform and/or supplement future evaluations. These frameworks are worthwhile contributions to
the literature base as well; recall from the rationale for the current study (Section 3.3) that
133
navigation is increasingly being employed across a range of health care settings, despite the lack
of literature around conceptualization and evaluation. This study has yielded both a conceptual
framework for family navigation and a mixed methods measurement framework for evaluation,
as well as a series of testable program theories to guide future research. Lastly, these frameworks
offer a model for family navigation that can help with implementation in other jurisdictions,
which was a stated program goal. The proposed navigation continuum of care may be
particularly useful for similar or emerging navigation initiatives as a means of locating their
particular initiative within the wider spectrum of navigation service models.
This study also contributes to the evaluation literature by applying a relatively novel framework
to a novel context. In doing so, it generates another example of how and why Realist Evaluation
can be appropriate for evaluating complex health interventions. As mentioned previously, this
study also contributes to the evaluation literature a clearly articulated method for constructing C-
M-O configurations.
Most importantly, this study thoroughly explains what family navigation is and demonstrates its
role in the system. The results of this study lend support to the value of the FNP’s service model
as viable way in which we can practically help families navigate the system in order to better
access the right care, in the right place, at the right time. It also provides an in-depth description
of who, in the GTA, is currently using navigation services and for what reasons. In doing so, it
highlights some of the major barriers families are facing in accessing appropriate mental health
and/or addiction care for their youth. This is important information for future service planning.
Further, the study contributes to a literature base in which academic studies on both navigation
and RE are very much limited. The hope is that readers of this study will develop a detailed and
nuanced understanding of the local youth mental health and/or addiction services systems; what
family navigation is; who is using it and for what reasons; and how and why it helps families.
134
References
Al-Abri, R., & Al-Balushi, A. (2014). Patient satisfaction survey as a tool toward quality
improvement. Oman Medical Journal, 29(1), 3-7.
Anderson, J.E., & Larke, S.C. (2009). Navigating the mental health and addictions maze: A
community-based pilot project of a new role in primary mental health care. Mental
Health in Family Medicine, 6, 15-19.
Armstrong, R.A. (2014). When to use the Bonferroni correction. Journal of the College of
Optometrists, 34(5), 502-508.
Bakas, T. (2014). Bakas caregiving outcomes scale. Encyclopedia of Quality of Life and Well-
Being Research, pp. 319-321. Netherlands: Springer.
Barr, J.K., et al. (2006). Using public reports of patient satisfaction for hospital quality
improvement. Health Services Research, 41(3 Pt 1), 663-683.
Battaglino, L. (1987). Family empowerment through self-help groups. New Directions for
Mental Health Services, 34, 43-51.
Bhaskar, R. (1979). The possibility of naturalism: A philosophical critique of the contemporary
human science. Atlantic Highlands, NJ: Humanities Press.
Boyd, R.N. (1989). What realism implies and what it does not. Dialectica, 43, 5-29.
Bryson, J.M. (2011). Strategic planning for public and nonprofit organizations: A guide to
strengthening and sustaining organizational achievement (Volume 1). New Jersey: John
Wiley & Sons.
Byng, R., et al. (2005). Using realistic evaluation to evaluate a practice-level intervention to
improve primary healthcare for patients with long-term illness. Evaluation, 11 69-93.
Cabin, R.J., & Mitchel, R.J. (2000). To Bonferroni or not to Bonferroni: When and how are the
questions. Bulletin of the Ecological Society of America, 81(3), 246-248.
Campbell, C., et al. (2010). Implementing and measuring the impact of patient navigation at a
comprehensive community cancer center. Oncology Nursing Forum, 37(1), 61-68.
Campbell, D.T. & Stanley, J.C. (1963). Experimental and quasi-experimental designs for
research. Chicago, IL: Rand McNally.
Canadian Institute for Health Information. (2013). Hospital mental health services for
concurrent mental illness and substance use disorders in Canada. Retrieved from
https://www.cihi.ca/en/hospital-mental-health-services-for-concurrent-mental-illness-
and-substance-use-disorders-in-canada
Canadian Institute for Health Information. (2017). Child and youth mental health in Canada –
Infographic. Retrieved from https://www.cihi.ca/en/child-and-youth-mental-health-in-
canada-infographic
Canadian Mental Health Association. (2006). Caring together: Families as partners in the
mental health and addiction system. Retrieved from
https://ontario.cmha.ca/documents/caring-together-families-as-partners-in-the-mental-
health-and-addiction-system/
135
Canadian Mental Health Association. (2013). Concurrent disorder services in Ontario: An
environmental scan. Retrieved from https://ontario.cmha.ca/documents/concurrent-
disorder-services-in-ontario-an-environmental-scan/
Center for Behavioral Health Statistics and Quality. (2015). Behavioral health trends in the
United States: Results from the 2014 National Survey on Drug Use and Health (HHS
Publication No. SMA 15-4927, NSDUH Series H-50). Retrieved from
http://samhsa.gov/data
Children’s Mental Health Ontario. (2016a). Breaking point – a system stretched beyond its
limits: A report on community-based children’s mental health centres. Retrieved from
http://cmho.org/education-resources/cmho-s-latest-work/item/511-breaking-point-a-
system-stretched-beyond-its-limits
Children’s Mental Health Ontario. (2016b). Residential treatment: Working towards a new
system framework for children and youth with severe mental health needs. Retrieved
from https://www.kidsmentalhealth.ca/fr/education-resources/cmho-s-latest-work
Chovil, N. (2009). Engaging families in child and youth mental health: A review of best,
emerging and promising practices. Vancouver, BC: F.O.R.C.E. Society of Kids’ Mental
Health. Retrieved from
http://www.excellenceforchildandyouth.ca/sites/default/files/resource/EIS_Family_Engag
ement_EN.pdf
Conference Board of Canada. (2014). Final report: An external evaluation of the Family Health
Team (FHT) initiative. Retrieved from http://www.conferenceboard.ca/e-
library/abstract.aspx?did=6711
Corporate Research Associates Inc. (2004). Cancer patient navigation evaluation: Final report.
Nova Scotia, Canada: Cancer Care Nova Scotia. Retrieved from
http://www.cancercare.ns.ca/site-
cc/media/cancercare/patientnavigationevaluationfindings.pdf
Curtin, F., & Schulz, P. (1998). Multiple correlations and Bonferroni’s correction. Biological
Psychiatry, 44, 774-777.
Curtis, W.J., & Singh, N.N. (1996). Family involvement and empowerment in mental health
service provision for children with emotional and behavioural disorders. Journal of Child
and Family Studies, 5(4), 503-517.
Dalkin, S.M., Greenhalgh, J., Jones, D., Cunningham, B., & Lhussier, M. (2015). What’s in a
mechanism? Development of a key concept in realist evaluation. Implementation Science,
10, 49.
Dillman, D.A., et al. (2014). Internet, phone, mail and mixed-mode surveys: The Tailored Design
Method. 4th ed. Canada: Wiley.
Driscoll, D.L., Appiah-Yeboah, A., Salib, P., & Rupert, D.J. (2007). Merging qualitative and
quantitative data in mixed methods research: How to and why not. Ecological and
Environmental Anthropology (University of Georgia), 3(1), 19-28.
Dunst, C.J., Trivette, C.M., Davis, M., & Cornwell, J. (1988). Enabling and empowering families
of children with health impairments. Children’s Health Care, 17(2), 71-81.
136
Field, A.P. (2009). Discovering statistics using SPSS: (and sex and drugs and rock ‘n’ roll).
Thousand Oaks, CA: SAGE Publications.
Fillion, L., et al. (2006). Implementing the role of patient navigator nurse at a university hospital
centre. Canadian Oncology Nursing Journal, 16(1), 11-17.
Fishman, K., & Levitt, A. (2014). Unpublished. Available upon request.
Freeman, H. (2011). As quoted in: Fayerman, P. (2011, February 25). The father of patient
navigation. The Vancouver Sun. Retrieved from
http://www.vancouversun.com/health/father+patient+navigation/4305453/story.html
Freeman, H.P., & Rodriguez, R.L. (2011). History and principles of patient navigation. Cancer,
117(suppl 15), 3527-2540.
Freund, P.D. (1993). Professional role(s) in the empowerment process: “Working with” mental
health consumers. Psychosocial Rehabilitation Journal, 16(3), 65-73.
Gill, T., & Renwick, R. (2007). Family quality of life and service delivery for families with
adults who have developmental disabilities. Journal on Developmental Disabilities,
13(3), 13-36.
Green, S.B. (1991). How many subjects does it take to do a regression analysis? Multivariate
Behavioral Research, 26, 499-510.
Haggerty, J.L., et al. (2003). Continuity of care: A multidisciplinary review. British Medical
Journal, 327, 1219-1221.
Hardin, J., & Hilbe, J. (2007). Generalized linear models and extensions. 2nd ed. College Station:
Stata Press.
Hoffman, L., Marquis, J., Poston, D., Summers, J.A., & Turnbull, A. (2006). Assessing family
outcomes: Psychometric evaluation of the beach centre family quality of life scale.
Journal of Marriage and Family, 68(4), 1069-1083.
Hook, A., Ware., L., Siler, B., & Packard, A. (2012). Breast cancer navigation and patient
satisfaction: Exploring a community-based patient navigation model in a rural setting.
Oncology Nursing Forum, 39(4), 379.
IBM Corporation. (2013). IBM Statistical Package for the Social Sciences, Version 11.0 for Mac.
Armonk, NJ: IBM Corp.
Institute for Clinical Evaluative Sciences. (2017). The mental health of children and youth in
Ontario: 2017 scorecard. Retrieved from https://www.ices.on.ca/Publications/Atlases-
and-Reports/2017/MHASEF
Isaacs, B.J. et al. (2007). Development of a family quality of life survey. Journal of Policy and
Practice in Intellectual Disabilities, 4, 178.
Kessler, R.C., et al. (2005). Lifetime prevalence and age-of-onset distributions of DSM-IV
disorders in the National Comorbidity Survey Replication. Archives of General
Psychiatry, 62(6), 593-602.
Kessler, R.C., et al. (2012). Lifetime comorbidity of DSM-IV disorders in the NCS-R
Adolescent Supplement (NCS-A). Psychological Medicine, 42(9), 1997-2010.
137
Klein, D.M., & White, J.M. (1996). Family theories: An introduction. Thousand Oaks, CA: Safe
Publications, Inc.
Kopp, J. (1989). Self-observation: An empowerment strategy in assessment. Social Casework,
70, 276-284.
Koren, P.D., DeChillo, N., & Friesen, B.J. (1992). Measuring empowerment in families whose
children have emotional disabilities: A brief questionnaire. Rehabilitation Psychology,
37(4), 305-321.
Koren, P.E., et al. (1997). Service coordination in children’s mental health: An empirical study
from the caregiver’s perspective. Journal of Emotional and Behavioural Disorders, 5,
162-172.
Lazear, K., Worthington, J., & Detres, M. (2004). Findings compendium: Issue brief 5,
Helpfulness of formal services, family organizations and informal supports. Tampa,
Florida: University of South Florida, Louis de la Parte Mental Health Institute, Research
Training Centre for Children’s Mental Health.
Leplin, J. (1981). Truth and scientific progress. Studies in History and Philosophy of Science, 12,
269-292
Lindsey, J.K., & Jones, B. (1998). Choosing among generalized linear models applied to medical
data. Statistics in Medicine, 17, 59-68.
McAllister, M., Dunn, G., Payne, K., Davies, L., & Todd, C. (2012). Patient empowerment: The
need to consider it as a measureable patient-reported outcome for chronic conditions.
BMC Health Services Research, 12, 157.
McCammon, S.L., Spencer, S., & Friesen, B.J. (2001). Promoting family empowerment through
multiple roles. Journal of Family Social Work, 5, 1-24.
McCullagh, P., & Nelder, J.A. (1989). Generalized linear models. 2nd ed. London: Chapman and
Hall.
McDonald, K.M., et al. (2007). Volume 7: Care coordination. In Shojania, K.G., et al. (eds).
Closing the quality gap: A critical analysis of quality improvement strategies (technical
review 9). Rockville, MD: Agency for Healthcare Research and Quality. Retrieved from
https://www.ncbi.nlm.nih.gov/books/NBK44015/
McPhee, J., Syed, T., Nunes, M., & the Mobilizing Minds Research Group. (2012). An
environmental scan of knowledge translation, mental health literacy, and policy/system
change initiatives aimed at improving the mental health of youth and young adults in
Canada. Retrieved from https://mindyourmind.ca/sites/default/files/Full%20Report%20-
%20Environmental%20Scan.pdf
Mental Health Commission of Canada. (2012). Changing directions, changing lives: The mental
health strategy for Canada. Retrieved from http://strategy.mentalhealthcommission.ca
Mental Health Commission of Canada. (2015a). Informing the future: Mental health indicators
for Canada. Retrieved from
https://www.mentalhealthcommission.ca/English/document/68796/informing-future-
mental-health-indicators-canada
138
Mental Health Commission of Canada. (2015b). The mental health strategy for Canada: A youth
perspective. Retrieved from
https://www.mentalhealthcommission.ca/English/document/72171/mental-health-
strategy-canada-youth-perspective
Mental Health Commission of Canada. (2017). Strengthening the case for investing in Canada’s
mental health system: Economic considerations. Retrieved from
https://www.mentalhealthcommission.ca/English/case-for-investing
Messick, S. (1990). Validity of test interpretation and use. Research Report 90-11. Education
Testing Service.
Miller, R.W. (1987). Fact and method: Explanation, confirmation and reality in the natural and
social sciences. Princeton: Princeton University Press.
Ministry of Health and Long Term Care. (2005). FHT guide to collaborative team practice.
Retrieved from
https://scele.ui.ac.id/berkas_kolaborasi/konten/MKK_2014genap/family_team.pdf
Ministry of Health and Long Term Care. (2009). Every door is the right door: Towards a 10-
year mental health and addictions strategy. A discussion paper. Retrieved from
http://ontario.cmha.ca/wp-content/uploads/2016/08/Every-Door-the-Right-Door-July09-
MH-discussion-paper.pdf
Ministry of Health and Long Term Care. (2011). Open minds, healthy minds: Ontario’s
comprehensive mental health and addictions strategy. Retrieved from
http://health.gov.on.ca/en/common/ministry/publications/reports/mental_health2011/men
talhealth.aspx
Mpinga, E.K., & Chastonay, P. (2011). Satisfaction of patients: A right to health indicator?
Health Policy, 100(2-3), 144-150.
Nguyen, C.T., Fournier, L., Bergeron, L., Roberge, P., & Barrette, G. (2005). Correlates of
depressive and anxiety disorders among young Canadians. Canadian Journal of
Psychiatry, 50(10), 620-628.
Nunnally, J.C. (1978). Psychometric theory. 2nd ed. New York: McGraw Hill.
Ontario Hospital Association. (2016). A practical guide to mental health and the law in Ontario.
Retrieved from
https://www.oha.com/Legislative%20and%20Legal%20Issues%20Documents1/A%20Pr
actical%20Guide%20to%20Mental%20Health%20and%20the%20Law%20in%20Ontari
o%20%282012%29%20%28PUBLICATIONS%29.pdf
Ontario Human Rights Commission. (2015). By the numbers: A statistical profile of people with
mental health and addiction disabilities in Ontario. Retrieved from:
http://www.ohrc.on.ca/en/numbers-statistical-profile-people-mental-health-and-
addiction-disabilities-ontario
Park, J., et al. (2003). Toward assessing family outcomes of service delivery: Validation of a
family quality of life survey. Exceptional Children, 68, 151-170.
Pautler, K. (2005). Annotated bibliography of collaborative mental health care. Mississauga,
Ontario: Canadian Collaborative Mental Health Initiative.
139
Pawson, R., & Tilley, N. (1997). Realist Evaluation. London: Sage.
Pearson, C., Janz, T., & Ali, J. (2013). Mental and substance use disorders in Canada. Health at
a Glance. (Catalogue no. 82-624-X). Ottawa, ON: Statistics Canada.
Pedersen, A., & Hack, T. (2010). Pilots of oncology health care: A concept analysis of the
patient navigator role. Oncology Nursing Forum, 37(1), 55-60.
Penchansky, R., & Thomas, W.J. (1981). The concept of access: Definition and relationship to
consumer satisfaction. Medical Care, 19(2), 127-140.
Putnam, H. (1982). Three kinds of scientific realism. Philosophical Quarterly, 32, 195-200.
QSR International Pty Ltd. (2015). N. Vivo qualitative data analysis software v.10 for Mac.
Renwick, R., Brown, I., & Raphael, D. (1998). The family quality of life project: Final report.
Report to the Ontario Ministry of Community and Social Services. Toronto, ON: Centre
for Health Promotion, University of Toronto. Retrieved from
http://sites.utoronto.ca/qol/projects/pwdd.htm
Rhemtulla, M., Brosseau-Liard, P.E., & Savalei, V. (2012). When can categorical variables be
treated as continuous? A comparison of robust continuous and categorical SEM
estimation methods under suboptimal conditions. Psychological Methods, 17(3), 354-
373.
Richardson, G.P., & Anderson, D.F. (1995). Teamwork in group model building. System
Dynamics Review, 11(2), 113-137.
Richter, P. (2017). Access to OCD treatment is limited. And that’s a problem. The Huffinton
Post. Retrieved June 14, 2017 from http://www.huffingtonpost.ca
Roberts, A., & Schmidt, N. (2012). The Family Navigation Project business case. Toronto, ON:
Author.
Robinson-White, S., Conroy, B., Slavish, K.H., & Rosenzweig, M. (2010). Patient navigation in
breast cancer: A systematic review. Cancer Nursing, 33(2), 127-140.
Rogers, P.J. (2008). Using programme theory for complicated and complex programmes.
Evaluation, 14(1), 29-48.
Samuel, P.S., Rillotta, F., & Brown, I. (2012). The development of family quality of life
concepts and measures. Journal of Intellectual Disability Research, 56(2), 1-16.
Samuel, P.S., Rillotta, F., & Brown, I. (2012). The development of family quality of life
concepts and measures. Journal of Intellectual Disability Research, 56(2), 1-16.
Sandelowski, M. (2000). Whatever happened to qualitative description? Research in Nursing &
Health, 23, 334-340.
Seek, A., & Hogle, W.P. (2007). Modeling a better way: Navigating the healthcare system for
patients with lung cancecr. Clinical Journal of Oncology Nursing, 11(1), 81-85.
Singh, N.N. (1995). In search of unity: Some thoughts on family-professional relationships in
service delivery systems. Journal of Child and Family Studies, 4, 3-18.
Sirotich, F., & Durbin, A. (2014). Identifying the needs of complex health populations receiving
community mental health and addictions services: An analyses of Ontario Common
140
Assessment of Need (OCAN) data for case management and supportive housing
programs. Final report. Toronto, ON: Canadian Mental Health Association.
Smith, J.P., & Smith, G.C. (2010). Long-term economic costs of psychological problems during
childhood. Social Science & Medicine, 71(1), 110-115.
Staples, L.J. (1990). Powerful ideas about empowerment. Administration in Social Work, 14,
p.30.
Statistics Canada. (2006). The human face of mental health and mental illness in Canada.
Retrieved from http://www.phac-aspc.gc.ca/publicat/human-humain06/index-eng.php
Studenmund, A.H. (2010). Using econometrics: A practical guide. New Jersey: Harlow Pearson
Education.
Tabachnick, B.L., & Fidell, L.S. (2007). Using multivariate statistics. 5th ed. Boston, MA:
Pearson Education, Inc.
Tannenbaum, L.G. (2001). Parent/professional perceptions of collaboration when viewed in the
context of Virginia’s Comprehensive Services Act system of care. Dissertation.
Blacksburg, VA: Virginia Polytechnic Institute and State University. Retrieved from
http://theses.lib.vt.edu/theses/available/etd-12192001-
122745/unrestricted/TannenbaumRevised.pdf
Turnbull, A.P., et al. (2000). Enhancing quality of life for families of children and youth with
disabilities in the United States. In Turnbull, A., Brown, I., & Turnbull, R.H. (eds).
Families and People with Mental Retardation and Quality of Life: International
Perspectives. Washington, DC: American Association on Mental Retardation.
VanVoorhis, C.R.W., & Morgan, B.L. (2007). Understanding power and rules of thumb for
determining sample sizes. Tutorials in Quantitative Methods for Psychology, 3(2), 43-50.
Vennix, J.A.M. (1996). Group model building: Facilitating team learning using system
dynamics. Chichester: John Wiley & Sons.
Ware, J.E., Synder, M.K., Wright, W.R., & Davies, A.R. (1983). Defining and measuring patient
satisfaction with medical care. Evaluation and Program Planning, 6(3-4), 247-263.
Wood, G.M. (2004). Health care reform tracking project (HCRTP): Promising approaches for
behavioral health services to children and adolescents and their families in managed care
systems. Family involvement in managed care systems (FMHI Publication #211 – p.6).
Tampa, FL: Louis de la Parte Florida Mental Health Institute, University of South
Florida.
Yong, A.G., & Pearce, S. (2013). A beginner’s guide to factor analysis. Tutorials in Quantitative
Methods for Psychology, 9(2), 79-94.
141
142
Appendix A Themes from the Family Navigation Project’s Focus Groups1
1. Access to accurate information: Parents want access to good, objective information
about the nature of the problem and about treatment options and various providers.
a. They want realistic information about their child’s difficulties and up-to-date
information about the range of treatment options and possible outcomes.
b. Parents also want help in weighing and evaluating the information and
recommendations that they are receiving from different sources.
c. Parents want to know about preventative resources earlier on and would like
guidance about when and how strongly to advocate for treatment. As examples they
posed these questions: “Do you wait until your child is in crisis? At what point do
you accept that there is a problem and that something must be done? How active do
you get in pushing the alternative of treatment?”
2. Guidance and support: Parents would like to have a “live person” involved in the person
of a mentor or guide who will help them to answer critical questions about the process.
a. They want the opportunity to review the information they are receiving with an
experienced and objective professional who can help them to assess the information
and figure out what would be a good fit for their child and themselves.
b. They would like to have someone on board whose job it is to listen to their concerns
and provide direction and support.
c. Parents feel that it is not enough to put a hot line in place – they need to be able to
talk to a responsive, interactive and caring person from the time they make the initial
call.
d. As a possible prototype of the kind of navigator they would like to see, parents
identified a professional group in the US known as Therapeutic and Educational
Placement Specialists (also known as Educational Consultants) who consult to
1 Sourced from the Family Navigation Project Business Case (Roberts, A. & Schmidt, N., 2012
143
parents about treatment possibilities and options within therapeutic programs such as
therapeutic boarding schools, wilderness programs and residential treatment centres.
3. Consistent, ongoing involvement: Parents describe feeling as if they are travelling on a
rudderless ship without a compass and with little experience or knowledge of what to
expect to guide their way. They wonder, “Where do you go? What is out there? To whom
do you turn?” They want a guide who is “hands on” and will “get in the boat with them” to
help them navigate their way through the process. They want someone who will stay
involved.
a. Parents want solid information about the barriers they are likely to meet along the
way and how they can go about circumventing them.
b. They want help in changing course if they or the youth discovers that the program or
therapist is not the right one. Parents want to know what to do next if the situation is
not improving.
4. Meaningful role and participation: Most importantly, most parents want to be actively
engaged in their child’s care. However, they struggle to know how to be involved as a
parent – e.g., whether to be more active or to back off and just be there for the youth.
They generally know that there is learning and healing that needs to take place on their
own part, but may not know where to start or who would help them.
a. Parents want someone to talk to about what they are learning as they go through the
process.
b. Parents want to be meaningfully involved in their child’s treatment program. They
want to participate in providing information and feedback to the evaluators and they
are deeply interested in receiving information about whether the child is attending
and progress is being made.
c. Most parents want support for themselves in order to cope with the ongoing
situation, and many want help in learning more about their own role in the situation
and potentially in the solution.
5. Tailoring and matching of approach: Parents and the people with the mental health
concerns want information about the nuance of a particular treatment approach or provider
144
and an understanding of whether or not a particular therapist and/or approach is likely to
be a good match for them or their child. Often they are looking for something different or
something more after trying things on their own or seeking help through the family doctor,
the school or a clinic. In spite of everyone’s best efforts, things are not improving. In
particular, they would like help in answering the following questions:
▪ How do we know whether a particular therapist, treatment provider or program is
right for our child?
▪ How will we as parents be involved and what kind of information about progress can
we expect to receive? Will we be informed regularly?
▪ Who will be our child’s advocate as we go through this? Who will talk to us?
▪ How will we know if things are not working out? How will we go about changing
course if a particular therapeutic approach or program is not working out?
6. Attention and support at critical junctures: Parents especially look for guidance and
support at critical junctures such as an inpatient admission or when, after all kinds of
efforts by the family and professionals alike, it is apparent that out-of-home residential
treatment needs to be contemplated.
a. With any kind of group therapy program whether in a residential or hospital setting,
parents want to know what the peer influences will be. They want to know who will
be in the program with their child and how to be sure that the peer culture is a
positive one and that the child is not going to get worse.
b. Parents also want to be sure that the program is not going to be punitive (e.g.,
wanting to be sure “that we are not sending our child to a prison”).
c. They want to know how visitation times/periods work, whether they will be involved
in family therapy sessions, and – in the case of a distant residential treatment
program – how they will be meaningfully involved (e.g., through telephone or Skype
family sessions and/or therapeutic letter-writing and/or on-site visits).
d. Especially in the case of a distant program, parents want assurance that they will be
informed if things are or are not working out and receive help in changing course, if
necessary.
145
e. Parents whose youth are in inpatient or residential programs would like to receive
help with aftercare or transition planning for when the youth is to leave the program
and/or go home.
7. Peer support: Parents want access to other parents who have been through similar
experiences and can envision groups of parents who have been through the process acting
as resources to others. In particular, contact with other parents can help people get past
stigma, denial and embarrassment so that these issues do not become barriers in their lives.
8. Accessibility: Parents want access to help outside of normal business hours; “crisis don’t
occur between 9-5pm” was a common sentiment. This means after-hours clinics or crisis
lines, mobile consultations, ability to communicate by any means available (Skype, email).
They also want a service whose offerings are not limited solely to those in the
neighbourhood. And lastly, they want a service facilitates equal access for everyone, not
just those with the financial resources for private treatment.
9. Options: Parents are looking for information about the full range of possible treatment
options including local private practitioners and private treatment programs and US and
international options.
a. Parents would like information about treatment programs that offer alternatives to
the current orthodoxies of the public system (e.g., they are interested in learning
about programs with abstinence-based rather than harm reduction approaches to
substance abuse treatment and about relationship-based residential treatment rather
than strictly behavioural approaches).
b. Parents would like to know about private and fee-for-service options.
c. They would like help in connecting to US Therapeutic and Educational Placement
Specialists when needed.
d. They would like to have information regarding costs of private programs within
Canada and the U.S., about applying for reimbursement from OHIP for US
programs, and about consent issues locally and in relation to US residential facilities.
10. Use of evidence and data in decision-making: Parents would like to see a navigational
service that makes intelligent use of data by tracking outcomes over time in order to create
146
a database of “collective wisdom”. In addition, they would like to see the development of
reliable, on-line resources for parents.
11. Acknowledgement and hope: Most importantly, parents want a place to go where they can
get acknowledgement that something is wrong and hope that there is a way forward. Part of this
will come from contact with other parents who have been through similar situations and have
found some answers and solutions. Part will come from learning that there is a range of options
and a critical pathway to help them move forward.
147
Appendix B Logic Model and Conceptual Framework
Figure 1. Logic Model for the Family Navigation Project
148
Table 1. Conceptual framework for family navigation
Sphere of
Influence Components Definition
Theorized
Mechanism
Family-
level
(service-
client
interaction)
Accessible expertise Services are organized to respond to families' needs; phone and email-based, extended and
flexible hours, meeting space; medical consultation and supervision available; Navigator
experience and expertise
Accessibility
Family education Navigators provide families with information and resources aimed at improving their
understanding of the problem itself, and the roles of the family, Navigators, and health care
system in the recovery process
Accessibility
Resource assessment
and information
sharing
Navigators provide appropriate and sufficient information on which resources are available
Accessibility
Resource matching Navigators identify the resources that best meet the family's identified needs Accessibility
Referrals Navigators directly connect families and facilitate relationships with identified resources or
services Accessibility
Connections to peer
support
Navigators provide families with connections to support networks of peers with lived
experience Accessibility
Family-based
perceived needs
assessment
Navigators discuss and determine the youth's and family's perceived needs and perceived
barriers to care Family
involvement
Family-based
collaborative care
planning
Navigators engage and work closely with families to develop care plans with clear steps and
supports that meet their perceived needs and overcome perceived carriers Family
involvement
Consistent family
engagement
Navigators consistently engage with families in all steps throughout the process Family
involvement
Information
dissemination and
Navigators ensure continuous and current information exchange across care settings so
families perceive transitions as seamless
Continuity of
care -
149
exchange informational
Ongoing follow up
and response to
changing needs
Navigators communicate with families regularly to continuously monitor progress and adjust
course as necessary throughout the process
Continuity of
care -
management
Relationship and
trust-building
Navigators strive to develop a relationship and build rapport with families by providing care
and communicating in a professional, respectful and responsive manner
Continuity of
care -
relational
Service/Syst
em-level
Facilitate service-
system relationships
Navigators visit and build relationships with service providers in the system
Out of scope
Promote evaluation
and accountability
Navigators evaluate their own service and other resources in order to promote efficiency and
effectiveness in the system through evidence-based care
Promote education
and awareness for
youth MHA
The Family Navigation Project conducts research and shares findings at educational events
targeted at a wide range of stakeholders and settings
Advocate for youth
MHA and
navigation services
The Family Navigation Project advocates for its cause on a variety of platforms using
primary data
Competency training Navigators must be highly skilled and experienced mental health and/or addiction
professionals.
150
Appendix C Measurement Summary
Table 1. Measurement summary: Context variables
Context measures
Variable Description Data type Collection
method Source
Client type Primary contact’s relation to youth Categorical
Chart
abstraction
Client
charts
Client status Clients are classified as active or inactive based on recency and frequency of
contact Dichotomous
Time since registration Number of weeks since initial registration Continuous
Age Age of youth for whom help is being sought Categorical
Gender Gender of youth for whom help is being sought Categorical
Mental health
Yes/no Parent-reported mental health concerns of youth for whom help is being
sought Dichotomous
Type(s) Parent-reported DSM-V category or dimension Categorical
Substance use
Yes/no Parent-reported substance use concerns of youth for whom help is being
sought Dichotomous
Type(s) Parent-reported DSM-V substance dependencies Categorical
Family/living situation Family structure and living arrangement of youth for whom help is being
sought Categorical
Education Education level of youth for whom help is being sought Categorical
151
Legal involvement Presenting legal issues requiring involvement of external stakeholders Dichotomous
Catchment area Whether the youth for whom help is being sought resides inside or outside of
the SHSC catchment area Dichotomous
Prior service
use Yes/no Whether the client has previously received youth MHA services elsewhere Dichotomous
Previous ED
visit(s) Yes/no
Whether the youth of the client has had previous emergency department
visit(s) Dichotomous
Previous
inpatient stay(s) Yes/no Whether the youth of the client has had previous inpatient stay(s) Dichotomous
History of
bullying Yes/no Whether the youth has a reported history of being bullied Dichotomous
School
avoidance Yes/no Whether the youth has reported school avoidance Dichotomous
Reason for contact Reported reason(s) for contacting the program
Ethnicity of youth Self-reported ethnic background Categorical
Self-report
survey
Original
questions
Household income Self-reported household income Categorical
Youth functional status
since registration
Whether the youth’s general wellbeing has worsened, stayed the same, or
improved.
Categorical
152
Table 2. Measurement summary: Mechanism variables
Mechanism measures
Variable Literature-based definition Original survey question Data
type
Accessibility
(Penchansky & Thomas,
1981)
Availability Adequacy of supply of existing services and
resources in relation to families’ needs.
To what extent do you feel you were able to
reach the Navigator whenever you needed?
Quan
tita
tive;
ord
inal
Accessibility
The relationship between the location of supply and
the location of families, taking account of factors
such as transportation resources and travel time,
distance, physical accessibility and cost.
To what extent do you feel it was convenient to
communicate or meet with the Navigator?
Accommodation
The relationships between how services and
resources are organized, such as appointment
systems, hours of operation, walk-in facilities,
telephone and web-based services; a family’s
relative ability to accommodate to these factors;
and their perception of appropriateness.
To what extent do you feel the Navigator
accommodated your schedule when making
arrangements with you?
Affordability
The relationship of cost of services (both the FNP
itself and the services to which it refers) to the
clients’ ability to pay. This includes “client
perception of worth relative to total
cost…knowledge of prices, total cost and possible
credit arrangements.”
To what extent do you feel the Navigator was
considerate of your financial resources?
Acceptability
The relationship between a family’s attitudes about
what the personal and practice characteristics
should be and the actual characteristics of the
service and its staff.
To what extent do you feel the Navigator met
your service expectations for family navigation?
Continuity of care
Informational
The use of personalized information on past events
and personal circumstances to make current care
appropriate for each family.
To what extent do you feel the Navigator
continuously communicated and coordinated
information with you and the service providers
to whom you were referred?
153
(Haggerty et al., 2003)
Management
The use of a consistent and coherent approach in
managing care plans that respond to a family’s
evolving needs.
To what extent do you feel the Navigator
continuously adequately responded to changes
in your family’s situation and needs?
Relational The development of an ongoing therapeutic
relationship between a family and their navigator.
To what extent do you feel the Navigator was
continuously committed to understanding and
helping your family until you no longer feel you
require their services?
Family involvement
(Wood, 2004)
Respect for families as experts on their children;
enlisting them as partners in their child’s care;
supporting them in their caregiver role; and
involving them as partners in decision-making at all
levels.
To what extent do you feel the Navigator
consistently involved you and your family in all
stages of care planning and decision-making?
154
Table 3. Measurement summary: Outcome variables
Outcome measures
Variable Definition Measure Data
type
Collection
method
Family
empowerment
(Koren,
DiChillo &
Friesen, 1992)
A family’s perception of itself
as having the knowledge, skills,
beliefs, attitudes and resources
required to successfully
navigate and negotiate the
service system and efficiently
utilize it to meet their needs.
Family Empowerment Scale (FES)
This bidimensional measure consists of 34 subjective statements scored on a 5-
point Likert-style scale which capture 1) the overall level of family empowerment;
and 2) and the way in which empowerment is expressed in daily life navigating the
service system to effectively and efficiently meet their needs.
The tool sums subscale item scores to produce a continuous total score for each
subscale.
Quan
tita
tive;
conti
nuous
Sel
f-re
port
surv
ey
Family quality
of life
(Hoffman,
Marquis,
Poston,
Summers &
Turnbull, 2006)
The degree to which families’
perceived needs are met, and to
which there are appropriate
opportunities to make active
choices that help them meet
their needs.
Modified Beach Center Family Quality of Life Scale (BCFQoLS)
Section A of this modified BCFQoLS consists of 20 items scored on a five-point
response scale ranging from “very dissatisfied (1)” to “very satisfied (5).” The
items span five conceptual domains of family quality of life: family interaction,
parenting, emotional well-being, physical/material well-being, and disability-related
support. Scores can be summed by domain and/or in total for a continuous measure.
A second section was been added to subjectively evaluate perceived change over
time since starting family navigation. This section consisted of 20 change items
which correspond to the 20 items in Section A and were rated on a scale of “not
true at all (1)” to “very true (3)” or “not applicable (4).” Scores can be summed by
domain and/or in total for a continuous measure.
155
Service
satisfaction
(FNP, 2014)
The extent to which families
were satisfied with the
navigation services directly
received, as well as satisfaction
with the resources to which
families were referred by the
navigation team.
NAVSAT
This 25-item measure includes 15 items that ask families to assess their satisfaction
with the treatment recommendations received, their navigator’s knowledge and
fluency in the MHA system, respect for confidentiality, and nature/frequency of
contact; and 10 items that evaluate families’ satisfaction with the referred resource
in terms of type, delivery method, location and effectiveness.
The scale yields four outcome variables including likelihood of recommending the
service, navigator helpfulness, and overall service satisfaction. Items are rated on
five- or seven-point Likert-style scales. These first three outcome variables relate to
satisfaction with navigation services and were summed to a continuous total
outcome score. The fourth outcome variable is overall satisfaction with referred
service; this seven-point item was also treated as a continuous total score.
156