incorporating outcome measures into fire department
TRANSCRIPT
Running head: INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 1
Incorporating outcome measures into fire department management practices
Holger Durre
Boulder Fire-Rescue, Boulder, CO
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 2
Certification Statement
I hereby certify that this paper constitutes my own product, that where the language of others is
set forth, quotation marks so indicate, and that appropriate credit is given where I have used the
language, ideas, expression, or writings of another.
Signed:
Date: 12/31/2018
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 3
Abstract
Governmental agencies have increasingly been called upon to be accountable to its
constituents (Han & Hong, 2016). One aspect of this has been the increasing use of outcome
measures in public agencies. The City of Boulder Fire-Rescue (BFR) department was
establishing a system to report outcome measures to improve operational efficiency and to
establish a foundation of data-driven decision making in the organization. The problem was that
the BFR had not identified an approach to incorporating outcome measures into its management
practices. The purpose of this research was to examine the incorporation of outcome measures
into the management practices of the fire service organization to establish options for BFR. The
research questions were (a) what are the best practices related to outcome measurement in the
public sector, (b) what are the factors associated with implementing outcome measures in fire
service organizations, and, (c) what approaches are fire departments utilizing to incorporate
outcome measures into management practices. The author used the descriptive research method
which included a review of the literature to establish current practice and the analysis of a survey
instrument.
The primary focus of the literature review was the exploration of systems-based
approaches to outcomes measurement in public sector organizations, particularly around the
logic model for strategic planning and outcome chain theory. Procedures included a survey to
identify current practices in the public sector and the fire service. The results of the research
identified several best practices for inclusion in BFR’s approach to integrating outcome measures
into its management practices. Recommendations included integrating scalable approaches to
output and outcome measurements as well as to integrate the logic model framework into
departmental business practices.
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 4
Table of Contents
Certification Statement ................................................................................................................... 2
Abstract ........................................................................................................................................... 3
Table of Contents ............................................................................................................................ 4
Introduction ..................................................................................................................................... 6
Background and Significance ......................................................................................................... 7
Literature Review.......................................................................................................................... 10
Best practices related to outcome measurement in the public sector ........................................ 12
Procedures ..................................................................................................................................... 15
Results ........................................................................................................................................... 18
Overview of Findings ................................................................................................................ 18
Detailed Results of Procedures .................................................................................................. 20
Discussion ..................................................................................................................................... 29
Recommendations ......................................................................................................................... 33
References ..................................................................................................................................... 36
Appendix A ................................................................................................................................... 40
List of Tables
Table 1 - Survey Question 1. Organization Type ....................................................................... 21
Table 2 - Survey Question 4. Aspect of Outcome Measurement Used ...................................... 22
Table 3 - Survey Question 5. Elements of incorporating outcome measures ............................. 23
Table 4 - Survey Question 9. Most influential factor contributing to outcome measurement .... 24
Table 5 - Survey question 10. Factors that proved challenging during implementation ............ 25
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 5
Table 6 - Survey Question 5. Factors related to the success of implementation ........................ 26
Table 7 - Survey question 12. Frequency of program performance discussion/reporting .......... 27
Table 8 - Survey Question 13. Statistical Validity by Program (n=144) .................................... 28
Table 9 - Survey question 14. Tools used to measure and report outcomes (n=144) ..................29
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 6
Incorporating outcome measures into fire department management practices
Introduction
One of the most effective ways for a fire department to communicate the relevancy of its
programs to its stakeholders is by objectively being able to demonstrate the impact on the public
it serves. However, despite the push for greater use of outcome measures, and its potential to
improve governmental operations, many municipalities have not yet widely adopted such
systems (De Lancer Jules & Holzer, 2002). This is important due to the increased reliance of
these approaches for funding justification as well as establishing the value of programs. One
example is the City of Baltimore, which has been using these approaches successfully for the last
15 years and subsequently has allowed the city “to fund high priority goals and to protect from
budget cuts the most effective services (“Baltimore case study,” 2018, para. 3).” Also, the same
case study found a direct positive correlation between the implementation of such systems and
the engagement of employees in public sector organizations as well as their general
understanding of organizational goals and priorities.
The City of Boulder and its fire department have been responsive to these developments
and have developed an innovation vision to improve the overall effectiveness of city services by
encouraging data-driven decision making and increasing operational efficiency. Also, the City of
Boulder Fire-Rescue (BFR) department is also pursuing accreditation through the Center for
Public Safety Excellence (CPSE) which requires the department to be able to objectively
demonstrate its ability to measure the effectiveness of its programs.
The problem is that BFR has not identified an approach to incorporate outcome measures
into its management practices. The purpose of this research is to examine the incorporation of
outcome measures into the management practices of fire service organizations to establish
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 7
options for the BFR to use in its approach. The research questions were (a) what are the best
practices related to outcome measurement in the public sector, (b) what are the factors associated
with implementing outcome measures in fire service organizations, and, (c) what approaches are
fire departments utilizing to incorporate outcome measures into management practices. The
author used the descriptive research method which included a review of the literature to establish
current practice and the analysis of a survey instrument.
Background and Significance
The City of Boulder Fire-Rescue Department (BFR) is a municipal fire department that
provides fire protection and emergency services to approximately 108,000 residents living in the
City of Boulder just northwest of the Denver Metropolitan Area (Boulder Economic Council,
2018). The BFR provides all-hazards response and prevention services to a service area that is
approximately 25 square miles in size. The agency is staffed by 117 uniformed personnel, 98 of
which staff seven fire stations. These personnel operate seven engine companies and one truck
company. The balance of the agency’s uniformed personnel, along with nine civilians, staff a
wildland division, training division, community risk division, and provide support and
administrative functions. The agency protects an estimated $21 billion in property (Boulder Fire-
Rescue [BFR], 2018, p. 43).
The City of Boulder is a hub for regional and national innovation as a result of a major
research university, several federal research labs, as well as a thriving entrepreneurial industry
being located in the city (Boulder Economic Council, n.d.). As an example, the city has been
home to the National Institute of Standards and Technology (NIST) since the 1960s which
created an influx of research scientists and a host of associated support industries (Engelke &
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 8
Paul, 2016). In addition, almost half of the members for Boulder’s city council have an
entrepreneurial or science-based background (City of Boulder Council website, n.d.).
In response to this environment, and at council direction, the City has embarked on an
innovation strategy to improve the overall effectiveness of city services. The overall architecture
of this strategy involves improving internal processes, becoming an incubator for ideas by
engaging community partners, and to amplify the impact of these initiatives by communicating
successes and progress ("City of Boulder Innovation Strategy," 2018). The city expects each
department to participate in achieving these initiatives and the BFR is beginning to contribute to
this process. To begin aligning with these initiatives the fire department budget has been
restructured to program-based budgeting. This budgeting method funnels funding to the program
level and forms a closer connection between the funding of a given service area and its
commensurate level of performance. To date, several improvements have been made in
measuring outputs, namely program activity levels in a more accurate way. In addition, the
department has hired a data and analytics project manager to assist with the implementation of
these initiatives.
However, the BFR does not currently have a cohesive approach to reporting outcomes of
the department’s program activity measures. This makes it difficult to connect the previously
mentioned program funding and related activity measures to outcomes that can objectively relate
the impact of a program on achieving the goals of the department. To connect these activities, the
BFR is in need of adapting its approach to performance management tracking to include the
ability to report the impact of the various programs it is responsible for administering. Without
the ability to tie the outputs and activities of programs directly to the impact that a program is
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 9
having, the department will not be able to realize the potential of progrtam-based budgeting fully
and will likely be at a funding disadvantage with other departments in the city.
Also, the BFR is currently an applicant agency under the Center for Public Safety
Excellence’s (CPSE) model of accreditation. Under the current model, agencies are required to
report the impact of program performance and do so on a regular basis. Performance Indicator
2D.2 is an example of this requirement as an applicant for accreditation must “continuously
monitor, assess, and internally report, at least quarterly, on the ability of the existing delivery
system to meet expected outcomes and identifies the remedial actions most in need of attention
(Center for Public Safety Excellence, 2015, para. 2D.2).” As written, a system for tracking these
outcomes does not currently exist at BFR, which creates a significant gap the agency must
overcome to be eligible for accredited status. This facet of accreditation intends to encourage
agencies to instill continuous improvement parameters into its operations in a sustainable
fashion. This would allow BFR to become a more resilient and sustainable agency. Therefore,
establishing an approach for integrating outcome measurement would allow the BFR to more
accurately track the impact of its programs, ensure a focus on continuous improvement and to
increase employee engagement by encouraging participation in these roles. The organization has
lacked an approach to implementing such practices, and as presented here the City needs to
overcome these limitations.
The BFR has encountered barriers to implementing outcome measures for a variety of
reasons. One reason is a lack of program-based tracking mechanisms that are focused on reliable
and reproducible data. This is in part related to the department not having tracked such measures
in the past and lacking the foundation to reduce the impediments to ease the access to such data.
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 10
The BFR’s goal in integrating outcome measures is therefore not only to report program impact
more reliably but also to improve the data tracking foundations on which these systems are built.
This applied research project (ARP) is directly linked to Unit 3: Thinking Systematically
as presented in the National Fire Academy’s RO125 Executive Leadership course, which states
as a terminal objective to “examine the systems within which the adaptive challenge exists, using
purposeful collection of data to help clarify and define what occurs within these systems. (United
States Fire Administration, 2015, p. SM 3-1).” This ARP is related to this curriculum by
examining the potential for systems of integrating outcome measures at the BFR and identifying
potential organizational change implications involved with the establishment of these practices.
Therefore, this paper uses the content areas of the R0125 course as the basis of examining and
evaluating the options available to the BFR for integrating these systems.
This ARP is relevant to United States Fire Administration (USFA) Goal One which is to
“reduce fire and life safety risk through preparedness, prevention, and mitigation. (United States
Fire Administration [USFA], 2014, p. 12).” This applied research project will directly support
this goal by helping the BFR become a more outcomes-oriented organization which will directly
reduce the commensurate risks in the City of Boulder.
Literature Review
The literature review sought to identify the current best practices related to outcome
measures in the public sector as well as to identify those areas in which fire service organizations
have adopted them into their operations. These areas are later explored within the research
questions of this paper. The review identified the major components relevant to outcome
measurement systems and provided context on how these can be applied in organizations. This
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 11
foundation of academic and industry literature supported the assembly of a survey that is
discussed later in this paper.
The primary focus of the literature review was the exploration of systems-based
approaches to outcomes measurement in public sector organizations. Much of this work revolved
around the logic model for strategic planning and outcome chain theory. The former being a
systematic and linear approach linked to establishing clear problem statements and the former
being focused on how outcomes are the product of a confluence of events and must, therefore, be
examined more broadly as often, not a single input itself can be attributed to the achievement of
a specific result.
To begin, it is important to establish the meaning of outcomes for the context of this
paper. In this case, outcomes will be referred to as the measurable impact of program activities
(Joly, Polyak, Davis, & Brewster, 2007). Lee, Johnson, and Joyce (2004, p. 111) posit further
that “when a government service has affected individuals, institutions [or] the environment, an
impact has occurred.” This paper will focus on specifically establishing those impacts that can be
reported to stakeholders. Performance management practices are not new to the public sector but
have been slow to evolve. Based on the work of Edward Deming, the roots of these systems are
found in the total quality management (TQM) approaches that found its way into government
and has influenced budgeting methods in this sense for many years (Lee et al., 2004). The roots
of these efforts began in the federal government under the Hoover Commission that first
established the foundation for performance budgeting (Milakovich & Gordon, 2004). These
efforts evolved into a broader discussion of budgeting as a central facet of performance
management in place today. Local governments have increasingly adopted performance
measurement approaches to its budgeting processes (Newcomer, 2007). These movements are all
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 12
intended to increase the ability of governments to report the direct impact of its activities to the
citizens that ultimately fund program activities.
Best practices related to outcome measurement in the public sector
Two primary methods were uncovered by the literature review in use today by public-
sector organizations currently integrating these into management practices. The first is the logic
model for strategic planning and the second is outcome chain theory. These two approaches each
tie program inputs and connect them to the various activities with the ability to report the outputs
as well as related outcomes. These will be explored separately to establish a baseline for the
assembly of the survey instrument.
The logic model for strategic planning “describes the logical linkages among program
resources, activities, outputs, customers reached, and short, intermediate and longer-term
outcomes (McLaughlin & Jordan, 1999, p. 65).” The system relies on the clear establishment of
a problem statement and associated goals that relate to solving these. Program inputs are clearly
defined which can be funding, people, or physical resources (Baxter, Blank, Buckley-Woods,
Payne, & Rimmer, 2014). These authors established in particular the importance of measuring
program outputs that are connected to goal statements developed during the establishment of a
program. These methods are intended to “make an explicit, often visual, statement of the
activities that will bring about change and the results you expect to see for the community and its
people (University of Kansas, 2018, para. 1).” Specifically, public sector organizations are
advised to take into consideration organizational context and complexity when establishing such
models for outcomes measurement (Heinrich, 2002). This is particularly true because the various
components identified earlier need to be assessed individually to ensure that the overall
organization can implement them successfully. The literature identified the benefits of using
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 13
logic models in helping an organization keep a focus on outcomes, building an understanding of
program accomplishments, providing a common language that focuses work, and enhancing
prioritization and allocation of resources (Taylor-Powell & Henert, 2008). These findings
influenced the research by identifying the specific components that were used during the
assembly of the survey instrument described in the procedures. The specific components of the
logic model framework were extracted for use in the survey instrument, especially outcome or
goal statement development, identifying program inputs, measuring reliable activity measures,
and developing key performance indicators.
The second major approach found by the literature review was the use of outcome chains.
ThiOutcome chains are also described as outcomes mapping, and in either case, involves the
systematic representation of the process of achieving outcomes in a manner that can be mapped
in its subcomponents. One key distinguishing characteristic of outcome mapping or outcome
chains is that the model can acknowledge that an individual program in and of itself may not be
solely for achieving a given output. In this case “a program is not claiming the achievement of
development impacts; rather, the focus is on its contributions to outcomes. These outcomes, in
turn, enhance the possibility of development impacts — but the relationship is not necessarily a
direct one of cause and effect (Earl, Carden, & Smutylo, 2004, p. 1).” Another example of this in
outcome chains. Like in the outcomes mapping example above, outcomes, or result chains,
“provide a theoretical model for defining the interrelated components of a project that are
required for its success (Parsons, Gokey, & Thornton, 2013, p. 7).” The primary feature then of
this approach is that multiple programs or factors can influence the outcome of a certain
program. This is a key distinguishing factor of outcome chains and outcome mapping when
compared to the logic model. “As such, outcome mapping assesses a program holistically and is
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 14
based on the premise that a program needs to know not only about development results, but also
about the processes by which these result were attained, and about its internal effectiveness (Earl
& Carden, 2002, p. 4).” A prominent place that these methods are currently being used is in
development work such as international outreach programs and community development. Given
that the fire service provides a service to the community that often attempts to reach across
boundaries of socioeconomics and to provide education, these models could be potentially
valuable for integrating outcomes into the management practices of a fire department.
One key aspects of outcome mapping and outcome chains that the literature identified are
that these are not always the best to use for smaller and contained programs (Earl et al., 2004).
This is due to the nature of outcome chains looking at the comprehensive impact of those
activities not under the direct control of a certain program. In other words, the activity of
complementary programs is taken into consideration which significantly increases the
complexity of the measurement activities around the program but also influences the ability of
smaller organizations to implement sophisticated enough measurement systems that can account
of the impact of those external factors sufficiently to be effective. For example, the
implementation of an outcome mapping approach in a community fish rearing program was
evaluated and this fond that “the inherent non-linear complexity of the development process and
the difficulties of monitoring impact and assigning attribution” was a limiting factor of the
implementation of the program (Sheriff & Schuetz, 2010, p. 4). These findings influenced this
research in describing the need for an organization to identify the ability to implement such a
complex system of assessing these factors adequately. Fire departments already deal with the
impact of incomplete internal data and expanding this problem to external factors may be a
limiting factor use in a medium-size urban fire department.
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 15
Finally, the literature review identified key factors related to staffing and deployment of
an outcomes-based measuring system (Pearson, 2005). Pearson indicates that the availability of
support staff, to include those dedicated to IT are crucial in ensuring these systems are
implemented effectively. This has been found in other cases as well such as the implementation
of program performance measurement systems (Iversen & Ngwenjama, 2006). A related
response to this has been the fire service’s recent increase in dedicated staff for performance
management related to accreditation and program-based performance measurement. These are
also supported as best practices in general fire service administration (Bruegman & Lindsey,
2009). This influenced this research to ensure that these factors were assessed in the research and
assembly of the survey instrument.
Procedures
The purpose of this research was to examine the incorporation of outcome measures into
the management practices of fire service organizations to establish options for the City of
Boulder Fire Department. To perform this analysis, a survey instrument was administered to both
public sector organizations with a specific portion focused on fire departments. The survey was
designed by using the literature review of this paper as a foundation to examine current best
practices in the industry. The procedures sought to answer the research questions identified.
The research questions asked (a) what are the best practices related to outcome
measurement in the public sector, (b) what are the factors associated with implementing outcome
measures in fire service organizations, and, (c) what approaches are fire departments utilizing to
incorporate outcome measures into management practices? The results are discussed in the
appropriate sections of this paper, and any applicable survey instruments are presented in the
appendices of this paper.
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 16
The survey was 14 questions in length and is represented in Appendix A of this paper.
The first three questions, as well as question 8, involved demographic background information
that was used to be able to categorize the responses and discern features based on agency type
and the level of application of outcomes-based practices in use by the particular agency.
Question 4 through 8 were posed to all participants and asked general questions that were aimed
at assessing general aspects of outcomes measurement in both public sector and fire service
entities. Questions 9 through 11 assessed what factors were associated with implementing
outcome measures in fire service organizations. Finally, questions 12 through 14 assessed the
specific approaches fire departments were utilizing to incorporate outcome measures into their
management practices.
The survey was distributed using the International Association of Fire Chief’s
KnowledgeNet, the Rocky Mountain Accreditation and Professional Credentialing Consortium
website as well as the LinkedIn professional networking application. Finally, the survey was
posted to the International City and County Manager’s Association (ICMA) performance
management discussion board. The survey was held open for 28 days on the Survey Monkey
platform. During this time, a total of 246 responses were received, and the overall completion
rate of the survey was 60%.
To address the first research question (a) what are the best practices related to outcome
measurement in the public sector, the survey instrument used the specific findings of the
literature review related to the outcome practices that were identified that the academic literature
identified as best practices for public sector organizations. The specific practices are represented
in question 4. Question 5 was focused on assessing the factors related to incorporating these
practices into the management practices of a specific organization. These were also obtained
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 17
from the literature review. During the literature review, staffing and budgeting were identified as
crucial factors related to the success of outcomes integration and, therefore, questions 6 and 7
assessed the level of these factors.
To answer the second research question (b) what are the factors associated with
implementing outcome measures in fire service organizations, question 8 determined if the
respondent was a member of a fire service organization. Questions 9 and 11 assessed directly
those items identified in the literature review that were correlated to the success of these
programs in the academic literature. The literature review identified barriers to implementation,
and these were assessed in survey question 10. The specific participation rates and applicable
results are presented in subsequent sections of this paper.
The third and final research question asked (c) what approaches are fire departments
utilizing to incorporate outcome measures into management practices? Once again, the survey
instrument was used as described for the previous research questions. In this case, questions 12
through 14 were related to how fire service organizations operationalized these practices.
Specifically, participants were asked to report on their activity in this area related to the
frequency of reporting out on the status of program activity, if these results were statistic valid,
and finally, what specific tools were used during the implementation of the processes. The last
question was primarily intended to identify potential tools for Boulder Fire-Rescue (BFR) to
consider implementing. The questions for this part of the survey were designed based on the
author’s knowledge of BFR administrative operations.
These procedures and the study itself was impacted by some limitations that are
important to discuss. The first of these was the lack of participation by non-fire service public
sector organizations. Despite several attempts to distribute the survey to distribution lists that
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 18
were generally focused on the public sector, such as the ICMA discussion boards, very few
responses were received. Therefore, the results are similarly affected and represent mainly the
experience of fire-service organizations. While this is still valuable as these are a largely a part of
the public sector, the scope of the results should be interpreted with this in mind.
Secondly, the academic literature was often broader than government service
organizations. Much of the literature review uncovered general practices and those in use by the
medical profession for patient management. While these still applied, the paper could have
benefited for more specificity related to the experience in the public sector.
Results
Overview of Findings
The original research posed three distinct questions. These were (a) what are the best
practices related to outcome measurement in the public sector, (b) what are the factors associated
with implementing outcome measures in fire service organizations, and, (c) what approaches are
fire departments utilizing to incorporate outcome measures into management practices? To
answer these questions, the results of a survey administered to both public-sector and fire service
entities were evaluated.
The first research question, aided by the literature review, discovered that best practices
in outcome measurement currently revolves around the implementation of a systematic approach
to outcome measurements such as those found in the logic model for strategic planning and
outcome chain or outcome mapping approaches. The survey indicated that the industry was
utilizing primarily measures of program activity in the form of outputs and activity (53.57%),
followed by outcome and goal statement development (36.90%), and the use of key performance
indicators (36.90%). This portion of the research also attempted to identify the most important
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 19
elements involved in incorporating outcome measures. The respondents indicated the three most
important elements were establishing clear goals to define program success (75.6%), clearly
identifying the problem being addressed by a program/activity (72.62%), and establishing short,
medium, and long-term activity and output measures (56.55%). Two key findings in research
question one were that 86.69% of respondents indicated they did not have staff dedicated to
manage their outcome measurement system and that only 33.33% connected outcome measures
to their budgeting process. This stands in contrast with the finding of the literature review that
these practices are encouraged. These results indicate the specific components in use by agencies
mirror those in the literature, although only some components are being used consistently.
The second research question attempted to answer what factors were associated with fire
departments during the implementation of outcome measurements. The survey assessed what
factors led to a department implementing such measures and the research found that this was
related to improving programs (42.28%), protecting funding or budget defense (27.52%), and
continuous improvement efforts such as accreditation (15.44%). The respondents indicated that
the most challenging aspects were a lack of organizational buy-in (27.52%), a lack of staffing to
support these functions (26.85%), and a lack of financial support (20.81%). One interesting
finding here was that very few agencies (6.71%) indicated that they felt that a lack of technology
was a limiting factor. Based on these results it was not surprising that the most critical factor
identified in ensuring the success of an outcomes measurement system was communicating the
value of the program (37.58%), clearly identifying the value of the program (37.58%) and
removing barriers to implementation (23.49%). These can be seen as directly linked to creating
buy-in as asked by the previous question.
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 20
To answer the third question, which assessed what approaches departments were using,
respondents were asked about the frequency of reporting activities, statistic validity, and the use
of measurement tools in the process. Respondents indicated that the reporting of progress and
outcomes measures is taking place at either monthly (38.89%) or quarterly (33.33%) intervals.
The remainder of agencies (27.78%) were reporting at either daily, weekly, or annual intervals.
The survey also asked if respondents felt that they could report statistically valid outcome
measures and if so, in which program areas this was being reported in. One interesting finding
here was that 93.75% of the respondents indicated they could do so in emergency response,
which is a very high percentage. A further 70.83% could do so in training, and 59.03% could do
so in code enforcement. However, the fewest respondents could report these in facilities
(15.97%) and fleet (36.11%). The question did not define statistic validity, and further research
would need to determine the reliability of these responses. Finally, the last question assessed the
tools or the manner in which, agencies were reporting outcome measures in their management
practices. These were mainly program performance reports (68.75%), data visualization
(42.36%), and statistical analysis software tools (36.11%).
Detailed Results of Procedures
To answer the first research question, a survey was issued to answer what the best
practices related to outcomes measurement were in the public sector. The survey was built using
results from the literature review. The survey was open for a total of 28 days and received a total
of 246 responses. The survey consisted of three distinct sections that each aimed to answer one
of the research questions. The survey assessed demographics information in questions 1 through
3 and in question 8. The latter was the determinant of whether or not the respondent was a
member of a fire service organization. Question 1 ascertained the make-up of the organization
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 21
regarding focus. The primary demographic of the respondents represented local government with
a total of 236 (95.93%) of the 264 respondents reporting this as their background. A far distant
second was non-governmental organizations at 2.44% or 6 of the respondents. The results of the
questions are presented in Table 1.
Table 1
Survey Question 1. Organization Type (n=264)
Rank Agency Type Count Percent of Total
1 Local Government 236 95.93%
2 Non-Governmental Organization 6 2.44%
3 State Government 2 0.81%
4 Federal Government 2 0.81%
5 Educational Institution 0 0.00%
Survey questions 2 and 3 sought to gain an understanding of how respondents were
utilizing outcomes in decision making. While these questions were not directly designed to aid in
answering research question 1, it served as demographic background. Of the 246 respondents,
39.84% (98) of the respondents reported that they were using outcomes to establish programs
while only 15.58% (39) reported having discontinued a program based on the lack of clear
outcome measures.
Survey question 4 assessed specifically which of the best practices identified in the
literature review that a particular entity was using in their outcome measurement systems. While
all of the practices are displayed in Table 2, the three primary practices identified were
measuring the level of program activity through outputs with 53.57% (90) respondents indicating
they were tracking these. Also, 36.90% (62) of the respondents reported developing outcome
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 22
goal statements. The same number of respondents reported the use of key performance
indicators. Of note was that respondents seemed to be using components of the systems based
best practices identified in the literature review, but only 2.89% (5) reported using outcome
chains, while 1.79% (3) used the logic model.
Table 2
Survey Question 4. Aspect of Outcome Measurement Used (n=168)
Rank Outcome Measure Count Percent of Total
1 Measuring Outputs 90 53.57%
2 Outcome or goals statement development 62 36.90%
3 Key Performance Indicators 62 36.90%
4 Dashboards 49 29.17%
5 Analytical software 44 26.19%
6 Defining inputs 34 20.24%
7 None of the above 33 19.64%
8 Reliability indicators 19 11.31%
9 Outcome chains 5 2.98%
10 The Logic Model 3 1.79%
To assess what respondents felt were the most important elements in their outcome
measurement system, 75.62% (125) of the respondents reported that establishing clear goals to
define program/activity success was key in their efforts while 72.62% (122) reported that clearly
identifying the problem being addressed by a program was vital. 56.55% (95) stated that
establishing short, medium, and long-term activity and output measures were important.
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 23
Table 3
Survey Question 5. Elements of incorporating outcome measures (n=168)
Rank Element Count Percent of Total
1 Establishing clear goals 127 75.62%
2 Clear identification of problem being
addressed
122 72.62%
3 Establishing short, medium, and long-term
outputs
95 56.55%
4 Establishing a link between outputs and
outcomes
62 36.90%
5 Dedicated time to report progress 52 30.95%
6 Accurately identifying inputs 33 19.64%
The literature review had identified the importance of dedicating support staff to
outcomes measurement as well as the fact that connecting these systems to the budgeting process
was important. In question 6, respondents reported that only 11.31% (19) had dedicated staff,
while 33.33% (56) reported a connection to the budget process. These two survey questions
concluded the questions designed to answer the first research question (a) what are the best
practices related to outcome measurement in the public sector?
These results indicate that while logic models and outcome chains are seen as best
practices in the literature, the industry is currently only using elements of each as indicated in
survey question 4. Output measurement is prevalent in organizations, which is not a surprising
result, but the use of these systems was only reported in a small number of cases. Also,
respondents indicated that clear goal statement development and understanding the problem each
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 24
program is addressing was seen as some other best practices to use in the deployment of such
efforts.
To answer the second research question, the same survey was utilized to determine the
factors associated with implementing outcome measures in fire service organizations. Question 8
was a demographics related question that ascertained whether or not a respondent represented a
fire service organization. As mentioned in the procedures, only 1.76% (3) of the respondents
indicated that they did not represent such an organization. This was a very significant result that
will be later explored in the discussion section of this paper. While this was a limitation to gain a
better understanding of public sector best practices, the results for the remaining research
questions were supported by a high number of responses from those in the fire service.
Survey question 9 was aimed at determining the most influential factor that led to a
particular fire service organization to start measuring outcomes. This was important to ascertain
as background for BFR to determine if the motivations of the organization were aligned or
differed from the general experience of the population. 42.28% (63) of respondents indicated that
their primary factor was to improve programs objectively, while 27.52% (41) reported the need
to protect funding with these efforts. A further 15.44% (23) stated that other continuous
improvement efforts motivated their deployment of these methods. Of note was that only 4.03%
(6) of the respondents reported that pressure from elected officials contributed to their pursuit of
such initiatives. Table 4 represents these results in full.
Table 4
Survey Question 9. Most influential factor contributing to outcome measurement (n=149)
Rank Factor Count Percent of Total
1 Improving Programs 63 42.28%
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 25
2 Budget Defense 41 27.52%
3 Related to other improvement efforts 23 15.44%
4 Meeting community expectations 16 10.74%
5 Political pressure 6 4.03%
6 Industry pressure 0 0.00%
Question 10 was designed to ascertain the challenges fire service organizations faced
with the implementation of these efforts. 27.52% (n=41) reported a lack of buy-in and 26.85%
(n=40) stated that a lack of staffing contributed to their challenges. Finally, only 6.71% (n=10)
stated that a lack of technology was a challenge for these agencies.
Table 5
Survey question 10. Factors that proved challenging during implementation (n=149)
Rank Factor Count Percent of Total
1 Lack of buy-in 41 27.52%
2 Lack of staffing 40 26.85%
3 Lack of financial support 31 20.81%
4 Lack of expertise 37 18.12%
5 Lack of technology 10 6.71%
The final question designed to address the second research question was related to what
factors were most contributory to the success of an agency’s efforts in outcome measurement.
37.58% (56) of the respondents reported that communicating the value of the outcome
measurement program was vital in ensuring overall program success. A further 28.86% (43)
reported that clearly identifying the parameters and intent of a program was key, along with
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 26
23.49% (35) of the respondents that indicated that removing barriers to implementation was most
related to the success of a program. On the other hand, only 1.34% (2) of the respondents
reported that celebrating success was key which contrasts most organizational change strategies.
Table 6 contains the complete results of this question.
Table 6
Survey Question 5. Factors related to the success of implementation (n=149)
Rank Factor Count Percent of Total
1 Communicating the value of the program 56 37.58%
2 Clearly identifying the parameters and intent of the
program
43 28.86%
3 Removing barriers to implementation 35 23.49%
4 Building an implementation team 10 6.71%
5 Dealing with setbacks 3 2.01%
6 Celebrating success 2 1.34%
These results, therefore, help to answer the second research question, which was to
determine the factors that were associated with implementing outcome measures in fire service
organizations. Respondents indicated that most initiated such efforts to improve programs and to
defend their budgets. These factors would be important at the BFR as well for consideration. The
surveys also revealed that the factors that most influenced these implementation efforts
negatively were a lack of buy-in, staffing, and financial support. These are also factors to
consider at BFR and will be discussed in greater detail in a later section. Finally, most agencies
reported that success was most connected to communicating the value of a program through
these initiatives, as well as clearly identifying the parameters of the program itself. Therefore, the
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 27
factors that are associated with implementing outcome measures in the fires service have been
identified in this sense and will be further discussed in the discussion section of this paper.
The final research question assessed what approaches fire departments are utilizing to
incorporate outcome measures into management practices. This was designed to determine the
specific frequencies, statistical grounding, as well the specific tools in use by departments. These
were assessed to determine if these would be valuable for inclusion at BFR in its effort to
establish a similar program. The survey used for the previous questions was utilized once again,
and the results were scored in the same manner as the previous components of the survey.
Survey question 12 examined the frequency at which respondents were reporting results
and discussing them during regular management activities. 38.89% (56) of the respondents stated
that they did so on a monthly basis, while 33.33% (48) stated that this was occurring quarterly.
The remainder of the respondents stated that they did so weekly, annually, or daily. These results
indicate that while some periodic examination of the progress of these measures is key, an
agency should determine its own approach based on internal restrictions and factors.
Table 7
Survey question 12. Frequency of program performance discussion/reporting (n=144)
Rank Frequency Count Percent of Total
1 Monthly 56 38.89%
2 Quarterly 48 33.33%
3 Weekly 24 16.67%
4 Annually 21 14.58%
5 Daily 4 2.78%
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 28
Survey question 13 assessed in which program areas an agency felt it could report
statistically valid outcome measures. This was meant to be a background question to determine
the state of sophistication of these efforts in the industry. The majority, 93.75% (135), of
respondents indicated that emergency response was an area that they could report this level of
outcome, while training at 70.83% (102) and code enforcement at 59.03% (85) was reported as
the third most common area of statistically valid outcomes. Of note is that no specific definition
was given of statistical validity and the respondent's interpretation of this may influence these
results.
Table 8
Survey Question 13. Statistical Validity by Program (n=144)
Rank Program Count Percent of Total
1 Emergency Response 135 93.75%
2 Training 102 70.83%
3 Code Enforcement 85 59.03%
4 Public Education 74 51.93%
5 Fleet 52 36.11%
6 Recruitment/Diversity 36 25.00%
7 Firefighter Wellness 34 23.61%
8 Facilities 23 15.97%
The final question assessed what tools agencies were using to report outcome measures in
their management practices. This was asked to ascertain the most common uses of tools by the
respondents and to understand what BFR should consider regarding implementing practices
related to deploying such a system. Most of the respondents, 68.75% (99) were using program
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 29
performance reports, while 42.36% (61) were employing data visualization methods. A further
36.11% (52) reported the use of statistical analysis software. Finally, these respondents reported
the use of meeting minutes in 31.25% (45) of the cases.
Table 9
Survey question 14. Tools used to measure and report outcomes (n=144)
Rank Tools Used Count Percent of Total
1 Program performance reports 99 68.75%
2 Data visualization 61 42.36%
3 Statistical analysis software 52 36.11%
4 Meeting minutes 45 31.25%
These survey questions related to research question three sought to answer what
approaches fire departments were using to incorporate outcome measures into their management
practices. It was clear that most agencies were utilizing either monthly or quarterly checkpoints
to ensure they stayed on track with their outcome measurement efforts and that most agencies
were utilizing program performance reporting and data visualization to report the progress of
these efforts.
Discussion
The research questions originally posed aimed to examine the incorporation of outcome
measures into the management practices of fire service organizations to establish options for the
City of Boulder Fire Department. The purpose was accomplished by carrying out the procedures
as outlined earlier which included a survey instrument administered to answer the three research
questions. To examine the implications for the BFR to integrate outcome measures into
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 30
management practices of the organization, these three questions will be examined to study the
relationship between the results of the survey and the literature review presented earlier.
The first research question attempted to answer what the best practices were related to
outcome measurement in the public sector. The literature identified two primary systems of
outcome measurement which included outcome chains as well as the logic model. Both of these
systems set out to establish an approach that took into consideration outcomes as a system of
components that interact with one another to create intended impacts (Earl et al., 2004). These
included the establishment of program parameters and goals, inputs, goal statements, as well
clearly defined impact endpoints that could be identified to ensure program success (Taylor-
Powell & Henert, 2008). This was contrasted in the findings of the administered survey that
revealed that while the various components of each of these systems were being utilized by the
public sector, as well as the fire service, not all the components were being used in conjunction
with one another.
This implied that BFR should consider evaluating the full implementation of such an
approach to attain the best practice scenario as identified in the literature review. The literature
review indicated that there is an inherent complexity associated with outcome measurement and
that these complexities must be considered in the implementation of such systems (Heinrich,
2002). Therefore, BFR should fully study the implications of implementing all the components
of such a system during the initial phases of such an effort to ensure that the various components
of the outcomes measurement system can be connected with one another effectively. This has
significant organizational implications as; currently, not all of the IT systems of BFR are
prepared to manage such an integration due to a reliance on legacy systems that have been in
place before the introduction of such measures. In addition, the literature indicated that outcome
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 31
chains are relatively complex to manage as they require the ability of an organization to track the
impact of external influences on outcomes (Earl & Carden, 2002). The BFR does not have the
ability to implement such sophisticated measurement systems and should consider this in the
selection of an approach.
Another significant finding in analyzing the results of the first research question was the
use of key performance measures (KPIs). These were previously not discussed in great length
but bear some significance concerning BFR’s implementation efforts. The third most utilized
component of outcome measures by the survey participants was the use of KPIs, and the
literature review indicated the general benefit of using such an approach to drive organizational
outcome focused results (Spekle & Verbeeten, 2014). Therefore, BFR should evaluate more fully
the use of KPI’s. The literature suggests that the primary benefit of using KPI’s is their tendency
to focus organizational energy toward the achievement of such measures due to the relative ease
of interpreting such metrics. This will involve an expansion of the current efforts underway to
create dashboards that are related to outcome achievement and will require an investment in the
information technology areas of the organization. Since BFR is in the initial stages of its
implementation, this should be considered in the integration process.
The second research question attempted to examine the specific factors associated with
implementing outcome measures in fire service organizations. The intent was to identify the
specific factors involved with such efforts to learn if there were considerations the organization
should consider before engaging in this effort. The most significant finding in this regard was
that the importance of communicating the value of outcome measurement, along with clearly
communicating the value of the program being measured, were key to such an implementation.
This is supported in the literature, particularly in the concept of return on investment. The idea
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 32
that the value of the program relates to an improvement of the standing of the organization as a
whole has been linked to the success of such implementation efforts (Phillips, 2011). This has
clear implications on BFR as this should be a central component of the implementation effort of
the new system of outcomes measurement in the department.
Another finding of the research that was related to this research question concerned
barriers to implementation. The survey participants indicated that a lack of buy-in was a barrier
that should be considered. While this was not directly found in the literature, general change
management practices indicate that this must be considered as a countermeasure. These findings
are supported by Kotter (2012) who posits that gaining buy-in for organizational change efforts
is a key to the overall success of change initiatives and new program implementation. This has
been important for BFR as a new administration has taken steps to alter the course of the
organization significantly. The author has recognized buy-in as being key to the success of any
change initiative in this organization.
The third research question attempted to answer what specific approaches fire
departments were utilizing to incorporate outcome measures into their management practices.
The survey participants were asked questions focused on reporting frequency, statistic validity,
as well as reporting tools in use by the agency being surveyed. The primary finding related to
answering this research question was related to the reporting frequency and manner in which the
results were reported. Most agencies were reporting monthly in the form of program reports that
were sent to a stakeholder such as a council. These findings are supported by the literature
review and the work of others. Particularly, the reporting to external stakeholders was seen as
vital in the literature (Thomas, 2006).
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 33
This finding is of importance to the BFR in its efforts as the current system of reporting is
already based on a streamlined process of monthly reporting of metrics through the use of a
community dashboard. However, the organization should make efforts to expand these initiatives
based on the findings of the research as well as the literature which indicate that external
accountability is directly related to the success of such initiatives. The regular reporting of
progress related to program performance should, therefore, be included in the design of such a
system.
A final finding of the research indicated that dedicated staffing was important in the
successful deployment of outcomes measurement systems. While BFR currently lacks a modern
information technology infrastructure, the staffing of a dedicated project manager for data and
analytics is not only relevant but should be a key change agent in this effort. While the findings
of the research indicated that most agencies were focused on emergency response, a key focus of
BFR is community risk reduction efforts, and as such, these will be a central component of the
new system of performance reporting at BFR.
Recommendations
Based on the results of the stated research questions several valuable recommendations
are being made for BFR to pursue during the implementation of its outcomes measurement
practices. These recommendations are based on the results of the survey administered and the
literature review as discussed. These recommendations are meant to help BFR design an
outcomes measurement system that integrates into the management practices of the organization,
which was the original purpose of the research.
It is recommended that the City of Boulder utilizes the logic model to integrate outcome
measures into the management practices of the organization. This integration should clearly
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 34
identify the problems each program is meant to address and develop commensurate goals for
solving these problems. To support these efforts, output measures that relate directly to short,
medium, and long-term outcome measures should be established. These recommendations are
based on the findings of surveyed organizations that indicate that these factors most contribute to
the success of such efforts.
It is also recommended that BFR clearly communicates the intended outcome of these
systems to program managers and to incorporate these into quarterly staff meetings that are used
to report the progress of these efforts. The BFR has an advantage in the implementation of these
systems as it currently employs a full-time data and analytics project manager that will need to
be the primary support for these efforts. This position will need to provide education on the
various components of the logic model as well to ensure that program managers are familiar with
the intent of each component. Also, the information technology resources of BFR will need to
support the implementation for these efforts and must be available to ensure that data tracking
and collection is not too cumbersome for individual programs to carry out.
Finally, a key finding of the research was that the regular reporting to external and
internal stakeholders, such as elected officials, was supported by the research and should be
considered in the establishment of an outcomes reporting program. The literature supported that
such integration aided in the accountability of these measures. Also, the respondents in the
survey instrument rated this as being directly related to their successful implementation of such
measures.
This paper was limited by the lack of participation of public sector organizations outside
of the fire service. Additional research should be aimed at identifying best practices outside the
fire service to ensure that the implementation of these measures is informed by a wider
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 35
experience in the public sector. In addition, future research should explore the statistic validity of
these measures to ensure that outcomes are linked with causation of program activities.
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 36
References
Baxter, S., Blank, L., Buckley-Woods, H., Payne, N., & Rimmer, M. (2014, May 10). Using
logic model methods in systematic review synthesis: describing complex pathways in
referral management interventions. BMC Medical Research Methodology, 14-62.
https://doi.org/doi.org/10.1186/1471-2288-14-62
Boulder Economic Council. (2018). Demographic and economic data: A snapshot of Boulder’s
key metrics. Retrieved April 2, 2018, from http://bouldereconomiccouncil.org/boulder-
economy/demographic-economic-data/
Boulder Economic Council. (n.d.). http://bouldereconomiccouncil.org/boulder-
economy/innovation-economy/
Boulder Fire-Rescue. (2018). Community risk assessment and standards of cover [Standards of
Cover]. Boulder, CO: Boulder Fire-Rescue Department.
Bruegman, R., & Lindsey, J. (2009). Fire administration. New York, NY: Pearson.
Budgeting for outcomes: Success in Baltimore. (2018). Retrieved from
https://www.innovations.harvard.edu/opex/research/budgeting-outcomes-success-
baltimore
City of Boulder Council website. (n.d.). https://bouldercolorado.gov/city-council/mayor-and-
city-council
Center for Public Safety Excellence. (2015). Fire and emergency services self-assessment
manual (9th ed.). Chantilly, VA: Author.
De Lancer Jules, P., & Holzer, M. (2002, December 17). Promoting the Utilization of
Performance Measures in Public Organizations: An Empirical Study of Factors Affecting
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 37
Adoption and Implementation. Public Administration Review, 61.
https://doi.org/https://doi.org/10.1111/0033-3352.00140
Earl, S., & Carden, F. (2002, August 1). Learning from complexity: The International
Development Research Centre`s experience with outcome mapping, https://
doi.org/10.1080/0961450220149852
Earl, S., Carden, F., & Smutylo, T. (2004). Outcome mapping: building and reflection into
development programs. Ottawa, ON: International Development Research Centre.
Engelke, P., & Paul, A. (2016). Boulder, Colorado: Innovation in a small town and a big state.
Retrieved from http://www.atlanticcouncil.org/blogs/futuresource/boulder-colorado-
innovation-in-a-small-town-and-a-big-state
Han, Y., & Hong, S. (2016). The impact of accountability on organizational performance in the
US Federal Government: The moderating role of autonomy. Review of Public
Administration, 1-21. https://doi.org/10.1177/0734371X16682816
Heinrich, C. (2002). Outcomes-based performance management in the public sector:
Implications for government accountability and effectiveness. Public Administration
Review, 62, 712-725. https://doi.org/doi/epdf/10.1111/1540-6210.00253
Innovation. (2018). Retrieved October 28, 2018, from https://bouldercolorado.gov/innovate
Iversen, J., & Ngwenjama, O. (2006). Problems in measuring effectiveness in software process
improvement: A longitudinal study of organizational change at Danske Data.
International Journal of Information Management, 26(1), 30-43.
https://doi.org/doi.org/10.1016/j.ijinfomgt.2005.10.006Get
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 38
Joly, B., Polyak, G., Davis, M., & Brewster, J. (2007, August). Linking Accreditation and Public
Health Outcomes: A Logic Model Approach. Journal of Public Health Management, 13,
349-356. https://doi.org/Retrieved from
Kotter, J. P. (2012). Leading change. Boston, MA: Harvard Business Review Press.
Lee, R. D., Johnson, R. W., & Joyce, P. G. (2004). Public budgeting systems (7th ed.). Sudbury,
MA: Jones and Bartlett.
McLaughlin, J., & Jordan, G. (1999). Logic models: a tool for telling your programs
performance story. Evaluation and Program Planning, 22(1), 65-72. https://doi.org/
Milakovich, M. E., & Gordon, G. J. (2004). Public administration in America (8th ed.). Belmont,
CA: Wadsworth/Thomson Learning.
Newcomer, K. E. (2007, January 17). Measuring government performance. International Journal
of Public Administration, 30, 307-329.
https://doi.org//doi.org/10.1080/01900690601117804
Parsons, J., Gokey, C., & Thornton, M. (2013). Indicators of inputs, activities, outputs,
outcomes, and impacts in security and justice programming. Retrieved from Department
for International Development: https://assets.publishing.service.gov.uk/
government/uploads/system/uploads/attachment_data/file/304626/Indicators.pdf
Pearson, S. (2005). Outcomes measures implementation best practice guidance. Retrieved from
National Institute for Mental Health in England: https://www.researchgate.net/profile/
Peter_Fonagy/publication/272824369_Outcomes_Measures_Implementation_
Best_Practice_Guidance/links/54f5dae80cf2ca5efefd3aaf/Outcomes-Measures-
Implementation-Best-Practice-Guidance.pdf
INCORPORATING OUTCOME MEASURES INTO FIRE DEPARTMEN 39
Phillips, J. (2011). Return on investment in performance and improvement training programs.
(2nd ed.). New York, NY: Routledge.
Sheriff, N., & Schuetz, T. (2010). Benefits and challenges of applying outcome mapping in an
R4D project. Retrieved from World Fish Center: https://www.outcomemapping.ca/
download/WF_2625.pdf
Spekle, R. F., & Verbeeten, F. H. (2014, June). The use of performance measurement systems in
the public sector: Effects on performance. Management Accounting Research, 25(2),
131-146. https://doi.org/doi.org/10.1016/j.mar.2013.07.004
Taylor-Powell, E., & Henert, E. (2008). Developing a logic model: Teaching and training guide.
Retrieved, from https://peerta.acf.hhs.gov/sites/default/files/public/
uploaded_files/Logic%20Model%20Guide.pdf
Thomas, P. G. (2006). Performance measurement, reporting, obstacles and accountability:
Recent trends and future directions. Canberra, Australia: Australian National University
Press.
United States Fire Administration. (2015). Executive leadership student manual. Emmitsburg,
MD: United States Fire Administration.
University of Kansas. (2018). Community toolbox: Developing a logic model or theory of
change. Retrieved from https://ctb.ku.edu/en/table-of-contents/overview/models-for-
community-health-and-development/logic-model-development/main