return on investment from public safety ... on investment from public safety leadership education:...
TRANSCRIPT
RETURN ON INVESTMENT FROM PUBLIC SAFETY LEADERSHIP EDUCATION:
An Impact Evaluation of the AIPM Graduate Programs
Victor ia Herr ington 2 0 1 4
A U S T R A L I A N I N S T I T U T E O F P O L I C E M A N A G E M E N T
1
PUBLICATION DETAILS © Australian Institute of Police Management ISBN: 978-0-9751397-2-1 Author: Dr Victoria Herrington [email protected] The Australian Institute of Police Management PO Box 168 MANLY NSW 1655 Telephone: AIPM Reception Switchboard Number: (02) 9934 4800 Web address: www.aipm.gov.au
ACKNOWLEDGEMENTS
This research could not have been completed without the assistance of a large number of people.
First, thanks to Mitch Pearson-Goff for his hard work and dedication as a research assistant on this
project. Mitch spent a great deal of time coordinating invitations to take part in the research,
downloading survey responses and interviewing participants, and undertook the work contained in
appendix 1 before leaving the AIPM to become a federal agent. Thanks also to all of the participants
who gave time in their busy schedules to take part in this research. And final thanks to the external
academics who peer reviewed this paper: Professor Mike Hough at Birkbeck, University of London,
and 2012 Professor in Residence at the AIPM; Professor Karl Roberts at the University of Western
Sydney; and Dr Curtis Clark, Assistant Deputy Minister of Alberta Correctional Services and member
of the Interpol Group of Experts on Police Training (IGEPT). Thanks also to Professor Bruce Avolio at
the Foster Center for Leadership at the University of Washington who generously provided advice
around the return on investment calculations presented at the end of this report.
2
SUMMARY
The AIPM delivers two graduate-level programs: the Graduate Certificate in Applied Management,
and the Graduate Diploma in Executive Leadership. These programs service the needs of police and
emergency services in Australia and New Zealand, as well as further afield, and have been running in
one form or another since 2002. Whilst there is considerable anecdotal evidence to suggest that
these programs are well received by participants, and valued by the jurisdictions that sponsor
participants’ attendance, these programs have never before been formally evaluated for their
impact on leaders’ behaviour, and the flow on benefits to organisations. This report presents data
from such an evaluation.
A mixed-methods approach was adopted and data were collected between March 2012 and
November 2013. Data were collected from course participants both before and after their
engagement in the graduate program; from former course participants who had completed their
studies between 12 and 24 months prior; from the managers of course participants; and from
jurisdictional stakeholders.
KEY FINDINGS The graduate programs were positively regarded by participants, managers and stakeholders, and
had notable impacts on individual behaviour and workplace practice. This led to significant benefits
for organisations. Specifically:
• The graduate programs had a statistically significant and positive impact on confidence in
one’s leadership skills.
• The graduate programs had a statistically significant and positive impact on self-reported
behaviour in the five domains of the Leadership Capability Framework: setting strategic
direction, achieving results, building and managing relationships, communicating with
influence, and personal drive and integrity.
• The graduate programs had a statistically significant impact on confidence and behaviour
associated with policing and industry-relevant skills.
• The small opportunity sample of managers interviewed largely concurred with participants’
behavioural self-assessments following their graduate program, suggesting that self-
assessments were accurate reflections of workplace behaviour. Two notable exceptions
were the achieving results and personal drive and integrity domains, where manager
assessments were more positive.
3
• Qualitative data from former students showed that they had implemented their learning
from the graduate program in a number of ways. The impact of which had led to enhanced
staff wellbeing, productivity, and improved outcomes against organisational goals.
• Qualitative data from former students also suggested that leadership development was an
ongoing process, and continued beyond the end of the graduate program, with workplace
experience marrying well with their formal learning to lead to exponential benefits over
time.
• The return on investment (ROI) to organisations for sending employees to AIPM graduate
programs was calculated using an established formula, but as with all ROI calculation caution
must be used in interpreting the results because of the number of assumptions involved (see
page 36 for a full discussion). Nonetheless the calculated ROI for the graduate programs
ranged from 164% to 3326% depending on the hypothesised duration of the training effect.
Organisations can expect a 164% return on investment if the impact of the training lasts for
one year, and a 3326% return on investment if it lasts for the remainder of the individual’s
career (estimated at 13 years).
In conclusion, this research represents a methodologically robust evaluation of the AIPM’s graduate
programs and draws on multiple sources of data. To our knowledge, there have been no similarly
robust evaluations of police leadership programs to date across the world, and as such this research
represents an important contribution to knowledge. That these findings indicate that AIPM’s
graduate programs have a significant impact on leadership behaviour in line with the expectations of
sponsoring organisations, and that there seem to be considerable organisational gains to be had
from this ongoing investment, should be of interest to sponsoring organisations, further highlighting
the importance of this work. Of course, and as with all research, there are limitations that need to be
considered. Accurately assessing the longitudinal impact of these programs is one area that requires
further investigation.
4
CONTENTS
PUBLICATION DETAILS ............................................................................................................................ 1
ACKNOWLEDGEMENTS ........................................................................................................................... 1
SUMMARY ............................................................................................................................................... 2
KEY FINDINGS ...................................................................................................................................... 2
INTRODUCTION ....................................................................................................................................... 6
EVALUATIONS OF TRAINING ............................................................................................................... 7
METHODOLOGY ...................................................................................................................................... 9
SAMPLE SELECTION ........................................................................................................................... 12
Current course participants .................................................................................................... 13
Former course participants .................................................................................................... 16
Managers of current course participants ............................................................................... 16
Jurisdictional stakeholders ..................................................................................................... 17
QUESTIONNAIRE DESIGN .................................................................................................................. 18
ANALYSIS ........................................................................................................................................... 21
ETHICAL CONSIDERATIONS ............................................................................................................... 21
METHODOLOGICAL LIMITATIONS ..................................................................................................... 22
FINDINGS ............................................................................................................................................... 24
SELF-REPORTED BEHAVIOUR ............................................................................................................ 24
Confidence .............................................................................................................................. 24
Changes in leadership behaviour ........................................................................................... 24
Flow-on benefits for operational behaviour .......................................................................... 27
Authority ................................................................................................................................. 27
Completion versus non-completion of the academic award ................................................. 27
MANAGER ASSESSMENTS OF BEHAVIOUR ....................................................................................... 29
IMPACT IN THE WORKPLACE ............................................................................................................ 30
Longer term impact in the workplace .................................................................................... 32
Sustainability of impact .......................................................................................................... 34
STAKEHOLDER ASSESSMENT ............................................................................................................. 35
RETURN ON INVESTMENT ................................................................................................................. 36
DISCUSSION AND CONCLUSIONS .......................................................................................................... 42
LIMITATIONS ..................................................................................................................................... 42
IMPLICATIONS OF THESE FINDINGS .................................................................................................. 43
5
FUTURE RESEARCH ............................................................................................................................ 43
CONCLUSION ..................................................................................................................................... 45
REFERENCES .......................................................................................................................................... 46
APPENDIX 1 - STUDENT SATISFACTION................................................................................................. 49
APPENDIX 2 – GRADE DISTRIBUTION .................................................................................................... 53
6
INTRODUCTION
The AIPM offers two graduate-level leader development programs: a Graduate Certificate in Applied
Management, and a Graduate Diploma in Executive Leadership. These programs are nested (one
precedes the other), and are offered to police and emergency service personnel at the inspector
rank or equivalent for the Graduate Certificate, and at the superintendent rank or equivalent for the
Graduate Diploma. The graduate certificate is comprised of four subjects. Two are offered through
distance learning, and two are offered face-to-face. The graduate diploma comprises eight subjects,
the four aforementioned graduate certificate subjects and four additional subjects. Of these
additional four, three are offered via distance learning and one face-to-face.
These programs have been running in their current guise since their most recent accreditation (with
the NSW Department of Education and Training) in 2010. Very similar programs were in place prior
to this, since 2002. There are between four and five enrolments for the graduate certificate program
each year, and two enrolments for the graduate diploma. Each enrolment is for between 30 and 40
participants. Australian and New Zealand police jurisdictions are allocated, pro rata, a set number of
places on each of these programs (equating to approximately 108 participants per year), with the
remaining students drawn from other law enforcement and regulatory agencies, allied emergency
services, and international partners.
There is considerable anecdotal evidence (e.g. through formal and informal feedback processes at
the AIPM) to suggest that these programs are well received by participants, and valued by the
jurisdictions that sponsor participants’ attendance. Nonetheless, the programs have never been
formally evaluated for their impact on leaders’ behaviour, and the flow-on benefits to their
organisations. This is not unusual. Recent reviews of the research in this field noted that there was a
lack of robust leadership development evaluations, with data relating to police leadership training
limited to perception evidence only; and that evidence from other public sector organisations
related to such a wide variety of leadership development activities it was difficult to draw definitive
conclusions (Pearson-Goff and Herrington, 2013; Neyroud, 2010; Kodz and Campbell, 2010). Outside
of the police and emergency service leadership field there are several comprehensive reviews of
leadership development initiatives (see for example Day et al., 2013), although even here the
longitudinal and multi-faceted nature of leadership and difficulties in accurately measuring it have
limited this work. Thus in leadership development in general, and police and emergency service
7
leadership development in particular, there remains a need for robust empirical research to evaluate
the impact that such initiatives have.
The purpose of this report is to present the findings of such an evaluation, and specifically an
evaluation of the graduate certificate and graduate diploma programs offered by the AIPM.
EVALUATIONS OF TRAINING Kirkpatrick’s seminal model provides a framework for approaching the evaluation of training
programs (1998). Kirkpatrick notes that any training evaluation needs to take account of four levels
of analysis:
• Student reactions to the training
• Student learning from the training
• Changes in behaviour emanating from the training, and
• The results (impact) of the training.
Traditionally leader development training has been evaluated against the first and second of these
items. Measures of student reactions to training are typically collected in the form of feedback
sheets completed after individual sessions, or on completion of the course. Such feedback sheets
concentrate on the ‘student experience’ and seek feedback on whether a training program was
enjoyable, engaging, and perceived to be valuable. Measures of student learning are typically
identified through assessment items, which are designed to ensure that students have achieved the
learning outcomes set for a program of study. If a student ‘passes’ an assessment item then the
learning outcome, and as such learning, can be regarded as having been achieved.
Both of these ‘measures’ are vital for education providers and are routinely collected, analysed and
evaluated at the AIPM (see appendix 1 and appendix 2 for a brief analysis). However, they provide
little information about the impact of a given training or education initiative on an individual’s
behaviour, or on their workplace once they return. This is particularly important in the AIPM context
wherein all students are sponsored and funded by their employers to attend the leadership
development opportunities on offer. Sponsoring agencies have, then, a right – and perhaps even an
obligation – to request data to ensure that they are receiving a return on this educational
investment. Despite this, comprehensive evaluations of any sort of organisationally sponsored
training are rare, and to our knowledge there are no systematic evaluations of police or emergency
service leadership training conducted to date anywhere in the world (Pearson-Goff and Herrington,
2013). With this in mind this this evaluation set out to answer one key question:
8
• What is the impact of the AIPM graduate certificate and graduate diploma on leadership?
In answering this question a mixed-methods approach was adopted and data were collected from a
range of sources including current program participants, participants’ managers, former students,
and jurisdictional stakeholders. These data were analysed using a range of statistical and qualitative
methods and mapped against data routinely collected by the AIPM. In doing this a comprehensive
picture of leadership development through the AIPM’s graduate programs emerged.
9
METHODOLOGY
As noted, the purpose of this document is to present the findings of a comprehensive impact
evaluation of the nested graduate programs offered by the AIPM. In assessing such impact it is
important to first consider what it is that the AIPM is hoping to achieve with these graduate
programs. That is, what impact should we be looking to measure? Two sources of information
inform this discussion. First is the Police Leadership Capability Framework. The police commissioners
of Australia and New Zealand collectively agreed a police leadership capability framework as part of
the Australian and New Zealand Police Leadership Strategy in 2010 (Victoria Police, 2010). This
framework is organised along six domains to describe the actions of an effective leader. Each of
these domains is supplemented by a number of sub domains, as set out below:
• Shapes strategic direction
o Inspires a sense of purpose and direction
o Focuses strategically
o Initiates and drives change
• Achieves results
o Drives service delivery
o Manages work area and resources
o Builds organisational capability
• Builds and manages relationships
o Treats everyone with respect
o Facilitates cooperation and partnerships
o Guides, coaches and develops people
• Communicates with influence
o Communicates clearly and effectively
o Adapts communications to audience
o Negotiates persuasively
• Personal drive and integrity, and
o Demonstrates and encourages professionalism
10
o Demonstrates self-awareness and a commitment to personal development
o Displays resilience
o Manages all official information appropriately and securely
• Policing skills
o Leads, directs and manages policing operations
Under each of these subdomains there are a range of behavioural descriptors articulating
expectations at each level of the commissioned ranks. For example, under shapes strategic direction
domain, and inspires a sense of purpose and direction subdomain is the behaviour descriptor
“Considers a broad range of options, their implications and ramifications”. Whilst under
communicates with influence domain, and communicates clearly and effectively subdomain is the
behavioural descriptor “Uses communication to motivate and inspire others to action”. This
framework forms a clear indication of the Australian and New Zealand police commissioners’
collective expectations of the actions of senior leadership in their organisations. This means that it is
a pivotal document for the AIPM in conceptualising what it is that needs to be achieved through
police leadership development1.
But the AIPM graduate programs are not only open to police, and are subscribed to by the broader
law enforcement family and allied emergency services. So how appropriate is this Police Leadership
Capability Framework in guiding the aims of the AIPM’s graduate programs? Importantly the DNA of
the Police Leadership Capability Framework can be traced back to the Australian Public Service
Commission’s Senior Executive Leadership Capability Framework (n.d.), which caters for public
servants from a wide range of disciplines. This Senior Executive Leadership Capability Framework
articulates the first five of the six domains also captured by the Police Leadership Capability
Framework, with the one (obvious) exception being policing skills. As such there is logic in the AIPM
utilising the Police Leadership Capability Framework to guide its leadership development work
across agencies with particular focus on the first five of these six domains, and perhaps a broader,
less police-specific (and more industry inclusive) approach taken when considering the sixth. This is
the approach taken in this document, and to signal the resulting hybrid between the Police
Leadership Capability Framework and the Australian Public Service Commission Leadership
1 The Australian and New Zealand Police Advisory Agency (ANZPAA) is currently developing a professionalization strategy, in which they are developing an agreed set of standards for performance at each rank. This work is ongoing, and not publically available, although draft Training and Education Guidelines for police manager and police executive suggests that it reflects closely the Police Leadership Capability Framework outlined above.
11
Capability Framework the term the Leadership Capability Framework is used. This framework
encompassed the first five domains of the Police Leadership Capability Framework, and a slightly
expanded sixth domain encompassing police and other emergency service industry skills.
Another source of information to consider when conceptualising what we mean by program impact
is the learning outcomes for the two graduate programs (and their constituent subjects). These
learning outcomes have been developed over a number of years to reflect both the needs of
industry, along with the academic requirements of the Australian Qualifications Framework (AQF)
for graduate certificates and graduate diplomas (i.e. level eight in the AQF). Thus they are designed
to marry the requirements of the Leadership Capability Framework with the intellectual
requirements of study at the graduate level. A recent analysis of the graduate program curricula (i.e.
learning outcomes) against the Leadership Capability Framework domains suggests that this is done
successfully, resulting in a well-balanced curriculum. See figures 1 and 2.
Figure 1: Curriculum mapping of the Graduate Certificate in Applied Management (% of course
learning outcomes addressing each capability)
Shapes strategic direction
34%
Achieves Results 25%
Builds and manages relationships
16%
Commmunicate with influence
11%
Personal drive and integrity
14%
12
Figure 2: Curriculum mapping for the Graduate Diploma in Executive Leadership (% of course
learning outcomes addressing each capability)
The industry-relevant skills domain is not represented in this analysis although that is not to say that
this is not represented in the curricula. Industry relevant skills content is embedded within learning
outcomes that set out to achieve other things. For example a learning outcome may concentrate on
achieving results – e.g. examine the links between organisational design, implementation and
accountability for quality service delivery – although the context in which that learning occurs
requires drawing on industry-relevant knowledge and skills. Similarly where a learning outcome asks
participants to diagnose organisational challenges, success relies on their drawing on their industry
context.
SAMPLE SELECTION We adopted a mixed-methods approach to assessing the impact of the AIPM graduate programs on
leadership. Quantitative and qualitative data were collected from:
• Current graduate program participants using a pre- and post-course survey instrument
• Former program participants who had completed their studies between 12 and 24 months
prior to the start of this research, using a semi-structured interview schedule
Shapes strategic direction
39%
Achieves Results 19%
Builds and manages relationships
14%
Commmunicate with influence
11%
Personal drive and integrity
17%
13
• Managers of current graduate program participants, using a structured interview schedule
designed to elicit qualitative and quantitative data, and
• Jurisdictional stakeholders using a semi-structured interview schedule.
A breakdown of the data collected from each sample by methodology is set out in table 1.
Table 1: Summary of data collected
Semi-
structured
interviews
(qualitative
data)
Mixed-methods
(structured)
interview
Quantitative
survey (pre-
course)
Quantitative
survey (post-
course)
Matched pairs
(both pre and
post course)
Current course
participants
161 83 70
Former course
participants
30
Stakeholders
12
Managers 9
Total 52 9 161 83 70
Current course participants
A questionnaire was sent to every enrolee on the AIPM’s graduate certificate and graduate diploma
program between 12th March 2012 and 2nd November 2013. This equated to 303 course participants
across 10 cohorts (205 on the graduate certificate program and 98 on the graduate diploma
program). Pre-course questionnaires were sent to participants with their enrolment packs three
weeks before the start of their studies. This allowed the benchmarking of individual performance
against the Leadership Capability Framework. A second (post-course) questionnaire was sent to the
same participants three months after the conclusion of their program (marked by the deadline for
their final assignment). During this intervening time 27 participants had withdrawn from the
program (9%). In addition, two cohorts (n=72) were not included in the post-program data collection
in time for this report. As such a request to complete a post-course questionnaire was sent to 204
course participants. A total of 161 participants returned pre course questionnaires, which represents
a response rate of 53% (of 303 prospective participants). A total of 83 course participants responded
14
to the post-course questionnaire, representing a response rate of 41% (of 204 participants). These
response rates are reasonable, and in line with similar surveys of police and emergency services
personnel (see Herrington and Pope, 2013). Seventy participants completed both a pre-course and a
post-course questionnaire (which is 23% of the 303 participants originally invited to take part in the
research, or 34% of the 204 participants contacted following completion of their course). Whilst
these matched pairs allow a robust measure of individual change for this group over time whilst
controlling for unsystematic variance as a result of individual differences2, it is important to consider
the impact of sample bias given that less than a third of eligible course participants completed both
pre and post course questionnaires. We can speculate as to the reasons for this, but cannot discount
that there was something unique about the sample involved in this research (e.g. they had a
particularly good, or indeed particularly bad experience on the course) which may affect the validity
of the findings from this research.
Most respondents were enrolled in the graduate certificate program (68%), and most of the
matched pairs were also drawn from this group (84%) which is unsurprising given the higher number
of students enrolled in that program during the course of this research. The small numbers from the
graduate diploma enrolments affects the analysis insofar as it limits the possibility to compare data
from the two (graduate certificate and graduate diploma) groups. This is not necessarily a problem,
however, particularly given the curricula analysis above that suggests that there is little difference
between the two programs in terms of the proportional focus of learning outcomes aligned with the
Leadership Capability Framework. This is in itself unsurprising given that both programs set out to
achieve, broadly, the same thing – positive movement in line with the Leadership Capability
Framework. That is not the same as saying that the content of the programs is similar, of course. At
the graduate certificate level the AQF requires that participants are able to demonstrate specialist
knowledge, whereas at the graduate diploma level the AQF requires advanced knowledge, so clearly
the depth with which the Leadership Capability Framework domains are dealt with differs between
programs, even if the curricula-proportion in which they are dealt with does not. Nonetheless, given
that the measurement of the impact of the AIPM’s graduate programs has been operationalised in
terms of movement in line with the Leadership Capability Framework, and both programs set out to
achieve this in (broadly) equal measure, we might expect little difference between the two programs
along these lines, and as such there is logic in dealing with the data from these programs in
aggregate. If one wanted to evaluate the qualitative difference between learning at the graduate
2 Both pre and post-course participants were sent a research reminder approximately two weeks after the initial invitation.
15
certificate and the graduate diploma then different data would be required, and this may be an area
for future research.
Most (76%) of participants were male, and the (pre-course) sample had an average of 22 years’
service with their organisation. There were no statistically significant differences between the
groups who returned matched questionnaires and those that only responded once in terms of
gender3, rank4 or years’ service5. This means that there are no demographic differences between the
matched and unmatched samples that may inadvertently skew the analysis of this sub-group. Sixty
per cent of respondents were drawn from Australian and New Zealand police jurisdictions, the
remainder were drawn from Australian and New Zealand emergency services (27%), and other
national and international law enforcement and emergency service agencies (13%). This is in line
with population data for participants on the graduate programs at the AIPM as a whole, where
during 2013, 55% of participants were drawn from Australian and New Zealand police and law
enforcement; 31% from Australian and New Zealand emergency services; 14% from international
law enforcement and emergency service agencies. This is an important consideration when
considering the generalizability of findings from this research to the broader AIPM participant
population, and suggests that at the level of host organisation, there is no difference in terms of
proportional representation between our sample and the population from which it is drawn.
Importantly, whilst post course questionnaires were sent out three months after the deadline for
the final piece of assessment for the program, not all respondents had completed their studies at
that time. Of the post-course responders 11 (16%) had not yet completed their studies (two in the
graduate certificate and nine in the graduate diploma program). There are a number of possible
reasons for this, including participants having been granted submission extensions for their
assessments. In addition, not all course participants choose to complete the entire graduate
certificate or graduate diploma program, and only attend the residential component of the course.
This group miss out on the learning embedded in the distance education subjects for the graduate
programs, of course, and may not benefit as much as those who complete the study opportunity (as
discussed on page 27).
3 χ2(1)=0.026, p>.05 4 χ2(3)=7.242, p>.05 5 Levene’s test is not significant and equal variances are assumed (F=.868, p>.05) t= -.203, df 154, p>.05
16
Former course participants
Thirty semi-structured interviews were undertaken with former AIPM graduate program
participants. These participants had all completed their studies between 12 and 24 months prior to
the start of this research in March 2012. The purpose of these interviews was to assess, qualitatively
and longitudinally, the impact of the AIPM’s programs on behaviour and workplace practices. These
interviews also allowed insight into how individual learning and associated behaviour changes
translate into organisational outcomes and capacity.
Letters of invitation to take part in the research were sent to former students of five graduate
certificate programs and four graduate diploma programs conducted in 2010 and 2011. Thirty
responded and were interviewed for this research. Reflective of the data collected from current
program participants (and reflective of the breakdown of senior leadership in police and emergency
service organisations across the world) most interviewees were male (n=24), with a broadly even
representation from inspector rank through to the senior executive levels. Seventeen had been
enrolled in the graduate certificate program and thirteen had been enrolled in the graduate diploma
program.
Managers of current course participants
A regular criticism of training evaluations that utilise a self-report mode of identifying behaviour
change is that these behaviours are easier to say than to do, and as such course participants may
over-estimate their performance. One way to counter this critique is to triangulate self-report data.
This can be done in one of three ways:
• By interviewing course attendees after they return to work to allow them to provide
examples of changed behaviour in practice (as detailed above under former course
participants)
• By observing course participants in the work place (which can be impractical and resource
intensive), or
• By garnering the views of the people working with course attendees in the workplace.
17
In an attempt to address the last of these points we undertook structured interviews with the
managers of current program participants who had completed both a pre- and a post-program
survey. These structured interviews elicited both qualitative and quantitative data about a course
participant’s performance at work. The quantitative data was collected using the same
questionnaire that current course participants completed, and allowed, then, a direct comparison
between participant assessments of behaviour, and manager assessments. Collecting these data
necessitated a complex administrative task and multiple opportunities for attrition. First we
contacted each of the jurisdictions to obtain approval to contact former students about their
managers. Two agencies did not respond. Where approval was granted we contacted course
participants who had completed both a pre- and post-course questionnaire. We asked these
participants for permission to contact their manager and discuss their performance following the
graduate program. We contacted 67 participants, of whom 31 responded. Unfortunately seven of
these noted that they had recently moved roles and/or their manager was not in a position to
comment on their behaviour pre- and post-course. We contacted the nominated managers of the
remaining 24 participants. Fifteen nominated managers were unavailable meaning that data were
collected from only nine. This means that this group can only be regarded as a convenience sample.
As such whilst the data from this group can be used reliably to assess differences between self- and
manager-assessments of behaviour for individual participants, such findings may not be
generalizable to other program participants.
Jurisdictional stakeholders
A second means of triangulating data collected from current and former students is through semi-
structured interviews with jurisdictional stakeholders. This provides an organisational-eye-view of
the graduate programs and in particular the perceived organisational benefit emanating from these.
The AIPM’s graduate programs are designed to fulfil the leadership needs of industry, and in
particular the needs of Australian and New Zealand policing. Thus whilst participants on AIPM
graduate programs may be drawn from a range of law enforcement and emergency service
organisations, the Board of Control – the overarching steering and accountability mechanism for the
AIPM – is comprised of the commissioners of each of the nine police jurisdictions in Australia and
New Zealand. It is important, then, to assess whether these stakeholders feel that their needs are
being met.
Letters of invitation to take part in this research were sent to each of the police commissioners,
wherein they were asked to nominate two senior officers with responsibility for leadership learning
and development to take part in a semi-structured interview. Ultimately twelve senior executive
18
leaders (typically at the assistant commissioner level) were interviewed, drawn from eight of the
nine police stakeholder jurisdictions.
QUESTIONNAIRE DESIGN Course participants were surveyed before and after their completion of an AIPM graduate program
in order to determine the impact this had on their leadership. This necessitated the development of
a pre- and post-course questionnaire. This questionnaire was also completed by the managers of
course participants, with reference to course participants, as detailed above. The questionnaire
utilised the Leadership Capability Framework as a starting point for determining items that might
effectively measure change. Each of the first five domains (shapes strategic direction, achieves
results, builds and manages relationships, communicates with influence, and personal drive and
integrity) were examined, and a small sample of behavioural descriptors from each of these domains
were converted into items for the questionnaire. These items related to self-reported behaviour.
(Industry-relevant skills were dealt with separately and are described later).
Of key importance was the creation of a questionnaire that was quick and easy to complete. Thus a
balance had to be achieved between fully representing the domain in question and parsimony.
These items were arranged along a five-point Likert scale and participants were asked to rate their
response from strongly agree to strongly disagree. The questionnaire was piloted with a small cohort
of participants (n=35) enrolled on a graduate certificate program in early 2012. Questions that
performed unexpectedly or were ambiguous were subsequently excluded.
Questions for each domain were designed to be scaled so that aggregate scores could be calculated
for self-reported behaviour relating to each domain. This is of benefit for a range of technical
methodological reasons. The items for each scale are listed below, along with their Cronbach’s α
(alpha) value, which is measure of the reliability of the scale6.
Shapes strategic direction (Cronbach’s α = .767)
1. I am effective at making strategic decision I am good at seeing the ramifications of my
decisions before I make them
2. I draw on the ideas of others when making decisions
3. I look for issues where my leadership can make a difference
6 A Cronbach’s α value identifies the extent to which items on the scale perform in ways that are consistent with each other (i.e. internal reliability). This is an important consideration when summing individual item responses together. A Cronbach’s α value can vary between -1 and +1, and a scale is regarded as good if it has a value that is positive and close to +1. A scale has good internal reliability if the α value is greater than +0.7.
19
4. I link strategic direction and operational outcomes easily
5. I look for connections between different issues and problems, that might not be
immediately obvious
6. I look for connections between current and future trends.
Achieves results (Cronbach’s α = .702)
1. I am effective at managing resources
2. I am effective at getting my team to work together
3. I find it hard to motivate my team (reverse coded)
4. When someone in my team is not performing well I address this with them
5. I communicate clear activity goals for my staff
6. I am good at making things (e.g. change and action) happen
Builds and manages relationships (Cronbach’s α = .742)
1. I look for development opportunities for my staff
2. I connect people in different organisations together to solve problems
3. I am good at building relationships with lots of different types of people
4. I empower staff to make decisions
5. I involve people from the community to identify problems
6. I involve people from the community to solve problems
Communicates with influence (Cronbach’s α = .686)
1. I am good at presenting complex information in writing
2. I find it hard to communicate my ideas (reverse coded)
3. I am good at listening to other people when they are talking
4. I am able to see both sides of an argument
5. I am good at presenting complex information verbally
6. I am a good mentor
Personal drive and integrity (Cronbach’s α = .627)
1. If I make a mistake or wrong decision I try and learn a lesson from it
2. I reflect on (think about) my decisions and behaviours afterwards
3. I find it difficult to remain calm when under pressure (reverse coded)
4. I am a good role model
20
5. I know, and play to, my strengths
6. I know, and work to address, my weaknesses
7. I make time to read research and academic literature about policing or emergency services
In addition to these scale items, several other questions of note were asked, including indicators of
confidence about one’s leadership skills, and questions about police and/or industry relevant skills.
These were analysed separately. Moreover, a central theme in leadership learning on the AIPM’s
graduate programs is a critical analysis of the impact of ‘authority’ on leadership. In particular
participants are encouraged to think about leadership as an activity, rather than a role, and as such
to be aware that because one has authority (most visible in policing and emergency service
organisations in terms of rank) this does necessarily equate to effectively exercising leadership. This
is an important consideration in stubbornly hierarchical organisations such as the police where,
because of their hierarchical structure, much can often achieve outcomes through an “I say: you do”
approach (Herrington, 2012: 95). Whilst command and control undoubtedly has its place, there are
limits to the effectiveness of such an approach, both in terms of organisational wellbeing (see for
example the rich literature on organisational justice e.g. Herrington and Roberts, 2013; Roberts and
Herrington, 2013) and achieving outcomes. Specifically, in a world where police and emergency
services tackle increasingly complex or wicked problems there may be no clear direction in which to
command one’s troops. In such a situation leadership is less about knowing the correct answer to a
problem and directing others towards that, and more about creating an environment in which
creative and innovative solutions can emerge. Thus leadership based solely on authority
inadvertently curtails a leader’s ability to achieve the outcomes that they and their organisation seek
(Heifetz et al., 2009).
Four items associated with authority were asked in the questionnaires, and created a provisional
authority scale (Cronbach’s α = .6247).
1. A leader must use their authority to get things done (reverse coded)
2. Staff must not challenge a leader (reverse coded)
3. A leader must not show weakness (reverse coded)
4. Staff must respect a leader’s authority (reverse coded)
7 Calculated based on post-course scores. Pre-course scores had a Cronbach’s α = .423, suggesting that post-course, and not unexpectedly, participants had a more cohesive appreciation of authority.
21
ANALYSIS Quantitative data were entered into a statistical software package (SPSS) to facilitate descriptive
analysis and statistical testing. Pre-post comparisons and self-manager comparisons employed
paired-samples t tests. T tests assess the extent that any observed difference between two sets of
scores is due to chance. They do this by calculating a probability value denoted as p. P values range
from 0 through to +1. The higher the value of p the more likely it is that the observed set of results is
due to chance. The lower the value of p, the lower the probability is that the observed difference
between two groups is due to chance. Where the p value is very low, a result is regarded as
statistically significant. Convention dictates that the p values of greatest interest to researchers in
determining statistical significance are p< .05, and p< .01. The former means that we can be
confident that only five results in every 100 will likely be due to chance (i.e. we can be 95% confident
that a true difference between two sets of scores - i.e. the pre and post conditions - exists), the
latter denotes that only one in every 100 results will likely be due to chance (i.e. we can be 99%
confident) (Herrington, 2011).
Qualitative data were entered into a thematic coding software package (NVIVO) and coded for
themes relating to the Leadership Capability Framework. This was a largely iterative process with
themes emerging from the data as it was coded. The overarching coding structure employed drew
on the aforementioned four facets of a training evaluation (Kirkpatrick, 1998):
• Reaction
• Learning (and associated gaps)
• Impact, and
• Results
ETHICAL CONSIDERATIONS This research was approved for ethical conduct by the AIPM’s Ethical Review and Research
Governance Committee (ERRGAC), which was comprised of six external (university-based) policing
scholars from four Australian universities8. Importantly, ethically, this research was regarded as low
risk insofar as it involved no more than inconvenience or mild discomfort (Australian Government,
2007: 18, paragraphs 2.1.6-2.1.7). Full voluntary and informed consent was secured at all stages of
the research process. Many of the ethical concerns in relation to this research are characteristic of
qualitative data collection in general.
8 See http://www.aipm.gov.au/research/ethical-review-and-research-governance-advisory-committee/ for further details.
22
A specific ethical concern was the matching of individual assessments with manager assessments of
behaviour. This was of course sensitive, and was dependent on individuals consenting to the
research team contacting their manager in the first place. Importantly, to guard against undue angst
the research team ensured that both managers and participants were aware that no information
pertaining to the other party was passed to the other. Nonetheless the attrition rate for this aspect
of the research may be a reflection - in part - of concern around the matching process. As noted
above, findings from this aspect of the research, whilst important, should be treated with due
caution because of the small numbers involved.
METHODOLOGICAL LIMITATIONS Research methodology is always a necessary compromise between the ideal and the humanly
possible, and as such every research project will have its limitations. Perhaps the key methodological
limitation of this study is that the research was undertaken by employees of the AIPM, and as such
the research team were not fully independent of the body being researched. Care was taken to
safeguard the independence of the research team through the external ethical review process, and
through the external peer review of the findings as set out in the acknowledgements section. The
professional integrity of the research team (to report findings honestly and without fear or favour)
will have to be taken on trust. Importantly there is an argument that this research could only have
been undertaken by a team within the AIPM. It necessitated approval from and engagement with
representatives from more than 20 organisations, and trust of the AIPM was a key facilitator in
securing consent.
Two other methodological issues are worth noting here, and both are a function of a voluntary (self-
selecting sample) survey approach. The first is sample bias. When individuals are asked to fill in a
questionnaire, some will say yes, and others will say no. Sample bias refers to the possibility that
there is something unique about those who take part that may influence the findings of the
research; for example they had a particularly good experience on an AIPM graduate course, or have
some other (unconscious) agenda for providing feedback, good or bad. With a survey response of
34% we cannot discount the possibility that the remaining 66% of course participants had a different
experience, although as we shall see in this report, findings seem to accord with the anecdotal
evidence and formal and informal feedback received by the AIPM across the last 12 years that these
programs have been running (see for example appendix 1).
The second concern relates to the accuracy of responses. Much of the data in the report relies on
self-reported assessments of behaviour and it is possible that respondents were either overly-
23
optimistic in their self-assessments, or wanted to portray themselves in a positive light (i.e. a social
desirability effect). We attempted to counter this through manager assessments, although only nine
managers were available to take part in the research. Importantly these managers had similar, or
better, perceptions of their employee’s behaviour, but that is not to say that this would have been
the case across the cohort. As such, the findings relying on self-reported behaviour must be taken
with appropriate caution.
24
FINDINGS
SELF-REPORTED BEHAVIOUR Data were collected from course participants before and after their graduate studies with the AIPM,
using the questionnaire previously described on page 18. These two data collection points are
referred to as time one (T1 - pre) and time two (T2 - post) respectively. The following section relates
the comparison of data between T1 and T2, using data from 70 matched pairs.
Confidence
We might expect that a leadership development program would lead to greater confidence about
one’s leadership skills. With this in mind participants were asked to rate their confidence in their
leadership skills at T1 and T2. A comparison of means was statistically significant9. That said,
confidence is a difficult concept to pin down, and one can be confident about one’s abilities even if
one is not particularly competent. This being the case leadership development can sometimes lead
to a reduction in confidence, as one becomes more aware of the limits of one’s own abilities.
Participants were therefore asked at T2 whether they thought they were an effective leader before
they started the course, and whether they felt they were an effective leader having completed the
course. (This allowed control for the possibility that some participants may have felt more effective
pre-course before truly understanding the role of a leader.) The difference between data for these
questions was statistically significant, suggesting that participants felt that the program had had a
beneficial impact on their leadership skills10.
Changes in leadership behaviour
Whilst felt change is an important individual consideration, a more robust measure of impact is
evidenced through changes in behaviour. Course participants were asked to self-report their
leadership related behaviour at T1 and T2 along the five scales created from the Leadership
Capability Framework (see page 18 for more details). Mean scores for T1 were compared with mean
score for T2 using paired t tests. All five domains showed statistically significant differences, with
self-reported behaviour at T2 more in line with the intentions of the Leadership Capability
Framework11. Figure 3 displays these findings.
9 t=-1.922, df 69, p=.059 (two-tailed), p<.05 (one-tailed). Justification for a one-tailed predication is the expected increase in confidence following training. 10 t=-3.176, df 82, p<.01 11 Shapes strategic direction (t=-5.841, df 69, p<.001); Achieves results (-3.249, df 68, p<.01); Build and manage relationships (t=-2.844, df 68, p<.01); Communicate with influence (t=-3.459, df 69, p<.01); Personal drive and integrity (t=-3.223, df 68, p<.01).
25
Figure 3: Leadership Capability Framework Domain Score - Pre-Post Comparison
In order to calculate global changes across all five domains pre and post training, each participant’s
pre- and post-training domain scores were summed. This produced a pre- and post-training total
score for each participant. The pre- and post-training means (average) for these scores were
calculated for the whole sample, and compared using a matched pairs t-test. The mean score was
greater post-training than pre-training and this was statistically significant. This indicates that post-
training participants were more positive about their abilities than before (see figure 4)12, which is
consistent with the aims of the program.
12 t=-5.227, df 68, P<.001
0
5
10
15
20
25
30
Shapes strategicdirection
Achieves results Builds &manages
relationships
Communicateswith influence
Personal drive &integrity
Mea
n of
scal
e su
m
Pre
Post
26
Figure 4: Leadership Capability Framework Total Score – Pre-Post Comparison
Another benefit of summing the domain scores into a total score is that it allows us to calculate an
overall effect size for the AIPM graduate program. An effect size demonstrates the magnitude of an
observed effect – in this case the impact of the AIPM graduate programs. By contrast probability
(significance) testing only allows us to conclude that there is a difference between groups that is not
the result of chance (i.e. a statistically significant difference). It tells us nothing of the size
(magnitude) of that difference. The effect size can be calculated in a number of ways. Here it is
calculated using Pearson’s r, which relies on the t value reported from the comparison above. It is
calculated as follows (Field, 2005)13:
𝑟 = �𝑡2
𝑡2 + 𝑑𝑓
Where t is the t value from the paired t test between total scale scores at T1 and T2 (i.e. -5.227) and
df is the degrees of freedom from that t test (i.e. 68). This calculation leaves us with an effect size of
0.54. This means that the pre-post survey data collected from course participants suggests that the
AIPM graduate program had a large effect on the (self-reported) leadership behaviour. We will
return to this effect size calculation later, when calculating return on investment (page 36).
13 Effect sizes range between 0 and 1, and anything over .5 can be regarded as a large effect (Field, 2005).
119
120
121
122
123
124
125
126
127
128
Pre Post
Mea
n of
tota
l dom
ain
scor
e su
m
27
Flow-on benefits for operational behaviour
In addition to the five aforementioned scales, the pre-post questionnaire also tried to tap into
behaviours that might be best classified under industry-relevant skills. The spirit of this domain in
the Police Leadership Capability Framework is that leaders should be able to effectively lead,
manage and direct (police and other emergency service) operations. Whilst operational activity is
not the focus of the AIPM’s graduate programs, the operational context is clearly embedded within
the programs’ learning outcomes and good leadership has positive flow on benefits for operational
activity. Thus we should expect some positive movement here. Three questions regarding
operational activities were asked of participants at T1 and T214.
1. How confident are you about your tactical and operational skills (whether in a sworn or
unsworn capacity)?
2. I am effective at making operational decisions, and
3. I am good at coordinating complex events.
For all three there was a statistically significant difference between T1 and T2 in the direction
anticipated (i.e. participants reported a positive increase in confidence and behaviour post-course)15.
Authority
Authority is a central concept in the leadership development training at the AIPM, with time spent
considering the difference between use of authority (i.e. use of one’s role/rank) and leadership (i.e.
engaging in leadership activities). Using the authority scale outlined in the methodology section self-
reported authoritative behaviour was assessed before and after the AIPM graduate program. There
was a significant difference between T1 and T2 in the direction expected, with participants at T2
noting a reduction in affiliation with behaviour associated with use of authority16. Qualitative data
too - discussed on page 32 - provides supporting evidence for this.
Completion versus non-completion of the academic award
Not all participants enrolled in AIPM graduate programs complete their studies, which raises the
question of whether a positive impact can be achieved through partial completion of the course17.
An assessment of the impact of the program for those who completed the full program, and those 14 These items were not comprehensive enough to form a scale in the same way that the other domains were, and must be analysed separately. 15 Confidence in tactical and operational skills (t=-2.304, df 69, p<.05); I am effective at making tactical and operational decisions (t=-2.304, df 69, p<.05); I am good at coordinating complex events (t=-3.579, df 68, p<.01). 16 t=-2.634, df 68, p<.01 17 Where participants only complete part of the course this is almost always the face-to-face element.
28
who did not can be made by calculating a change score for each individual for each domain (by
subtracting scores for the five domains at T1 from scores at T2). These change scores were then
compared for the completer and non-completer groups using an independent samples t test. Only
the shapes strategic direction domain was significantly different between the two groups, with those
who completed their graduate program reporting a significantly higher score in this domain than
those who did not complete18. This finding complements the analysis presented on page 11 that
notes a large proportion of the graduate curricula dedicated to this domain (34% in the graduate
certificate and 39% in the graduate diploma). It would follow, then, that this is a particular focus for
the graduate programs and one would expect that those who complete their studies to perform
better in this regard. It is interesting that in interviews with former students, several noted the value
of the distance education components of the program, despite initial thoughts. For example:
I thought academia was a crock before I got involved in it. When I came in I was not exactly negative because I wanted the [face-to-face] experience, but I didn’t think I would get anything out of the academic side of things. But I came out the other side with a greater understanding of the processes and how to truly research and create something that is complete. Before I thought I knew how to create something, but the course took me to a new level of understanding. (Graduate diploma participant, Australia and New Zealand emergency service member).
Of course not everyone had a positive view of the distance learning components of the program, and
some participants simply felt that academic development was not something that they would
benefit from. This is not surprising given that police and emergency services are still grappling with
the issue of professionalization and all that that means, including reference to an established body
of knowledge and the relative value of tertiary qualifications. Tertiary study requires the
demonstration of high level critical thinking skills, the ability to synthesise complex information, and
a familiarity with an evidence base for decision making. Undoubtedly such skills can be achieved
without the completion of a formal tertiary award, but are they? The requirement to test (through
assignments) these skills, ensure that this learning is done (see analysis in appendix 2). Whether
participants value completion of an academic qualification or not, the pre-post data suggests that
there are additional benefits for (self-reported) leadership behaviour from achieving this. Thus one
could argue that it is in the jurisdictions interests to further encourage participants to complete both
the face-to-face and distance learning aspects of the AIPM program to ensure that they maximise
their return on investment. Interviews with former course participants suggested that there was
more that jurisdictions could do in respect to this. For example:
18 t=3.150, df 38.183, p<.01 (Levene’s test was significant meaning that equal variance could not be assumed).
29
No one spoke to me about the program, and there was limited support and accountability to complete the diploma. I’ve not finished the [final] paper…[and] if I don’t do it, it has no bearing on my work. (Former graduate diploma participant, Australian and New Zealand police service)
No I didn’t feel supported. Our organisation does not seriously value this program. The commander wants people to do it but there is a lack of regard for the program. They don’t acknowledge the impact or support it enough. I think I am only the second person to finish it! If there was more encouragement from the organisation then maybe the completion rate would increase. (Former graduate diploma participant, Australian and New Zealand police service).
It is of course a matter for the jurisdictions – the sponsors of participants on these programs – as to
how they might maximise completion rates and as such their organisational return, but a monitoring
of completion rates and tying this to some sort of organisational accountability mechanism may be
appropriate.
MANAGER ASSESSMENTS OF BEHAVIOUR Individual self-assessments of behaviour along the five leadership domains at T2 were corroborated
with manager assessments of the same. The aim of this analysis was two-fold: to triangulate self-
assessments of behaviour, but also to identify whether self-reported changes in behaviour were
observable in the workplace. This is a particularly important consideration for industry sponsored
development programs where there is an assumption that this individual development will benefit
the organisation in some way on return to the workplace. As such it is not sufficient that individuals
change, they must be seen to change in the eyes of those that sponsor their development
opportunity. Matched data were collected from nine managers of participants (difficulties in
collecting this data and the implications of a low response rate have been detailed on page 17).
These data were compared with course participants’ self-assessed scores across the five Leadership
Capability Framework domains using a paired t test. Two domains returned a statistically significant
result: achieves results19, and personal drive and integrity20. In both cases managers had a higher
opinion of an individuals’ behaviour in that domain than they did. In the remaining three domains
there were no significant differences between manager- and self-assessments of behaviour, which is
consistent with there being general agreement between the two.
19 t= -3.000, df 7, p<.05 20 t=-3.309, df 8, p<.05
30
When asked about notable changes in their charges following their AIPM program, managers noted
increased confidence, communication and a greater breadth of thinking. For example:
[The participant’s] self-confidence has grown and confidence in their management style has grown and is still growing. Writing skills are also improving, they were never bad, but [s/he] is now writing in a more succinct way which is obviously important when communicating with the executive. [S/he] is certainly more reflective, though I think [s/he] probably always was, and [s/he] verbalises and talks through his/her decisions more. (Manager of graduate diploma participant, Australia and New Zealand police service)
[The participant] is more open and understands one needs other viewpoints to bounce ideas off, so is now trying to understand from multiple dimensions and advance [himself/herself] there. [S/he] introduced emotional intelligence to the organisation and after the program was more driven to use it to achieve broad outcomes. Has a better understanding of contextual issues facing public safety and a broader perspective. The program enabled [him/her] to identify blind spots. (Manager of graduate certificate participant, Australia and New Zealand emergency service)
This is important insofar as it demonstrates that for these participants at least, self-documented
change was noticed in the workplace, and was seen to be contributing to organisational outcomes.
IMPACT IN THE WORKPLACE Participants were asked at T2 whether they had planned to make any changes to their leadership
behaviour in the workplace on completion of their graduate program. Most (93%) said that they did
(6% were unsure and 1% had no plans). Participants were then asked what changes they planned to
make. These free-text answers were coded along the lines of the Leadership Capability Framework,
where possible, and illustrate that most planned to better build and manage relationships (including
better supporting and empowering their staff, and developing others), and to engage more fully in
self refection21. Figure 5 sets these findings out.
21 Participants could note more than one planned change, and as such numbers may exceed the total number of respondents.
31
Figure 5: What changes to your leadership do you plan to make?
Whether course participants planned to make changes or not, all reported that they had done so by
the end of the program, with 14% having made all their planned changes, and 86% some. Making
such changes had not necessarily been smooth sailing, and most (83%) noted some difficulty in
doing so, which revolved largely around the demands of their current role22 (see figure 6).
22 Participants could nominate more than one inhibitor and as such numbers may exceed the total number of respondents.
0 10 20 30 40 50 60 70
Shape strategic direction
Achieve results
Build and manage relationships
Communicate with influence
Personal drive and integrity
Leadership rather than authority
Self-reflection
Undertake deeper analysis
32
Figure 6: Inhibitors to enacting leadership change
Longer term impact in the workplace The longitudinal impact of the program on individuals and their workplace behaviour was examined
through interviews with former participants. Of this group six (out of 30) had been promoted since
finishing their graduate course, and eight had acted at higher levels. All except one participant noted
some positive impact on their leadership behaviour which they attributed to the program, and 23
(three quarters) felt that they were now better equipped to deal with leadership challenges than
they would otherwise be. For some, the impact of the program had been profound. For example:
It wasn’t just a script or a set of proscriptions that I got out of the graduate diploma, but a whole way of policing. The self-reflection was good…and the course was life changing, because I learnt to open up to myself. (Former graduate diploma participant, Australian and New Zealand police service member)
That the program had been life changing for some is important when estimating the longevity of the
training impact demonstrated with the pre-post data above. All training tends to have a decay effect,
wherein after a while one forgets the new knowledge one has and old ways reassert themselves. In a
leadership sense if an individual learns a new way to be a leader, they may implement this new
approach on their return to work, although we would expect to see a decay of this over time. This
may be the result of organisational or structural pressures to work in the ‘old way’, or because of a
lack of time to think consciously about behaving in the ‘new way’. However, if a program
fundamentally changes one’s view or knowledge about oneself, then there may be greater resilience
0 5 10 15 20 25 30 35 40
Other people!
Current position
Timing - waiting for right opportunity
Larger structural changes required
Time/workload
Reverting to old habits
External factors
33
to the decay of enacting associated changes over time. The data collected thus far cannot assess
whether there is a decay effect of training from the AIPM graduate programs. Longitudinal data,
collected in the same way over many years, are necessary for that. Although the qualitative
comments of former course participants suggest that this may be an interesting dynamic to consider
further:
[The graduate program] is not something where the significance of it is clearly evident. It is something that you live with. (Former graduate certificate participant, Australia and New Zealand police service member).
Self-awareness was an oft-cited outcome from the graduate programs, and the noted flow on
benefits of being a more reflective practitioner included an ability to “listen more and put myself in
their [my staffs’] shoes” to “learn the balance between delegating and fobbing off” and “to not beat
myself up to a pulp and learn to forgive myself” (Former graduate diploma participant, Australian
and New Zealand police service member). Changing one’s relationship with one’s colleagues was
writ large, and was seen to have benefits for the organisation not only in terms of individual
wellbeing, but also in terms of the team dynamics of the group for whom the participant was
responsible for. Morale, productivity, sick leave, and generally having people turn up to work who
wanted to be there were attributed by former participants to their changed leadership as the result
of their studies. Several noted that this included letting go and empowering staff to come up with
solutions as well as identify problems. With better delegation skills some noted that they were no
longer working excessively long hours, with flow on benefits for their own health, well-being and job
satisfaction too.
By being more reflective, participants felt in a better position to communicate with their staff, and as
such had a better chance of dealing with issues that could become troublesome. For example:
My biggest takeaway was trying to understand how other people view you. I try not to be aloof or anything like that, and I am now more relaxed and approachable so that people can come and talk to me about their issues. If they talk to you about their issues you can sometimes nip potential problems in the bud. (Former graduate certificate participant, Australia and New Zealand police service member)
And in some cases the time away to reflect about one’s workplace allowed participants to make
relatively small but significant changes to their own behaviour that helped solve long standing and
festering workplace concerns. One powerful example of this is set out below, and relates to
addressing a long standing toxic culture in one command, ameliorated by a relatively small change to
behaviour on return to work:
34
When I used to go to work, I used to drive in the back gate, walk up the back stairs and then go into my office where I stored my gun in a safe and then get ready for my 8am meeting. So the first time that any of my command staff or anyone else saw me was at this 8am briefing. When I went back to work after the graduate diploma I decided that instead I would walk in the front door, past all of the troops and say hello to everyone. I also decided that I would store my gun in the communal armoury instead of in my office so that I had to pass the troops every morning. This was a hard thing to do, and I had my heart in his mouth because the environment on the floor was toxic. So it was uncomfortable to be amongst the troops downstairs, and that is why I had sought refuge in my office…I instigated these changes, and whilst they were hard, I also found that they worked. I know this because one of the older sergeants came to me one day and said “no one else will probably tell you this, but it is really good that you come and talk to us every day”. This changed my relationship with the troops and opened up opportunities for them to talk to me in an informal context instead of only in the context of a meeting, or when they were in trouble. (Former graduate diploma participant, Australia and New Zealand police service member)
Other tangible organisational outcomes from the graduate programs included the development of
organisational strategies, development of stakeholder and partnership operating procedures, and
the implementation of new policies. Many of these emanated from former participants’ academic
work – and specifically their strategic policy review paper which is the capstone piece of work in the
graduate diploma. One participant noted the organisational impact of a simple decision to extend
the amount of time allocated to meeting colleagues about a given issue. This allowed the
organisation to move from a superficial analysis of a concern to being able to “really thrash it out”
(former graduate diploma participant, Australia and New Zealand police service member). This
encouraged strategic thinking, and allowed time for greater participation in the conversation so that
different perspectives could be garnered. Ultimately this ensured that the outcomes from such
meetings were more considered, and were more likely to have the desired result. One participant
was able to attribute a belief that there had been a rise in business satisfaction levels for external
clients because s/he made a conscious effort to build “a climate and culture that allowed people to
have their say” (former graduate diploma participant, Australia and New Zealand police service
member).
Sustainability of impact As noted above, it is not possible to calculate the longevity or sustainability of the impact of the
AIPM graduate programs on behaviour over time, although interviews with former participants
provide compelling evidence that behavioural changes are still evident between 12-24 months
following their course. Moreover, there is also evidence for a compound effect, wherein an
individual can take their learning, apply it in the workplace, and learn further from that experience.
35
This could lead to exponential improvements in leadership over the longer term, as suggested in the
following quote:
Since the graduate diploma I have done a number of organisational strategies, including developing a strategy for the strategy department as well as a crime scene strategy, and IT related strategy and others. In undertaking these I used stuff from the graduate certificate as well as the graduate diploma. I used some of the direct tools, intertwined with using research skills and drawing on the AIPM library...There is no way that I would have been able to get these five strategies endorsed without doing the program. I wouldn’t have had the broad strategy development process down, and my approach would have been crude. In doing each strategy I have refined my processes and learnt my craft better. (Former graduate diploma participant, Australia and New Zealand police service member)
It is suggested that it takes at least ten years, or 10,000 hours of dedicated practice to become an
expert in a given field (Day et al., 2013), and as such there is logic that one’s leadership would get
better over time. Of course some participants may not be well supported on their return to their
workplace and may have difficulty implementing the personal and behavioural changes they would
like. Others may question whether leadership is really the right road for them and/or leave their
organisation altogether. And still others may experience the aforementioned training decay, and
revert to old habits over time. Interestingly, if one starts to slip back into one’s old ways, appropriate
jurisdictional accountability and workplace monitoring and support can help counter this, as
evidenced below:
I was pulled up by one boss who said that my colleagues had noticed a lot of good changes following the graduate diploma, and that they saw me starting to slip back into old habits, and that perhaps I should read my self-reflection paper again… On occasion I slip back into old habits, and when I do this I read my self- reflection paper and try and remember why I thought all of those things (listening etc) were so important.(Former graduate diploma participant, Australia and New Zealand police service member)
Nonetheless it remains difficult to estimate the longer term impact of the graduate programs, and
this is an area that requires further research.
STAKEHOLDER ASSESSMENT Interviews with 12 senior leaders in eight of the nine police jurisdictions identified that whilst the
AIPM’s graduate programs were generally well regarded, the programs were competing with other
leadership development opportunities, both internal and external. Some jurisdictions, for instance,
had adopted in house programs similar to the AIPM’s graduate programs, and/or had partnered with
local universities to create tertiary leadership study opportunities. Stakeholder interviews
36
highlighted that this meant that the added-value of the AIPM had to be clear to those who were
making budgetary decisions about staff development. A unique facet of the AIPM programs, noted
by stakeholders and ripe for further investigation, was networking. Participants and stakeholders
alike noted the value of the networking achieved through the multi-jurisdictional programs offered
by the AIPM. Police and emergency services operate in a multi-jurisdictional space, and getting
leaders from multiple agencies together to discuss leadership in the context of public safety has a
number of beneficial effects. These include expanding one’s professional network for dealing with
wicked problems, better understanding jurisdictional differences and how to work with one another,
and appreciating a greater diversity of ideas and perspectives. All of which was regarded as having
the potential to positively impact on multi-agency working in a operational sense, and make a
contribution to effective public safety.
Another reported benefit of the graduate programs at the AIPM was that it was considered a ‘safe
space’ insofar as it was jurisdictionally neutral and allowed participants the space to think about and
discuss issues that they might not be able to talk about within their jurisdictions. The AIPM graduate
programs gave them the “opportunity and freedom to…talk about their experiences” (police service
stakeholder), and the face-to-face components of the graduate program allowed them to do so in a
location that was “closed off from family and friends” (police service stakeholder) and the trials and
tribulations of everyday life.
The ‘Manly Experience’ is something else that you can’t quite put your finger on. By bringing together future and current leaders there is a greater capacity to share…The AIPM is a mentor to future leaders. They need to be the leading institution for leadership, and people feel that when they go there. (Police service stakeholder)
RETURN ON INVESTMENT Calculating an organisational return on investment (ROI) from leadership development is notoriously
difficult. It is very difficult to quantify, in a dollar figure, the impact that a change in leadership
behaviour has on an organisational bottom line. Particularly in policing and emergency services
where operational activity is not measured in dollar terms in the first place. Moreover, how does
one quantify the impact of having the strategic vision to predict the development of new crime types
and position the organisation appropriately? How does one link leadership development with a
reduction in unplanned leave whilst taking into consideration the multitude of intervening variables
that contribute to sickness? How does one assess the impact that a commitment to personal
integrity has in saving the organisation from embarrassing inquiries? Nonetheless, return on
37
investment is a seductive construct for those facing difficult budgetary decisions, and as such
scholars have dedicated time and effort to coming up with a workable formula. In basic terms all of
these formulas set out to subtract the costs of the program from the benefits. In leadership
development in particular, the main points of contention revolve around how one conceptualises
these two values, what data is relied on to reach these values, and what assumptions are contained
therein. Moreover, the scope of these measures is necessarily limited to evaluating interventions
within “a set of simplifying assumptions” (Fulmer and Ployhart, 2014: 163) and as such can only ever
provide a partial picture of the value of the program. Nonetheless, and exercising appropriate critical
caution, ROI formulas can offer a transparent and logical means to calculate a conservative estimate
of return (Phillips and Phillips, 2007).
Some ROI models are more detailed than others in attempting to quantify the impact of a leadership
development program. One highly structured model is the Brogden-Cronbach-Gleser Model. This
has been used widely in the leadership development literature (Cabrera and Raju, 2001; Macan,
Lemming and Foster, 2013). Recently Cascio and Boudreau (2011) used it in their financial
assessment of human resource initiatives, and Avolio et al. (2010) used it in their estimate of return
from leadership development. The model proposes the following formula:
𝑅𝑂𝐼 =𝑁𝑇𝑑𝑆𝐷𝑦 − 𝐶
𝐶
Where:
N = number of participants in the development intervention
T = expected duration of the change in leadership behaviours (converted to fraction of
years)
d = effect size of the intervention
SDy = 40% of the annual salary of a participant23
C = total cost of training the expected number of participants.
First, this model requires that the impact of the program be assessed. This can be done through
before and after measures of behaviour from a leadership development program to calculate the
program’s impact. This impact is mathematically transformed into an effect size (denoted by d). (For
this research this was calculated on page 26. Importantly in the context of this calculation, and
because d is effectively based on self-reported behaviour, we must be mindful of the possibility of
23 This is referred to as the “40% rule” (Cascio and Boudreau, 2008: 234) and is used as a conservative estimate of and individual's dollar value to the organisation in terms of performance.
38
inadvertent inflation of this figure - due to overly optimistic self-assessments - and the impact that
this might have on the ROI calculation (B. Avolio, personal communication, 28th February 2014). The
effect size is then multiplied by the number of participants on the program (N), the number of years
that the effect lasts for (T), and by 40% of the average wage of a participant on the program (SDy)24.
It is this latter figure – the 40% figure – that allows the impact of the training to be turned into a
dollar impact figure. Central to this calculation is the assumption that 40% of salary is a conservative
estimate of an individual’s dollar value to the organisation in terms of performance (Avolio et al.,
2010; Cascio and Boudreau, 2011; Schmidt and Hunter, 1983). Thus by multiplying this figure by the
size of the effect that the leadership development program has (i.e. the effect size d), the number of
people we want to calculate the ROI for (i.e. N), and the length of time the effect lasts for (i.e. T) we
are able to estimate the net gain to the organisation25.
This first part of the equation represents, then, the program’s benefits. The costs of the program are
subtracted from this figure to calculate net gain, and this is divided by the costs once again to
provide the ROI figure. Using the findings from this research we can use this formula to calculate an
estimate of the return on investment for one participant on a graduate program at the AIPM.
Therefore:
N = 1 (i.e. we want to calculate the ROI for one participant)
T = 1 (i.e. we know that there is a leadership training effect that lasts the length of the
program, about a year, as evidenced by the pre-post questionnaire data)
d = 0.54 (the effect size as calculated on page 26)
SDy = $54,250 (which is the 40% of the average salary of course participants – inspectors and
superintendents - at the AIPM)26
C = $11,117 (which is the average cost to police jurisdictions per person attending a
graduate program at the AIPM)27
24 Whilst there is ongoing debate about the best way to calculate SDy, from supervisor estimates of job-worth, to more concrete measures of employee productivity (Greer and Cascio, 1987; Mathieu and Tannenbaum, 1989; Barrick, Day and Lord, 1991; Becker and Huselid, 1992; Cascio and Boudreau, 2011) for the purposes of this evaluation and the public safety context, using the 40% rule - with appropriate caution - is sufficient (Avolio et al., 2010; B. Avolio, personal communication, 28th February 2014). 25 In organisations with a commodity to sell, this gain can be regarded as an indication of the net increase in profits following a training intervention. In public safety organisations it is perhaps best regarded as the net value to the organisation from an increase in (leadership) performance. 26 This figure was calculated by taking the average inspector and superintendent salaries for one police force (New South Wales Police) for 2013 (Police Association New South Wales, n.d.), and applying this number respectively to the inspectors and superintendents in our pre-post sample. These salaries were averaged out across the pre-post sample to calculate a mean salary for this group, and this figure is 40% of that mean.
39
Importantly this cost (C) figure does not take into consideration the indirect costs of a participant’s
attendance at the AIPM, including their travel to and from the AIPM for the face-to-face
components, the cost of backfilling a position whilst a participant is at the AIPM, and the additional
costs incurred by the family of a participant during their absence, such as for child care. Nor does
this figure accurately reflect the true cost to the AIPM in running these programs, which is
considerably higher when staff time is factored in. It is nonetheless a useful, if crude, rule of thumb
to assist in these calculations.
𝑅𝑂𝐼 =1 𝑥 1 𝑥 0.54 𝑥 $54250.40 − $11,117
$11,117= 1.64
The calculation above means that if the positive effect of attending the AIPM graduate programs last
for one year (i.e. the approximate duration of the pre-post evaluation time frame) the return on
investment to a jurisdiction is 1:1.64, or 164%. To put a dollar figure on this, for every $11,117
invested by a jurisdiction, they receive $18,231 in organisational benefit as a result.
Importantly the data collected from former course participants (page 32) suggests that the impact of
the AIPM graduate programs last longer than a year. And as such this ROI calculation becomes
particularly interesting if we hypothesise that the changes identified will be with the participant for
the rest of their career. Of course it is important to note that typically ROI calculations are limited to
the time period for which concrete data has been collected (Avolio, personal communication, 28th
February 2014), but in the absence of such longitudinal data here, we nonetheless believe it is
instructive to demonstrate the potential ROI possible from involvement in the graduate programs
over the longer term. This suggestion would need to be corroborated by longer term data collection
and analysis, of course, although it is an interesting place to start.
Arguably police and emergency service workers, particularly those in senior roles, will choose to stay
with their organisation until retirement – typically at 35 years of service. By subtracting the number
of years an officer has already served in their organisation when they attend the AIPM from this
27 This figure is relevant only for Australian and New Zealand police. It has been calculated by summing the annual police jurisdictional contributions to the AIPM, with the (discounted) tuition fees for this group, and dividing this number by the minimum participant commitment identified by the jurisdictions for 2014. One figure is presented for both the graduate certificate and the graduate diploma, although in reality the graduate diploma costs more (because it involves eight rather than four subjects). This has been done for parsimony and the resulting calculation can therefore be regarded as a guide only. Non-police and international organisations are catered for differently in the AIPM pricing model.
40
expected career span of 35 years, one finds that the officers in the pre-post data set had an average
of 13 career years remaining post-course. Thus we change N from 1 to 13 in the ROI calculation:
𝑅𝑂𝐼 =1 𝑥 13 𝑥 0.54 𝑥 $54250.40 − $11,117
$11,117= 33.26
The return on investment if behavioural changes are maintained over 13 years is 1:33.26, or 3326%.
The dollar figure for this level of organisational return is $369,751.40, per person, for an upfront
investment of $11,11728.
Of course, and noted repeatedly, the actual duration of the training effect is unknown, and may lie
anywhere between one and 13 years29, or it may last 13 years but diminish in magnitude over time,
or it may even increase over time as suggested in the quote on page 35 noting a compound effect.
Comprehensive longitudinal research - using the same pre-post measures - is needed to assess this.
Nonetheless the return on investment calculations provided here, outline the potential scale of
return from leadership development. And in actual fact there is a strong case to be made that these
estimates are on the conservative side. For instance:
• These calculations are based on self-report, and as such all the usual caveats with self-report
- including the veracity of such behavioural claims - must apply (as noted previously).
However, analysis of the small number of matched manager assessments suggested that in
some domains program participants may actually under-estimated performance. Thus whilst
an effect size of 0.54 has been used in the calculations based on self-report, the true effect
size may be higher. (It may also, of course, be lower as a function of over-optimistic self-
assessments and social desirability effects as noted on page 22).
• These calculations do not take into consideration the dollar value of significant
organisational ‘wins’ emanating from improved leadership (as opposed to the value of a
generalised improvement in leadership) which ultimately reduce the demand for certain
services and a reallocation of resources to other pressing matters. For example, successful
and significant criminal investigations, successful and evidence-based policy decisions,
28 When these figures are scaled up to account for police jurisdictional investment as a whole, for every $1,200,645 that the nine police jurisdictions of Australia and New Zealand collectively invest in the AIPM for graduate programs per annum, they received an organisational return of $39,933,452.70 over 13 years. 29 The ROI for the halfway point between these two extremes (i.e. an effect duration of 6.5 years) is 16.13 (1613%), with a dollar return to the organisation of $179, 317.21 for an initial $11,117 investment.
41
development of effective crime solving partnerships, all of which have a positive dollar
implication for an organisation over the medium to long term.
• These calculations start and end with the individual who has attended the training program,
and do not consider the trickle-down benefits of better leadership. The qualitative data
collected provides evidence for former participants better dealing with, communicating
with, and developing their staff. These staff could in turn – one could argue – increase in
terms of productivity, with a commensurate positive dollar implication for organisations
Moreover these staff may go on to better treat those who report to them, or may better
engage with the public, all with flow on organisational benefits not captured in this
calculation (see Roberts and Herrington, 2013, for a review of the theoretical base and
literature on which this argument rests).
• These calculations do not take into consideration the possibility of a compound effect.
Perhaps the graduate programs are the starting point for participants, and perhaps, as
evidenced through at least one former participant, this grounding allows one to capitalise on
the learning and commit to ongoing growth, with exponential gains for the organisation in
terms of a dollar value. The AIPM learning philosophy (AIPM, 2013) adopts a 70:20:10
approach to leadership development, wherein only 10% of leadership is hypothesised to be
developed in the classroom, 20% through mentoring, and 70% through on the job
experience. A central aim of the 10% delivered at the AIPM is to prepared individuals to be
better prepared to benefit from the 20% and 70% that they undertake back in the
workplace. Whilst the qualitative data suggests that this has an important effect, it has not
been taking into consideration in this calculation.
Therefore, there is a compelling argument that even if some participants slipped back into old
habits, or left the organisation, or did not gain as much from their studies, the calculations above
may as likely be an underestimate as an overestimate the true value of the ROI that organisations
receive following participants’ engagement in the AIPM’s graduate programs.
42
DISCUSSION AND CONCLUSIONS This research demonstrates that when ‘impact’ is conceptualised along the lines of the Leadership
Capability Framework there were statistically significant improvement in participants’ self-reported
behaviour by the end of their studies30. Moreover, the magnitude of this improvement was large,
and as such police organisations can expect a return on their investment of somewhere between
164% (if the effect holds for one year) and 3326% (if the effect holds throughout the participant’s
remaining career). Of course these calculations may be an under-estimate of the true value of these
programs to organisations, as there are numerous flow-on, trickle-down and compound
improvements that are not effectively captured here. Thus there is space for further research on
police leadership development, and a need to develop more comprehensive measures of leadership
impact. The following sections outline the limitations of this research, its implications, and highlight
areas ripe for further research.
LIMITATIONS As with every piece of research, this project has its limitations, and many of these were set out at
the start of this document. First, the above findings are based on a self-selecting sample of course
participants. Whilst this sample appears to be broadly reflective of the whole population (i.e. all
participants on AIPM programs) it is not possible to discount the possibility that there is something
peculiar about this group that chose to complete questionnaires. They may be intrinsically more
conscientious, for example, or have had a better experience on the program and be more inclined to
‘give something back’ and take part in this research. Attrition rates for managers are of concern too,
which is why these manager findings, whilst positive, should be treated with appropriate caution.
Another clear limitation, well documented throughout this report, is that this research tells us only
about the quantitative impact of the program after approximately one year, (notwithstanding the
longer term qualitative data from interviews with former students). In order to glean the kind of
longitudinal data required to better calculate return on investment, regular and repeated
assessments of behaviour should be made. Ideally this pre-post cohort could be followed up in two,
five, 10, and 13 years’ time to assess whether the impact of their studies increases, decreases or
stays the same.
30 Where participants completed their studies, they were significantly better off in terms of self-reported behaviour in the setting of strategic direction domain than those who did not finish their studies.
43
A final limitation to note is that ‘impact’ has been operationalised here in terms of the Leadership
Capability Framework. There are good reasons for doing this, particularly given the clear signal from
the Australian and New Zealand commissioners of police, the Public Service Commission, and
ANZPAA that these domains and associated behaviours are what one is looking for in effective senior
police and emergency service personnel and as such what leadership development should seek to
instil. However, do these domains reflect the reality of senior executive roles in these organisations?
And are they what the profession truly needs if it is to face the leadership challenges head?
Moreover if these behaviours are the hallmark of effective leadership, then how well do our
organisations recognise and reward them through the performance appraisal process? These
questions are being addressed by other research being undertaken at the AIPM, due for completion
in 201431.
IMPLICATIONS OF THESE FINDINGS There are several implications from these findings for the AIPM and police and emergency services
more broadly. The first relates to completion rates. There is evidence that completing the graduate
program leads to higher gains in self-reported behaviour in the setting strategic direction domain. If
this is a desirable outcome for sponsoring organisations then there is benefit in these organisations
better encouraging participants to complete. It is of course up to each organisation as to how they
might do this, but accountability mechanisms, tying completion to workplace progression, or
providing time for participants to complete the distance learning study during work hours, may be of
benefit.
This point links to the second implication: for the professionalization agenda. Part of the thinking
around professionalization must involve the relative merit of tertiary qualifications. Does
professionalization necessitate academic qualification? There are significant additional learning
benefits to be had from completing formally recognised tertiary level studies (as set out in page 27).
If this learning is considered an important part of the professionalization plan, then this underscores
the import of sponsoring organisations ensuring completion rates.
FUTURE RESEARCH Research is an iterative process and the answers to one set of questions inevitably give rise to new
ones. This research project is no different and findings suggest several areas of further research.
First, and documented throughout this report, is the need for a longitudinal follow up of participants
to assess, quantitatively, the longevity of the impact of the graduate programs. Does this diminish
31 See http://www.aipm.gov.au/research/research-projects/ for further details.
44
after a year or two? Does it increase? Longitudinal research is notoriously difficult, but in a
leadership context it is an important piece of the puzzle, particularly if it takes at least 10 years of
practice for leaders to consolidate their learning and master their craft (Day et al., 2013). This would
be a significant piece of research, and would necessitate the continued tracking and matching of
participants over the long term. It would require a long term commitment from all of those involved.
A second area of research that presents itself is the comparative value of the AIPM’s graduate
programs to similar leadership development opportunities available. Within the AIPM this might
involve a comparison between the academic-style graduate programs and non-academic, shorter,
development opportunities. External to the AIPM this might involve a comparison between AIPM
graduate programs and similar programs conducted in house, or in partnership with local
universities, by each of the jurisdictions. Whilst comparative effect sizes could be translated into
comparative return on investment calculations, allowing jurisdictions to decide where they achieve
best bang for their buck, the hidden value of programs must also be factored in, including their
contribution to the professionalization agenda, their multi-agency format, their jurisdictional
neutrality, and the relative costs-benefits of being away from work, family, and friends.
Further research might like to consider - qualitatively and quantitatively - the relative benefits of the
graduate certificate and the graduate diploma. These two distinct, but nested, courses have been
analysed in this report in aggregate for two reasons: the comparative paucity of data collected from
graduate diploma participants, and the fact that if ‘impact’ is operationalised in terms of the
Leadership Capability Framework self-reported behaviours along a five-point scale might not
accurately reflect the relative depth of learning gleaned from each program of study. To assess the
relative difference between the graduate certificate and the graduate diploma ‘impact’ would need
to be measured in terms of depth of understanding about key concepts.
The final area of research to identify here has already been noted (on page 43), being the need to
critically reflect on whether the Leadership Capability Framework is the most appropriate means to
identify the components of effective leadership. Does this accurately reflect what organisations need
from their senior leaders? Or is it what they want? Attempts to address this question are ongoing at
the AIPM.
45
CONCLUSION In conclusion, this research represents a methodologically robust evaluation of the AIPM’s graduate
programs, drawing on multiple sources of data. Findings indicate that these programs have a
significant impact on leadership behaviour in line with the expectations of sponsoring organisations.
Whether this impact is sustained over the long term is not yet known, although there is some
evidence that it is, and may even develop further. The financial implication for organisations
considering whether to enrol their leaders in one of these graduate programs is dwarfed by the
return on this investment over the long term. This coupled with the ongoing debate about the
professionalization of the police and emergency services, should be a compelling argument in favour
of these programs for those tasked with ensuring the leadership development of employees.
46
REFERENCES Australian Government (2007) National Statement on Ethical Conduct in Human Research. Canberra:
Australian Government.
Australian Institute of Police Management (2013) Learning, Teaching and Assessment Strategy 2013-
2016. Sydney: AIPM.
Australian Public Service Commission (n.d.) Senior Executive Leadership Capability Framework.
Available at http://www.apsc.gov.au/__data/assets/pdf_file/0006/4020/framework.pdf. Accessed
18th December 2013.
Avolio, B., Avery, J. and Quisenberry, D. (2010) Estimating return on leadership development
investment. The Leadership Quarterly, 21, 633-644.
Barrick, M., Day, D. and Lord, R. (1991) Assessing the utility of executive leadership. Leadership
Quarterly, 2(1), pp 9-22.
Becker, B. and Husel;id, M. (1992) Direct estimates of SDy and the implications for utility analysis.
Journal of Applied Psychology, vol 77 (3), pp 227-233.
Cabrera, E. and Raju, N. (2001) Utility Analysis: Current trends and Future Directions. International
Journal of Selection and Assessment, 9 (12), pp 92-102.
Cascio, W. and Broudreau, J. (2011) Investing in people: financial impact of human resources
initiatives. New Jersey: Pearson Education Inc.
Day, D., Fleenor, J., Atwater, L., Sturm, R. and McKee, R. (2013) Advances in leader and leadership
development: A review of 25 years of research and theory. The Leadership Quarterly,
http://dx.doi.org/10.1016/j.leaqua.2013.11.004.
Field, A. (2005) Discovering statistics using SPSS. London: Sage.
47
Fulmer, I. and Ployhart, R. (2014) “Our most important asset”: A Multidisciplinary/Multilevel Review
of Human Capital valuation for Research and Practice. Journal of Management, vol 40 (1), pp161-
193.
Greer, O. and Cascio, W. (1987) Is cost accounting the answer? Comparison of two behaviourally
based methods for estimating the standard deviation of job performance in dollars with a cost
accounting based approach. Journal of Applied Psychology, vol 72 (4), pp 588-595.
Heifetz, R., Grashow, A. and Linsky, M. (2009) The practice of adaptive leadership: Tools and tactics
for changing your organisation and the world. Cambridge, MA: Harvard Business Press.
Herrington, V. (2011) Police as critical consumers of research: Informing practice through research.
In P Birch and V Herrington (Eds) Policing in Practice. South Yarra: Palgrave MacMillan.
Herrington, V. (2012) Inter-agency cooperation and joined-up working in police responses to persons
with a mental illness: Lessons from New South Wales. Policing: A Journal of Policy and Practice. 6 (4),
pp 388-397.
Herrington, V. and Pope, R. (2013) The impact of police training in mental health: an example from
Australia. Policing and Society,
http://www.tandfonline.com/doi/abs/10.1080/10439463.2013.784287.
Herrington, V. and Roberts, K. (2013) Organisational Justice: Implications for police and emergency
service leaders. AIPM: Sydney. Available at http://www.aipm.gov.au/wp-
content/uploads/2013/08/Org-Justice-Final.pdf. Accessed 20th December 2013.
Kirkpatrick, D. (1998) Evaluating training programs: The four levels. Second edition. San Francisco:
Berrett-Koehler Publishers Inc.
Kodz, J., and Campbell, I. (2010) What works in leadership development? A rapid evidence review.
London: National Policing Improvement Agency.
Maca, T., Lemming, M. and Foster, J. (2013) Utility analysis: Do estimates and format matter?
Personnel Review, vol 42 (1), pp 105-126.
48
Mathieu, J. and Tennenbaum, S. (1989) A process-tracing approach towards understanding
supervisors’ SDy estimates: Results from five job classes. Journal of Occupational Psychology, 62, pp
249-256.
Neyroud, P. (2010) Review of Police Leadership and Training. London: Home Office.
Pearson-Goff, M. and Herrington, V. (2013) Police Leaders and Leadership Development: A
Systematic Literature Review. Sydney: AIPM. http://www.aipm.gov.au/wp-
content/uploads/2012/09/Police-leaders-and-leadership-development-A-systematic-review.pdf
Phillips, J. and Phillips, P. (2007) Measuring return on investment in leadership development. In K.
Hannum, J. Martineau, and C. Reinelt. The handbook of leadership development evaluation. San
Francisco: Wiley and sons.
Police Association New South Wales (n.d.) General Salary Increases. Available at
http://www.pansw.org.au/sites/default/files/public/Circular%2050%20Salary%20Rates%20Attachm
ent.pdf Accessed 1st March 2014.
Roberts, K. and Herrington, V. (2013) Organisational and procedural justice: A review of the
literature and its implications for policing. Journal of Policing, Intelligence and Counter Terrorism,
8(2), pp 115-130.
Schmidt, F. and Hunter, J. (1983) Individual differences in productivity: An empirical test of estimates
derived from studies of selection procedure utility. Journal of Applied Psychology, vol 68 (3), pp 407-
414.
Victoria Police (2010) Australian and New Zealand Police Leadership Strategy. Project Document.
Melbourne: Victoria Police.
49
APPENDIX 1 - STUDENT SATISFACTION
By Mitch Pearson-Goff
Data for this analysis were matched with the cohorts who took part in the pre-post course survey for this research project. In total 252 students (165 Grad Cert; 87 Grad Dip) from the nine available cohorts32 completed an overall course evaluation. At the graduate certificate level a total of 140 participants completed the overall evaluation (85% of the 165 students enrolled). At the graduate diploma level a total of 56 participants completed the overall evaluation (64% of the 87 students enrolled).
Was the information you were provided with sufficient?
A total of 114 participants (95% of 120 responses) from the graduate certificate and 47 participants (92% pf the 51 responses) from the graduate diploma agreed that information provided was sufficient to allow them to know what to expect from the programs.
Figure 1 & 2: The information provided to me was sufficient for me to know what to expect
*(Neither agree nor disagree scores excluded)33
Was the program handbook provided to you useful?
In terms of providing students with an understanding of the graduate certificate and graduate diploma programs 132 participants (100% of 132 responses) from the graduate certificate and 48 graduate diploma students (96% of 50 responses) agreed that the program handbook provided sufficient understanding.
32 One graduate certificate cohort did not complete an overall evaluation n=24, the remaining cohorts matched those included in the research project. 33 Grade cert: 17 responses excluded. Grad dip: 4 responses.
Agree 95%
Disagree 5%
Graduate Certificate
Agree 92%
Disagree 8%
Graduate Diploma
50
Figure 3 & 4: The program handbook provided me with a sufficient understanding of the program
*(Neither agree nor disagree scores excluded)34
Was the online format for the distance learning element user friendly?
Distance education is a significant component of both the graduate certificate and graduate diploma. In total 113 graduate certificate participants (98% of 115 responses received) agreed that the current online format is clear and user friendly. As for the graduate diploma there was no comparable question that assessed the clarity of the distance education format.
Figure 5: The online format for the distance education was clear and user friendly
*(Neither agree nor disagree and NA scores were excluded)35
34 6 responses excluded (neither agree nor disagree) 35 19 responses exclude (13 NA scores; 6 neither agree nor disagree scores)
Agree 100%
Disagree 0%
Graduate Certificate
Agree 98%
Disagree 2%
Graduate Certificate
Agree 96%
Disagree 4%
Graduate Diploma
51
Did the distance education component prepare you for the face-to-face element of your course?
In total 90 graduate certificate participants (96% of 94 responses) and 33 graduate diploma students (97% of 34 responses) agreed that the completion of distance education subjects assisted them in preparing for the residential component of their respective programs.
Figure 6 & 7: Completion of the distance education assisted in my preparation for the residential component of the program
*(Neither agree nor disagree and NA scores were excluded)36
Did you receive feedback on your assignments in a timely manner?
The facilitation of the graduate certificate and graduate diploma is comprised of many factors including the feedback provided for assessment items. A total of 127 graduate certificate participants (96% of 132 responses) and 44 graduate diploma students (90% of 49 responses) agreed that this feedback was received in a timely manner.
36 Grad cert: 42 responses excluded (28 NA scores; 14 neither agree nor disagree scores). Grad dip: 16 responses excluded (13 NA scores; 3 neither agree nor disagree scores).
Agree 96%
Disagree 4%
Graduate Certificate
Agree 97%
Disagree 3%
Graduate Diploma
52
Figure 8 & 9: The feedback from assessments was timely
*(Neither agree nor disagree scores were excluded)37
Was the quality of the teaching good?
The quality of the teaching or delivery is a valuable area to measure when evaluating our programs, overwhelming, 133 participants from the graduate certificate (100% of 133 responses) and 52 participants (100% of 52 responses) from the graduate diploma agreed that the quality of the teaching was good.
Figure 10 & 11: The overall quality of teaching was good
*(Neither agree nor disagree and scores were excluded)38
37 Grad cert: 3 responses excluded. Grad dip: 3 responses excluded. 38 Grad cert: 2 responses excluded.
Agree 96%
Disagree 4%
Graduate Certificate
Agree 100%
Disagree 0%
Graduate Certificate
Agree 90%
Disagree 10%
Graduate Diploma
Agree 100%
Disagree 0%
Graduate Diploma
53
APPENDIX 2 – GRADE DISTRIBUTION
Grade data for all cohorts between February 2011 and July 2013 were analysed at the subject level.
This equated to data from 423 graduate program participants.
Graduate Certificate in Applied Management
601S1 – Foundations of Strategic Management
601S2 – Strategic Analysis and Planning
Pass 26%
Credit 42%
Distinction 31%
High Distinction 1%
Pass 31%
Credit 44%
Distinction 23%
High Distinction 2%
54
601 S3 – Leadership and organisational change
601 S4 – Contemporary issues in public safety and emergency services
Pass 9%
Credit 52%
Distinction 36%
High Distinction 3%
Pass 6%
Credit 57%
Distinction 37%
55
Graduate Diploma in Executive leadership
701 S1 – Contemporary Leadership
701 S2 – Power, politics and policy development
Pass 16%
Credit 44%
Distinction 40%
Pass 30%
Credit 51%
Distinction 19%
56
701 S3 – Executive Leadership Development
701 S4 – Strategic Policy Review
Credit 55%
Distinction 40%
High Distinction 5%
Credit 22%
Distinction 45%
High Distinction 33%