how evaluation made a difference to planning and decision making in a small community australasian...
TRANSCRIPT
How evaluation made a difference to planning and decision making in a small community
Australasian Evaluation Society Queensland Seminar 11 July 2006
Ellen VasiliauskasDirector d-sipher pty ltd
Evaluation Development Award Winner 2005
© d-sipher pty ltd
Ph: 07 5471 1330
• AES Award Criteria
• The Noosa Community Governance & Planning Project case study:– Use of evaluation & evidence based methods– What outcomes were achieved as a result?– Who has benefited and who has not?
Overview
Quality of the methodology that could relate to more general initiatives to promote evaluation
AES Evaluation Award Criteria
Usefulness and impact of the contribution
Cost-effectiveness and feasibility of the contribution
Consideration of ethics and social justice issues
Overseas trends
The key elements of shifts in rural policy in OECD countries overseas are around:
“ - decentralisation of policy administration and, within limits, policy design to those levels;
- increased use of partnerships between public, private and voluntary sectors in the development and implementation of local and regional policies.”
The Future of Rural Policy Conference in Siena, Italy July 2002
From sectoral to place-based policies in rural areasOECD 25-06-2003
Local context - Community participation placed on government reform agenda
• Changing Australian policy with framework of federal, state & local govt. microeconomic reform.
1990’s:
• Local government IPA specifies areas of community consultation.
2001:
2002: • Who will take care of the planning at the local level across all the other sectors? – Social, Environmental and Economic.
• Government recognised it could not solve the increasingly complex social, economic, environmental & attitudinal factors when planning for a sustainable future.
1980’s:
Changing role of Local government
• Closer to communities than other levels of government
• An emphasis on local implementation of federal & state policies
• Changing role from roads, rates & rubbish to facilitators (1980’s).
Local government:
Private sector businessIndustry groupsEducation
Private sector businessIndustry groupsEducation
The Private SectorThe Private Sector
FederalStateRegionalLocal
FederalStateRegionalLocal
The Public Sector
The Public Sector
All members ofthe community
All members ofthe community
The CommunityThe Community
Third sector (not for profit)Voluntary organisationsLocal Services & Clubs
Third sector (not for profit)Voluntary organisationsLocal Services & Clubs
The Community SectorThe Community Sector
Localgovernment as
strategic facilitators of place based
initiatives
Localgovernment as
strategic facilitators of place based
initiatives
What is community governance?
‘Governance has to do with institutions, processes & traditions for dealing with issues of public interest.’
Understanding Community Governance1999 Local Government NZ Conference
Use of evaluation & community governance –
The Noosa case study
Some information about Noosa
• Small regional coastal location on Sunshine Coast in Queensland – 1.5hrs North Brisbane
• Population of 44,000 (2003)
• Main industries: tourism, construction, property & agriculture
• Balance of built & natural environment –high % of national park
• Village atmosphere
Council recognised: -
‘…the quality of life and wellbeing of the local Shire community relies on many sectoral interests, which are outside of Council and often driven by other competing priorities.’
Why did Noosa Council undertake community governance?
Develop plans for the Noosa community to achieve sustainability & acceptable quality of life in:
– Social– Arts & Heritage– Environment &– Economic sectors.
A vision to the year 2015 but with recommendations on actions, responsibility for action and funding for the first 2-4 year increment.
Brief from Noosa Council for community governance project
Before the Community Governance Project
• Noosa Council has a history of community consultation - 20 yrs ago Council meetings opened to the community
• Existing consultative methods:– Limited to conventional methods – ‘eg community based
workshop’– Consultation processes for development of Noosa Plan
under Integrated Planning Act relied on community meetings, which had low attendances
– Multiple committees on topic based issues
• Main responsibility for decision-making, prioritising and action rested with Council
Involvement & influence of a few – the squeaky wheels, the power brokers
Involvement & influence of a few – the squeaky wheels, the power brokers
Conflict v consultation
Sectoral interest groups competing for resources, and priorities
Deals done and trade-offs made
Control by misinformation/selective information
Conflict v consultation
Sectoral interest groups competing for resources, and priorities
Deals done and trade-offs made
Control by misinformation/selective information
What happened as a result of conventional consultative methods?
Little understanding or overview of local issues – information was fragmented
Little understanding or overview of local issues – information was fragmented
‘Everyone is well-off in
Noosa’
‘Everyone is well-off in
Noosa’
‘There are no
housing/social issues’
‘There are no
housing/social issues’
‘There’s more need elsewhere’
In government & outside of Noosa
In government & outside of Noosa
Few facts, lots of commonly held myths
Few facts, lots of commonly held myths
‘Rents here are high’
‘Rents here are high’
‘It’s the commercial fishermen that are doing
the harm’
In the CommunityIn the Community
‘The cost of living here is
high’
‘The cost of living here is
high’
‘We need one of those, some of this, & more of that’
‘We need one of those, some of this, & more of that’
We need a University
Council should build a sustainable house
We need a 500 seat auditorium
Focus was on solutions rather than issues, resulting in ad hoc & often costly approaches
Focus was on solutions rather than issues, resulting in ad hoc & often costly approaches
Limited consideration of regional or State level issues, directions and
planning
Limited consideration of regional or State level issues, directions and
planning
Limited local level data suitable for broad planning purposes
• Data collection:– lack of quality data– lack of timely data (1996 ABS Census)
– is a significant task to collect
Many datasets, reports & information held by Council & State government, &
non-government sector
BUT these are generally poorly used, coordinated, or applied at the local level
Many datasets, reports & information held by Council & State government, &
non-government sector
BUT these are generally poorly used, coordinated, or applied at the local level
Limited resources: financially & timeLimited resources: financially & time
Relationship between Community Sector Boards, Council & its committees
governance
governmentSector Boards identified
initial issues, focused the discussion,&
provided community led direction
EconomicBoard
Environment Board
SocialBoard
Arts & Heritage Board
Collaborative TourismBoard
Strategic Reference Group
ROLE: Develop co-operative & innovative arrangements within
the Community Governance Model
SRG membership: Council committee reps, staff & Board
reps, Project Manager& CEO
CouncilCommittee
Council
Nine Community Board Members plus one elected
Councillor & Senior Manager
ROLE: “Develop plans to guide and direct the development of
Noosa and its community to 2015”
Evaluation methods – The Noosa case study
• Use of multiple convergent evaluation methods.
• The methodology and framework developed as a result of the project provide many useful lessons with generic application to planning and community engagement in local communities and for local government.
AES Award Criteria: Quality of the methodology that could relate to more general initiatives to promote evaluation
1. A framework and methodology underpinned by a values-led planning process:• This provided the ethical basis for Sector Boards, Council staff, Councillors, the local Shire
community, as well as state government stakeholders to engage with each other.
2. For the planning element of the project:• Local level data reanalysis and information summaries were complemented and strengthened
with deliberative methods such as Board meetings and open community workshops.
3. For the consultative component:• Issues papers developed by Sector Boards based on evidence from data• Community comment on papers & content through workshops, open invitation for comment and
a quantitative based survey• Refinement of Sector papers & development of draft plans based on community feedback. Use
of evidence based methods to inform discussion and decision making.
4. Overall evaluation of methods used:• Community evaluation of the consultation methods included in survey.• Overall evaluation of governance framework and outcomes by an independent consultant.
AES Award Criteria: Quality of the methodology that could relate to more general initiatives to promote evaluation
Use values led processesUse values led processes
Honesty, openness, equity & procedural justiceHonesty, openness, equity & procedural justice
Values defined & agreed up frontValues defined & agreed up front
Adherence to ethical practicesAdherence to ethical practices
Uphold procedural justice to ensure other voices are heard
Uphold procedural justice to ensure other voices are heard
Use processes that reflect agreed community valuesUse processes that reflect agreed community values
Sectoral & factional interests increasingly disenfranchised
Sectoral & factional interests increasingly disenfranchised
Allows representation of broader community opinion & balanced discussion
Allows representation of broader community opinion & balanced discussion
OUTCOMESOUTCOMES
Establish decision making processesEstablish decision making processes
Will it be by consensus or voting?Will it be by consensus or voting?
Provides protocols & guide for all stakeholders
Provides protocols & guide for all stakeholders
Identify & agree on principles up front – but remain flexible
Identify & agree on principles up front – but remain flexible
Set the climate, build group identitySet the climate, build group identity
ETHICS – part of the
quadruple bottom line
ETHICS – part of the
quadruple bottom line
Limits effectiveness of power brokersLimits effectiveness of power brokers
AES Award Criteria - Consideration of ethics and social justice issues
Analysis of Australian Bureau ofStatistics data
Analysis of Australian Bureau ofStatistics data
Focus on the issue and not the mythFocus on the issue and not the myth
Raised level of community debate & discussion of issues
Raised level of community debate & discussion of issues
OUTCOMESOUTCOMES
Analysis of existing government, community sector & Council dataAnalysis of existing government, community sector & Council data
Broadens discussion across issuesBroadens discussion across issues
Provides supporting evidence to government agencies for local needs
Provides supporting evidence to government agencies for local needs
Identifies where data is incompleteIdentifies where data is incomplete
Use of evidence based methods
Review & analysis of existing data & literature
Use of evidence based methods
Review & analysis of existing data & literature
Literature review to scope sectors & align with current theory
Literature review to scope sectors & align with current theory
Councillors commented that they had never seen anything like this before about Noosa. It also helped them to understand what was going on in their area.
‘It took issues out of the political arena and allowed them to be examined in a more open
atmosphere and environment, which is conducive to more creative thinking.’
- Councillor
Data & reference sources1. Population & Australian Bureau of Statistics data (1996 & 2001);
2. Market Facts survey of Noosa Shire residents (2000);
3. Health indicators of SE Qld (2001);
4. Housing data (rental & bonds 2002);
5. Noosa Council Reports- Demographic Report 2002, Choosing
Futures Report 2002;
6. Consultation with Council staff;
7. “A guideline for integrating community wellbeing in planning”
(LGAQ, Dec 2001);
8. “Just, vibrant & sustainable communities”
(A framework for progressing & measuring community wellbeing)
LGCSAA Townsville 2001;
9. Anecdotal information and feedback from community service
providers .
Use of modelling Use of modelling
Makes the discussion manageableMakes the discussion manageable
OUTCOMESOUTCOMES
Makes communication easier & builds shared understanding
Makes communication easier & builds shared understanding
Focuses, defines parameters of discussionFocuses, defines parameters of discussion
‘Model for progressing social cohesion & community wellbeing
in Noosa Shire’
Social cohesion & community
wellbeing
Wel
lnes
s
Learnin
g
Comm
unity
Governance
Social Capital
Provides structure & can elevate the discussion to the strategic level
Provides structure & can elevate the discussion to the strategic level
INDICATORSStrategies
Actions & projects
Objectives
Sector Values & Principles
How will we get there?
Sector model
Where are we now?
Background and discussion in each Sector Plan
Shire VisionWhere do we want to be in 2015?
Sector VisionSector VisionSector VisionSector Vision
‘We’re not locked into Council silos and operational plans, we can think more
divergently across the whole of the issue.’- Councillor
BenchmarkingBenchmarking
OUTCOMESOUTCOMES Provides a relative & ‘realistic’ picture of situation & performance
Provides a relative & ‘realistic’ picture of situation & performance
Benchmark local area to broaden understanding & provide national & global
context
Benchmark local area to broaden understanding & provide national & global
context
Acknowledgement of regional, national & global influences
Acknowledgement of regional, national & global influences
Introduced to local government community
planning the concept of ‘local area benchmarking’
Broadening participants’ perspective and understanding generally
Broadening participants’ perspective and understanding generally
NoosaOverall
NoosaHinterland
NoosaCoastal
Noosa- Noosaville
Tewantin
Sunshine-Peregian
Cooroy
Cooran
Pomona/ Boreen Pt.
Kin Kin
Compared with national, regional & other similar
local government areas
Internally benchmarked & compared with national & regional profiles
How Noosa’s indicators were benchmarked
Growth rates - Benchmarked
Whitsunday, Airlie Beach
Byron Bay
Coffs Harbour
Douglas, Port Dougals
Gold Coast
Kempsey, Sth West Rocks
Maroochy
Richmond, Evans Head
Caloundra
Shoalhaven, Ulladulla
Noosa
Tweed, Murwillumbah
Eurobodalla, Batemans Bay
Hastings, Pt Macquarie
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
% change in population 98-99
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
% c
han
ge
in p
op
ula
tio
n 9
7-98
Localities selected on basis of similarities in terms of lifestyle, population, & physical location
Localities selected on basis of similarities in terms of lifestyle, population, & physical location
Average annual income - 1999
av ge personal income - 1999Buderim 30466Yandina 30234Noosa Hds 28804Alex Hds 28050Kaw ana 27968Mooloolaba 27310Nambour 27252Noosav ille 26991Caloundra 26805Cooroy 26788Woomby e 26622Glass House Mtns 26368Mooloolah 26164Coolum Beach 26084Maleny 26059Eumundi 25883Landsborough 25750Palmw oods 25676Maroochy dore 25629Gy mpie 25595Beerburrum 25557Eudlo 25268Beerw ah 25124Tew antin 24960Kenilw orth 24945Pomona 24680Cooran 24544Marcoola 23443Kin Kin 22263
Brisbane = $32,406Gold Coast = $28,375Sunshine Coast= $26,251
Sunshine Coast avge
NOOSALOCATIONS
OESR, 2001
0%1%2%3%4%5%6%7%8%9%
10%
0 t
o 4
yrs
5 t
o 9
10
to
14
15-
19
20-
24
25-
29
30-
34
35-
39
40-
44
45-
49
50-
54
55-
59
60-
64
65-
69
70-
74
75-
79
80-
84
85-
89
90
yrs+
Noosa Noosaville Sunshine PeregianTewantin Noosa RuralAustralia
Age distribution profile Noosa Shire Segments Vs Australian Average
ABS 1996 Census, Community Profiles
Development of indicatorsDevelopment of indicators
OUTCOMESOUTCOMESProvides initial benchmarks from existing
or other routine data sources (eg ABS Census) to measure comparative
performance over time
Provides initial benchmarks from existing or other routine data sources (eg ABS
Census) to measure comparative performance over time
Initially limited as based on available dataInitially limited as based on available data
Reflected key themes in model or main strategy areas
Reflected key themes in model or main strategy areas
Social Component Initial indicatorsWellness • Age of population
• Community health index
• Hospitalisation rates
• Welfare index
• Average annual income
Social capital • Population mix, growth & stability (length of residence)
• Housing affordability
• Number of volunteer groups
• Crime rates
• Infrastructure
Community governance
• Participation in Council elections
• Community perceptions of Council
• Evaluation of participant satisfaction with community governance
Learning • Dropout rates in schools
• Levels of literacy
• Council staff survey
• Cultural change – how would we measure this?
Local media drew on issues raised by governance process to encourage a balanced discussion on the population cap,
housing, the knowledge economy etc.
Community consultations held as workshop discussions
Community consultations held as workshop discussions
OUTCOMESOUTCOMES
Boards developed issues papers for informed community discussion
Boards developed issues papers for informed community discussion
Partnered with Arts QLD policy consultations – used Minister as drawcard,
attracting >200 local participants & significant media attention
Partnered with Arts QLD policy consultations – used Minister as drawcard,
attracting >200 local participants & significant media attention
Synchronising with other projects made best use of limited resources
Focused discussion, built on existing knowledge
Community added, affirmed/rejected issues & directions
Raised community awareness of project
Evaluation used to provide Boards with community
feedback on issues papers
Evaluation used to provide Boards with community
feedback on issues papers
OUTCOMESOUTCOMES
Community formally evaluated each Sector Board’s issues paper through a tick the box
survey (quantitative)
Community formally evaluated each Sector Board’s issues paper through a tick the box
survey (quantitative)
Builds community trust & confidence in process
Builds community trust & confidence in process
Sectoral interests increasingly disenfranchised
Sectoral interests increasingly disenfranchised
Identifies & highlights broader community opinion & enables balanced discussion
Identifies & highlights broader community opinion & enables balanced discussion
Community consultation workshops stimulated discussion of each Sector
Board’s issues paper (qualitative)
Community consultation workshops stimulated discussion of each Sector
Board’s issues paper (qualitative)
Builds transparency, limits ‘personalised opinions’
Builds transparency, limits ‘personalised opinions’
Builds accountabilityBuilds accountability
Economic Board
Arts & Heritage Board
Environment Board
Social Board
Average all
3 3.5 4 4.5 5
coverage of most important issues
3
3.5
4
4.5
5
rele
van
ce t
o N
oo
sa's
nee
ds
Relevance & coverage of Sector Board proposals
goodpoor average
good
poor
average
very good
good
average
very goodgoodaverage
Average results
“It has enabled us to work outside of the political arena and to work with people who look at the
evidence without bringing personalities to the table.” - Councillor
• Structure provided free access to local expertise:– community representatives with planning expertise as Board Members– senior academics from local university and consultants– Council only paid for one consultant plus admin support to project manage the
whole project.
• Review and reanalysis of existing data-sets and sources:– provided a fresh local level information without need for costly additional research
• Built on past knowledge, findings and consultation outcomes where possible:– To avoid re-inventing the wheel– Avoided community exhaustion from over-consultation– Acknowledgement of the contribution of previous evaluations, consultations, and
planning activities undertaken by the various departments in Council.
• Synchronisation with other projects and consultations:– Arts sector consultations conducted in conjunction with Arts Queensland (AQ)
policy consultations
• Strategies and outcomes developed were generally considered to be more sustainable and hence more cost effective in the medium and long term.
• Since 2001, creative local area solutions with strong community contribution and support continue to be developed.
AES Award Criteria - Cost-effectiveness and feasibility of the contribution
Arnstein’s Ladder (Arnstein, Sherry 1969)
Level 1 Manipulation
Level 2 Education
Passive audience: information given, but partial or constructed
Level 3 Information People told what is going to happen, is happening or has happened
Level 4 Consultation People given a voice & consulted but no power to ensure views are heeded
Level 5 Involvement Some people's views have influence, but traditional power holders still make decisions
Level 6 PartnershipBeginnings of negotiation with traditional power holders (agreeing roles, responsibilities & levels of control)
Level 7 Delegated power Some power is delegated
Level 8 Citizen Control Full delegation of all decision-making & action
Noosa’s community governance
Over time, it has become evident that the evaluation and community governance process has built community capacity to participate in more informed debate and discussion.
Evaluation & evidence based methods provide powerful tools for use in community engagement & planning processes.
They are a sound complement to deliberative methods such as committees & workshops, providing a sound foundation for ethical, values-led planning, policy and economic development.
Conclusion
How evaluation made a difference to planning and decision making in a small community
Australasian Evaluation Society Queensland Seminar 11 July 2006
Ellen VasiliauskasDirector d-sipher pty ltd
Evaluation Development Award Winner 2005
© d-sipher pty ltd
Ph: 07 5471 1330
THANK YOU