o the office of the auditor general for western australia g a monitoring and evaluation of a...

o

The Office of the Auditor General for Western Australia

g a

MONITORING AND EVALUATION OF AMULTI-AGENCY RESPONSE TO

HOMELESSNESS

Paper to the Ireland Evaluation Network 23 May 2008 Peter Wilkins

o


g a

Evaluating the complex

• A research theme of the International Evaluation Research Groupwww.inteval-group.org/

• A forthcoming book ‘Evaluating the Complex: Attribution, Contribution and Beyond’ Edited by Robert Schwartz, Kim Forss and Mita Marra

• A new interest in effectiveness evaluations of overarching policy initiatives

o


g a

Evaluating the complex (2)

• Complex policy initiatives mostly fail standard evaluability assessment.

• Objectives problematic, little evidence of causal linkage

• Non-linear links of interventions, contextual variables and outcomes

• Attribution a major challenge, even when macro-level outcomes are measurable

• Editors have identified 5 approaches that might be helpful: cluster evaluation; multi-level evaluation; quantified logic models; comparative community studies; thematic evaluations

o


g a

Origins of Homelessness Initiatives

• Task Force created in July 2001 to develop a plan of action• The Chairperson of the Taskforce was from the not-for-

profit sector• Also to provide advice on the development of benchmarks

and social indicators relating to homelessness and recommend a plan to monitor the progress of the implementation strategy”.

• Taskforce reported in January 2002 and made 68 recommendations relating primarily to nine State agencies.

• It recommended that implementation include tracking the extent to which the Actions have been implemented. It also called for Independent Review “The non-government sector is keen to see an independent review undertaken after a two-year period to assess the outcomes for people who are homeless. This is viewed as part of open government and the commitment to work in partnership with the community.”

o


g a

Origins of Homelessness Initiatives (2)

• State government response in May 2002, accepting majority of recommendations.

• Committed to a further $32 million over the four years 2001-02 to 05-06 for a set of initiatives

• Report recognised the complex environment and circumstances surrounding homelessness. For instance, it noted that people who are homeless have typically experienced difficulties in their lives and that often homelessness arises because of these difficulties including family conflict, mental illness, interruptions to stability through imprisonment, long term care or institutional living, family violence, and problematic alcohol and other drug use.

• Three themes of the Government's response to the Taskforce's recommendations: Better options for housing; Vulnerability and transition and Stability in housing.

o


g a

Monitoring Committee

• Established a monitoring committee to oversee and report on the implementation of the State Homelessness Strategy, and to – Develop performance indicators to measure Government

performance in responding to the Taskforce recommendations.– Prepare an annual report to Cabinet Standing Committee on

Social Policy, against these performance indicators, with the firstreport due in May 2003.

– Prepare an evaluation report on the implementation of therecommendations for submission to Cabinet in December 2003.

• Monitoring Committee membership included representatives of five key departments, the Chairperson of the Taskforce, from ATSIC and the Office of the Auditor General [1]. It met regularly and its major outputs were 17 reports: 12 quarterly progress, 3 performance indicators & 2 evaluations.

[1] For much of its life the OAG member who participated as an observer was the author of this paper.

o


g a

First Evaluation

• Focused “primarily on formative issues, ie improving processes rather than evaluating success”

• Sought to address 6 key questions: Implementation; Timeliness; Short-falls; Impacts; Program logic; and Performance indicators, and found:– Initiatives requiring the development of strategies or the forging of

agreements had generally made satisfactory progress but that progress on initiatives involving legislation, policies or systems has been a little more mixed.

– Nearly all the performance indicators developed by the Monitoring Committee were useful

– Areas for further consideration included: no mechanism for showing what other initiatives might be beneficial; developmental performance indicators; and no single measure of the level of homelessness

– draft Program Logic Model (Figure 1) & PIs associated with a Hierarchy of Outcomes (Figure 2)

• Four main recommendations: continue to ensure that agency and community support & engagement continue; cross-government strategies to further breakdown the "silo" effect; review of two of the PIs; and a further review of the Strategy

o


g a

Program Logic for Government Response to Homelessness

Source: Cummings, R and Moir, S (2003).

o


g a

The Hierarchy of IndicatorsSource: WA State Homelessness Strategy - Report Against Performance Indicators - As at 30 June 2005.

o


g a

Second Evaluation• Sought to assess impact & effectiveness of the initiatives in

the strategy and addressed: the extent to which commitments had been implemented; effects of the initiatives on other major service deliverers; outcomes for clients and other affected groups; whether funded initiatives should be continued, ceased or changed & the service system’s response had improved; effectiveness of the strategy in addressing the priority outcomes; how well coordinated across Government.

• Report identified that the evaluation was atypical in that it is reviewing outcomes of the individual initiative and strategic levels.

• Noted components of a monitoring and an impact evaluation:Monitoring Evaluation is carried out to ensure that a project remains on track and to guide continuous improvement. – Impact Evaluation is used to assess the impact of a settled project

• Main findings were generally positive: majority of initiatives successfully implemented or underway, good outcomes for clients.

o


g a

Second Evaluation (2)

Extracts of the 2006 Evaluation Report: Major Findings against some Key Evaluation Questions

• To what extent were the commitments made in the Government Response to the State Homelessness Taskforce (Putting People First) implemented?Work on thirty of the initiatives completed, progress cf 25 initiatives completed in 2003. Overall, 58 of the 67 are reported to be progressing or completed satisfactorily. The results of the evaluation reflect a general satisfaction with the Strategy as a whole amongst stakeholders.

• How effective has the Strategy been in addressing the Priority Outcomes?A program logic analysis confirmed that the initiatives are consistent with the priority outcomes. The data collected in the study, as well as client feedback and stakeholder views reported here also support this conclusion

• How has the Strategy been Implemented in the context of the Government’s Social Policy Agenda and how well has it been coordinated across Government? Widespread agreement among stakeholders that Strategy fits within the Government’s social policy agenda. Some success, still a long way to go. Still clients who are missing out on services, waiting too long for services & being passed from department to department.

o


g a


CONCLUSION• The evidence suggests beginning to break the cycle of homelessness

although not possible at present to assess their cost effectiveness. Where current funding or policies do not provide for effective servicing, a cycle of homelessness remains as clients reappear in the service system.

• Overall, the Strategy is beginning to show some positive measurable outcomes in the headline indicators….The improvements so far, however, are modest. This is to be expected because of the magnitude of the issue being addressed. Perhaps biggest weakness is improving accommodation services for individuals suffering from mental illness…..While there has been a small improvement in indicators such as Homeswest waiting lists per head of population, the absolute level of homelessness is still high and more needs to be done if it is to be substantially improved….

• The evaluation indicated that while much was achieved, much remained to be done.

o


g a


• As the Strategy was coming towards an end, it also became apparent that there were significant pressures on the availability of affordable housing including public housing stock linked to a natural resources driven economic boom in the State. The Government has acknowledged these pressures and indicated that it is seeking to address them

• As the website for the 2006 evaluation notes: “In 2006, the booming economy is placing more people in work…. Costs are increasing to buy and build properties and rents are rising. People with minimal resources, both financial and personal are being squeezed out at the bottom end of the housing market. This places an added strain on the safety net services”

o


g a

Evaluation Linked to Monitoring

• Distinctive feature of the implementation process has been the intertwining of monitoring & evaluation and a synergistic relationship between the two

• 2006 evaluation specifically addresses effectiveness by reviewing performance indicator information collected and reported as the "Hierarchy of Indicators"

• The relative sophistication of the approach can be identified from the findings of a recent report of the Auditor-General of New South Wales (2007) who found that “… we were unable to determine how well the government is responding to homelessness statewide. This is because there are no statewide performance measures or targets on homelessness. Also there is limited benchmarking, and no formal means of spreading information on homelessness initiatives and projects.”

• The approach entails risks of over-simplifying a complex situation & creating a view that attribution has been resolved. Other analytical frameworks (Figure 3).

o


g a

Housing Analytical Framework for Planning

Source: WA Department of Housing and Works (2005).

o


g a

Evaluation and Attribution (2)• The 2nd report observes that the evaluation is more challenging than

that of a single program as many factors are at play in the same arena as the strategy

– “Political, cultural, social and economic trends act on service delivery agencies at a macro level. In addition, there are a multitude of different projects and programs at work, a multitude of organisations at work independently of any particular initiative. Should changes in social outcomes occur, the challenge is to be able to distinguish between at least three possibilities: the change is occurring because of the Strategy; the change is occurring for another reason apart from the Strategy; or the changes occurring partly due to or inspite of the Strategy.”

– It further notes that “while positive change in social indicators may indicate a successful strategy, it is generally impossible to map such a one-to-one relationship between activity and result. The use of professional judgment of stakeholders can be useful to help interpret data.”

• It makes a contribution to understanding causality without establishing it. Based on all the evidence it was reasonable to conclude that the program was an important contributing factor to the changes observed.

• The report noted that the evaluation “has attempted to identify measurable impact where possible, and where this is not possible, it has used the program logic analysis and Hierarchy of Indicators to demonstrate changes in headline indicators which in turn that are expected to impact on the priority outcome”.

o


g a

Evaluation and Attribution

• Attribution in simple terms is about causality, about answering the question: Are observed results the consequence of the intervention rather than other programs or external factors?

• Mayne 2001 talks of "the problem of attribution" and notes that more difficult than measuring outcomes is determining just what contribution a specific program made to the outcome, and how much of the success (or failure) can be attributed to the program.

• Pinpointing causality between individual interventions and outcomes is in most cases problematic, and doing so for a multiplicity of interventions is nigh on impossible.

• The State Homelessness Strategy is complex as a result of multiple interventions operating at different levels in different ways, and having multiple stakeholders

• The second evaluation sought to address “whether funded initiatives should be continued, ceased or changed” which implies a need for some understanding of causality

o


g a

Effectiveness and Cost-Effectiveness

• Research group is looking at “The Effectiveness and Cost-Effectiveness of Homelessness Prevention and Assistance Programs” using the Western Australian experience as its area of study.

– Effectiveness the extent to which the programs improved the outcomes of those they assist; cost-effectiveness looks at the outcomes delivered for the net dollar spent including ‘net return’ compared with alternative uses of the same resources.

– Primary field work a longitudinal client-based survey and a one-off survey of agencies primarily looking at costs. Costs determined using one of two scenarios: clients of the homelessness assistance service compared with the population average; and clients of each service in target categories compared with clients who did not fall into one of these categories

• Preliminary findings: the programs produced positive outcomes for clients in the period immediately following the provision of support and did so at low cost of delivery relative to the delivery of other services

• "…. if homelessness programs were able to reduce the utilisation of health and justice facilities by clients of homelessness programs down to population rates of utilisation the savings achieved would pay for the homelessness programs several times over. This suggests that there is potential for homelessness programs to be dramatically cost-effective."

o


g a

Effectiveness & Cost-Effectiveness (2)• Outcomes assessed across a range of dimensions: changes in

clients status measures such as labour force, level of income etc; changes in the utilisation of homelessness programs services and non-homelessness programs services; changes in self-assessed satisfaction and Program-specific client outcome indicators– Programs supporting those leaving prison proving beneficial in lowering

rates of recidivism and making positive contribution to the lives of individuals who have previously returned to prison

– On average, the annual cost of health services for a community centre client is $10,217 per person per year greater than the population average, and for Justice services $3810 per person per year greater than the population average

• Value of annual "population offsets" is at least 2.7 times greater than the annual program costs, a significant potential for cost savings from providing assistance.

• The preliminary report notes that the Final Report "will include an examination of client outcomes 12 months down the track when most clients will have left support."

Flateau, P et al (2007). “The Effectiveness and Cost-Effectiveness of Homelessness Prevention and Assistance Programs: A Report of Preliminary Findings” Australian

Social Policy Conference, Sydney. www.sprc1.sprc.unsw.edu.au/ASPC21007/

o


g a

A Client Based Model

Source: Flateau, P et al (2006). The Effectiveness and Cost-Effectiveness of Homelessness Prevention and Assistance Programs Positioning Paper, Melbourne: Australian Housing and Urban Research Institute (AHURI). www.ahuri.edu.au

o


g a

A Non-Client Model

Source: Flateau, P et al (2006). The Effectiveness and Cost-Effectiveness of Homelessness Prevention and Assistance Programs Positioning Paper, Melbourne: Australian Housing and Urban Research Institute (AHURI). www.ahuri.edu.au

o


g a

Concluding Comments

• Intertwining of monitoring & evaluation

• The second evaluation enabled reflection by the evaluator and stakeholders on the question of causality without making purity of approach a barrier to forward movement of the Strategy

• Ongoing research has the potential to build understanding of cost effectiveness

• See paper at www.aes2007.com.au/papers.php

o


g a

Contact details

Dr Peter Wilkins

Assistant Auditor General

Office of the Auditor General4th floor Dumas House

2 Havelock St

WEST PERTH

Western Australia 6005

Phone: +61 8 9222 7588

Mobile: +61 407 380 772

Fax: +61 8 9322 5664

Email: [email protected]

o the office of the auditor general for western australia g a monitoring and evaluation of a...

Documents

western australia g

auditor general

complex policy initiatives

homelessness paper

state homelessness strategy

complex environment

state government response

set of initiatives report