popular government

COVER ARTICLE

2 Measuring the Performance of Emergency Homeless SheltersIngrid K. Flory

In homeless shelters as in local government, feedback on performance is crucial to sound management and better services. But shelters mustindividualize their services for an often transient population. This articleoffers a practical approach to measuring performance in a setting in which time and resources are limited.

FEATURE ARTICLES

11 Performance Measurement in North Carolina Cities and TownsDavid N. Ammons

A 2001 Institute of Government survey confirms that performancemeasurement is widely practiced by local government in North Carolina,but the extent of its use varies with the size of the community.

18 Public Problems, Values, and ChoicesPhillip Boyle

Before people can decide how to solve a public problem, they must be able to see it in terms of four public values: liberty, equality, community,and prosperity.

24 How to Read Governmental Financial Statements, Part 2Gregory S. Allison

Financial reporting requirements for state and local governments havechanged. What are the new features and when must governmentsimplement them?

32 Local Government Contracts with Nonprofit Organizations:Questions and AnswersFrayda S. Bluestein and Anita R. Brown-Graham

Contracts are the most common vehicles for partnerships between localgovernments and nonprofits. Both sectors need to understand thesecontracts’ requirements and limitations.

45 North Carolina Leaders at the National LevelA. John Vogt

Several local officials are making notable contributions nationally, serving on the national executive boards of professional organizations and other groups.

DEPARTMENTS48 At the Institute

Knapp Foundation Announces $1 Million Challenge

Popular GovernmentFall 2001 • Volume 67, Number 1

Page 2

8

Page 24

Page 32

NEW

S&

OBS

ERV

ER/ J

OH

NL.

WH

ITE

NEW

S&

OBS

ERV

ER/ K

EITH

GRE

ENN

EWS

& O

BSER

VER

/ CH

RIS

SEW

ARD

ON THE COVER

Photograph by Steve J. Benbow

Cots at Wake County’s largest emergency homeless shelter awaittheir occupants for the night. Today’s shelters measure success in part by number of cots or beds filled, also by number of mealsserved, number of residents placed in permanent housing, andresidents’ progress toward independent living.

p o p u l a r g ov e r n m e n t f a l l 2 0 0 1 3

A s emergency homeless sheltersacross the country adopt a modelof service delivery that encom-

passes more than offering just food andshelter, their objectives become complexand difficult to measure. The result is thatfew of them have established the level ofperformance measurement necessary togauge their success. Current literatureemphasizes the need for performancemeasurement at shelters but offers fewrecommendations on how to develop andimplement measures that are practicaland meaningful to shelter management.This article proposes such measures foruse at the largest emergency shelter inWake County, North Carolina, the SouthWilmington Street Center. It also recom-mends a practical, accurate system fordata collection in a setting where timeand resources are limited. Although theselected measures are tailored to the ob-jectives of this shelter, the recommendedsystem of measurement can be adaptedfor use elsewhere.

Background

In the past, many emergency homelessshelters served as nothing more thanplaces where people could find a mealand a bed—“a hot and a cot,” in shelterparlance. Thus they measured their suc-cess by the number of beds filled andmeals served. In recent years, however,many emergency shelters have adopted a“one-stop shop” model. Under thismodel a shelter’s residents can obtain avariety of services at the shelter. Themodel defines success not only by thenumber of beds filled and meals servedbut also by the number of residentsplaced in permanent housing and theprogress each resident makes towardindependent living.

Since adoption of the one-stop shopmodel, shelters have been struggling to

find practical ways to measure their suc-cess. Much of the challenge is due to thenature of the target population: home-less people are transient and thereforedifficult to locate for the purpose ofdetermining long-term outcomes.

In addition, homeless service providersand their residents often define “success”as completion of individual service plans.These plans are agreements between resi-dents and their case managers on specificgoals to be met in order to attain indepen-dent living—for example, that residentswill seek substance abuse or vocationalcounseling and save a certain percent-age of their earnings. Such qualitativeand individually based goals make uni-form measurement of progress espe-cially difficult.

A third hurdle is the lack of resources.Collecting data for performance measure-ment requires allocation of staff time, andfrontline service providers often viewsuch data collection as a low priority.

Despite these challenges there aregood reasons to measure performance atemergency homeless shelters. First, fund-ers and constituents are increasinglydemanding evidence that their funds areaccomplishing the targeted goals. Dataprovided by performance measurementcan be a powerful tool for retaining cur-rent resource levels, acquiring additionalfunding, or securing community sup-port. Equally practical is the need tobase management strategies on concretedata. Through performance measure-ment, shelter managers can continually

P O P U L A R G O V E R N M E N T

The author, a 2001 graduate of theInstitute of Government’s Master of PublicAdministration (MPA) Program, currentlyresides in Easthampton, Massachusetts.Contact her at [email protected] article won the Deil S. WrightCapstone Paper Award, a competitionamong graduating MPA students.

Measuring the Performance of Emergency Homeless Shelters Ingrid K. Flory

COVE R ART IC LE

NE

WS

& O

BS

ER

VE

R/ C

HR

ISS

EW

AR

D

4 p o p u l a r g ov e r n m e n t f a l l 2 0 0 1

Design of a Data Collection System

The potential pitfalls in implementingperformance measurement at emergencyhomeless shelters are many, and re-searchers debate how best to minimizepossible inaccuracies. The following dis-cussion addresses what the common

monitor program quality, demonstrateprogram effectiveness, and modify pro-grams to improve their effectiveness.1

The South Wilmington Street Center

The South Wilmington Street Center isan emergency homeless shelter ownedby Wake County and operated by UrbanMinistries, a nonprofit organization.The shelter serves single men and is oneof the few in the county that have no cri-teria for entry.2 It is outfitted withapproximately 200 beds but regularlyaccommodates 350 residents with theaddition of cots and sleeping mats.

The shelter has two programs: transi-tional and emergency. Residents in thetransitional program receive case man-agement in exchange for pledging toobtain work, stay sober, and save atleast 70 percent of their earnings. At thisshelter and most others, “case manage-ment” refers to the services provided toa resident by a social worker or othershelter staff, based on an individual ser-vice plan (described earlier). Residents inthe emergency program may receivereferrals from the shelter’s staff for spe-cific services, but they do not formallyreceive case management. Unlike transi-tional residents, who regularly stay atthe shelter until they secure other hous-ing arrangements, emergency residentsoften use the shelter erratically.

Selection of Performance Measures

Selection of performance measures forthe South Wilmington Street Center in-volved linking organizational objectivesdefined by the shelter’s staff and man-agement to performance indicators drawnfrom the literature and from interviewswith four staff members (see Table 1).These indicators gauge the extent to whichthe shelter is reaching its objectives.

When an objective is as individuallybased as “reduce residents’ use of alco-hol and other drugs,” performance mea-surement is notoriously difficult. As oneresearcher comments, “Existing scalesare generally disappointing since theylack the scope and sensitivity needed todemonstrate small and fluid changes, orto account for modest gains.”3 Thus,

although the selected indicators measureeach of the shelter’s objectives to thehighest degree practical, variations willbe inherent in the resident population.Such variations do not negate the valueof performance measurement for thepurpose of informing management deci-sions, but they would affect a scientificstudy’s margin of error.

Table 1. Organizational Objectives Linked to Performance Indicators

Objective of Shelter Indicator of Performance

Assist residents in locating, Percentage placed in long-term housingobtaining, and keeping or rehabilitative settings

Percentage living in housing of their choosing for up to one year after leaving case management

Enhance economic and Percentage unemployed but seeking work

Percentage employed, or receiving or seeking disability

Percentage employed receiving pay increase(s)

Percentage reporting successful budgeting of finances

Reduce residents’ use of Percentage of identified substance abusersalcohol and other drugs receiving treatment for alcohol and other

drug abuse

Percentage reporting heavy or regular use of alcohol or drugs

Increase residents’ access to Percentage fully or somewhat engaged in services and other agencies their case management

Percentage rating shelter services as “very . . .”or “somewhat helpful”

Percentage reporting being “very . . .” or “somewhat satisfied” with shelter services

Percentage of emergency residents applying for acceptance into transitional program

Assist residents in obtaining Percentage acquiring additional benefits andentitlements for which they service referrals while in case managementare eligible

Prepare residents to Percentage of emergency residents movingreenter society into transitional program

Percentage reporting arrests or probation or parole violations during or after case management

Among those for whom psychotropic medications have been prescribed, percentage taking them as directed

Percentage participating in community activities such as church, sports events, and meetings

acceptable housing

employment status of residents


racy.8 This level of accuracy can be ex-trapolated to other indicators for whichthere are no objective means of measure-ment, to assess loosely the overall accu-racy of the self-reporting.

Another concern about self-reportingis that the relationship between the inter-viewer and the interviewee can com-pound the risk of inaccurate responses.Typically, researchers and practitionershave relied on case managers to conductfollow-up surveys because case managersare most likely to have the knowledgeand the drive to locate former residents.Many researchers, however, have expres-sed concern that clients may exaggeratetheir successes to their case managers.On the other hand, one study found thatwhen case managers administered follow-up surveys, the self-report bias was actu-ally negative. That is, former residentsroutinely overreported problems andunderreported successes. Regardless of

eliminate bias due to attrition, it will pro-vide shelter managers with informationthat can be analyzed to identify commoncharacteristics or experiences of clientswho leave case management.

Self-Reporting BiasA second potential pitfall is reliance onself-reported outcome data. The concernsurrounding this issue is twofold. First,reliance on self-testimony carries an in-herent risk of dishonesty. In the contextof homeless shelters, however, replacingself-reporting is nearly impossible be-cause so many of the outcomes are mea-surements of lifestyle and independence.Although the recommendations in thisarticle do rely on self-reporting, they in-clude several indicators that can be peri-odically cross-referenced to ascertain thelevel of self-reporting bias. For example,one questionnaire asks residents to re-port if they have had any convictions orparole violations while receiving casemanagement. These self-reports can beperiodically cross-referenced with localauthorities to gauge their level of accu-

obstacles to data collection in social ser-vices programs are and how the systemrecommended for the South WilmingtonStreet Center surmounts them.

Attrition RatesAlthough many researchers recommendconducting exit interviews when resi-dents leave a homeless shelter,4 othersexpress concern that exit data skew out-comes because they do not account for“attrition” (residents leaving the shelterwithout notice, reducing the shelter’spopulation).5 Some researchers have at-tempted to account for this by trackingattrition rates,6 but others contend that it is impossible to interpret attrition ratesas indicative of solely negative or positiveoutcomes.7 To avoid that pitfall, this ar-ticle recommends a method of data col-lection that will measure performance repeatedly during a resident’s case man-agement and first year out of the shelter.The method will help ensure that even ifa resident leaves without warning, theshelter will have some data on the resi-dent. Although having such data will not

A heavily used service at homelessshelters is hot meals, often prepared andserved by volunteers.


the direction in which the bias leans, itcan be minimized through frequent con-tact between client and case manager.9 Inother words, the more often a case man-ager and a client speak, in either regularmeetings or follow-up conversations, thegreater the client’s tendency to be honest.Given these considerations, the recom-mended method of data collection fortransitional residents is for case man-agers to conduct follow-up surveys withthe residents in addition to having regu-lar contact with them after they leavethe shelter. The use of case managerswill increase the response rate, and therequirement of regular contact willminimize the bias resulting from thecase manager/client relationship.

Recommendations for the South Wilmington Street Centerand Other Shelters

As noted previously, the South Wilming-ton Street Center has two programs:emergency and transitional. Althoughthe populations in these programs arehoused in the same space, they receivevery different levels of service. Therefore

the shelter should use different criteriato measure its success with them. Fol-lowing are recommendations on howthe center and shelters in general canbegin collecting data for performancemeasurement with three populations:emergency residents, transitional resi-dents, and former transitional residents.The recommendations are based on a lit-erature review and on interviews withthree directors of comparable emergencyshelters in Wake County and one nation-ally recognized emergency shelter in Bos-ton. The questionnaires created for datacollection were pretested for validity andpracticality at the South WilmingtonStreet Center.10

Emergency ResidentsLike many other shelters, the SouthWilmington Street Center expects themajority of its residents to need casemanagement or other support services inorder to gain economic self-sufficiency.The shelter’s staff therefore assume thatbefore a resident finds permanent orsupportive housing, he will receive casemanagement through the transitionalprogram.11 Given that all residents enter

as emergency residents and must applyto be accepted into the transitional pro-gram, application to and acceptance in-to the transitional program should bethe principal outcome measures for emer-gency residents. The percentage whoapply will indicate the shelter staff’s suc-cess in encouraging emergency residentsto seek services. The percentage who areaccepted will reveal the shelter staff’ssuccess in preparing residents to meetthe transitional program’s requirementsof sobriety and willingness to obtainemployment.

These data can be easily obtained bykeeping a record of all emergency resi-dents, emergency residents who apply tothe transitional program, and emergen-cy residents who are accepted. Thus thecenter and other shelters need only cal-culate the percentage of emergency resi-dents who apply to the transitional pro-gram and, of those, the percentage whoare accepted.

The South Wilmington Street Center’sbus delivers homeless people to theshelter on a night that is expected to be cold.

NEW

S&

OBS

ERV

ER/ M

ELN

ATH

AN

SON


A second way in which the center andother homeless shelters can measure theirsuccess in encouraging emergency resi-dents to apply to the transitional pro-gram is to document their attempts toengage emergency residents in services orthe transitional program. “Engagement”is defined as “establishing a relationshipwith a prospective client for the purposeof developing interest and involvementin treatment.” The level of engagement isan indicator of how active a client is inhis or her own treatment. The higher thelevel of engagement, the greater the suc-cess of the organization in empoweringits clients to meet their own goals.Management can promote documenta-tion of engagement by periodically se-lecting a sample of emergency residents,inquiring whether staff have attemptedto engage them in services, and notingwhether they have accepted or refusedassistance. Not only will this provideinformation on the rate of engagementfor emergency residents, but also it willpromote outreach among the shelter’sstaff as a performance indicator for theshelter. In addition, management can

cross-reference such information withdemographic and personal data to deter-mine differing levels of engagementamong various subgroups of the popula-tion.12 This will help identify subgroupsthat are not being reached because oflanguage barriers, length of homeless-ness, or other reasons. Such periodic sur-veys also could be designed to help assessclients’ satisfaction with services.

Transitional ResidentsThe structure and the goals of the tran-sitional program allow for a range ofperformance measures to be gatheredthroughout a resident’s participation inthe program. Thus, performance measuresfor both current and former transitionalresidents should emphasize individual-level progress toward independent living,in addition to more traditional outcomes,such as economic gains and placement inpermanent housing.

Taking into consideration both thesize of the population served and theperceived time constraints of the shel-ter’s staff, a survey was developed to beadministered orally to a sample of tran-

sitional residents (see the sidebar onpage 8 for sample questions). Approxi-mately 120 transitional residents shouldbe randomly selected each year to re-ceive the survey. This sample size willensure that the survey results will beaccurate within a 10 percent margin oferror.13 The case managers of these resi-dents should conduct the survey orallywith them during three regularly sched-uled appointments. Conducting the sur-vey at these times will ensure a fairlyhigh rate of response. In addition, sincethe survey will require only about sevenminutes to conduct,14 it will infringe lit-tle on the client’s appointment. Casemanagers who pretested the survey re-ported that this time requirement wasnot unduly burdensome and that thesurvey provided them with an opportu-nity to reflect on their client’s progress todate. Thus such an instrument alsomight contribute to the quality of casemanagement.

People sleep on the streets for manyreasons—for example, lack of space in shelters, fear of authority, and mental illness.


1. When was the last time you maintained independent housing? Please enter month and year:

2. Please indicate how many months you have been in the transitional program:

_______________ months

3. Please indicate your current employment status: (Please check only one).

Employed, day labor (contracted daily) � (If checked, go to item 4)

Employed, temporary (contracted weekly or monthly) � (If checked, go to item 4)

Employed, permanent � (If checked, go to item 4)

Unemployed, seeking work � (If checked, go to item 6)

Unemployed, not seeking work � (If checked, go to item 6)

Receiving/seeking disability � (If checked, go to item 6)

4. If employed, have you received a pay increase since you were hired?

� Yes � No

If yes, how many pay increases have you received?________________

5. Over the past four weeks, have you successfully saved 70% of your earnings?

� Yes � No � Not sure

6. In the past four weeks, have you participated in any community activities, such as church, sports events, orhouse meetings?


7. In the past four weeks, have you attended an alcohol or other drug treatment program (including AA orNA meetings)?

� Yes � No � Not applicable

8. How helpful has each of the following services been to you? Please check all that apply.

Very Somewhat Not Don’t Helpful Helpful Helpful Know N/A

HIV education and testing � � � � �

Veterans services � � � � �

Case management � � � � �

Mental health services � � � � �

Substance abuse counseling � � � � �

Health services � � � � �

Job counseling � � � � �

Recovery dynamics classes � � � � �

Provision of birth certificate or NC ID � � � � �

Tuberculosis testing and/or treatment � � � � �

Referral to housing � � � � �

Other � � � � �

Specify:_________________________________________________________

SAMPLE QUESTIONS FROM THE TRANSITIONAL RESIDENT QUESTIONNAIRE


The first round of the survey shouldbe conducted within the first month of aresident’s joining the transitional pro-gram; the second before the time atwhich the average resident leaves theshelter for other housing; and the thirdat an interval after which most residentshave left the shelter for other livingarrangements.15 Conducting the surveyat these intervals will provide informa-tion about services or situations thatmay speed or slow the process of reach-ing desired outcomes. It also will helpoffset any bias created through attrition,because there will be some data on resi-dents who leave unexpectedly, whichcan be analyzed for trends. Such infor-mation can help identify common char-acteristics or experiences of clients wholeave case management.

While conducting the survey, the casemanager should enter the answersdirectly into a database,16 thereby avoid-ing spending time on data entry at alater date. Then, at a regularly scheduledinterval (six months is recommended),management can aggregate responsesand analyze them for trends. This will

provide valuable data for managementdecisions and for new goals for staff andresidents.

Former Transitional ResidentsAfter the selected transitional residentsleave the shelter, their case managersshould conduct a follow-up survey atincreasing time intervals for one year (asample survey is available from the MPAProgram). The purpose of repeatedlyconducting this survey is for manage-ment to assess the long-term outcomesof former residents while case managershelp their clients make the transition toindependent living.

Case managers should conduct thefirst and second rounds of the surveyover the telephone or in person at fourand eight weeks after the client leavesthe transitional program. (Managementshould encourage case managers to con-tact their clients within the first weeks ofdeparture, though, to offer support andto maintain contact.)17 This emphasis onthe client’s first two months out willoffer additional support in a difficultperiod of transition and increase the

9. Overall, how would you rate your satisfaction with the services you have received at the SouthWilmington Street Center?

Very Somewhat Not Very Not at All NotSatisfied Satisfied Satisfied Satisfied Sure

� � � � �

10. Since entering the transitional program, have you obtained additional benefits, such as Social Security,disability, food stamps, or Section 8 assistance?


11. Have you ever been convicted of a crime other than a traffic violation?


12. Have you had any arrests or probation or parole violations since entering the transitional program?


13. Are you taking any mental health medications?


If yes, are you taking the medicine as directed?


14. In your opinion, how have you grown since entering the transitional program?

SAMPLE QUESTIONS FROM THE TRANSITIONAL RESIDENT QUESTIONNAIRE (CONT.)


case manager’s ability to maintain cur-rent contact information for the client.After two months, clients should be sur-veyed three more times before their one-year anniversary of leaving the shelter.18

If clients do not have a telephone or can-not otherwise be reached by telephone, a postcard should be sent requestingthat they call or drop by the shelter forfollow-up with their case manager. Theshelter might consider providing anincentive such as grocery coupons or buspasses to encourage former residents toparticipate in the follow-up surveys.

Again, the case manager should entersurvey responses directly into a databaseat the time he or she conducts the survey,and management should aggregate theresponses and analyze them for trends atleast every six months.

Conclusion

Although the challenges to conductingperformance measurement at emergencyhomeless shelters are many, the feedbackthat a good set of performance measurescan provide is crucial to making soundmanagement decisions and improvingservices. Given the complexity of a shel-ter’s objectives and the transience of itsclientele, no performance measurementsystem is likely to be flawless. The sys-tem for performance measurement rec-ommended in this article seeks to takeinto account the transience of the popu-lation, the need for individual-level data,and the limited resources of shelters.The information generated by such asystem should give shelters valuableinsight into their strengths and weak-nesses without placing an undue burdenon their staff.

Notes

1. William Hargreaves & CliffordAttkisson, Evaluating Program Outcomes, inEVALUATION OF HUMAN SERVICE PROGRAMS

303 (Clifford Attkisson et al. eds., New York:Academic Press, 1978).

2. Even active drug abusers, who areroutinely banned from a majority of shelters,are allowed to stay at the South WilmingtonStreet Center.

3. Celine Mercier et al., Program Evalua-tion of Services for the Homeless: Challengesand Strategies, 15 EVALUATION AND PROGRAM

PLANNING 417, 422 (1992).

4. Debra Rog, The Evaluation of theHomeless Families Program: Challenges inImplementing a Nine-City Evaluation, inEVALUATING PROGRAMS FOR THE HOMELESS 47(Debra Rog ed., San Francisco: Jossey-Bass,1991).

5. Alexander Young et al., Routine Out-come Monitoring in a Public Mental HealthSystem: The Impact of Patients Who LeaveCare, 51 PSYCHIATRIC SERVICES 85 (Jan. 2000).

6. Mercier et al., Program Evaluation ofServices.

7. One study of mental health patientsfound that the group that had unexpectedlyleft treatment actually had better outcomesthan the group that had remained in treat-ment. Thus, attrition rates often may indicatea positive outcome. Young et al., RoutineOutcome Monitoring.

8. Residents of the South WilmingtonStreet Center sign a waiver to permit theshelter’s staff to have access to their court and service records. Such information is kept strictly confidential and is primarily used to increase the effectiveness of casemanagement.

9. Robert Orwin et al., Pitfalls in Evalua-ting the Effectiveness of Case ManagementPrograms for Homeless Persons, 18 EVALUA-TION REVIEW 153 (Apr. 1994).

10. Two questionnaires were developed for data collection. Shortage of space pre-cludes their publication with this article. In the pretest, case managers administeredfive copies of each questionnaire to currentand former transitional residents. Mostfeedback regarding questionnaire content and design was positive. Various suggestionshave been incorporated into the finalrecommendations.

11. Although this assertion lacks empiricalevidence to back it up, it is the predominantphilosophy among homeless service providers.

12. Carol Mowbray et al., The Challenge of Outcome Evaluation in Homeless Services:Engagement as an Intermediate OutcomeMeasure, 16 EVALUATION AND PROGRAM

PLANNING 337 (1993).13. To achieve a smaller margin of error, the

shelter would have to survey the entire popu-lation of transitional residents. This may ormay not be administratively feasible, givencurrent resources and staffing levels. It istherefore recommended that the shelter beginwith a sample of transitional residents and, ifpossible, expand the survey to the entirepopulation at a later date.

14. The estimated time required to admin-ister the survey is based on the pretest.

15. These intervals will minimize the effect of attrition and allow for trend analysis based on length of treatment. Hargreaves & Attkisson, Evaluating Program Outcomes,at 308. The South Wilmington Street Centerhas yet to determine the average length ofparticipation in the transitional program.

16. A critical component in adopting these measures is the development of a casemanagement database that will facilitateaggregating and coordinating data.Specifically the database should allow forcomparison between services received andservices planned for residents receiving casemanagement. This will aid case managers inassessing their clients’ level of engagement and fidelity to treatment. In addition, thedatabase should have trigger capacities toremind case managers when surveys are to be administered to specific clients. Last, theshelter will be relying on the database to select a random sample of clients to survey. To spread the survey work among casemanagers, the database could be set to selectfor survey a determined number of newclients per case manager.

17. According to one case manager, it iscommon for case managers to contact formerclients soon after their departure, although thecase managers often do so in their free time.

18. These surveys should be evenly distrib-uted throughout the remainder of a client’sfirst year out of the shelter. Suggested timesare 5, 8, and 11 months out.

Nominate Your Favorite Institute ofGovernment Teacher

Public officials who haveparticipated in Institute ofGovernment schools or

courses are encouraged to nomi-nate Institute faculty membersfor the Albert and Gladys HallCoates Term Professorship forTeaching Excellence. The awardhonors a member of the Instituteof Government faculty for excel-lence in teaching. The recipient,selected biennially, receives astipend of $5,000 per year plusexpenses during the two-yearterm.

All Institute faculty membersexcept the director, the associatedirectors, holders of permanentchairs, and prior recipients of theaward are eligible for nomina-tion. Additional information andnomination forms may be ob-tained from the Institute’s Website, http://www.iog.unc.edu/award, or by calling John L.Saxon at (919) 966-4289.

popular government

Documents