dont know refused

32
Don’t Know/Refused: A Researcher’s Perspective on HMIS Data Collection Challenges By Susan Walker Rhode Island Coalition for the Homeless Bowman Systems Collaborate 2016

Upload: susan-walker

Post on 13-Feb-2017

43 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Dont Know Refused

Don’t Know/Refused:A Researcher’s Perspective on HMIS Data Collection ChallengesBy Susan Walker Rhode Island Coalition for the HomelessBowman Systems Collaborate 2016

Page 2: Dont Know Refused

What do I Know About Homelessness and Data?

Occupy Providence 2011-2012, back when I used to call constituents “friends”.

Antiques Dealer since 1998, alongside homeless and marginalized people.

Studied Medical Billing & Coding at a for-profit online university. Learned about records and predatory lending.

Currently enrolled at Brown School of Public Health, learning and doing data collection and analysis.

Recently hired by my dream employer. I’m new! If you pay close attention, you’ll

probably catch my mistakes!

Page 3: Dont Know Refused

Data Collection Concepts

Hard to Reach Population Question Ordering Coding Skip Patterns Prompting with Answer Choices Probing Look backs

Data Collection Points Non-Verbal Communication Interviewer Bias Client disposition/Setting Sample Size Bias ERROR!

Page 4: Dont Know Refused

Personality

Data Quality is influenced by the personalities of…

The HMIS end user The client The provider’s office culture The funding managers Government officials The HMIS administrators Variable itself

Page 5: Dont Know Refused

Form Versus Questionnaire

The HUD Intake form has issues. You ask a client for their name. You don’t ask a client for their name data quality. “Has the client been continuously homeless (i.e., on the street, in

an emergency shelter, or safe haven) for at least one year? … Still on the HUD Template.

Our Entry in HMIS is in a very different order. A form gives the impression that the data is somehow uniform,

and fits easily into the boxes. A questionnaire, with instructions, communicates that gathering

data requires conversation, and entering data requires judgment.

Page 6: Dont Know Refused

Entry Date The Entry Date and data entry

date are often different: a source of error.

There can be overlapping ES entries on the same date.

Backdating is great, but can lead to absurdities.

I emphasize the importance of the relationship between date and data:

By entering a date, they are confirming that all the data in the record is true.

If they haven’t taken care to verify it, there is a problem.

Page 7: Dont Know Refused

Name Street Names… Duplicate Clients Data Quality Descriptors? We have found out that we

sometimes have 2 clients to one ID because they have the same name.

We train Case Managers to enter a name data quality descriptor, but not to ask about it.

It is unclear which HMIS Data Elements need to be asked of the client and which do not.

ALIAS

Page 8: Dont Know Refused

Social Security Number

There is a disincentive for sharing this data.

HUD’s rationale: unduplicated client counts, and SSN necessary for applications. Data Quality Descriptor errors: If the Case Manager enters “doesn’t know” or “refused”, then the blank SSN shouldn’t flag.

Some shelters were using 4 digits as a matter of procedure. In the event of a security breach, our clients would be vulnerable.

Page 9: Dont Know Refused

Date of Birth

Date of Birth is a variable with personality! It’s a number does not express a quantity. We use it to calculate age. If it’s not entered properly, it’s hard to calculate age. This is about the only trouble free variable. Our case managers find “Date of Birth Type” confusing, which is

about the only thing I would consider changing in ServicePoint.

Page 10: Dont Know Refused

Veteran Status

• Self report might be in conflict with actual Veteran Status.• Often missing, but I don’t understand the skip pattern. • Why do you think people skip this question?• In Rhode Island, Veteran Status as defined in the data standards

would exclude people from programs. • We train to accept self report. The VA verifies later on.

Page 11: Dont Know Refused

Primary Race Frustration with these categories is widespread. Positive –They allow for meaningful comparison with all

other race data. Negative- Our clients do not like categorizing themselves

this way. The HMIS Race Data Standards are based on the OMB’s

Standards. Unless the Census changes, these categories are unlikely

to change.

Page 12: Dont Know Refused

Secondary Race

“Allow clients to identify as many racial categories as apply (up to 5).”

The 2000 Census was the first to offer more than one race category, with a view to understanding multi-racial populations.

The secondary race option is a good faith effort to assess racial diversity.

Stay tuned for revised OMB standards which will likely impact HUD data standards.

Page 13: Dont Know Refused

Ethnicity

“In the 2010 census, about 20 percent of Hispanics left the race question completely blank.”

Ethnicity is about culture, customs and language.

Different races can have the same ethnicity. People of one race can be many different ethnicities.

We discovered in training that people answer “Latino” for race, and trained to enter the ethnicity and probe for primary and secondary race.

Page 14: Dont Know Refused

Gender

A simple, yet awkward question. Prefacing the whole interview with “I

need to ask every question, even though some may seem obvious” can help.

The data standards changed recently. It’s highly likely that case managers

nationwide answer this question without asking clients.

Page 15: Dont Know Refused

Disabling Condition

We can’t decide where to put this question. Personality: Often in conflict with Disability HUD Verifications. Case managers report hearing “I don’t know”. Case managers need training on how to probe by specific

conditions. How do you ask, and preserve the dignity of the client? Disability is a huge term, but the HUD criteria are specific. There’s a conflict if your program prohibits asking about

disability before housing placement. Confusion surrounding the “self report” standard, and the

documentation required for housing eligibility.

Page 16: Dont Know Refused

Residence Prior to Project Entry

It’s not clear from HMIS that this needs to be updated every entry.

It often does not get updated every entry. Our Case Managers want a “getting

evicted” option. We need an error report that shows that

this was not updated at entry. HMIS Data Standards 5.1 devotes 10 pages

to this element. New Option “Interim Housing” It can be very challenging to code this

information properly even if the client is properly interviewed.

Page 17: Dont Know Refused

Length of Time We tried to put this into plain

English. This field is only as good as the

previous field. It would be nice if error reports

would flag when these fields are neglected.

There is no way to match the length of time to the previous place to see that the length of time actually corresponds to the previous place.

Page 18: Dont Know Refused

Head of Household

Frequently skipped. What household? Even our plain English questions do not make sense in most

settings. Long instructions in the Data Standards. Skipping seems to be associated with the awkwardness of the

question, and the amount of judgment needed to answers the question.

In our COC many case managers enter the client with the disability as HoH.

After that, we select the woman before the man, as men commonly exit PH-FAM.

Personality-How case managers answer this question reflects uncomfortable truths about our society.

Page 19: Dont Know Refused

Entering from Street or Emergency Shelter

Our data completeness is good with this field.

No way of knowing if it is updated at each entry.

Length of Time Homeless is an extremely important metric.

We don’t have Safe Havens in RI. “Look Back” data is challenging.

Page 20: Dont Know Refused

How Many Times?

Our data is complete for this field. Hard to know if it’s accurate. In training we discovered that more

than one case manager would count 4 years as 4 times.

We trained them to count the breaks. We had a case manager act like a client, and he did a

fantastic job of not answering any question straight. We can train to collect and enter data, but training in

judgment is very difficult.

Page 21: Dont Know Refused

Total Number of Months 1 month, ah ah ah…2 months ah ah ah… 1 day = 1 month???

Page 22: Dont Know Refused

Income “Projects collecting data through

client interviews should ask clients whether they receive income from each of the sources listed rather than asking them to state the sources of income they receive.”

We struggle to get PH providers to update this information annually.

Our case managers are not entering “Under the Table” income.

We are encouraging copious use of Updates so that the data is already there at Annual Assessment.

Clients are not sure what the “right” answer is.

Page 23: Dont Know Refused

Non-Cash Benefits

TANF: A variable with an alias. Some benefits have amounts and

some don’t. Plagued by duplicates. I used this screenshot in a

newsletter to show what perfect end dates look like.

We are training that it is ok to use files and case notes to make these updates in the Annual Assessment.

Page 24: Dont Know Refused

Health Insurance

Page 25: Dont Know Refused

Domestic Violence The accuracy of this data element is

entirely dependent on the situation. What do you do when a client answered

YES at an earlier entry, and answers NO at a current entry interview?

People enter NO, then enter “Client doesn’t know”

The purpose of this element is to determine if the client needs protection: a unique personality trait.

The data standards have a specific interview protocol, unique to this element.

Also unique data entry protocols.

Page 26: Dont Know Refused

Is the Client Permanently Housed?• Personality: Answered in

error all over our HMIS. • Too much data rather than

not enough.• A cut and dried answer that

is easy to determine. • It seems like it should be

more important than it is. • A misunderstood variable. It

does not serve the purpose people think it does.

Page 27: Dont Know Refused

Error(s)• Error and analysis are two

sides of the same concept. • Individual errors are

addressed.• I don’t know if Confidence

Intervals and Standard Deviations are accounted for in ART Reports.

• Our data is used for eligibility, funding, reporting and understanding, but the limitations of the data are rarely discussed.

• A calculation for average Length of Time Homeless needs a Confidence Interval.

Page 28: Dont Know Refused

Validity and Reliability

Page 29: Dont Know Refused

Motivation & Data Quality

Winning buy-in from end users is probably the biggest challenge.

Federal and local funding depend on data. Eligibility depends on data. Providing the right services depends on data. Entering high quality data makes you a “good citizen of the

HMIS”. HMIS gives voice to people who have no voice. HMIS tells their story, and people really do listen.

Page 30: Dont Know Refused

ReferencesBunzli, L. (2016). The Definition of Chronic Homelessness: Applications and Implications for Policy and Practice Part II: Mixed Methods Study.

Cohn, D. (2015). Census Considers New Approach to Asking About Race - By Not Using the Term at All. Pew Research Center. Retrieved from http://www.pewresearch.org/fact-tank/2015/06/18/census-considers-new-approach-to-asking-about-race-by-not-using-the-term-at-all/

HUD. (2014). HMIS Data Collection Template for Project ENTRY – CoC Program.

HUD. (2016). 2014 HMIS Data Standards Data Manual . HUD.

Matt Warfield, B. D. (2016). How has the ACA Medicaid Expansion Affected Providers Serving the Homeless Population: Analysis of Coverage,Revenues, and Costs. Retrieved from http://kff.org/medicaid/issue-brief/how-has-the-aca-medicaid-expansion-affected-providers-serving-the-homeless-population-analysis-of-coverage-revenues-and-costs/

Prewitt, K. (2013, August 21). Fix the Census’ Archaic Racial Categories. The New York Times. Retrieved from http://www.nytimes.com/2013/08/22/opinion/fix-the-census-archaic-racial-categories.html?_r=0

Page 31: Dont Know Refused

Image Credits

Adams, S. Dilbert.com. Retrieved from http://dilbert.com/terms/.

Hargreaves, R. Mr. Men. Fabbri.

Sesame Street.

Page 32: Dont Know Refused

Questions?