program evaluation of the colorado consortium on differential response final … · 2019. 10....

Social Work Research Center | Colorado State University i

April 2014

Program Evaluation of the

Colorado Consortium on

Differential Response

Final Report Appendices

Social Work Research Center | Colorado State University ii

Program Evaluation of the

Colorado Consortium on Differential Response

Final Report Appendices

Prepared by:

Marc Winokur Rebecca Orsi

Helen Holmquist-Johnson

Social Work Research Center

School of Social Work

Colorado State University

Raquel Ellis George Gabel John Rogers

Summer Brenwald Marneena Evans

Prepared for:

National Quality Improvement Center on Differential Response in Child Protective Services

The recommended citation for this report is:

Winokur, M., Ellis, R., Orsi, R., Rogers, J., Gabel, G., Brenwald, S., Holmquist-Johnson, H., & Evans, M. (2014). Program evaluation of the Colorado Consortium on Differential Response: Final report. Fort Collins, CO: Social Work Research Center, School of Social Work, Colorado State University.

Social Work Research Center | Colorado State University iii

TABLE OF CONTENTS

Appendix A: Agency Response Guide ............................................................................................. 1

Appendix B: CONSORT Diagram...................................................................................................... 5

Appendix C: Instrument Development ........................................................................................... 6

Appendix D: Data Sources ............................................................................................................... 9

Appendix E: Family Exit Survey ..................................................................................................... 13

Appendix F: Case-Specific Report ................................................................................................. 18

Appendix G: Caseworker General Survey ..................................................................................... 24

Appendix H: Administrative Data and Survey Weighting Procedures .......................................... 30

Appendix I: Family Exit Survey Question Crosswalk ..................................................................... 35

Appendix J: Case-Specific Report Question Crosswalk ................................................................. 39

Appendix K: Study Logistics .......................................................................................................... 41

Appendix L: Colorado DR Model Fidelity Assessment Matrix ...................................................... 45

Appendix M: Case-Specific Report Tables .................................................................................... 51

Appendix N: Family Exit Survey Tables ......................................................................................... 77

Appendix O: Administrative Data Tables .................................................................................... 110

Appendix P: Survival Analysis Curves .......................................................................................... 127

Appendix Q: Cost Study Tables ................................................................................................... 135

Social Work Research Center | Colorado State University 1

Appendix A

AGENCY RESPONSE GUIDE

1. WHAT IS THE ALLEGED MALTREATMENT IN THIS REFERRAL? 2. IS ADDITIONAL INFORMATION AVAILABLE OR NECESSARY TO DETERMINE

AGENCY RESPONSE? a) Prior DHS (Department of Human Services) Involvement (7.202.4 E) b) Obtaining information from collateral sources, such as schools, medical

personnel, law enforcement agencies, or other care providers. (7.202.4 E) c) Criminal and/or Agency History d) Child Age and Vulnerability e) Protective Factors & Strengths

3. DOES THE ALLEGED MALTREATMENT MEET CRITERIA FOR AGENCY RESPONSE? The following definitions were taken from Colorado Children’s Code Title 19, 19-1-103: _Any case in which a child exhibits evidence of skin bruising, bleeding, malnutrition, failure to thrive, burns, fracture of any bone, subdural hematoma, soft tissue swelling, or death and either: Such condition or death is not justifiably explained; the history given concerning such condition is at variance with the degree or type of such condition or death; or the circumstances indicate that such condition may not be the product of an accidental occurrence. _Any case in which a child is subjected to unlawful sexual behavior. _Any case in which a child is a child in need of services because the child's parents, legal guardian, or custodian fails to take the same actions to provide adequate food, clothing, shelter, medical care, or supervision that a prudent parent would take. _Any case in which a child is subjected to emotional abuse. As used in this subparagraph (IV), "emotional abuse" means an identifiable and substantial impairment of the child's intellectual or psychological functioning or development or a substantial risk of impairment of the child's intellectual or psychological functioning or development. _Any case in which, in the presence of a child, or on the premises where a child is found, or where a child resides, a controlled substance, as defined in section 18-18-102 (5), C.R.S., is manufactured or attempted to be manufactured. _Any case in which a child tests positive at birth for either a schedule I controlled substance, as defined in section 18-18-203, C.R.S., or a schedule II controlled substance, as defined in section 18-18-204, C.R.S., unless the child tests positive for a schedule II controlled substance as a result of the mother's lawful intake of such substance as prescribed. _No, does not meet criteria (screen out/I&R only).

http://www.michie.com/colorado/lpext.dll?f=FifLink&t=document-frame.htm&l=jump&iid=COCODE&d=18-18-102&sid=c944bf2.6dc265b6.0.0#JD_18-18-102http://www.michie.com/colorado/lpext.dll?f=FifLink&t=document-frame.htm&l=jump&iid=COCODE&d=18-18-203&sid=c944bf2.6dc265b6.0.0#JD_18-18-203http://www.michie.com/colorado/lpext.dll?f=FifLink&t=document-frame.htm&l=jump&iid=COCODE&d=18-18-204&sid=c944bf2.6dc265b6.0.0#JD_18-18-204


4. DOES THE REFERRAL REQUIRE A RESPONSE? ___ Yes (continue onto #5)

PA5 – Child in Need of Protection PA4 – Youth in Conflict

___ No

Additional follow up needed? Provide or refer for additional services?

5. DETERMINE RESPONSE TIME (7.202.4 K)

1. Immediate and/or same day response is required when the report indicates that: a. Without immediate response, the child is in danger of moderate to severe

harm, or b. The child’s vulnerability or factors such as drug and alcohol abuse, violence,

isolation, or risk of flight from one county to another county or state, increase the need for immediate response. (If the report is received after hours and the time frame is immediate, the response needs to happen immediately and/or up to eight hours)

2. End of the third calendar day following receipt of the report when the report indicates that:

a. W/out a response within three days, the child is in danger of moderate to severe harm, or

b. The child’s vulnerability or factors such as drug and alcohol abuse, violence, isolation, or risk of flight from one county to another county or state, increase the need for intervention in the near future.

3. Within five working days from the date the report is received when the report indicates maltreatment or risk of maltreatment to a child and indicates an absence of safety concerns.


6. BASED ON THE ALLEGATION(S) IDENTIFIED IN THE REPORT, SELECT ONE OF THE FOLLOWING: (Reference Differential Response Alleged Maltreatment Guide)

1. HIGH RISK ASSESSMENT RESPONSE (SELECT REASON)

Mandatory Reason: o Allegation of serious harm o Allegation of sexual abuse o Suspicious child fatality or

homicide o Institutional referral o Allegation of egregious incident of

abuse or neglect

Discretionary Reasons: o Currently open investigation response o Frequent, similar, recent referrals o Violent activities in the household o Caregiver declined services in the past o Caregiver unwilling/unable to achieve safety o Past safety concerns not resolved o Previous serious child harm offenses o Credible RP alleges high safety concern o High child vulnerability o Substance Abuse not manageable through FAR o Drug exposure not manageable through FAR o Domestic Violence not manageable through

FAR o Court ordered investigation o FAR Eligible, approved exemption – staffing o Not in FAR County jurisdiction o Insufficient info to assess for FAR eligibility o Other (Describe):

2. THIS REFERRAL IS FAR ELIGIBLE.


DIFFERENTIAL RESPONSE ALLEGED MALTREATMENT GUIDE

Track assignment determined by presence of imminent danger, level of risk, number of previous reports, source of the report and / or presenting case characteristics such as type of alleged maltreatment and age of the alleged victim.

NEGLECT

High Risk Assessment Family Assessment Response

Failure to provide medical care in life endangering situations Child inadequately supervised, imminent danger Child abandonment Child exposed to Meth Manufacturing Child access to drugs Alleged PRAN assaults non-offending victim while holding child Weapons are being used in vicinity of child Child intervenes in an incident of DV and is injured Alleged DV PRAN has made suicidal or homicidal statements

Vulnerable child without supervision Medical diagnosis of failure to thrive Housing conditions pose health, safety or harm to children Domestic issues between adults in residence endanger safety and

welfare of child

Educational neglect Child born exposed to chemicals / drugs Caregiver involved in possession, use, sale of a controlled

substance in the presence of child

Caregiver driving under influence with child Child has a physical, mental or emotional condition requiring care

that is not being received

Failure to protect from conditions/actions which endanger child Child present or involved with parent committing a criminal act Items thrown in vicinity of child Child intervenes in an incident of DV and is not injured

PHYSICAL


Life threatening injury Report of non-accidental injury which requires medical attention Severe injury (i.e. broken bone, injury to head, torso, genitals) Third degree burns – non-accidental Physical punishment to child less than 3 y/o involving shaking,

throwing or hitting head or trunk

Excessive Physical Injuries Unreasonable confinement or restraint including tying, locking, caging,

chaining Child has injuries

Child fatality with other children living in home

Threatened physical abuse Present visible injury First or Second degree burns - non-accidental Report of non-accidental injury; not currently present Unexplained injury Excessive/extreme/severe punishment without injury.

SEXUAL


All allegations of intra-familial sexual abuse

EMOTIONAL MALTREATMENT


Child is restrained or restricted from leaving during the DV incident Unreasonable confinement or restraint including tying, locking,

caging, chaining, or unreasonable force or cruelty.

Verbal acts or omissions which have an observable adverse effect on the child (i.e. name-calling and language; constant yelling) unreasonable force or cruelty

Caregiver shows little or no attachment to child Child intervenes in an incident of DV and is not injured Child shows behavioral changes that appear to be related to DV

and is fearful of physical injury or death to self or others


Appendix B

CONSORT Diagram – Colorado Consortium on Differential Response

Assessed for eligibility (n = 10,799)

Excluded (n = 5,408) Mandatory investigation (n = 1,939) Discretionary investigation (n = 3,469)

Selected for data collection (n = 1,099) Administrative data analyzed (n = 3,194) Case Report – Analyzed (n = 871) Excluded from analysis (n = 228)

Non-response (n = 218) Case still open (n = 10)

Family Survey – Analyzed (n = 257) Excluded from analysis (n = 842)


Allocated to FAR (n = 3,428) Received allocated intervention (n = 3,194) Did not receive allocated intervention (n = 234)

Screened out (n = 103) Required investigation (n = 89) Wrong county referred (n = 41) No one participating as a child (n = 1)

Selected for data collection (n = 981) Administrative data analyzed (n = 1,802) Case Report – Analyzed (n = 801) Excluded from analysis (n = 180)


Family Survey – Analyzed (n = 206) Excluded from analysis (n = 775)


Allocated to IR (n = 1,963) Received allocated intervention (n = 1,802) Did not receive allocated intervention (n = 161)

Screened out (n = 75) Required investigation (n = 49) Wrong county referred (n = 36) No one participating as a child (n = 1)

Allocation

Analysis

Randomized (n = 5,391)

Enrollment


Appendix C

Instrument Development

Cognitive Testing

Colorado was given the task to revise the family exit survey by Walter R. McDonald &

Associates (WRMA). In September 2010, Westat’s Instrument Design, Evaluation and Analysis

(IDEA) Services, an expert team of questionnaire design and survey pretesting methodologists,

completed cognitive testing of the family exit survey. Twelve one-hour in-person interviews

were conducted in Colorado’s Jefferson and Larimer counties. Nine interviews were conducted

in Jefferson County on September 16, 2010 at a local church and three interviews were

conducted in Larimer County on September 17, 2010 at the Larimer County Department of

Human Services (LCDHS), Child, Youth and Family (CYF) division offices. For the Jefferson

County interviews, recruiting was done by the Parent Partner coordinator from Jeffco

Community Connections/Jefferson County Department of Human Services. Respondents were

recruited from the Parent Partner Program whose participants had an open child welfare case

that closed successfully (3 or more years prior) and now work as volunteers to mentor families.

In Larimer County, recruiting was done by the DHS CYF coordinator, who recruited respondents

whose cases had recently closed (less than a year prior). Respondents took the survey using pen

and paper while the interviewer administered probes concurrently. Respondents were given

$25 for participating. Two respondents in Larimer County were also compensated $15 for

transportation. The observations from the cognitive testing were shared with WRMA, in

addition to recommendations and suggested revisions for the family exit survey.

Respondents’ Comprehension

The survey overall was easy for respondents to read and there were few if any comprehension

problems. One respondent, who revealed her 4th – 8th grade reading level, commented that the

survey was easy to follow. Others noted that they have taken a lot of surveys through DHS and

this survey was particularly easy for them understand and complete.

Multiple Caseworkers

Some respondents had more than one caseworker assigned to them while their case was open.

One respondent thought she had at least 8 different caseworkers in the 6 years her case was

open. Others described having an “intake” caseworker, who may have been the first person

they met with, and an “ongoing” caseworker with whom they met subsequently, and their

experiences with the two differed. Respondents in these situations had difficulty determining

which caseworker to think about when answering the survey items.


RECOMMENDATION

In the survey introduction, include instructions about which caseworker to think about

when answering.

SUGGESTED REVISION

As mentioned in the letter, [agency name here] contacted you one or more times in the

last year concerning the well-being of a child. Please answer the following questions

about your experience with {AGENCY NAME} and with the {AGENCY NAME} caseworker

who first contacted you.

Caseworker versus Program

For some items, respondents wanted to answer for their experience with the program overall

rather than just the caseworker. This may have been because they felt the question did not

really apply to just the caseworker, or that the caseworker did not influence whatever aspect of

their experience the question is asking about.

RECOMMENDATION

The memo indicates for which items it may be more useful to use wording such as “the

program,” “your experience,” or “your case that just ended” instead of “your

caseworker.”

Difficulty Associating Help with Caseworker

Many respondents were reluctant to give the caseworker credit for the progress or success

they had made in improving their lives since their case first opened. Somewhat related to this,

some respondents had trouble associating the help they had received with their caseworkers.

Some had only been referred to services, but not actually provided them, by the caseworker

and others had sought them out on their own. Respondents often spoke of their own personal

agency to achieving success and felt discouraged that there was no opportunity to indicate that

on the survey.

RECOMMENDATION

This issue will be addressed with the suggested revisions to the survey introduction and

to those items that respondents felt lent themselves more to their overall experience

with the program while their case was open rather than the individual caseworker.

Survey Length

Although the interviewers did not time how long it took respondents to complete the survey

without stopping, there are at least two indicators that it does not require a large time

commitment. During the interviews respondents often moved quickly through the survey, with


interviewers struggling to keep up with them. In addition, the cognitive interviews were

scheduled for one hour, while most only took about 30 minutes, which included the

interviewers interrupting the respondents taking the survey to administer cognitive probes.

Based on this and past experience, the evaluation team estimated it would take respondents

10-15 minutes at most to complete the survey.

RECOMMENDATION

Besides indicating the estimated amount of time to complete the survey in the cover

letter, the survey should be formatted as a booklet or in some other way that visually

conveys its short length when respondents open the survey packet.

Lessons learned

During the pilot period, the evaluation team refined a procedure to encourage family

participation in the family survey. Using a weekly list of closed study cases, the evaluation

teams called families prior to mailing of the survey to confirm mailing address and identify the

need for a Spanish survey instrument. The evaluation team also called families after the mailing

of the survey to confirm receipt of the survey and incentive and to encourage completion and

return of the survey.

Incentive Study

After all standard operating procedures were followed for the family exit survey,

including any follow up responses, an incentive follow up was initiated for non-completes,

cases that did not receive a second survey, and non-refusals previously sent 6 weeks prior to

the current date. An experiment to determine the worthiness of this process was implemented

in October of 2010 through December 2010. It was determined that with an increase in

returned surveys that the Evaluation Team would proceed with this additional follow up. Thus,

the incentive process was initiated in January 2011 and continued through October 2012.

Eligible cases were assigned for the ongoing incentive weekly. The family exit surveys

designated as ongoing incentives were returned to a particular researcher at Westat for ease of

tracking. Returned surveys were receipted in the management system by receipting the case

and updating the case as complete. Respondents then received an additional $20 check for

returning their completed survey.


Appendix D

Data Sources

Administrative Data

The Colorado Statewide Automated Child Welfare Information System (SACWIS) is

known as Trails. The evaluation team worked closely with agency administrators, county and

state information technology staff, and QIC-DR staff to develop practices for extracting and

submitting data from Trails and for tracking families through the system over time.

The QIC-DR administrative data was compiled from seven sections of data from Trails:

1. IDs and Submission Dates of the Cases: this section includes the State ID and ends with Subdate. Variables in this section are primary IDs. These IDs match with the IDs used in

the other files delivered to the QIC-DR.

2. RCT Pathway Assignment and the RCT Report: variables in this section are designed to collect key information about the RCT pathway assignment, and events that occur

during the RCT report such as a pathway change, safety assessments, and transfer to

ongoing services. This section also includes variables for allegations of various different

forms of child abuse, as well as single variable to denote if any of the allegations were

substantiated/indicated.

3. Caregiver Characteristics: variables in this section are focused on the number of caregivers in the household and the demographic characteristics of the primary

caregiver.

4. Child Characteristics: this section collects information on the number of children in the household as well as demographic and removal information for up to 6 children. Seven

demographic and removal variables are repeated for each child. Children are to be filled

in DOB order unless there are more than 6 children. When there are more than 6

children we ask that information from the oldest five and the youngest child be used.

5. Case Characteristics: variables in this section include number of contacts, duration of contacts, and Core Services received.

6. Outcomes: this section has one variable for tracking the number of screened-in referrals prior to RCT pathway assignment. The remaining variables are designed to track the

next 10 re-referrals that occur after the RCT pathway assignment. This includes re-

referrals that come in while the RCT Report is still open. The same three questions are

asked for each re-referral resulting in 30 variables. This section also includes subsequent


(within 365 days from referral and any time after referral until the data collection end

date) assessments, founded assessments, D&N, removals, and traditional case opening.

7. Summary Data: the final section consists of two variables designed to collect in aggregate the number of referrals to ongoing services and whether an open case still

existed 365 days after RCT pathway assignment.

Case-specific Reports

The caseworker case-specific report was a closed-ended web-based survey that included

questions in the following five areas:

1. Case Characteristics: including whether the case was FAR or IR; whether the responding caseworker was the worker originally assigned to the case; how well the primary

caregiver spoke English; and number and types of contacts with the family.

2. Family Functioning Scale: for eight areas of need (material; substance abuse; physical health; mental health; parenting skills; domestic violence; education; and social

supports), the caseworker indicates whether need was present and if so, whether it was

addressed while the case was open and how much, if any, improvement there was in

the area.

3. Safety Threats: for five areas of threat (neglect or abandonment; physical, sexual, or emotional abuse; lack of supervision or proper care; damaging adult-child relationship;

and other threat), the caseworker indicates the threat level at case opening and case

closing; whether the threat was addressed and, if so by whom; and the reason why any

present threat was not addressed.

4. Services, Providers, and Supports: including whether information about or referral to services was given; whether family actually received services and, if yes, how soon after

initial report. If any referrals or services were given, worker indicates for each of nine

possible service areas whether referral or service was provided or in place at case

opening and family’s level of participation in each. Caseworker indicates whether

friends and relatives outside the household provided support, whether no-cost

community resources were used, and reasons why family may not have been fully

served. Caseworker checks all that apply of 19 service providers from which s/he helped

family obtain services and rates how well provided service matched family needs and

how effective services were in producing needed change.

5. Family Engagement: caseworker rates family at case opening and closing on extent to which they were cooperative, receptive to help, engaged, uncooperative, and difficult.


Caseworker General Survey

The caseworker case-specific report was a closed-ended web-based survey that included

questions in the following eight areas:

1. Tenure and Duties: including time in child welfare profession; CPS functions included in present position; caseload and number of which are FAR cases; and rating of whether

the introduction of FAR as affected caseload size, workload, and paperwork.

2. Skills: ratings on both interpersonal skills and case skills 3. Job Satisfaction: ratings on current level of job satisfaction and whether the

introduction of DR made it more or less likely that they would continue in this line of

work.

4. DR Knowledge and Attitudes: ratings on the degree to which DR affects child safety and comparisons on DR and IR on seven dimensions relating to family involvement.

Following this comparison, the respondent checked all that apply on a list of factual

statements about DR, then rated “the current” child welfare system on its effectiveness

in handling 10 common family issues. The respondent rated his or her own satisfaction

with FAR and reported on anything they thought is keeping FAR from working as well as

it should.

5. Training: the respondent indicated whether they have received and/or need training in six areas, including a general introduction to DR; core DR training; refresher of DR

concepts; individual training in areas such as coaching; engagement and interviewing;

and specialized training such as domestic violence.

6. Services: ratings on availability of 20 different services provided by community providers.

7. Demographic Characteristics: including age, sex, race, ethnicity, and education. 8. Values: comparisons on eight sets of statements to indicate values on the desired goals

of CPS process such as child safety and keeping the family together.

Family Exit Survey

The family exit survey was a closed-ended paper and pencil instrument that asks the

family respondent to use rating scales to answer questions in the following five areas:

1. Satisfaction: including satisfaction with the caseworker and the help received from the caseworker as well as likelihood of calling the caseworker if needs should arise in the

future.

2. Relationship with Caseworker: including affective response to caseworker’s first visit; number of times family met with the caseworker; how carefully the caseworker listened


and understood the family’s needs; whether the caseworker addressed all of the

respondents concerns, took the family’s opinions into consideration, was sensitive to

the family’s culture and values; whether the caseworker focused on family strengths as

well as weaknesses; and how easy it was to contact the caseworker.

3. Services and Needs: including whether the caseworker helped the family to receive a number of specific services; whether there were family needs for which the family did

not receive help; and whether the caseworker assisted in involving other family and

friends as support.

4. Family Outcomes: including respondent’s assessment of whether, as a result of work with the caseworker, the family is better off; respondent is a better parent, can better

provide for the safety of her or his children, and better able to provide necessities. The

family also rated the extent to which eight life stresses increased, decreased, or stayed

the same after experience with the caseworker.

5. Demographic Characteristics: including respondent’s education, sex, ethnicity, race, preferred language; and the household income. Respondent also reported on whether

caseworker spoke preferred language and services were offered in preferred language.


Appendix E

Family Exit Survey

As mentioned in the letter, Child Protective Services has contacted you in the past several months

concerning one or more children in your home. Please answer the following questions about your

experience with Child Protective Services and the caseworker who contacted you. If more than one

caseworker visited your home, please answer the questions about the person you saw the most.

Instructions:

Please use a blue or black pen to complete this form.

Mark ‘X’ in the response box to indicate your answer:

Option 1 Option 2 Option 3

1. How satisfied are you with the way you and your family were treated by the caseworker who visited your home?

Very satisfied Somewhat satisfied Not at all satisfied

2. How satisfied are you with the help you and your family received from the caseworker?

Very satisfied Somewhat satisfied Not at all satisfied

3. How likely would you be to call the caseworker or Child Protective Services if you or your family needed help in the future?

Very likely Somewhat likely Not at all likely

4. How did you feel after the first time a Child Protective Services caseworker came to your home? Check all that apply.

Relieved Respected Encouraged

Angry Worried Thankful

Hopeful Comforted Stressed

Afraid Disrespected Discouraged

X


5. About how many times did you or other members of your family meet with the caseworker?

1 2-5 6-10 More than 10

6. Overall, how carefully did the caseworker listen to what you and other members of your family had to say?

Very carefully Somewhat carefully Not at all carefully

7. Overall, how well do you feel the caseworker understood your and your family’s needs?

Very well Somewhat well Not at all well

8. Were there things that were important to you or your family that did not get talked about with the caseworker?

Yes

No

9. How often did the caseworker consider your opinions before making decisions that concerned you and your family?

Always Sometimes Never

10. Did the caseworker recognize the things that you and your family do well?

Yes

No

11. How easy was it to contact the caseworker?

Very easy Somewhat easy Not at all easy


12. Did you or your family get any of the following help or services during your experience with Child Protective Services?

Check all that apply.

Emergency shelter Car repair or transportation assistance

Housing assistance Food or clothing for your family

Money to pay your rent Appliances, furniture, or home repair

Help paying utilities Welfare/public assistance services

Medical or dental care for you or your family Any other financial help

Help for a family member with a disability Legal services

Assistance in your home, such as cooking or cleaning Help with child care or day care

Help getting mental health services Respite care for time away from your children

Help in getting alcohol or drug treatment Meetings with other parents about raising children

Parenting classes Help in getting into educational classes

Counseling services (individual, family, mental health) Help in looking for employment or in changing jobs

Domestic violence services Job training or vocational training

Education services

13. Was there any help that you or your family needed but did not receive?

Yes No

If yes, what?

___________________________________________________________________________

___________________________________________________________________________

14. Overall, are you and your family better off or worse off because of your experience with Child Protective Services?

We are better off We are the same We are worse off


15. Are you a better parent because of your experience with Child Protective Services?

Yes

No

16. Are your children safer because of your experience with Child Protective Services?

Yes

No

17. Are you better able to provide necessities like food, clothing, shelter, or medical services because of your experience with Child Protective Services?

Yes

No

18. What is your highest level of education?

Less than 8th grade

8th – 11th grade

High school diploma or GED

Some college or trade school

Two-year college degree

Four-year college degree

Some graduate school or graduate degree

19. What was your total household income last year?

$0 - $9,999

$10,000 – $19,999

$20,000 – $29,999

$30,000 – $39,999

$40,000 – $49,999

$50,000 – $59,999

$60,000 or more

20. What is your gender?

Male

Female


21. Are you of Hispanic, Latino, or Spanish Origin?

Yes (please specify) ______________________________________

No

22. What is your race? Check all that apply.

Black or African American

White

Alaska Native

American Indian

Asian

Native Hawaiian or other Pacific Islander

Other (please specify) ____________________________________

23. Were you offered services in your preferred language?

Yes – in English

Yes – in another language

No


Appendix F

Case-Specific Report

Family/Case Name: _______________________________________ Trails Referral ID: _____________

County: __________________________ Response Track: FAR IR

Part One

1. Was this: an Alternative Response case an Investigation Response case 2. Are you the original caseworker assigned to this family?

yes no 3. How well did the primary caregiver speak English? very well

well not well not at all

4. Number of contacts with family (estimate if necessary): # of contacts

a. Face-to-face meetings with members of the family? ________ b. Telephone contacts with members of the family? ________ c. Other contacts with family members (court visits, etc.)? ________ d. Contacts with others on behalf of this family? ________ e. Face-to-face contacts between other agency providers and family? ________

5. Family Functioning

Check all family needs present at case opening

Then for every need checked, complete (2) and (3)

Condition addressed while the case was

open?

Improvement (check one)

No Yes None Little Moderate Much

Material Needs (e.g., housing, food/clothing, income, employment, etc.)

Substance Abuse (e.g., alcohol, prescription drugs, illicit drugs, etc.)

Physical Health (e.g., adult or child disability, developmental delay, etc.)

Mental Health Parenting Skills/Discipline Domestic Violence Education (e.g., school attendance, progress, etc.)

Social Supports (e.g., extended family, friends, neighbors, etc.)


6. Threats to Child Safety:

Check all safety threats present in this case first.

Then for every threat checked, complete (2) and (3)

(2) Indicate whether level of safety threat was mild, moderate or severe.

(3) Was the safety threat addressed?

At first contact (circle)

At

Closure (circle)

Yes, by: No, because: Don’t know/

not sure

County staff

Private agency provider

Unpaid community resource

Family/ kin

Other Funds unavailable

Provider unavailable

Uncooperative family

Other

Neglect or abandonment (e.g., child lacked basic needs, the home was unsafe or unclean, medical or educational neglect, etc.)

mild mod sev mild mod sev none

Physical, sexual, emotional abuse

(e.g., excessive discipline, violence in the home, sexual or emotional maltreatment, etc.)


Lack of supervision or proper care

(e.g., child left unsupervised, burns, fractures, etc.)


Damaging adult-child relationship

(e.g., verbal or physical fights, rejection, etc.)


Other threat (specify) mild mod sev mild mod sev none

Social Work Research Center | Social Work Research Center 20

7. Was information about or referral to services given to the family?

yes no uncertain 8a. Were any services (traditional or non-traditional) or supports provided to this family (caregivers or children)?

yes no uncertain whether family actually received services

8b. If yes, how soon after the initial report date did the family receive services? within one week within two weeks within three weeks within four or more weeks family was not offered services uncertain

If you answered yes to either 8a or 8b, complete the chart on SERVICES TO FAMILY INSERT. If you answered no or uncertain to both of these questions, do not complete the chart.

9. Since the case opened, were relatives and friends outside the household involved in providing needed

support and/or assistance to this family?

not at all very little moderately extensively

10. Were no-cost neighborhood/community resources (i.e. churches) used to assist this family?

not at all very little moderately extensively

11. Check any of the following reasons why the family may not have been fully served:

size of worker caseload limited staff time to work with family other pressing cases on caseload problems beyond scope of CPS to remedy limited funds for needed services other____________________________

12. Family Characteristics during the First Meeting:

Rate the characteristics of the family members at the first time you met with them:

Very Moderately A Little Not At All

Cooperative Receptive to help Engaged Uncooperative Difficult


13. Family Characteristics during the Last Meeting:

If you met with members of the family more than one time, rate these characteristics the last time you met with them.

does not apply

Very Moderately A Little Not At All

Cooperative Receptive to help Engaged Uncooperative Difficult

If you did not have to complete SERVICES TO FAMILY INSERT, STOP HERE. Otherwise, please continue with Part Two.

Part Two

14. Did you help members of this family in obtaining services from any of the following? (check all that apply)

school neighborhood organization mental health provider alcohol/drug rehab agency/program MR/DD provider youth organization health care provider job service/employment security employment & training agency (JTPA etc.) legal services provider support group childcare/preschool provider/Head Start community action agency domestic violence shelter emergency food provider church or religious organization recreational facility (e.g. YMCA) neighbors/friends/extended family other


15. Overall, how well were the services that were actually provided matched to the service needs of the family? very well matched somewhat matched not very well matched not at all matched

16. Overall, how effective were the services provided to the family in solving their problems or in producing needed changes? very effective somewhat effective not very effective not at all effective


SERVICES TO FAMILY INSERT

The following is a list of services that are sometimes provided to families. 1) Place a check after any service to indicate:

(1) service provided during the case - services were provided to a family member(s) while the case was open and had not been in place at the time of the first visit.

(2) information/referral provided – service information was given or referrals were made. (3) service in place at start - services were in place at the time of first visit.

2) For any service received by the family, give us some idea of the level of services received or used from very little (1) to very much (5).

For each service check all that apply

(1) (2) (3) Level of participation or use by family (check)

Service provided

Info/ referral

provided

Service in place at start

Very little < --------------- > Very much

Services to address Material Needs

(e.g., help with housing payments, emergency shelter or food, TANF, employment assistance, etc.)

1 2 3 4 5 uncertain

Substance Abuse Services

(e.g., alcohol or drug abuse treatment) 1 2 3 4 5 uncertain

Health Services

(e.g., medical or dental care, etc.) 1 2 3 4 5 uncertain

Mental Health Services 1 2 3 4 5 uncertain

Parenting Classes 1 2 3 4 5 uncertain

Domestic Violence Services 1 2 3 4 5 uncertain

Educational Services 1 2 3 4 5 uncertain

Social Support Services

(e.g., marital/family counseling, support groups, etc.)

1 2 3 4 5 uncertain

Other (specify) 1 2 3 4 5 uncertain


County: _____________________

Primary Responsibility: Case Worker Supervisor Other: _______________

Fully Describe

Appendix G

Caseworker General Survey

Tenure and Duties 1. How long have you worked in child welfare? ___ years____ months

2. Check all work areas that are currently part of your job.

Intake Out-of-home placement/ custody cases Screening Family preservation/prevention services Case assessment– Investigative Response (IR) Adoption Case assessment– Alternative Response (AR) other _____________________________ Ongoing casework/ intact families

3. If you are a case worker:

3a. How many cases are on your current caseload? ___________

3b. How many of these are AR cases? ___________

4. If you are a supervisor, do you supervise any AR workers? Yes No

5. Has AR in any way caused an increase or decrease in your: large small no small large decrease decrease change increase increase

a. caseload size b. overall workload c. paperwork


Skills 6. Please rate each item by circling the number that best describes your perception of your skill level in

each area.

INTERPERSONAL SKILLS Basic Advanced a. Interviewing 1----2----3----4----5----6----7 b. Listening 1----2----3----4----5----6----7 c. Counseling 1----2----3----4----5----6----7 d. Non-verbal communication 1----2----3----4----5----6----7 e. Reasoning 1----2----3----4----5----6----7 f. Empathizing 1----2----3----4----5----6----7 g. Interpersonal relationships 1----2----3----4----5----6----7 h. Cultural sensitivity 1----2----3----4----5----6----7

CASE SKILLS i. Fact finding skills 1----2----3----4----5----6----7 j. Evaluating case facts 1----2----3----4----5----6----7 k. Gathering complete and quality information 1----2----3----4----5----6----7 l. Effectively having clients complete case plans 1----2----3----4----5----6----7 m. Decision making skills 1----2----3----4----5----6----7 n. Accuracy of judgments 1----2----3----4----5----6----7 o. Developing case plans with families 1----2----3----4----5----6----7 p. Involving families in the assessment process 1----2----3----4----5----6----7 q. Identifying client and family strengths 1----2----3----4----5----6----7 r. Connecting families with needed resources 1----2----3----4----5----6----7 s. Collaborative decision making with families 1----2----3----4----5----6----7

Job Satisfaction 7. Overall, how satisfied are you with your current child welfare job? very very dissatisfied satisfied

1 --- 2 --- 3 --- 4 --- 5 8. Has the introduction of AR made it any more or less likely that you will remain in this field of work? much no much less likely effect more likely

1 --- 2 --- 3 --- 4 --- 5


Alternative Response Knowledge and Attitudes 9. For cases that are appropriate for AR, in your opinion how does the AR approach compare to the IR

approach regarding child safety? AR much no IR much safer difference safer

1 --- 2 --- 3 --- 4 --- 5 10. In your view, what are the major differences between AR and IR in your county? much more no much more likely with difference likely with AR IR

a. Families receive services they need .............. b. Families receive services quickly ..................... c. Families referred to other resources or agencies in community ................................. d. Separate interviews of child and caregiver ....... e. Family members present at initial assessment f. Cooperation of caregivers/family members ...... g. Participation in decisions and case plans .........

11. In reviewing the following statements, please check all the statements that apply to AR: Check all that apply

a. Families assigned to AR may receive a formal investigation if they prefer. b. Families receiving AR may refuse services without consequence if there are

no major safety concerns. c. When families receive AR, no one is formally identified as a perpetrator. d. Families receiving AR are transferred to the IR pathway when safety issues make it

necessary. e. Families receiving AR are approached in a friendly, non-accusing manner. f. Families are consistently assigned to AR vs. IR pathways, based on clear criteria. g. When families receive AR, there is no formal determination of child maltreatment. h. AR is formalized in statute, policy, and protocols. i. All CPS cases are eligible for AR.

12. In your experience how effective is the current child protection system in working with client

families in which there is: not at all very effective effective a. drug abuse 1 ----- 2 ----- 3 ----- 4 ----- 5 b. alcohol abuse 1 ----- 2 ----- 3 ----- 4 ----- 5 c. domestic violence/intimate partner 1 ----- 2 ----- 3 ----- 4 ----- 5 d. extreme poverty 1 ----- 2 ----- 3 ----- 4 ----- 5 e. extreme child behavior problems 1 ----- 2 ----- 3 ----- 4 ----- 5 f. mental illness 1 ----- 2 ----- 3 ----- 4 ----- 5 g. mental retardation/developmental disability 1 ----- 2 ----- 3 ----- 4 ----- 5 h. extremely poor parenting skills 1 ----- 2 ----- 3 ----- 4 ----- 5 i. educational neglect 1 ----- 2 ----- 3 ----- 4 ----- 5 j. parent-child conflict 1 ----- 2 ----- 3 ----- 4 ----- 5


13. Overall, how satisfied are you with the AR program in your county? not at all completely satisfied satisfied 1--- 2 --- 3 --- 4 --- 5

14. Is there anything that is preventing the AR from working as well as you think it could or should be working?

Training 15. Indicate what type of training opportunities you have received regarding AR and training opportunities

you could benefit from Training Training No training needed

Received Needed or received a. General introduction/overview of AR b. Core AR Training c. Refresher of AR concepts d. Individual training (i.e. direct

coaching, mentoring, shadowing) e. Training on engagement and interviewing f. Specialized trainings (i.e. domestic

violence, mental health)

Services 16. Indicate how much you agree with each statement below, in terms of your experience with obtaining

services in the communities where you work. do not strongly agree agree

a. I can usually find services in my community that can help keep children safe in their home. 1 ----- 2 ----- 3 ----- 4 ----- 5

b. It is easy to work with most of the service providers in my community. 1 ----- 2 ----- 3 ----- 4 ----- 5 17. The following is a list of specific services and service providers. For each service listed below, please

indicate agreement with the following statement:

I have confidence that when a family has one of the following needs, these needs are able to be met by a local community provider: not confident very confident

a. child care/day care 1------2------3------4------5 b. early childhood services 1------2------3------4------5

c. respite care/crisis nursery 1------2------3------4------5 d. mental health services 1------2------3------4------5 e. substance abuse treatment 1------2------3------4------5 f. developmental disability services 1------2------3------4------5 g. medical services 1------2------3------4------5 h. dental services 1------2------3------4------5 i. transportation services 1------2------3------4------5 j. domestic violence services/shelter 1------2------3------4------5


k. food services/food pantry 1------2------3------4------5 l. housing assistance 1------2------3------4------5 m. utilities & other household assistance 1------2------3------4------5 n. employment services 1------2------3------4------5 o. adult education/vocational services 1------2------3------4------5 p. parenting classes, household management 1------2------3------4------5 q. youth, recreational activities 1------2------3------4------5 r. legal services 1------2------3------4------5 s. support groups (e.g. parents anonymous) 1------2------3------4------5 t. mentoring 1------2------3------4------5 Demographic Characteristics 18. What is your age? ____ (years)

19. What is your gender? Male Female

20. Are you of Hispanic, Latino, or Spanish Origin?

Yes (please specify) ______________________________________ No 21. What is your race? (check all that apply) Black or African American Asian (please specify) ____________________________________

Native Hawaiian or other Pacific Islander Alaska Native White

American Indian Other (please specify) ____________________________________

22. What is the highest educational level you have attained?

Less than high school education High school graduation (or GED) 1-2 years college(no degree) Community college associate degree 3-4 years college(no degree) Bachelors’ degree Graduate study(no degree) Master’s degree Doctoral degree


23. In the following items you will be presented with a pair of statements. We want you to choose between them. We understand that you might endorse both statements but try to choose the statement that best reflects your general work focus and beliefs. Indicate your preference by circling A or B. You will see a statement more than once, but each pairing is different. There are no right or wrong answers.

Please rate the strength of your preference on the following scale of one (Very Weak) to five (Very Strong).

Items Which statement? Strength of preference?

a. Work should be focused on keeping the family together.

A

1 2 3 4 5 Child protection workers should be willing to be an advocate for the child.

B

b. The client is the child and all other work is secondary.

A

1 2 3 4 5 Work should be focused on keeping the family together. B c. Work should be focused on protecting the child.

A

1 2 3 4 5 Work should be focused on keeping the family together. B d. Families are the best place for children to achieve their full

potential. A 1 2 3 4 5 There is a need to ensure the physical and emotional well being of all children. B

e. Children’s rights should be safeguarded so they achieve

their full potential. A 1 2 3 4 5 The family’s right to guide the development of their children should be safeguarded. B

f. Families are the best place for children to achieve their full

potential. A 1 2 3 4 5 The state has a responsibility to protect children. B

g. There is a need to ensure the physical and emotional well

being of all children. A 1 2 3 4 5 The state should not be responsible for families or their children. B

h. Families are the best place for children to achieve their full

potential. A 1 2 3 4 5 Children’s rights should be safeguarded so they achieve their full potential. B

THANK YOU FOR YOUR TIME!


Appendix H

Administrative Data and Survey Weighting Procedures

Calculation of the survey weights for data analysis The Colorado Consortium on Differential Response evaluation involved randomly assigning eligible cases to either FAR or IR follow-up, collecting administrative data for all cases, and follow-up data for a subset of cases. The follow-up data included case specific information provided by case workers and family exit survey responses. Survey weights were prepared for the analysis of the administrative data, case reports, and the family exit survey. The analysis compared outcomes between the FAR and IR approaches. The project covered five counties in Colorado. The families were randomly assigned to FAR or IR response by the counties. The probabilities of assigning a case to FAR or IR differed among counties and over time. The administrative weights adjusted for the differing probabilities of assignment and the changing assignment probabilities over time. A subset of cases was randomly selected for follow-up data collection. The probability of selecting a case for follow-up depended on the county, the assignment, and on time. The survey weights adjusted for the differing probabilities of selection for follow-up data collection. However, because case reports were received for only 80% of cases and family surveys were received for only 22% of cases selected for follow-up, the survey weights were adjusted for non-response creating a set of weights for the analysis of the case reports and another set of weights for the analysis of the family survey data. The analysis weights can be thought of as the number of similar cases represented by the responding case. The weights were adjusted to sum to the number of eligible cases assigned in the study (4,996). The weights were used in programs designed to use survey weights and calculating estimates from survey data. Calculation of weights for the administrative data The random assignment for FAR and IR and the random selection of cases for follow-up was implemented through the WRMA randomizer. In order to balance workloads, the randomizer used block randomization to sequentially assign cases such that, at any time, the fraction of cases assigned to FAR (or IR) was close to the target fraction. This was achieved by setting the block length (NN) and the number of cases within the block (NA) to be assigned to FAR. Within a block of NN sequential cases, the probability of assignment to FAR is NA/NN. To make the outcomes harder to predict, the length of the blocks was randomized to be either 1 or 2 times


NN (while keeping the probability of assignment constant). The block randomization algorithm was also used to select cases for follow-up. Table 1 shows the probability of assignment to FAR and the probabilities of assigning a FAR or IR case for follow-up data collection, by county and time period. The probabilities were set to get approximately the same number of FAR or IR cases selected for follow-up. The assignment probability for each case was determined based on the assignment dates and times stored in the output file from the randomizer (obtained from WRMA). Table 1. Assignment probabilities by county and time period

County Start Date

Probability of FAR

assignment

Probability of follow-up selection

for FAR cases

Probability of follow-up selection

for IR cases

Arapahoe 10/1/2010 50.0% 34.8% 34.8%

Arapahoe 1/7/2011 60.0% 29.4% 42.9%

Arapahoe 12/1/2011 70.0% 25.0% 57.1%

Arapahoe 1/3/2012 80.0% 22.2% 85.7%

Arapahoe 2/1/2012 90.0% 20.0% 100.0%

Fremont 10/1/2010 50.0% 100.0% 100.0%

Fremont 12/1/2011 75.0% 66.7% 100.0%

Garfield 10/1/2010 50.0% 100.0% 100.0%

Garfield 12/1/2011 75.0% 66.7% 100.0%

Jefferson 10/1/2010 50.0% 40.0% 40.0%

Jefferson 12/1/2011 63.6% 30.8% 55.6%

Jefferson 1/3/2012 75.0% 26.7% 80.0%

Jefferson 2/1/2012 85.7% 23.1% 100.0%

Larimer 10/1/2010 80.0% 25.0% 100.0%

Larimer 12/1/2011 90.0% 22.2% 100.0%

For all combinations of county and start date, the fraction of cases assigned was very close to the target fraction shown in Table 1. Based on a chi-square test, the differences between the target probabilities and the fractions assigned were not at all close to being significant (in part due to the blocking used in the randomization). The cases randomized to FAR represented a random sample from the population. Call this the FAR sample. For that sample, the probability of selection is shown in the third column of Table 1. Using P(.) to indicate the probability of an event, the probability of a case being selected for FAR is ( ). The base weight (the weight before any non-response adjustment) is the inverse of the probability of selection. Thus the base weight for the FAR sample is:


( )

The weighted estimates for the FAR sample represented the population of all cases assigned during the study (i.e., the outcome for the population if all cases had been assigned to FAR follow-up). The cases randomized to IR also represented a random sample from the population. Call this the IR sample. Since cases not assigned to FAR were assigned to IR; the base weight for the IR sample is:

( )

The weighted estimates for the IR sample represented the population of all cases assigned during the study (i.e., the outcome for the population if all cases had been assigned to IR follow-up). Across all five counties, there were 1,802 cases assigned to IR and 3,194 cases assigned to FAR, for a total of 4,996 cases. The sum of the FAR weights (and the IR weights) for the administrative data are close to but not equal to 4,996 (they would equal 4,996 if the probability of assignment was fixed). The administrative weights were rescaled to sum to 1,802 for the IR cases and 3,194 for the FAR cases so that counts for the FAR and IR samples would sum to known totals and could be used for population estimates. After the rescaling, it is still true that the weighted estimates of means and percentiles for the FAR sample represented the population as if all cases had been assigned to FAR follow-up, and similarly for IR cases. Thus analysis of differences between FAR and IR cases estimated the difference between using FAR for all cases versus using IR for all cases. Calculation of weights for the cases selected for follow-up The results from the randomizer were entered into the Trails system. In some cases, errors were made when entering the randomizer results into Trails. These are called “randomizer data entry errors.” Ultimately, whether follow-up was attempted was determined by the information in Trails rather than the original randomizer selection.

In addition, there were 18 IR cases in Jefferson County for which no follow-up selection was entered into Trails. Follow-up was not attempted for these “missing follow-up” cases. Based on the data (excluding the 18 cases with no Trails Follow-up entry), the probability of an incorrect entry into Trails was 2.72% for cases randomized for follow-up and 1.29% for cases randomized for no follow-up. These two percentages are significantly different at the 5%


significance level. The probability of a correct entry into Trails is 1 minus the probability of an incorrect entry into Trails. The probability of “follow-up” designation in Trails (for all but the 18 cases with missing follow-up information) was calculated using the following formula: ( )

( ) ( ) ( ) ( )

In this equation, the randomizer probability of “follow-up” for the FAR sample comes from the fourth column in Table 1. The randomizer probability of “follow-up” for the IR sample comes from the fifth column in Table 1. The probability of no “follow-up” from the randomizer is 1 minus the probability in Table 1. With the exception of 18 cases, the base weight for the follow-up data collection is:

( )

The SurveyWeight for the 18 IR Cases in Jefferson County with no Trails entry sums to 50. The SurveyWeight for the other cases in Jefferson County sums to 1700. To adjust for the 18 Jefferson County cases with no survey entry in Trails, the SurveyWeight for IR cases in Jefferson County were increased by the factor (1700+50)/1700. This adjustment is equivalent to assuming the 18 cases are similar to the other IR cases in Jefferson County. As with the administrative weights, to provide consistent estimates for the population, the follow-up survey weights were rescaled to sum to 1,802 for the IR cases and 3,194 for the FAR cases. Adjusting the survey weights for non-response The adjustment for non-response used the following steps:

For cases designated for follow-up in Trails, define a non-response variable: 1 for respondents and 0 for non-respondents.

Starting with a set of candidate variables from Trails, use stepwise logistic regression to identify predictors of non-response. The subject-specific predicted probabilities of non-response were saved from the final stepwise model.

Sort the predicted probabilities and classify then into non-response adjustment groups. Within each non-response adjustment group, calculate the response proportion, and

divide by the non-response proportion to calculate the non-response adjusted weight.


Adjust the weights to have a total of 1,802 for IR cases and 3,194 for FAR cases. For the family survey, the overall response rate was 22.5%. Based on the stepwise regression, the probability of non-response depended only on whether there was a neglect allegation (variable = allegation_neglect). As a result, two non-response adjustment groups were used, one for each level of allegation_neglect (Yes/No), with response rates of 20.4% and 27.7% respectively. For the case reports, the overall response rate was 81.3%. The logistic regression step identified seven predictors of response for the case report, with variable names allegation_neglect_env_injurious, County, trails_track, number_children, TNumDatsf2f, number_safety_concerns, and AssignDate. The predicted probabilities of non-response were sorted and divided into 10 equal sized groups of 208 cases. The response rates in these groups ranged from 56.7% to 91.8%.


Appendix I

Family Exit Survey Question Crosswalk

Research Questions Related Survey Questions Preparation for Analysis

Do families assigned to FAR receive different services and in different quantities than do families assigned to IR?

Question 12

Any Services Received? If respondent did not select any services, they were given a 0 (0 = no) If respondent selected at least one service, they were given a 1 (1 = yes). Number of services received? sum of all services received in question 12 Types of services received (Educational, Financial, Mental Health, Parenting, Medical). 1. Recode the categories based on the condensed list in the family exit survey demographics document 2. Compute sum of the condensed service categories for each respondent

Are FAR caseworkers more likely to be perceived as having style and skills reflecting family-centered practice than are IR caseworkers?

Family-centered practice: Questions 6- 11

1. Reverse score question 8 2. Recode question response options for the questions below as “positive” or “less positive” responses Questions 6 and 7 Very carefully/well = 1 Somewhat or not at all very carefully/well= 0 Question 9 Always = 1 Sometimes or Never = 0 Question 11 Very easy = 1 Somewhat or not at all easy = 0 Question 23 Yes in English or Yes in another language =1 No = 0 Step 3: Compute a score for each respondent by computing a sum of the recoded questions 6- 11 and 23 Factor analysis confirmed scale is measuring a distinct construct; scale has good reliability (.84)



Are family satisfaction with the caseworker for families assigned to FAR different from families assigned to IR?

Family satisfaction: Questions 1, 2, 3

1. Recode question response options for the questions as “positive” or “less positive” responses Very satisfied/likely =1 Somewhat satisfied/likely or not at all satisfied/likely = 0 2. Compute a score for each respondent by computing a sum of the recoded questions 1-3 Factor analysis confirmed scale is measuring a distinct construct; scale has good reliability (.81)

Do families assigned to FAR believe that more of their needs were met satisfactorily than do families assigned to IR?

Question 13

This negatively worded question was recoded to make it a positive item.

Do families assigned to FAR perceive more improvements in family well-being than do families assigned to IR?

Family well-being: Questions 14, 15, 17

1: Recode question response options for the questions below as “positive” or “less positive” responses Question 14 We are better off = 1 We are the same or we are worse off =0 Questions 15 and 17 Yes = 1 No = 0 2: Compute a score for each respondent by computing a sum of the recoded questions 14, 15 and 17 Factor analysis confirmed scale is measuring a distinct construct; scale has fair reliability (.69)

Do families assigned to FAR perceive more improvements in child safety than do families assigned to IR?

Question 16

No preparation needed



Do families assigned to FAR have different feelings after the first time a CPS caseworker came to their home than do families assigned to IR?

Feelings: Question 4 1: Create a scale by assigning positive feelings a“1” and less positive feelings a “-1” to the feelings in the list Less positive feelings = angry, afraid, worried, disrespected, stressed, discouraged More positive feelings = relieved, hopeful, respected, comforted, encouraged, thankful 2: Compute a score for each respondent by computing a sum of the recoded feelings Factor analysis confirmed scale is measuring a distinct construct; scale has good reliability (.80)

Do families assigned to FAR experience more face-to-face meetings with the caseworker than do families assigned to IR?

Question 5 Create a 3 level categorical variable by collapsing the “6-10” and “more than 10 categories” into one category labeled “6 or more”

Are family characteristics associated with perceptions of family satisfaction, family-centered practice, family well-being, and child safety?

1) Highest level of education, total household income, and race/ethnicity (Questions 18, 19, 21 and 22) 2) Family satisfaction: Questions 1, 2, 3 Family-centered practice: Questions 6-11 Family well-being: Questions 14, 15, 17 Family safety: Question 16

1) Highest level of education: condense categories (see demographic categories document) Total household income: condense categories (see demographic categories document) Race: Using questions 21 and 22, derive a variable with the 2 categories below: a) White only (meaning the respondent only chose White as a response to question 22) b) “Other” (meaning the respondent chose any other category or combination in question 22 and/or chose “White” and at least one other race; and/or also selected “yes” for question 21) 2) Same as above



Are family characteristics associated with having needs met satisfactorily?

1) Highest level of education, total household income, and race/ethnicity (Questions 18, 19, 21 and 22) 2) Met needs: Question 13

1) Same as above 2) Same as above

Is there a difference in perceptions of family satisfaction, family-centered practice, family well-being, and child safety for families who completed the survey earlier in the study than for families who completed the study later in the study?

1) Randomization date 2) Family satisfaction: Questions 1, 2, 3 Family-centered practice: Questions 6-11 Family well-being: Questions 14, 15, 17 Family safety: Question 16

1) No preparation needed 2) Same as above


Appendix J

Case-Specific Report Question Crosswalk


1. Are there differences in the number of face-to-face and telephone contacts between families assigned to FAR and IR?

Question 4 No preparation needed.

2. Are there differences in the addressing of and improvement in family functioning between families assigned to FAR and IR?

Question 5 Code each of the eight presenting needs as yes/no. For cases with presenting need = yes: 1. Code ‘addressed need’

as yes/no; 2. Code ‘improvement’ as

low=none/little and high=moderate/much

3. Are there differences in the addressing of and improvement in safety threats between families assigned to FAR and IR?

Question 6 Recode each of five safety threats as yes/no. For cases with safety threat = yes: 1. Code safety level at

first/closure as none=0, mild=1, moderate=2 and severe=3

For cases with safety threat = yes: 1. Code ‘addressed threat’

(any means) as yes/no/don’t know

4. Are there differences in the provision of service referrals and actual services between families assigned to FAR and IR?

Questions 7, 8a None, except treat uncertain responses as missing.



5. Are there differences in the timing of services received between families assigned to FAR and IR?

Question 8b Recode response options: within 3 weeks = within 1 or within 2 or within 3 and more than 3 weeks = within 4 or more; Treat not offered or uncertain as missing.

6. Are there differences in the provision of support and resources by relatives/friends or by the community between families assigned to FAR and IR?

Question 9, 10 Recode response options as low = not at all/very little and high = moderately/extensively.

7. Are there differences in positive engagement with the caseworker between families assigned to FAR and IR?

Questions 12, 13

Recode all ten (2 time periods and 5 characteristics) response options as low = not at all/a little and high = moderately/very

8. Are there differences in matching of service to needs and in effectiveness of services between families assigned to FAR and IR?

Questions 15, 16 Recode both question response options as low = not at all/not very and high = somewhat/very.


Appendix K

Study Logistics

Institutional Review Board

The evaluation team submitted protocols to the CSU and Westat Institutional Review

Boards in 2010. CSU gave IRB approval on November 23, 2010. On November 18, 2010 the

Westat IRB and the CSU IRB agreed to an IAA, which assigned CSU as the IRB of record. A

reapplication with full CSU IRB board was completed after each year of the study. An

amendment was completed during the first reapplication to gain approval for the revised focus

group protocol. The reapplications also included adverse event reports pertaining to child

fatalities, which were reviewed by the IRB board and deemed to be not research related.

Agreements

A data sharing agreement was completed between CDHS and American Humane

Association/Kempe before the study began to cover the sharing of data between CDHS, CSU,

and the cross-site evaluation conducted by WRMA. An evaluation agreement between the

evaluation team from CSU and Westat with each participating county also was completed (see

appendix J).

Workgroups

The project formed multiple workgroups very early in the process to determine the

practices, policies and philosophical underpinnings of DR as it would be practiced in Colorado.

These groups met from the beginning of the project, and more were formed over time. The

workgroups were instrumental in enhanced communication between and among counties,

particularly during implementation. They provided a venue for problem solving and sharing of

successes and challenges. The practice based workgroups were screening and referral, FAR

practice, learning development. The evaluation specific workgroups were data and cost study,

while the DR leadership team had oversight for both practice and evaluation issues.

Data Workgroup – the evaluation team facilitated monthly meetings of the data workgroup,

which was charged with ensuring that Trails has the capabilities and functionalities to capture

all data elements related to the project and that data assurance and survey reports were

developed. The workgroup included the evaluation director (ED), Westat evaluation staff,

project director (PD), state and county data and practice experts, State Administrative Review

Division (ARD) representatives, and the Trails manager.


Cost Analysis Workgroup – the evaluation team facilitated meetings of the cost study

workgroup, which was charged with designing the data collection and analysis plan for the cost

study. The workgroup included the ED, PD, data and practice experts, and financial experts

from CDHS and the five counties.

DR Leadership Team – the workgroups were overseen by the leadership team, which was

comprised of the PD, ED, administrators from all five counties, the evaluation team, the

Colorado Disparities Resource Center (CDRC), and various Colorado Division of Child Welfare

(DCW) staff. The team met monthly to assist in the evaluation process, specifically reviewing a

monthly ad hoc report generated by the Jefferson County data analyst, which allowed for

ongoing monitoring of eligibility, assignment, and outcome information.

Evaluation Partnership Agreement

Colorado Consortium on Differential Response Evaluation Partnership Agreement

Arapahoe, Fremont, Garfield, Jefferson, and Larimer County Department of Human Services,

Colorado Department of Human Services, Social Work Research Center at Colorado State

University, and Westat

The following information will serve as our partnership agreement. This documentation of our

individual and joint responsibilities will aid everyone involved, and was compiled from our joint

conversations by phone and through email. This agreement describes our common objective to

better understand the impact of differential response practices in child protective services as it

relates to child, and family safety and wellbeing.

Overview: In October 2008, the U.S. Department of Health and Human Services,

Administration for Children and Families, Children’s Bureau awarded a grant to the American

Humane Association, Walter R. McDonald and Associates, Inc., and the Institute of Applied

Research to operate the National Quality Improvement Center on Differential Response in Child

Protective Services.

Colorado was selected to participate through a competitive process and the counties

participating include Arapahoe, Fremont, Garfield, Jefferson and Larimer counties. The research

design includes a random assignment methodology with each county. Using Trails and an

external WRMA website, the demonstration will randomly assign eligible families to receive

either a traditional investigation response (IR) or family assessment response (FAR). The Social

Work Research Center at Colorado State University and Westat will conduct a rigorous program


evaluation to determine whether the Colorado Consortium for Differential Response (CCDR) is

an effective approach for child protective services. The evaluators will conduct process,

outcome and cost evaluations to study the impact and implementation of differential response.

Responsibilities of Partners: Major agreement issues are listed below for both partners.

Additional information and details are included in the attachment entitled

The Evaluation Team will:

1. Assign a Site Liaison to assist with data collection, scheduling and other evaluation

procedures

2. Provide sample scripts to caseworkers and supervisors for the county to use when

notifying families of the study

3. Work with the state and counties to modify the letters accompanying the family survey.

4. Make every effort to ensure the monitoring of the sampling ratio so that it does not

affect caseload size. We have requested that randomized blocking be used to balance

the families receiving support.

5. Make every effort to reduce the burden of the evaluation on the Counties by, for

example, helping with administrative tasks that are directly caused by the DR project,

and taking majority of the responsibility needed to keep in contact with the families

participating in the study (see #5 below).

6. Ensure that all information collected in the course of the study will be kept confidential.

7. Ensure that all evaluation activities are reviewed by relevant institutional review boards

and that all activities comply with federal requirements for protection of human

subjects, as required by the relevant IRB’s.

8. Ensure that families are notified about the evaluation in materials provided at the time

of survey and that passive consent is obtained.

9. Provide participating families with a prepaid monetary incentive of $10 cash.

10. Notify counties when the sample size for random assignment is reached and will work

with the county on a transition plan.

11. Produce an Acceptance Report about whether randomizer reports were collected

properly.

County Departments of Human Services will:

1. Assign a General Liaison to the data committee. Duties include:

working with the Evaluation Team to monitor evaluation procedures setting up interviews with staff and other stakeholders for yearly site visits


reviewing reports and tracking missing information (Discrepancy Reports) identifying case workers that were assigned to specific cases help to identify the address and telephone number along with other missing

information

2. Agree to the random assignment of families as described in the case flow (attached)

3. Agree not to enroll IR cases in the FAR track or place families on a waiting list for the

track. (Families assigned to FAR can be re-tracked to IR within the first 30 days)

4. Adhere to the attached case flow and contact the Evaluation Team if issues arise

5. Report any errors/problems around the random assignment procedures to the

Evaluation team immediately

6. Follow procedure for the randomizer and train workers for new Trails procedures

7. For study cases, workers will update family addresses (as consistent with Trails prompt)

8. Remind staff to inform the family about the evaluation at the time of closing (using the

script developed by the Evaluation Team)

9. Encourage caseworkers/supervisors to participate in the caseworker survey, sharing

their perceptions of the initiative

10. Ensure that caseworkers complete the case-specific report for selected families

11. Inform the liaison about when a new worker is assigned and provide the contact

information for workers (when needed)

12. Participate in evaluation training and make sure that new staff participate in evaluation

training

13. Provide copies of county materials developed for the CCDR

program evaluation of the colorado consortium on differential response final … · 2019. 10....

Documents