Massachusetts Healthy Families Evaluation-2 (MHFE-2)A Randomized Controlled Trial

of a Statewide Home Visiting Program for Young Parents

ANNUAL PROGRESS REPORT TO THE MASSACHUSETTS CHILDREN’S TRUST FUND

FISCAL YEAR 2010

Prepared by:

Jessica Goldberg, PhD, Project DirectorFrancine Jacobs, EdD, Co-Principal InvestigatorJayanthi Mistry, PhD, Co-Principal Investigator

Ann Easterbrooks, PhD, Co-Principal Investigator

Eliot-Pearson Department of Child DevelopmentDepartment of Urban and Environmental Policy and Planning

Tufts University

Medford, MassachusettsNovember 2010

2

TABLE OF CONTENTS

......................................................................Massachusetts Healthy Families Evaluation Team 3

.................................................................................................................Background and Context 4

..................................................................................Healthy Families Massachusetts (HFM) 4

Massachusetts Healthy Families Evaluation (MHFE) Framework: The Five-tiered ......................................................................................................................................Approach 5

.............................................................................................................MHFE-2 Evaluation Design 5

.....................................................................................................Recruitment and Enrollment 6

............................................................................MHFE-2 Participants and HFM Utilization 9

..............................................................................................................................Data Sources 11

....................................................Data Collection Activities through June 30, 2010 (FY10) 13

...........................................................................................Preliminary Data Analysis Activities 13

.....................................................................Establishing HVS/RIO Equivalence at Time 1 14

..................................................................................Creating Community Context Profiles 15

..............................................................................Evaluation Timeline and Activities for FY11 19

....................................................Publications, Presentations, and Graduate Student Projects 20

................................................................................................Published Papers and Reports 20

..................................................................................................Presentations and Testimony 20

............................................................................Unpublished Student Papers and Projects 21

...........................................................................................................................Student Grants 22

3

MASSACHUSETTS HEALTHY FAMILIES EVALUATION TEAM

Co-Principal Investigators Ann Easterbrooks, PhD Francine Jacobs, EdD Jayanthi Mistry, PhD

Project Staff Meg Brady, MA, Research Coordinator John Boisvert, BA, Project Administrator Jana Chaudhuri, PhD, Data Analyst Cynthia R. Davis, PhD, Data Analyst Jessica Goldberg, PhD, Project Director

Interviewers and Research Analysts Alex Chewning, MA Mariah Contreras, BA Akira Gutierrez Diaz, BA Jessica Dym Bartlett, MA, MSW Claudia Miranda Julian, MS Maryna Vashchenko, MA Jessica Greenstone, MA Jen Elise Prescott, BA

4

For the past decade, our team of independent researchers from the departments of Child Development and Urban and Environmental Policy and Planning at Tufts University (the Massachusetts Healthy Families Evaluation, MHFE) has been contracted by the Massachusetts Children’s Trust Fund (MCTF) to evaluate the Healthy Family Massachusetts newborn home visiting program (HFM). The first cohort evaluation (the Massachusetts Healthy Families Evaluation [MHFE-1]) was completed in 2005; the second cohort evaluation (MHFE-2) began in 2007.

This second annual MHFE-2 report, produced for the Massachusetts Children’s Trust Fund, details data collection and analysis activities conducted between July 1, 2009 and June 30, 2010 (Fiscal Year 2010 [FY10]). The first annual report (for FY09; released 10/09), describes data collection and analysis activities through June 30, 2009; it also provides a comprehensive overview of the MHFE-2 design and implementation process. While this current report includes brief summaries of selected evaluation design components, please refer to the FY09 report for a comprehensive overview of the study.

This Year 2 report includes:• A brief overview of HFM and its evaluation;• A summary of evaluation activities completed through the end of FY10; • Analysis of Time 1 data, including:

• descriptive analyses for the entire sample,• equivalence analyses between program and control group, and• preliminary analysis of community context data; and

• Project-related publications and presentations.

BACKGROUND AND CONTEXT

Healthy Families Massachusetts (HFM)HFM is a comprehensive, voluntary, newborn home visiting program for all1 first-time parents ages 20 and under in the state of Massachusetts. Based on the Healthy Families America (HFA) model for home visiting, Healthy Families Massachusetts (HFM) provides parenting support, information, and services to young parents, beginning prenatally or until the child turns one, and continuing until the child’s third birthday. Since its inception, HFM has provided services to over 26,000 families. The stated goals of the program are:

To prevent child abuse and neglect by supporting positive, effective parenting; To achieve optimal health, growth, and development in infancy and early childhood; To encourage educational attainment, job, and life skills among parents; To prevent repeat pregnancies during the teen years; and To promote parental health and well-being.

HFM program services include home visits, goal-setting activities, group-based activities, and linkages and referrals to other resources. Eligible participants can receive services from the time they find out they are pregnant to when their child turns three years old; the potential enrollment period for an HFM participant, then, is three years and eight months.

1 HFM is a universal program, meant to serve every first-time parent under 21 in Massachusetts, and enrolls participants on a first-come, first-serve basis (unlike other programs modeled on Healthy Families America, which target “high-risk” families only). However, HFM has never had sufficient funding to meet this goal of universal services; in FY02, which represents the height of funding to date, HFM had enough funding to serve only 60% of eligible families. In subsequent years, HFM has only been funded to serve 40% of new eligible families each year.

5

Massachusetts Healthy Families Evaluation (MHFE) Framework: The Five-tiered Approach Our evaluation of HFM is rooted in Jacobs’s Five-Tiered Approach to evaluation,2 a developmental model that moves evaluation activities from a primary focus on descriptive and process-oriented information to an emphasis on program effects. Tier One activities produce needs and demand assessments, and usually are conducted prior to the program’s implementation. Evaluation activities at Tiers Two and Three are directed at program processes: they describe program staff, services, clients, and costs; examine program implementation compared to model standards; and provide feedback to programs for improvement. Tiers Four and Five focus on outcome evaluation activities, assessing the extent to which a program is meeting its short-term and long-term goals. The primary difference between Tier Four and Tier Five is the use of an experimental design in Tier Five; when such scientific rigor is possible, researchers are more confident that changes they observe in participants are the result of the intervention being studied.

MHFE-1 focused on Tiers Two, Three, and Four: program monitoring and accountability, quality review in relation to model and program standards, and measurement of outcomes. It employed a quasi-experimental design, relying on sources of comparison data that included state and nationwide historical data on key indicators and extant data from studies of adolescents and young parents. Using a mixed methods approach, data were collected from a sample of 361 HFM participants, at six-month intervals, at four different time points over a period of 18 months. An ethnographic substudy, conducted in three communities, explored participants’ beliefs about parenting, childrearing, and help-seeking, and the extent to which HFM services were consonant with those beliefs. While the findings from the first evaluation phase were promising, 3 the quasi-experimental design precluded our ability to definitively attribute positive outcomes to the HFM program. In MHFE-2, the project implemented an experimental design (Tier Five) to determine, with the certainty that such a design confers, the actual effects of the program.

MHFE-2 EVALUATION DESIGNThis six-year randomized, controlled trial (RCT), for which data collection began in February 2008, consists of two main components: the Impact Study, through its experimental design, will illuminate the net outcomes that HFM has yielded for its participants. The more focused Integrative Study will elucidate the multi-level influences (e.g., community context, program character, individual mental health challenges and school experiences, etc.) in the lives of participants that move them along their particular developmental trajectories, as they transition both to parenthood and adulthood. Participants are interviewed at three time points: enrollment, 12 months post-enrollment, and 24 months post-enrollment. Increased attention to the community ecology in which these HFM programs

2 See Jacobs. F. (2003). Child and family program evaluation: Learning to enjoy complexity. Applied Developmental Science, 7(2), 62-75; Jacobs, F. H. (1988/2008). The Five-Tiered Approach to evaluation: Context and implementation. In H. B. Weiss & F. H. Jacobs (Eds.), Evaluating Family Programs, New York: Aldine DeGruyter (reissued 2008); Jacobs, F., & Kapuscik, J. (2000). Evaluating family preservation services: A guide for state administrators. Medford, MA: Tufts University.

3 For final evaluation report, see Jacobs, F., Easterbrooks, M. A., Brady, A., & Mistry, J. (2005). Healthy Families Massachusetts final evaluation report. Medford, MA: Massachusetts Healthy Families Evaluation.

6

operate — in essence, enhanced activities at Tier Three — also are being integrated into this second cohort study as the Program Community Study.4

Recruitment and Enrollment The paragraphs below provide a brief description of the MHFE-2 sampling design, and the process and status of study recruitment and enrollment from early February 2008 through November 2010.

Selecting the Evaluation SitesSeveral months before recruitment was slated to begin, the Tufts team, in cooperation with MCTF, chose eight programs from the twenty-six program sites across the state to participate in the evaluation. These programs were chosen on the basis of three criteria:

As a group, they represented each of the Department of Health and Human Services regions in Massachusetts;

As a group, they offered a mix of urban and exurban/suburban communities with diverse populations; and

As individual programs, each was large enough to accommodate evaluation enrollment within a 6-8 month period (based on FY06 MCTF referral and enrollment data).

Based on its size, each of these eight programs, or “evaluation sites,” was assigned a target recruitment number, and expected to continue recruitment and enrollment until this target had been reached. The aim was to collect baseline data from at least 650 people, the number that would provide sufficient statistical power for the planned analyses. Assuming that only between 75% and 80% of those who were recruited would actually participate in data collection, Tufts and MCTF decided programs should recruit a total of 860.

Recruitment began in early February 2008. Within a few months of this date, it became apparent that the eight evaluation sites were enrolling participants at a much slower rate than anticipated.5 In response to this problem, MHFE/MCTF added program sites to the evaluation, thereby lowering the target recruitment rates for all participating sites. Over the course of 2008, based on MHFE’s continued monitoring of recruitment rates across the programs, ten HFM programs were added to the original eight, bringing the final evaluation site total to 18 by the end of the 2008 calendar year (see Appendix A for evaluation sites, the months they entered the evaluation, and their target enrollment numbers).

Recruiting Participants into the StudyThere were two “levels” of recruitment; the first took place, as mentioned above, at the HFM program level, and the second was conducted by the Tufts team. Both are explained below.

Level 1: Participants Were Recruited by HFM Intakers into the EvaluationAfter each program’s recruitment start date, every participant referred to that program, provided she was eligible, was asked by the HFM Intaker6 to participate in the study. Eligible participants were

4 See FY09 Report for detail about the Impact and Integrative Studies’ designs.

5 This situation resulted from a variety of factors, including regional changes in teen birth rates, decreases in agency operating budgets, and fewer referrals from community agencies. It should be noted that these lower enrollment rates characterized all of the HFM programs—not only those involved in the evaluation.

6 All staff members involved with the evaluation were trained by Tufts and MCTF (see below for more detail).

7

those who were: 16 years or older, female,7 have received no HFM services in the past (i.e., no transfers or re-enrolls), able to speak either English or Spanish, and cognitively able to provide informed consent.8 If the participant agreed to enroll in the study, the HFM Intaker entered the participant’s name and information into the program's management information system, the Participant Data System (PDS). At this point, an algorithm randomly assigned her to either the “Home Visiting Services Group” (HVS) (the program group), or the “Referrals and Information Only Group” (RIO) (the control group). This assignment was immediate.

If the participant did not meet these study eligibility requirements, but was eligible for HFM (e.g., was 15, spoke only Haitian Creole, etc.), the HFM Intaker contacted MCTF and arranged for the participant to be enrolled directly into the HFM program.

If the participant was assigned to HVS, she would receive home visiting services as usual. If assigned to RIO, she would not be able to receive any HFM home visiting services, but rather was referred to other community services. In order to determine where each RIO participant should be referred, the Intaker conducted a MCTF-designed intake interview that assessed the participant's need for services. Based on this interview, the Intaker provided referral information and/or resources related to the participant's needs (e.g., mental health, WIC, housing, TANF, etc.). RIO participants also receive monthly mailings from MCTF about child development.

Everyone who declined to participate in the evaluation was given the option of receiving the same intake interview and referral to services that the RIO participants received. They could also opt to go on a wait list; once the HFM program had enrolled its quota of participants they could let people into the program from this waitlist (provided there was space).

Level 2: Participants Were Enrolled by Tufts into the Impact and Integrative StudiesOnce HFM concluded the first phase of recruitment (assigning participants to HVS or RIO), the Tufts team assumed all further responsibility for recruitment and data collection. Every participant was invited to: 1) sign a release allowing Tufts to access her administrative data from the Massachusetts Departments of Elementary and Secondary Education (DESE), Transitional Assistance (DTA), Public Health (DPH), and Children and Families (DCF); and 2) to participate in a half hour phone interview (called the Intake Interview).

Participants who consented to at least one of these activities (i.e., sign a release and/or participate in the Intake Interview) were considered to be part of the Impact Study sample. At this point of first contact by MHFE, the participant also was given the option of participating in an additional two and a half hour research visit. Participants who consented to this research visit were considered to be part of the Integrative Study sample.

MHFE-2 SampleParticipants who completed the baseline Intake Interview and/or signed the agency consent at baseline are considered to be in the Impact Study sample. Participants who agreed to, and completed,

7 While HFM does serve fathers, they constitute only (4%) of the statewide HFM population; random recruitment would not produce a sample of fathers large enough for results to be statistically meaningful.

8 The Tufts team relied on the experience and expertise of the HFM Intaker to determine whether or not the participant appeared to fully understand the study. If the HFM Intaker felt that the participant was not cognitively able to provide fully informed consent, she contacted MCTF and asked for the participant to be enrolled directly into HFM. There were several instances in which a participant was enrolled by HFM but later withdrawn by Tufts because of an apparent cognitive delay.

8

the research visit at baseline are considered to be in the Integrative Study sample. In other words, everyone in the Integrative Study sample is also in the Impact Study sample, but not vice versa.

As seen in Figure 1, a total of 840 young women were recruited by HFM for the evaluation. Of those HFM had recruited, the Tufts team was able to successfully contact and enroll 691 participants (83%) into the Impact Study. Ninety-nine percent of the Impact Study participants (n=688) signed a consent form allowing us to gain access to their agency records. The majority of participants (665) completed a Time 1 Intake Interview, and 440 (63%) also agreed to participate in the Integrative Study. MHFE-2 uses an intention-to-treat approach; women who were assigned to HVS (the program group) remain in this group regardless of their actual level of participation in the HFM program.

There are three categories of people who were recruited into the evaluation at the HFM level, but are not currently enrolled in the study: withdrawn, ineligible, and still on outreach. Of the 164 women who, as of June 30, 2010, were not enrolled in the Impact Study, 50 (31%) had either asked to be withdrawn or were withdrawn by Tufts; 40 (25%) were ineligible, and 57 (35%) were still on outreach (see Figure 1). Explanations of each category follow:

“Withdrawn” The majority of people who fell into this category were people who were enrolled by HFM into the evaluation, but then, when contacted by Tufts, declined to participate in any research activities. There were several participants, however, who were withdrawn for reasons other than lack of interest. Examples include: two participants, both minors, who were withdrawn at their parents’ requests; one participant who withdrew after the Tufts interviewer filed a suspected maltreatment report with DCF on her; and three RIO participants who were withdrawn by the Tufts team because their need for immediate intervention ethically precluded their remaining in the control group (e.g., cases of suicidal ideation).

“Ineligible” There were two categories of ineligibles: participants who were ineligible for the HFM program, either at enrollment or shortly thereafter; and participants who were ineligible for the study. Each category is explained below

‣ IneligibleforHFM:This included people who lost their baby within weeks of enrollment, either through miscarriage or termination; turned out to have had a previous child; and/or moved to another state within weeks of enrollment.

‣ Ineligible for the evaluation: While, overall, the HFM staff did an excellent job at screening referrals for evaluation eligibility, programs did enroll some participants who did not qualify for the study. Several participants were enrolled despite having had received HFM services in the past; some participants were enrolled despite having been already assigned to RIO in a different evaluation site; some were enrolled despite having no working knowledge of either English or Spanish; and finally, some participants appeared, when visited by Tufts interviewers, to be cognitively incapable of providing informed consent. All participants deemed ineligible by Tufts were withdrawn from the study.

“Still on Outreach” In order to be in the Integrative Study, a participant must be contacted and interviewed by Tufts within three months of study enrollment. However, even after that three-month window has passed, a

9

participant still could be enrolled in the Impact Study: participants contacted after the window are asked to sign an agency consent, and to participate in a truncated, one-time-only version of the Intake Interview that asks basic status questions such as race, living configuration, etc. The Tufts team will continue to pursue those who are still on outreach for the duration of data collection (slated to end in December 2011).

Figure 1: MHFE-2 Recruitment and Enrollment

MHFE-2 Participants and HFM UtilizationThis section presents a demographic profile of the participant sample, using selected individual characteristics; see Appendix B for a more detailed description of this group. We also provide information about participants’ initial program use.

Participant Demographics and Other Selected CharacteristicsParticipants were, on average, 18.4 years old at the time of enrollment into the evaluation. As shown in Table 1, the ethnic distribution of the MHFE participants (40% White Non-Hispanic, 32% Hispanic, 15% Black Non-Hispanic, and 13% categorized as “Other” (this includes anyone who identified as more than one race or ethnicity).

51 (34% of unenrolled) are withdrawn → 36 (72%) HVS→ 15 (28%) RIO

40 (27% of unenrolled) are ineligible → 21 (53%) HVS→ 19 (47%) RIO

56 (39% of unenrolled) still on outreach → 39 (70%) HVS→ 17 (30%) RIO

147 (18% of total recruited) not enrolled in the study

→ 96 (67%) HVS→ 51 (35%) RIO

688 (99% of Time 1 Sample) signed an Agency Data Release

→ 406 (61%) HVS→ 259 (39%) RIO

476 (69% of Time 1 Sample) also did a Time 1 Research Interview†

This is the Time 1 Integrative Sample→ 277 (58%) HVS→ 199 (42%) RIO

MHFE enrolled 693 (83% of total recruited) in the study*

This is the Time 1 Impact Study Sample→ 424 (61%) HVS→ 269 (39%) RIO

665 (96% of Time 1 Sample) did a Time 1 Intake Interview

→ 421 (61%) HVS→ 267 (39%) RIO

840 recruited by HFM into MHFE-2

*This number exceeds enrollment goal of 670.†This number exceeds enrollment goal of 420.

10

Table 1: Distribution of Teen Pregnancies by Race: MA Total and MHFE-2 Sample

MA Total Teen Pregnancies (ages 10-19) in 2007*

MA Total Teen Pregnancies (ages 10-19) in 2007*

MHFE-2 SampleMHFE-2 Sample

Count Percent Count PercentBlack Non-Hispanic 638 13% 96.00 15%White Non-Hispanic 2283 46% 256.00 40%Hispanic 1731 35% 203.00 32%Other (includes Multi-racial) 338 7% 84.00 13%

*Data were derived from Massachusetts Community Health Information Profile (MassCHIP).9

As seen in Figure 2, 58% of the participants were living with a parent, guardian, or older relative at the Time 1 Interview, and slightly more than half (51%) of the participants reported being in some kind of committed relationship (this could be with the baby’s father or with someone other than the baby’s father). One-quarter of the participants was receiving TANF benefits (cash assistance) at Time 1. The overwhelming majority of women (97%) reported having health insurance, and 77% said that they had a primary care provider. The majority (88%) of the mothers was born in the United States, and 69% of the sample was born in the state of Massachusetts. English was the preferred language of 75.1% of participants.

Figure 2: Selected Characteristics of MHFE-2 Participants at Time 1 Interview

Program Utilization InformationThe extent and nature of participation in the HFM program represent critical data for MHFE-2, both to understand how the programs are operating and are being accepted by the young mothers, and to document the relationship of program participation to the achievement of desired outcomes. These analyses will include both quantitative and qualitative data. Indicators of service utilization

9 Massachusetts Community Health Information Profile (MassCHIP). Massachusetts Department of Public Health (Version 3.0) [Software]. 2007 Births (Vital Records): Adolescent health report for Massachusetts total (2007). Retrieved from http://www.mass.gov/dph/masschip.

11

(quantitative) are largely derived from the PDS, and include such variables as duration in program, number of visits completed, number of groups attended, and number of goals completed. Qualitative data about topics such as the nature of the home visit and the home visitor/client relationship are drawn from the Research Interviews and PDS home visit memo fields. As explained below, however, these datasets cannot be fully constructed until the HVS sample has had the opportunity to complete the full period of programming.

MHFE-2 recruited participants between February 2008 and November 2009. Consequently, as of June 30, 2010, even those participants assigned to HVS at the very beginning of the study had not had the opportunity to participate in the program for the maximum possible enrollment of three years and eight months. Therefore, until the full HVS sample of MHFE-2 participants have either had the opportunity to complete the full duration, and/or have been discharged, it would be premature to draw any conclusions about many aspects of program usage (e.g., number of total visits, duration in program, use of groups, etc.). Such analyses will be possible only after the last enrolled cohort (those enrolled in November 2009) have completed the maximum possible period of time for program participation (this will be June 2013). At present, then, we can report only limited information about participants’ program use:

‣ Pregnancy status at referral: The majority of participants (68%) were pregnant at referral.

‣ Number of days between referral and first contact: Programs took, on average, 15 days to contact participants once they had been referred to HFM.

‣ Retention: To determine how many participants stayed in HFM for at least one year, we first selected only those participants for whom that would have been possible (i.e., had at least 12 months between date of enrollment and June 30, 2010). Of the 403 participants for whom this was the case, 163 (41%) stayed in HFM for at least one year. Of the 124 participants who could have been enrolled for two years as of the end of FY10, 27 (22%) participants stayed in the program for at least that long.

Data SourcesThis section begins with brief descriptions of the data sources, by study component (Impact or Integrative). See Appendix C for a more complete overview of data sources, organized by construct, or sub-category of information that is required.

Impact Study Data SourcesThe Impact Study includes data generated from multiple sources, including public agency administrative data; program data, and telephone interviews with individual participants.

Administrative Data Public agency data (e.g., DCF: supported cases of child maltreatment by participant; DPH: rate of repeat birth as teen; DESE: high school graduation rate; DTA: participant receipt of public assistance) (see Appendix D for requested data points) are utilized to answer the primary research questions about the effectiveness of HFM on the five program goals. These data are collected on all MHFE-2 participants, program and control. Data dumps from the agencies were slated to occur annually, beginning in calendar year 2010. We have received an initial dataset from DCF; datasets from the other three agencies are forthcoming.

12

Participant Data SystemThe Participant Data System (PDS) is the web-based management information system administered and maintained by MCTF. Documented by home visitors and supervisors, data in the PDS provide information about the following areas of service utilization:

Referrals, enrollment, and service level; Pregnancy and birth information; Service encounter records (frequency of home visits, content of home visits, etc.); Assessments; Status reports (completed at six-month intervals); IFSP goal-setting and goal-attainment records; and Discharge records.

Intake InterviewThe Intake Interviews consist of a 30-minute semi-structured phone interview. At each of the three annual data collection time points, we collect information that will help us to understand characteristics of MHFE participants, their contexts, and their use of social services. Participant characteristics include demographics (e.g., age, ethnic background, relationship status), family resource, residential and financial situations, involvement of the baby’s father, and well-being (e.g., stress, depression). Information about participants’ use of public assistance and social services other than HFM will help “contextualize” the impact of HFM services relative to the array of other services that mothers in both HVS and RIO groups may receive.

Geographic Information SystemsThe purpose of conducting Geographic Information Systems (GIS) based analysis and maps is to facilitate an examination of whether and how community context influences program implementation and utilization, as well as of participants’ attainment of intermediate and distal outcomes. Towards this end, data are derived from public databases (e.g.MassGIS, U.S. Census, Home Mortgage Disclosure Act, etc.), that help us to characterize the communities in which participants live according to potential indicators of community based assets and risk (e.g., socioeconomic stratification, risks in public health, public safety and environment domains, attributes such as human and social capital, and community resources—infrastructure, public services, recreational and cultural facilities, etc.).

Integrative Study Data SourcesIn addition to the Impact Study Intake Interview that is conducted over the phone, MHFE participants who have enrolled in the Integrative Study are invited to participate in a research visit in their homes. The research visits include a semi-structured interview, completion of written questionnaires (see Appendix E for psychometric information), and observations of mother-child interactions, and provide data for both distal outcomes (see Appendix C, Table 1) and intermediate objectives (see Appendix C, Table 2).

During these visits, which typically last two hours, qualitative and quantitative methods are used to collect in-depth information about program services (HFM and other programs); social relationships and support networks (family/friend, father of baby, neighborhood/community); mothers’ history of childhood care and victimization, and more recent history of intimate partner violence; educational history; and personal functioning/well-being (e.g. depression, trauma history, stress and coping). These characteristics and contextual factors may influence how, for example, HFM program services are utilized and incorporated by participants. Since child maltreatment represents only one component of

13

parenting – and it has negative valence – other indicators of positive, effective parenting in this protocol, such as parenting attitudes and beliefs, and observations of mother-child interaction (e.g., maternal sensitivity and child responsiveness) are included.

Data Collection Activities through June 30, 2010 (FY10)Data collection began in the spring of 2008, and will proceed through the end of calendar year 2011 (see Table 2). Data are collected from participants at three different time points: enrollment (Time 1), 12 months post-enrollment (Time 2), and 24 months post-enrollment (Time 3). As of June 30, 2010, Time 1 data collection was complete, Time 2 data collection was almost complete, and Time 3 data collection had just begun.

Table 2: Data Collection Timetable

Figure 3 presents the status of data collection efforts as of the end of FY10; at this point, Time 1 data collection was complete, and Time 2 data collection would proceed for another 6 months. (At the time of this writing, still three months before the end of the Time 2 data collection, approximately 80% of the Impact Sample, and 83% of the Integrative Sample have been interviewed.) Because Time 3 data collection began shortly before June 2010, those numbers are not reported here.

Figure 3: Data Collection Status as of June 30, 2010

!

!"#$%&%

!"#$%'%

665 completed a Time 1 Intake Interview ! 421 (61%) HVS ! 267 (39%) RIO

472 also completed a Time 1 Research Interview ! 276 (58%) HVS ! 199 (42%) RIO

500 (75% of Time 1 Sample) completed a Time 2 Intake Interview

! 300 (60%) HVS ! 200 (41%) RIO

361 (76% of Time 1 Integerative Sample) completed a Time 2 Research Interview

! 209 (58%) HVS ! 152 (42%) RIO

PRELIMINARY DATA ANALYSIS ACTIVITIESAs of the end of FY10, all quantitative data from Time 1 (i.e., data from the Intake Interview and Research Interview measures) had been entered and cleaned; a series of preliminary analyses had been undertaken. Two of these baseline analyses that are critical to the analytic work that will proceed in FY11 are presented below. The first details the equivalence analyses of the MHV and RIO groups, and the second pertains to the categorization of participants’ communities.

Data Collection Time Points

Year 12008

Year 12008

Year 12008

Year 12008

Year 22009

Year 22009

Year 22009

Year 22009

Year 32010

Year 32010

Year 32010

Year 32010

Year 42011

Year 42011

Year 42011

Year 42011

Time 1 Time 2

Time 3

14

Establishing HVS/RIO Equivalence at Time 1Random assignment of potential program participants to treatment and control groups is a hallmark of experimental investigation, and as reported earlier, was implemented in MHFE-2. This is the ideal circumstance for a study of the efficacy of an intervention. However, perfect randomization is difficult to achieve. Even in the most rigorously designed studies, researchers often come across group differences that need to be addressed before they are able to draw conclusions regarding treatment efficacy. With that caveat in mind, MHFE conducted analyses to establish the equivalence at baseline of these two groups – in essence, to validate the assignment process. These analyses are presented in Tables B.1 and B.2 in the Appendices.

As can be seen in these tables, HVS and RIO groups, indeed, were equivalent on a broad range of variables that measure participants’ functioning in the domains relevant to HFM goals (positive and effective parenting; optimal health, growth, and development; educational attainment, job, and life skills among parents; preventing repeat pregnancy in the teenage years; parental health and well-being). In four areas, however—depression, engagement in risky behaviors, the use of mental health services, and child care vouchers—the two groups differed. RIO participants were found to have significantly higher levels of depression (as assessed by the Center for Epidemiological Studies Depression Scale [CES-D]) to engage in more health risk behaviors (as assessed by the Youth Risk Behavior Scale [YRBS]), to be more likely to have a child care voucher, and to be more likely to receive mental health services than HVS participants.

Recent statistical and methodological advances have provided techniques that allow researchers to continue to make comparisons between RIO and HVS participants while accounting for these baseline differences. The approach propose for MHFE-2 is explained below

Examining Group Differences in Light of Baseline Equivalence A method for addressing treatment and control group differences that has been gaining attention in a variety of disciplines is the use of propensity scores.10 The theory behind propensity scores is straightforward. For each participant, a score is created that reflects his/her probability of being a member of the treatment group (here, we will predict the probability that an individual is enrolled in HVS). This propensity score, once created, can be used as a control variable in statistical procedures that are aimed at determining differences between the two groups.

To create propensity scores for all participants, we will conduct a logistic regression with the individual’s group status (i.e., treatment or control) as the outcome. Predictors in the logistic regression will include variables for which the two groups were not equivalent at baseline (e.g., depression, health risk behaviors) as well as demographic variables that may account for group differences in the outcomes of interest. The propensity score that is generated can act as a single-variable summary of covariates for each individual. The propensity score is a simple and versatile method for controlling for baseline differences. In fact, it is even recommended for use in experimental designs where no baseline differences were identified.11 Once created, the propensity score can be used in a variety of analyses (e.g., multiple regression, structural equation models, and hierarchal linear models). Consider, for example, a multiple regression analysis comparing positive

10 Rosenbaum, P.R., & Rubin, D.B. (1983). The central role of propensity score in observational studies for causal effects. Biometrika, 70, 41-55.

11 Rubin (1997). Estimating causal effects from large data sets using propensity scores. Annals of Internal Medicine, 2, 757-63

15

parenting attitudes between RIO and HSV participants. We will use positive parenting as the outcome and group status and the propensity score as predictor variables. If the group status variable is a significant predictor of positive parenting, the presence of the propensity score in the regression equation will allow us to say that even after accounting for baseline disparities between the two groups, there are significant differences in positive parenting attitudes between RIO and HVS participants.

In addition to being used as a predictor in a regression equation, the propensity score can also be used to create matched samples12. In the ideal situation, when randomization occurs and no baseline differences between groups are detected, a matched sample is not necessary. However, given that baseline differences have been detected in the MHFE sample, a matched sample will increase the likelihood that our conclusions are unbiased and enhance our ability to make causal statement. To create a matched sample, we will use the propensity score of each member of our treatment group to identify a member of the control group who “matches” the treatment individual. In using the propensity score to create a matched sample, we will essentially be matching individuals in terms of depression, health risk behaviors, and relevant demographic variables. We will then make comparisons using statistical procedures designed specifically for matched samples such as the paired-sample t-test and the McNemar chi-square test.

Creating Community Context ProfilesIn MHFE-1, differences in both service offerings and participant behavior with regard to the full array of services appeared to be influenced by both program characteristics (e.g., agency auspices) and community characteristics (e.g., demographic make-up and population density), suggesting that there is contextual variation in service implementation. We intend to explore these intersections comprehensively in MHFE-2, examining how particular types of programs, located in particular communities, engage particular constituencies. We plan to use program data (e.g., staff, auspices, outreach strategies, policies, supervision practices) and community data (e.g., demographics, neighborhood stability, social capital, access to resources) to determine and document differential patterns of program implementation, utilization, and effects in varying community contexts. For example, findings from our first study suggest that participants in more densely populated, lower-income communities are more likely to “no-show.” However, at this point, we do not know why this is the case. The explanation may lie in features of community context that we did not assess in the first cohort study, such as the availability of other social services, easy access to competing activities and resources, or the availability of social support in community/neighborhood. Conversely, in communities that are more spatially dispersed, without public transportation and with, perhaps, fewer other social services, HFM may seem more of a novelty, less stigmatizing, and more useful and positive to the young parent.

Indeed, home visiting evaluations increasingly are attending to the community contexts in which the programs are located and services are provided, however the measures typically used to characterize the spatial, institutional, and demographic dimensions of communities remain limited in a variety of ways. A “community” spatially defined by census tract designations alone, for example, may bear little resemblance to the actual neighborhood parameters as experienced by residents, or, as Nicotera

12 See, for example, Hill, J.L., Waldfogel, J., Brooks-Gunn, J., & Han, W.J. (2005). Maternal employment and child development: A fresh look using newer methods. Developmental Psychology, 41, 833-50.

16

refers to it, to the community as a “place”13 described from an insiders’ view. Such data would likely identify a host of community assets, even amidst significant disadvantage, that might help explain why some families in some communities achieve outcomes while others do not. Additionally, there is increasing concern among experts that essentializing any given demographic characteristic (e.g., race or income) among residents of a community, and then establishing the relation of that characteristic to a particular outcome (e.g., maltreatment), does disservice to the multiplicity of interconnected community factors that influence individual behavior.14 Community configurations that are based on a bundle of characteristics hold promise in this regard.

One such approach to characterizing communities is the Environmental Justice (EJ) model, in which a set of indicators is used by environmental activists, researchers, and policy-makers to identify neighborhoods that are disproportionately at risk for environmental health problems (Environmental Protection Agency, 1998). EJ populations are defined as those segments of the population that EOEA has determined to be most at risk of being unaware or unable to participate in environmental decision-making or to gain access to state environmental resources. The Massachusetts Executive Office of Environmental Affairs characterizes these EJ neighborhoods as having a “majority population of any or a combination of people who are foreign born, linguistically isolated, or a minority, and of low-income”.15 Specifically, neighborhoods (i.e., census block groups) that meet one or more of four criteria are identified as EJ communities: median annual household income is at or below 65% of the state median; 25% or more residents are minority; 25% or more residents are foreign-born; 25% or more residents lack English proficiency.

We decided to use the EJ approach for our study, adapting it to suit our conceptual model and population. Our final set of community indicators includes two of the EJ descriptors (percent minority and median income), to which we added population density. The cut-off for percent minority, as described above, is 25 %. In MA, 65% of the statewide median household income is $30, 515. For population density, we use the cut-off recommended by Lopez and Hynes16 of 3500 people/square mile.Using ArcGIS (software, v. 9), each participant was assigned a geographic coordinate with the U.S. Census Topologically Integrated Geographic Encoding and Referencing system (TIGER) based on her address at Time 1, we were then able to designate participant U.S. Census defined blockgroups IDs through spatially reconciling participant coordinates within surrounding blockgroups, at which point we integrated MassGIS17 data on income, minority, and population density at the blockgroup level derived from Summary File 3, Summary Level 150, 2000 U.S. Census)

To create the Community Context Profile (CCP), each participant’s neighborhood at Time 1 categorized according to the criterion for each indicator. The initial eight groups, based on all possible combinations, appear in Table 3. Participants who had moved out of Massachusetts by the time we collected their Time 1 data (2%) were categorized as missing. Groups with fewer than 99 participants

13 Nicotera, N. (2007). Measuring neighborhood: A conundrum for human services researchers and practitioners. American Journal of Community Psychology, 40, 26-51.

14 Rogoff, B., & Angellilo, C. (2002). Investigating the coordinated functioning of multifaceted cultural practices in human development. Human Development, 45, 211-225.

15 Massachusetts Executive Office of Energy and Environmental Affairs. Environmental justice policy of the Executive Office of Environmental Affairs, Retrieved May 2009 from www.mass.gov.

16 Lopez, R., & Hynes, H.P. (2003). Sprawl in the 1990s: Measurement, distribution, and trends. Urban Affairs Review, 38(3), 325-355.

17 http:www.mass.gov/mgis/massgis.htm

17

in them (10% of the total sample) were dropped from further analyses involving the CCP variable. The final four groups used in analyses (henceforth referred to as CCP-A, CCP-B, CCP-C, and CCP-D, respectively) are listed first and highlighted in Table 3, and visually presented in Figure 4. Equivalence analyses confirmed that HVS and RIO participants were equally represented in each of the four CCPs.

Table 3: Initial Distribution of Community Context Profiles

Community Context Profile Count Percentage(CCP-A) Low Income,* High Population Density,† High Percent Minority‡ 188 27.1(CCP-B) High Income, Low Population Density, Low Percent Minority 151 21.8(CCP-C) High Income, High Population Density, High Percent Minority 176 25.4(CCP-C) High Income, High Population Density, Low Percent Minority 99 14.3High Income, Low Population Density, High Percent Minority 17 2.5Low Income, High Population Density, Low Percent Minority 32 4.6Low Income, Low Population Density, Low Percent Minority 8 1.2Low Income, Low Population Density, High Percent Minority 8 1.2

*Low income is defined as 0-$30,515; high income is $30,516 or higher.†Low population density is 0-3.5 thousand per dry square mile; high population density is 3.51 thousand or higher‡Low percent minority is 25% or less; high percent minority is 25.1% or higher.

Figure 4: Community Context Profiles A, B, C, and D

Bivariate analyses were run to determine the extent to which selected individual-level demographic characteristics of the participants aligned with the characteristics of the aggregate-level community context profiles (i.e., are the participants who live in block groups that fall into CCP-A more likely to be non-White than those who live in CCP-B?). As can be seen in Table 4, participant demographics did appear to align with the CCP. When compared with CCPs B and D, CCPs A and C had significantly

18

higher numbers of participants who spoke other languages instead of or in addition to English, were born outside of the US, and were Hispanic or African American. It is notable that participants in profile D had a slightly (not statistically significant) higher percent of multilingual, non-US born and Hispanic participants than participants in profile B; these differences probably are attributable to the higher population density in CCP-D, which perhaps facilitated the program’s outreach efforts among minorities.

Table 4: Select Demographic Characteristics of Participants in Each Community Context Profile CCP-A (N=188)

CCP-B (N=151)

CCP-C (N=176)

CCP-D (N=99)

Preferred Language English 59% 93% 72% 76%Preferred LanguageSpanish 8% 1% 5% 4%

Preferred Language

Other language or multiple languages 33% 7% 23% 20%Place of Birth Massachusetts 61% 78% 67% 78%Place of Birth

Other state 21% 20% 17% 11%Place of Birth

Other country 18% 2% 16% 10%Race/Ethnicity Black/African American 19% 4% 23% 8%Race/Ethnicity

White/Caucasian 16% 78% 17% 60%Race/Ethnicity

Hispanic/Latina 55% 11% 38% 20%

Race/Ethnicity

Other 10% 8% 21% 12%

19

EVALUATION TIMELINE AND ACTIVITIES FOR FY11

As is shown in Table 5, the evaluation to date mainly has been dedicated to preparation and data collection. Evaluation activities in FY11 will include: the conclusion of Time 2 data collection and continuation of Time 3 data collection; implementation of the Program Community Study (PCS) at the program level; coding and analysis of the home visitor/client relationship data; preliminary analyses of the agency datasets; and preliminary analyses of first year program outcome data.

Table 5: MHFE-2 TimelineYear 1(FY07)

Year 2(FY08)

Year 3(FY09)

Year 4(FY10)

Year 5(FY11)

Year 6(FY12)

Year 7(FY13)

Year 8(FY14)

Recruit and train HFM Evaluation Sites X XDesign and pilot interview protocols X XUse interagency data, census tract information, PDS data, and GIS to create community/program profiles

X X X

Pilot interagency data-use by conducting longitudinal data analyses on first-cohort sample

X X

Obtain data sharing agreements with state agencies

X X

Participant recruitment X X XTime 1 data collection X X XTime 2 data collection X X XTime 3 data collection X X XAnnual data dumps from state agencies X X X XData coding and analysis X X X X XFinal Evaluation Report X

20

PUBLICATIONS, PRESENTATIONS, AND GRADUATE STUDENT PROJECTSBelow are listed all MHFE products from 2007 to date (the years of this current study).

Published Papers and Reports Easterbrooks, M.A., Chaudhuri, J.H., Bartlett, J.D., & Copeman, A (in press). Resilience in parenting

among young mothers: Family and ecological risks and opportunities. Children and Youth Services Review.

Diez, V. & Mistry, J. (2010). Early childbearing and educational attainment among mainland Puerto Rican teens. Journal of Adolescent Research, 25(5), 690-715.

Chaudhuri, J. H., Easterbrooks, M. A. & Davis, C. R. (2009). The relation between emotional availability and parenting style: Cultural and economic factors in a diverse sample of young mothers. Parenting: Science and Practice, 9(3), 277-299.

Goldberg, J., Jacobs, F.H., Mistry, J., Easterbrooks, M.A., Davis, C.R., & Vashcehnko, M. (2009). Massachusetts Healthy Families Evaluation-2: A randomized controlled trial of a statewide home visiting program for young parents. Annual data report to the Massachusetts Children’s Trust Fund, Fiscal Year 2009. Medford, MA: Tufts University.

Mistry, J., Jacobs, F., & Jacobs, L. (2009). Cultural relevance as program-to-community alignment. Journal of Community Psychology, 37(4), 487 – 504.

Jacobs, F & Goldberg, J. (2008). Evaluating contemporary social programs: Challenges and opportunities. In M.E. Kenney, L.E. Reese, A.M. Horne, & P. Orpinas (Eds.), Handbook of Prevention: Promoting Health and Social Justice. Washington, DC: American Psychological Association.

Riley, S., Brady, A., Goldberg, J., Jacobs, F., & Easterbrooks, A. (2008). Once the door closes: Understanding the parent-provider relationship. Children and Youth Services Review, 30(5), 597-612.

Driscoll, J. R. & Easterbrooks, M.A. (2007). Young mothers’ play with their toddlers: Individual variability as a function of psychosocial factors. Infant and Child Development, 16 (6), 649-670.

Easterbrooks, M. A., Barrett, L. R., Brady, A. E., & Davis, C. R. (2007). Complexities in research on fathering: Illustrations from the Tufts study of young fathers. Applied Developmental Science, 11(4), 214-220.

Mistry, J.M., Jacobs, F., Goldberg, J., Easterbrooks, M.A., Davis, C.R., & Jimenez, I. (2007). A longitudinal study of repeat births in a sample of adolescent mothers: Follow-up results from the Massachusetts Healthy Families Evaluation. Medford, MA: Tufts University.

Presentations and TestimonyPrescott, J.E., Jacobs, F.H., & Goldberg, J. (accepted for presentation). A friend, but not exactly: An

exploration of the home visitor/client relationship. Poster to be presented at the 2011 Biennial Meeting of the Society for Research in Child Development, Montreal, CA,

Easterbrooks, M.A., Bartlett, J.D., & Miranda-Julian, C. (2010, June). Resilience in parenting among young mothers with a history of childhood maltreatment. Presented at the Pathways to Resilience II Conferences: The Social Ecology of Resilience, Halifax, Nova Scotia, Canada.

21

Bartlett, J.D., Miranda-Julian, C., & Easterbrooks, M.A. (2010, April). Identifying resilient pathways among young mothers. Presented at the NASW Massachusetts Chapter Symposium 2010, Boston, MA.

Bartlett, J.D., & Easterbrooks, M. A. (2010, March). When ghosts and angels meet: Childhood contributions to neglect by very young mothers. Poster session presented at the 13th Biennial Meeting of the Society for Research on Adolescence, Philadelphia, PA.

Bartlett, J.D. (2009, September). Distinguishing neglect in maltreatment research: Findings from a Massachusetts evaluation of a prevention program for young mothers.  Presented at the UC Davis National Child Abuse and Neglect Conference, Sacramento, CA.

Bartlett, J.D., & Easterbrooks, M.A. (2009, April). Who’s not minding the child and why? Correlates of child neglect among the children of young mothers. Poster session presented at the Biennial Meeting of the Society for Research in Child Development, Denver, CO.

Davis, C. R., Miranda-Julian, C., Goldberg, J., & Easterbrooks, M. A. (2008, March). Resilient functioning across competing developmental demands of young mothers. Poster session presented at the annual meeting of the Society for Research on Adolescence, Chicago, IL.

Diez, V. & Mistry, J. (2009). Motherhood in the borderlands: Trajectories into adulthood among Puerto Rican teen mothers on the mainland. Paper presented at the Biennial Conference of the Society for Research in Child Development, April 1st – Apr. 4th, 2009, Denver, CO

Jacobs, F. (July, 2008). Keynote address: First National Research Conference on Child and Family Programs and Policies, Bridgewater, MA. Working together: Integrating home visiting research, practice, and policies.

Jacobs, F., & Goldberg, J. (September, 2009). The Massachusetts Healthy Families Evaluation: A multilevel investigation of a home visiting program for young parents. Presentation to Rep. Kay Khan, State Representative for the 11th Middlesex District; House Chair of the Joint Committee on Children, Families and Persons with Disabilities. Massachusetts State Legislature, Boston, MA.

Miranda-Julian, C., Davis, C. R., & Easterbrooks, M. A. Grandmother caregiving support among a sample of adolescent mothers. (2009, August). Poster session presented at the annual conference of the American Psychological Association, Toronto, Canada.

Tan, E. T., Davis, C. R., Easterbrooks, M. A., & Goldberg, W. A. (2009, March) Young fathers' involvement: Individual and contextual antecedents. Poster session presented at the biannual meeting of the Society for Research in Child Development, Denver, CO.

Unpublished Student Papers and Projects Chewning, A. (2010). Parenting cognitions and parenting stress among young mothers. Unpublished

master’s thesis, Eliot-Pearson Department of Child Development, Tufts University, Medford, MA

Decosta, N. (2010). The relation between young mothers’ childhood histories and emotional availability in mother-child interaction. Unpublished master’s thesis, Eliot-Pearson Department of Child Development, Tufts University, Medford, MA.

Dym Bartlett, J. (2010). Young mothers, child neglect, and discontinuities in intergenerational cycles of child maltreatment. Doctoral dissertation proposal, Eliot-Pearson Department of Child Development, Tufts University, Medford, MA.

Miranda-Julian. C. (2010). Cumulative risk: An examination of adolescent parenting in the context of trauma. Unpublished qualifying paper, Eliot-Pearson Department of Child Development, Tufts University, Medford, MA.

Vashchenko, M. (2010). The role of young mothers’ coping with parenting stress in the quality of their parenting. Doctoral dissertation proposal, Eliot-Pearson Department of Child Development, Tufts University, Medford, MA.

22

Dym Bartlett, J. (2009) When ghosts and angels meet: childhood contributions to neglect by very young mothers. Unpublished qualifying paper, Eliot-Pearson Department of Child Development, Tufts University, Medford, MA.

Driscoll, J. (2008). Is what young mothers do more important than how they feel? An exploration of relationships among maternal depressive symptoms, maternal-child emotional availability and child persistence. Unpublished doctoral dissertation proposal, Eliot-Pearson Department of Child Development, Tufts University, Medford, MA.

Student GrantsDym Bartlett, J. (2010-2012). Young mothers, infant neglect, and discontinuities in intergenerational cycles of

maltreatment. Doctoral fellowship awarded by the Quality Improvement Center on Early Childhood (QIC-EC), (The QIC-EC was created by the Children’s Bureau, Administration for Children & Families, USDHHS as a five-year cooperative between the Children’s Bureau and the Center for the Study of Social Policy, in partnership with ZERO TO THREE and the National Alliance of Children’s Trust and Prevention Funds to prevent maltreatment among infants and young children.)

Miranda-Julian, C. (2008-2010). Posttraumatic stress and parental functioning in a sample of adolescent mothers. Pre-dissertation grant awarded by the Eunice Kennedy Shriver National Institute of Child Health and Human Development, USDHHS.

2323

2424

2525

2626

2727

2828

2929

3030

3131

3232

3333

3434

3535

3636

3737

3838

3939

4040

4141

4242

4343

4444

45