Measuring Interorganizational Collaboration in Communities: Baltimore’s Strategy to Improve Birth Outcomes as a Case Study

Donna Strobino and Meredith MatoneDepartment of Population, Family and Reproductive HealthJohns Hopkins Bloomberg School of Public Health

Infant Mortality Rates: Maryland & Baltimore

Gaps in Current Efforts to Improve Birth Outcomes

• Individual initiatives are funded inadequately and separately

•Poor coordination of services•Lack of adequate primary health care•Lack of health insurance •Lack of minimum standard of care•Limited community mobilization to

promote healthy behaviors prior to or between pregnancies

General Overview

• The Strategy to Improve Birth Outcomes (SIBO) is a citywide and community-led strategy

• Reframes the way people, providers, services, and community agencies interrelate around the wellbeing of women throughout their reproductive life cycle (preconception, pregnancy and postpartum) and their infants

• Takes a population-based approach to solving intractable disparities in birth outcomes

High Impact Service Areas

(1) Primary health care in a medical home

(7) Smoking cessation

(2) Obstetric care (8) Family planning

(3) Home visiting (9) Nutrition support

(4) Drug and alcohol treatment (10) Breastfeeding promotion

(5) Domestic violenceInterventions

(11) Safe sleep education

(6) Mental health care

Family League of Baltimore City RFP

•SIBO challenges communities to mobilize around improving birth outcomes in a community and increasing demand for and utilization of the eleven high-impact services.

•The Family League is funding:▫a citywide communications campaign▫an evaluation and performance monitoring

strategy▫resident and community mobilization around

infant mortality and collaboratives capable of connecting all women in a community with appropriate care and services.

Five Critical Components: 1

• Policy mandate emphasizing

▫ Improved birth outcomes as a priority of the Baltimore public health landscape.

▫ Funding streams to facilitate systems and services to address gaps in coverage, improve effectiveness of practices and policies, and replace inefficient implementation of existing practices with evidenced-based alternatives.

Five Critical Components: 2

• Reorganize or retool health care delivery systems and practices for mothers and infants (including fathers, partners, families and communities who care for and support them).

▫ Assess and refine service provider practices and policies, especially related to risk assessment and create new operational models of service provision, where needed.

Five Critical Components: 3

•City-wide social marketing intervention

▫to influence the behavior of community residents, with a focus on expecting mothers and their partners and families, caregivers, service providers, and policy makers.

▫ Target all citizens

Five Critical Components: 4

• Community-led focused initiative designed to coordinate already existing, culturally relevant, local health and social services within target areas▫ Customized approach requiring strong

community input and collaboration, leading to transferable models for other neighborhoods.

▫ NO new services

Five Critical Components: 5

• A collaborative strategy driven by a set of partnership principles defining the quality and character of stakeholder interactions which guide the tenor of policies, agreements and protocols.▫ Important in defining the community

led component▫ Applies to CIT and Steering Committee

FLBC RFP for Community Collaboratives: Must Demonstrate the Capacity to:

•Outreach to all pregnant women in the community to connect them with appropriate care and services

•Participate fully in the new citywide triage system for pregnant women into home visiting programs;

•Link pregnant women with appropriate supports and services (home and center based)

•12 targeted communities with 3 awards of $400,000 - 600,000 for the 21 month period April 2010 – December 2011

Expected Impact in RFP

SIBO’s long term success: improvements in rates for pre-term birth, LBW, and deaths from unsafe sleep in targeted communities and in Baltimore City

Tracking of several indicators related to the activities of collaborative teams that contribute to the achievement of the key outcomes.

Success also measured by whether communities develop partnerships collaborations to assure access to the 11 high-impact services and link the various components of the health care system and human resources in the community to assure this access.

Evaluation Objectives

• Assess implementation of SIBO core components related to policy, services, communities, and families and individuals;

•Monitor implementation throughout the duration of SIBO to allow for modifications and trouble-shooting in real time (including the collaboration strategy); and

• Produce data to understand the impact of the first four core components on infant mortality in Baltimore City.

SIBO Evaluation Design

•Quasi-experimental design to evaluate changes in indicators in communities funded for the community-led initiative with the other communities with high infant mortality rates not initially funded: funded communities compared to remaining targeted communities

Data Source for Collaboratives

Data from community partners about collaborations among agencies and community leaders involved in implementation of the community-led focused intervention in targeted communities

Overview

•Significance

•Measurement problem

•Collaboration definition, models, research

•Evaluation strategies for collaboration

•Case Study: The Strategy to Improve Birth Outcomes in Baltimore City

Significance

Collaboration between health and social service agencies within a community that creates a population-based system of care are ideal because complex issues are inseparable from social and physical determinants of

health, which cluster by community.

Significance

•Proposed benefits of collaboration:▫Promote cost-effectiveness

▫Reduce duplication of services

▫ Improve communication

▫Comprehensive service provision

▫ Increase community capacity

•What is the evidence to support these claims?

A Measurement Problem: What is collaboration?

•Conceptually: logical and “feel-good” results indicator

•Reality: plagued with measurement issues

Collaboration Definition“Collaboration occurs when a group of autonomous

stakeholders of a problem domain engage in an interactive process, using shared rules, norms, and structures, to act or decide on issues related to that domain.”

Wood and Gray 1991

Collaboration in public health…

“A relationship between two organizations that involves exchanging information and sharing resources in order to coordinate services for mutual benefit and to achieve common goals, such as facilitating care of treatment of clients.”

Friedman et al. 2007

Collaboration Models

•“Level of Collaboration” models

•Attempt to classify the stage, or intensity, of relationships between organizations

• Indicators generally include purpose (goals), structures (central body, leadership) and behaviors (communication, budget)

Levels of Community Linkage Model, Hogue 1993

State of the Research• Most community-level research to date has

focused on costs and benefits of collaboration to organizations and/or accounts of the collaboration formation process

▫Stressors of collaboration in public sector

▫Characterization of nonprofit organizations likely to participate in formal collaboration

▫Effective arrangements of collaborative efforts

State of the Research

•Collaboration research in community health promotion, specifically grant-funded initiatives and public and nonprofit organizations, is underdeveloped

•Done little to advance measurement tools

Evaluation Strategies

•Social Network Analysis•Survey•Thomson, Perry, and Miller Method•Qualitative Analysis

Social Network Analysis

Social Network Analysis•Social network analysis is a

method of studying the linkages among individuals, groups, organizations, and other units (termed actors)

•Patterns of relationships are assessed and used to understand the structure, function, and performance of a network

SNA in Public Health

•Broad applications: ▫Transmission networks: disease and

information▫Social networks: social support and social

capital▫Organizational networks: health systems

SNA Methodology: Respondents

•Create “sampling frame” of organizations of relevance in evaluation

▫Known upfront, or

▫Formative research: snowball sampling, etc.

•Designate organization representatives: individual or group

▫Validity

**This methodology is applicable to all evaluation methods

SNA Content

•Rating scales of relationship type or intensity

•Questionnaires often draw heavily from the collaboration models for indicators of relationships

SNA Sample Questionnaire

SNA Data•Longitudinal

•Numeric▫Network density, centrality, geodesic

distances, directionality

•Graphical▫Sociograms: visual representations of

numeric data

▫Matrices

Sociograms: Cross et al., 2009

Surveys

Survey Content• Measures more variable in type and complexity

• Like SNA, can draw heavily from the collaboration models for indicators of linkage levels or,

• Frequency of action measures

▫ e.g. number of communications in the past year

Survey Data

•Longitudinal

•Numeric▫Conventional statistical methods

•Graphical▫Microsoft PowerPoint: models using circles

to represent partners and lines to represent linkages (Frey et al. 2003)

Thomson, Perry and Miller

Thomson, Perry, and Miller 2007

•Goal: to make collaboration quantifiable with an empirically tested measurement tool

•Built on the Wood and Gray (1991) definition

•Categorize collaboration into five dimensions: Governance, Administration, Mutuality,

Norms, Organizational Autonomy

TPM Content• 56 item closed-ended Likert scale questions on

organizational attitudes and behaviors surrounding each of the five dimensions of collaboration

• From the 56-indicator questionnaire, 17 indicators were empirically validated in statistical structural equation modeling

• 17 indicators cross-validated in a 2004 study of 133 collaborators in a Family Preservation Program

TPM Data

•The Thomson, Perry, and Miller strategy measures solely the presence of collaboration using a fixed model of attitudinal and behavioral indicators developed as proxies for the dimensions of an operational definition of collaboration

•Collaboration is standardized▫More readily compared across initiatives

Qualitative

Qualitative Methodology

•Facilitated focus groups or in-depth interviews

•Standard data analysis methods

Qualitative Data•Formative▫Who are the players?

•Process/Outcome▫Lists of collaborators, collaborative process info

▫Contextual information: costs/benefits, barriers/facilitators to collaboration

•Mixed-method approaches▫In addition to SNA, survey, or TPM

▫Validate numeric data

▫Provide explanatory data

Degree of Complexity: instrument development, data analysis techniques and software, and required expertise

Cost: material and non-material resources required by the data collection strategy, and the associated burden on the research team and respondents

Evaluation Strategies Matrix

The Strategy to Improve Birth Outcomes in Baltimore City

Choosing an Evaluation Method

•Data▫Ideal…and,▫Realistic

•Timeframe•Degree of complexity•Cost

Data• Matches ideal data

Timeframe • Case studies of similar scale

Complexity • SNA Evaluator on evaluation team

Cost • Software for large analyses

• Quick rating-scale questionnaire administered at time of other evaluation activities (e.g. community surveys)

SIBO Community Collaboration Component Proposed Evaluation Design

How will we measure success?

Cognitive Mapping• Formative

•Who are the players?

•What should the system look like?

•What should the exchanges between agencies look like?

•3 Perspectives▫Core Implementation Team▫Service Providers▫Steering Committee

Cognitive Mapping

1. Inform sampling frame

2. Inform data instrument construction

3. Inform Technical Assistance Toolkit for Collaboration

Evaluation Next Steps

•Finalize questionnaire

•Finalize sampling frame

•Select agency representative respondents

• Issue baseline survey

Thank you.