READY - A Youth

Development

Outcomes Measure

U N I V E R S I T Y O F

ROCHESTER M E D I C A L C E N T E R

Project Team

United Way of Greater Rochester– Kathy Lewis– Elizabeth Ramsay

Rochester-Monroe County Youth Bureau– Chris Dandino

Univ. of Rochester Medical Center, Div. of Adolescent Medicine– Jonathan D. Klein, MD, MPH – Melissa Matos Auerbach, MA– Shannon M. Smith– Sheryl Ryan, MD– Cheryl Kodjo, MD, MPH– Premini Sabaratnam, MPH

Representatives from youth serving agencies in Rochester

Participating Agencies included:

Baden Street Settlement Big Brothers, Big Sisters of Greater Rochester Boy Scouts of America, Otetiana Council Boys & Girls Club of Rocehster Center for Youth Services Charles Settlement Community Place of Greater Rochester Girl Scouts of Genesee Valley Hochstein Music School Metro Council for Teen Potential Southwest Area Neighborhood Association (SWAN) Urban League of Rochester YMCA of Greater Rochester

Youth Development

Philosophy or approach - a set of principles emphasizing active support of the growing capacity of young people by individuals, organizations, and institutions

Characterized by a positive, strength building orientation

Occurs at home, in school, among peer groups, and in community-based programs

Has gained importance nationally, in states, and in local communities

Community Outcomes

Local funders are increasingly concerned with demonstrating effective progress toward outcomes

Existing measures of youth development are lengthy and complex

Youth Development Outcomes Measurement Project

GOAL:– To develop an evaluation tool for YD

programs that met the following criteria:

Easy to Use Easy to AdministerApplicable to a Variety of Youth Development

ProgramsUseful for Assessment of Impact of Program on

Youth Development of Participants

Rochester, New York

City in Monroe County, Western New York

Population 219,773 52% non-white 37% ages 19 or less Person under age 18 in 34%

of households 32% of families with children

under 18 below poverty level

Source: US Census, 2000

Youth-Serving Agencies

Serve the Youth of Rochester through: case management counseling homework assistance sports programs life skills building leadership programs music lessons provision of safe, open recreational spaces

Three Phase Project

Phase I Instrument development via a consensus process

Phase II Piloting to test validity and reliability of instrument

Phase III Field tests and dissemination

Phase I: Instrument Development - Dec. 2000 - May 2001

Meetings with representatives from youth-serving agencies and funders

Identification of core and optional outcome measures and questionnaire items

Establishment of face validity of core measures and measurement strategies

Identification of Core Outcomes

Three meetings resulted in list of 54 indicators and 10 outcomes

peer and adult relationships constructive use of leisure time basic social skills community service health maintenance decision making process responsibility understanding boundaries/rules positive identity independent/daily skills

Narrowing List of Outcomes

Agency representatives were asked:

What impact does your program have?

What would you like to learn to improve the quality of your program?

Programs Wanted to Know

Effectiveness of staff Effectiveness of services they provide Impact on youth and their families Impact of youth involvement in more than one

program Youth Development philosophy of staff Gaps and what programs can do about them

Consensus Process

Programs used nominal iterative process to identify consensus priority areas for youth that they could impact

First Round: 18 constructs

Second Round: 7 constructs– Top 2 retained

Third Round: 4 constructs– Top 2 retained

Outcomes for Operationalization

Basic Social Skills

Caring Adult Relationships

Decision Making Process

Constructive Use of Leisure Time

Candidate Questions

Questions adapted from instruments by Add Health Boys and Girls Club of America Girl Scouts of America Metro Council for Teen Potential Worcester Youth Development Initiative YMCA

Phase II: Piloting the Draft Instrument - May 2001 - March 2002

Piloting in two phases:

A. Cognitive interviews to test validity of items

B. Field test of internal consistency of items and feasibility

A. Cognitive Interviews

48 urban and suburban adolescents aged 10 to 17 Mean completion time: 11 minutes 70% had no suggestions 67% reported survey was “easy” to complete 81% understood everything in the survey 98% did not mind answering the survey Items re-worded to increase readability (now at 4th grade

level) and to simplify concepts

B. Field Test

389 urban and suburban adolescents Ages 10 to 19 Large drop-in programs and smaller, structured

programs Findings:

– Feasible for program staff to administer– Large groups required more staff time– Easy for older adolescents– Some issues remain for younger adolescents

Field Test Results

Youth more attached to programs did better on measures

Four constructs have several good factors for program use in evaluation

Instrument consists of six factors, corresponding to three outcomes

Internal reliability scores (s from .5782 to .8557)

Factor Analysis: Core Outcomes

Self Control

Empathy Basic Social Skills

Communication

Staff Relationships Caring Adult Relationships

Program Effect

Decision Making Decision Making

Constructive Use of Leisure Time

Factor Analysis - all participants OUTCOME

CONSTRUCTStaff

RelationshipsProgram

EffectivenessSelf Control Empathy Communication

ITEMS 1 2 3 4 5 6Usually feel connected 0.094 0.657 0.111 0.121 -0.174 0.163Listen to w hat others say 0.156 0.104 0.176 0.735 0.035 0.041Think about how others see things 0.038 0.000 0.074 0.659 0.277 0.195Ask others questions to get ideas 0.070 0.164 0.167 0.354 0.576 0.031Stay out of f ights 0.073 -0.017 0.830 0.095 0.137 -0.043Keep anger under control 0.120 0.008 0.647 0.359 -0.016 0.090Stay out of trouble 0.064 0.034 0.715 0.015 0.145 0.305Ask for help if needed 0.143 0.044 0.107 0.012 0.668 0.182Program helped make/keep friends 0.214 0.647 -0.077 0.169 0.134 0.026Program helped make better decisions 0.135 0.540 0.045 -0.003 0.398 0.149Since coming to prog., relations w / family adults better 0.200 0.559 -0.008 -0.128 0.173 -0.034Talk about thoughts and feelings 0.131 0.225 0.041 0.318 0.502 0.045Would talk to program staff about problem 0.567 0.280 -0.025 0.048 0.175 0.067Program staff expect me to try my best 0.724 0.131 0.169 0.012 -0.049 0.006Program staff care about w hat happens to me 0.742 0.197 0.073 0.023 0.108 0.094Program helped identify caring adults 0.616 0.372 -0.060 0.109 0.210 -0.020Program staff listen to me 0.681 0.137 0.069 0.084 0.074 0.189Program staff act like young people are important 0.761 0.033 0.102 0.171 0.009 0.040Young people can make difference at program 0.724 -0.112 -0.014 0.152 0.129 0.094Think about how decision w ill affect my future 0.099 0.120 0.128 0.320 -0.121 0.678

Think about how decision w ill affect others 0.054 0.121 0.240 0.266 0.128 0.583

Listen to others, then make ow n decision 0.219 -0.026 0.095 0.529 0.104 0.279Think about how made decision afterw ards 0.113 0.100 -0.019 -0.014 0.241 0.722

Think about consequences of decision afterw ards 0.194 -0.323 0.065 0.077 0.402 0.467

ROTATED COMPONENT MATRIX: ALL CASES (n=389)

Caring Adult Relationships Social SkillsDecision Making

Factor Analysis - 13-19 year olds OUTCOME

CONSTRUCTStaff

RelationshipsProgram

EffectivenessSelf Control &

EmpathyCommunication

ITEMS 1 2 3 4 5

Usually feel connected 0.076 0.596 0.352 -0.069 0.042Listen to w hat others say 0.134 0.101 0.527 0.496 -0.139Think about how others see things 0.026 -0.035 0.292 0.631 0.155Ask others questions to get ideas 0.017 0.203 0.085 0.611 0.184Stay out of f ights 0.086 -0.051 0.730 0.064 0.158Keep anger under control 0.185 -0.032 0.659 0.142 0.117Stay out of trouble -0.016 0.080 0.566 0.119 0.371Ask for help if needed 0.135 0.030 0.045 0.678 0.226Program helped make/keep friends 0.240 0.634 0.008 0.268 -0.142Program helped make better decisions 0.122 0.626 -0.003 0.239 0.128Since coming to prog., relations w / family adults better 0.149 0.650 -0.121 -0.028 0.072Talk about thoughts and feelings 0.249 0.291 0.007 0.531 0.109Would talk to program staff about problem 0.473 0.364 -0.057 0.115 0.151Program staff expect me to try my best 0.713 0.132 0.251 0.014 0.001Program staff care about w hat happens to me 0.744 0.158 0.106 0.060 0.183Program helped identify caring adults 0.681 0.386 -0.057 0.080 0.020Program staff listen to me 0.689 0.185 0.101 0.143 0.197Program staff act like young people are important 0.763 0.043 0.180 -0.011 0.045Young people can make difference at program 0.716 -0.078 -0.037 0.222 0.118Think about how decision w ill affect my future 0.180 0.053 0.399 0.000 0.588Think about how decision w ill affect others 0.035 0.160 0.300 0.256 0.613Listen to others, then make ow n decision 0.192 0.000 0.171 0.181 0.518Think about how made decision afterw ards 0.113 0.137 -0.004 0.088 0.750Think about consequences of decision afterw ards 0.114 -0.226 -0.025 0.399 0.572

ROTATED COMPONENT MATRIX: AGES 13+ (n=248)

Caring Adult Relationships Social SkillsDecision Making

Factor Analysis - Reliability

FACTORStaff

RelationshipsProgram Effectiveness Self Control Empathy Communication Decision Making

0.856 0.609 0.659 0.584 0.578 0.640

ALPHA COEFFICIENTS FOR FACTOR COMPONENTS: ALL CASES (n=389)

FACTORStaff

RelationshipsProgram Effectiveness

Self Control & Empathy

Communication Decision Making

0.845 0.614 0.648 0.660 0.707

ALPHA COEFFICIENTS FOR FACTOR COMPONENTS: AGES 13+ (n=248)

Phase II: Con’t

Slight revision of individual items and rearrangement of questions leading to final instrument:

– Pencil and paper survey

– 40 questions addressing four core outcomes, program participation, connectedness to program, and sociodemographic information

– Requires between 10 and 15 minutes to complete

Development of training modules, scoring templates, and score report generating software

Phase III: Dissemination in Rochester - May 2002 - May 2003

Summer and Fall 2002 - 11 youth serving agencies in the Rochester area were trained to use the instrument and the report generating software

TA provided to agencies to develop appropriate sampling plans

During the program year of 2002-2003, over 1,000 youth participating in YD programs in the Rochester area completed surveys

Current Steps: May 2003 - present

Software and score reports revised based on qualitative feedback from Year 1 implementation

Continued training and TA to current users Dissemination to various other youth

development programs through ACT for Youth Center of Excellence

Validation studies

Additional sites

Alaska - Residential School System served as a beta test site in school year ‘02-’03

Erie County, NY - over 75 programs including youth bureaus and youth boards, and UW and Department of Youth Services funded programs

Hawaii - Children’s Alliance of Hawaii Oswego County, NY - Oswego City-County Youth Bureau funded

programs Salamanca, NY - 21st CCLC program Syracuse, NY - Catholic Charities of Onondaga County sites Queens, NY - Queens Child Guidance Center Beacon & OST

sites

Use of Data by Programs

Internal quality improvement. Examples include:– Reviewing and discussing score reports with staff and with Boards– Comparing program scores within one agency to identify opportunities

for improvement– Reviewing curricula and current programming strategies– Discussing program strategies with other similar programs– Identifying training and technical assistance needs

Reporting to funders Proposal writing Sharing data back allowing the creation of an aggregate

community level score report using de-identified data

Training and Technical Assistance

Training, TA, and use of the READY Toolkit are available to interested users

Fees are based on the number of users and the level of training and TA required

Options include:– Training and TA provided directly to end users– One time training provided for end users, and

continuous training and TA provided to a lead agency which then agrees to provide first line TA to end users

READY Toolkit

READY Toolkit includes a CD which contains:– A Personalizable Instrument Template– READY Analysis Program– Toolkit Instructions Manual– User’s Agreement

Instrument template may be personalized to contain program names and staff titles

READY Analysis program allows community programs to enter their own survey data, and generate a score report

Score report contains summary measures for core YD outcomes and frequencies for all survey items

SAMPLE Score Report

Pg 1 of 17

SAMPLE

Score Report Pg 2 of 17

Publications

Klein JD, Sabaratnam P, Matos Auerbach M, Smith SM, Kodjo C,

Lewis K, Ryan S, Dandino C. Development and factor structure of a

brief instrument to assess the impact of community programs on

positive youth development: The Rochester Evaluation of Asset

Development for Youth (READY) tool. Journal of Adolescent Health 2006;

39: 252-260. Sabaratnam P, Klein JD. Measuring youth development outcomes for

community program evaluation and quality improvement: Findings from

dissemination of the Rochester Evaluation of Asset Development for

Youth (READY) Tool. Journal of Public Health Management and

Practice 2006; 6(suppl): S88-S94.

For more information about the READY tool, please contact:

Premini Sabaratnam, MPHSr. Health Project Coordinator

Div. of Adolescent Medicine, University of Rochester

(585) 273-4616

Premini_Sabaratnam@urmc.rochester.edu

or

Jonathan D. Klein, MD, MPHAssociate Professor of Pediatrics and of Community & Preventive Medicine University of Rochester

(585) 275-7760

Jonathan_Klein@urmc.rochester.edu

Or visit...

The University of Rochester, Division of Adolescent Medicine, Leadership Education in Adolescent Health

website atwww.urmc.rochester.edu/gchas/div/adol/leah/resources.htm

or

The ACT for Youth, Center of Excellence website at www.actforyouth.net