katewinterevaluation.com evaluating the ram program an introduction october 22, 2015; farmingdale,...

KATEWINTEREVALUATION.com

Evaluating the RAM Program

An Introduction

October 22, 2015; Farmingdale, NY


Overview• RAM Logic Model• Research Question• Summary of Intervention and Preparation• Impact Evaluation Approach

• Selection and Assignment• Headcounts• Cohort Progression and Timing• Attrition

• Impact Evaluation Plan• Implementation Evaluation Plan• Evaluation Timeline• Q&A


Research Question• Does the Research-Aligned Mentorship (RAM) program (i.e.,

the intervention) result in significantly increased 4-year graduation rates of high needs students, as compared to the business-as-usual condition?


Summary of Intervention• Objectives: Increase participating students’:• Self-perception of ability to succeed in college• First year GPA (third year for transfers)• Rate of credit completion (min. 30 cr. per year)• 1st to 2nd (3rd to 4th) year retention (by 20%)• 4-year graduation rate (by 20%)

• Target population: full-time first-time entering freshmen and transfer students with at least 60 credits,* meeting at least 1 of 4 aspects of “high need” but not EOP• Duration: 2-years for freshmen, 1-year for transfers


Summary of Intervention (Cont.)• Activities:• First-year & sophomore-year/junior-year experience courses• Special events: a welcoming reception, speakers (often

distinguished researchers), and social occasions/group-building exercises• Required counseling (RAM counselors: 4x within first 2 years for

freshmen, 3x in junior year for transfers; Dept. counselors 1x/yr)• Digital Roadmap• Partial block scheduling• Teaching and learning through hands-on research


Program Preparation (Year 1)• Faculty professional development• Networking with off-site research opportunities• Preparation of RAM counseling, partial block scheduling, and

website• Finalization of evaluation design, data collection plan, and

schedule• Assignment of cohort 1 RCT treatment and control groups


Impact Evaluation Approach• Randomized Controlled Trial (RCT)• Random selection and random assignment of students• 2 sets (treatment and control) per institution (entering

freshmen and transfers, separately)• Treatment conditions get RAM program elements; control

conditions get “business as usual”• Analysis at school and aggregate levels, with comparisons by

student demographic within-group as appropriate


Analysis Plan• Analysis will be conducted for each school, for each group

within a school, for each group across schools, and for all students across all schools. • “Design and Development” educational research


Contrast Mapping

ControlTreatment

TransferFreshmen

GPA, Gender, SES, 1st Gen, race, adult learner

GPA, Gender, SES, 1st Gen, race, adult learner

Across and within all institutions:• T(all) vs. C(all)• T(all-lgpa) vs. C(all-lgpa)• T(all-w) vs. C(all-w)• T(all-m) vs. C(all-m)• T(all-lses) vs. C(all-lses)• T(all-urm) vs. C(all-urm)• T(all-nonurm) vs. C(all-nonurm)• T(all-lap) vs. C(all-lap)

• T(all-ef) vs. C(all-ef)• T(all-tr) vs. C(all-tr)

• T(all-ef-w) vs. C(all-ef-w)….


Selection and Assignment of Students• Determination of eligibility as entering freshmen or transfers at

each partner institution• Random selection of needed number of students (+) for both

conditions of both sets • Random assignment of both sets to either treatment or control

conditions• Determination of baseline equivalence for each set and of

representation of population• Monitoring of intervention participation by cohort for cross-

overs and no-shows, during intervention and until graduation


Student Headcounts*Institution Needed

AnnuallyPer

ConditionTreatment

Total/yr4-yr Total

Headcount4-yr

TreatmentFarmingdale State College (SUNY) – NY 500 125 250 2,000 1,000

Bowie State U – MD 200 50 100 800 400

Central Connecticut State U – CT U 200 50 100 800 400

Kean University – NJ 200 50 100 800 400

SUNY College at Old Westbury NY 200 50 100 800 400

TOTAL 1,300 325 650 5,200 2,600


Cohort Progression and TimingYear Entering Freshmen TransfersPY1 Program Development and PreparationPY2 Cohort 1 Year 1 Cohort 1 Year 3PY3 C1 Y2; C2 Y1 C2 Y3; Monitor C1 (Y4)PY4 C2 Y2; C3 Y1; Monitor C1 (Y3) C3 Y3; Monitor C2 (Y4)NCE Y5 C3 Y2; Monitor C1 (Y4) & C2 (Y3)

Control Group for C3 Y2 includedMonitor C3 (Y4)


Key Attrition Points:• This RCT design can only

meet WWC standards without reservations with low or moderate attrition

• With moderate attrition, we must show baseline equivalence and verify random assignment


To Meet WWC Standards without Reservations:

Attrition must be as low as possible, overall, and similar for both the treatment and control groups.


Impact Evaluation

Formative and Summative Assessment


Impact Evaluation Questions1. Analyses of treatment students compared to control students:

a. Is first year GPA higher (third year for transfers)? b. Are more credits earned each year (min. 30)?c. Is 1st to 2nd year retention (freshmen) and 3rd to 4th year retention

(transfers) 20% higher?d. Is the percentage of students graduating within 4 years 20% higher?

2. Within treatment group analyses:a. To what extent do program impacts differ between freshman

students and transfer students? b. Do program impacts differ by other key student demographics? c. To what extent do program impacts differ across sites?


Impact Evaluation IndicatorsIndicator Contrast Timing Source4 year graduation rate improves by 20%

Treatment versus control groups, by cohort and set

Annual, once cohorts have progressed through (PY3)

Registrar data/ IR

1st to 2nd (frosh) and 3rd to 4th year retention (tr) improves by 20%


Annual, starting end of PY2

Registrar data/ IR

100% of treatment students complete 30 cr/yr



Registrar data/ IR

First year GPA (third year for transfers) improves



Registrar data/ IR


Impact Evaluation Indicators (Cont.)Indicator Contrast Timing SourceStudent self-confidence improves

Pre- to post-intervention scores

Annual, starting PY2

Student survey

Quantity of academic and support services serving high need students increases

# of counseling sessions and % of students receiving counseling before and after program

Quantity of active learning experiences, research experiences, and co-curricular supports increase

# of opportunities and % of students participating before and after program

RAM Program model is tested across five partner sites and prepared for further replication

N/A


Implementation Evaluation

Formative and Summative Assessment


Implementation Evaluation Questions1. To what extent are program activities implemented with fidelity

and quality (participants, timing, content)? 2. What kinds of experiences will RAM program participants and

program staff members have as part of receiving/administering the program?

3. What is the program participants’ perceived level of satisfaction with program services?

4. What are the RAM program strengths and areas for improvement?

5. To what extent does RAM program implementation differ across sites?


Implementation Evaluation Indicators (PY1)

Activity Output Timing SourceOffer faculty development for enhanced courses (CL), additional research, and research funding

# of faculty ready to support CL# of enhanced courses

Network with research opportunities # of on- and off-site research slots available

Prepare RAM counseling # of counselors

Implement block scheduling # of blocked courses

Develop website Resources shared on website/tracking analytics

Randomly select and assign cohort 1 First cohort of RCT ready


Implementation Evaluation Indicators (PYs 2-5)Activity Indicator/Outcome Timing Source

FYE & SYE/TYE courses Evidence of experiences, # of students participating, student perceptions

Foundational courses have CL # of foundational courses with CL, students in CL courses, student perceptions

Special events # of events, students participating, perceptions of usefulness

Counseling # of hours of counseling received, % of students meeting reqs., perceptions of Digital Roadmaps

Block scheduling # and percentage of courses blocked

Research mentors/ opportunities

# of students matched, hours of mentoring, research placements (#offsite, #onsite), student presentns/pubs

Resource sharing via website Site analytics show high rate of resource utilization and engagement, user perceptions

Monitoring and assessment Timely feedback, annual formative reports


Evaluation Timeline• Regularly scheduled check-in conference calls• Formative assessment after each term• Summative assessment after each year• Cumulative summative assessment after NCE Y5


Questions or [email protected]

703-574-3746

mailto:[email protected]

katewinterevaluation.com evaluating the ram program an introduction october 22, 2015; farmingdale,...

Documents