Closing the Loop: A case study in the use of formative program evaluation to direct the successful growth of an
intensive summer research program
Closing the Loop: A case study in the use of formative program evaluation to direct the successful growth of an
intensive summer research program
5-WEEK SUMMER RESEARCH PROGRAM
Research Components
Small groups conduct laboratory research with a faculty member (40 hrs/wk)
Two program-wide research meetings
End of program professional-style conference in which students make PowerPoint presentations
Social Components
Residence in apartment suites with access to all campus facilities including a gym and an art museum.
Field trips involving hiking, canoeing, and visits to an art museum and a sculpture garden.
.
Karen Singer-Freeman, Ronnie Halperin & Joseph Skrivanek
ANNUAL FORMATIVE EVALUATION METHODOLOGY
Individual interviews with each studentFocus group meeting with facultyEnd-of-the-summer student survey Direct observation of key program eventsReports include findings, conclusions, and recommendations
THREE PROBLEMS IDENTIFIED
Problem #1:Academic ReadinessStudents enter program with limited coursework in scienceFaculty expect students to be more prepared
Corrective Actions Faculty review of student applications leads to appropriate lab
assignments and the development of appropriate projects Spring orientation program begins to prepare students for summer
research program Faculty presentations during first week of program provide conceptual
introduction to all projects for all students
OutcomesFaculty report greater satisfaction with the students in their lab groupsStudents report that they feel more prepared for the program
Problem #2: Student Engagement in Science: Students lacked vision of selves as serious students as manifested by:
Attendance and behavior problemsEngagement at research meetings
Corrective Actions Contractual agreements clearly describing obligationsProactive instructions about desired group behaviorsIntegration of program alumni to serve as role models
OutcomesVirtual elimination of attendance and behavioral problemsMore active participation by students in research meetings Increased student productivity as evidenced by conference
participation
Problem #3: Navigating Route through CollegeStudents lack roadmap to becoming college graduates and professionals
Uninformed about the admissions and financial aid systemsUnaware timeline for successful transferUnaware of career opportunities
Corrective ActionsCreation of a new faculty position dedicated to advising and mentoring Each student is provided with:
Individual advising Workshops on selecting appropriate 4-year-institutions, applying and
transferring, selecting majors, financial aid, and careers in STEM fields
OutcomesThe majority of students reported the workshops were enriching and
enjoyable.
CONCLUSIONSThe use of formative evaluations, not simply to document process, but
to guide program modification, has the as yet unrealized potential to effect meaningful change.
This presentation demonstrates how closing the loop through formative evaluation contributes to the development of a successful program.
Baccalaureate and BeyondSummer Research Program
4-Year Liberal Arts CollegePurchase College, SUNY
6 Community College Partners
0
10
20
30
40
50
60
% o
f st
ud
en
ts r
ep
ort
ing
Low Med High
Research Meetings: Student Ratings of Enrichment Value
2005 2008
0
20
40
60
80
100
AA/AS Degree BA/BS Degree
% Completed Degrees
ProgramParticipantsNationalComparison
0
10
20
30
40
50
% o
f stu
dent
s re
port
ing
Not at all Somewhat Very much
Student Ratings of Orientation Helpfulness
Outcomes over nine years
The program has enrolled more than 200 students from surrounding community colleges.
42% of program participants transfer to Purchase 33% of participants are currently pursuing post-baccalaureate workRates of degree completion for all participants are significantly higher
than national averages for community college students
Supported in part by NIH grant #06210 and NSF grant #0524965
0
10
20
30
40
50
60
70
% o
f st
uden
ts r
epor
ting
Not at all Somewhat Very much
Student Ratings of Workshops
Enriching
Enjoyable
0
20
40
60
80
100
% o
f la
bs
rep
rese
nte
d
2004 2006 2008
Student Conference Presentations
National Conferences ABRCMS
