the creation of two comparison groups to facilitate evidence-based decision making

The Creation of Two Comparison Groups to Facilitate Evidence-Based Decision Making

Prepared and Presented by Keith WurtzDean, Institutional Effectiveness, Research and Planning

Crafton Hills College

Session Objectives

• Provide a brief introduction to the Left Lane Program

• Explain where the idea for two comparison groups came from

• Describe the two comparison groups and the limitations of the study

• Describe the findings• Review the process for Continuous Quality

Improvement and how the results were used to inform decision-making

The Left Lane Program (LLP)

• Purpose of the LLP is to reduce the amount of time it takes students to earn an AA/AS Degree at Crafton from 5.3 years to 4 years

• Program funded by the San Bernardino Community College District Chancellor

• First Cohort in Fall 2012


• Students who participate in Left Lane receive Priority Registration B

• To keep the Priority Registration B status students are required to do the following– Attend a Summer Bridge Program– Participate in SOA3R – Student Orientation, Application,

Assessment, Advisement, and Registration– Complete and closely follow their Student Education Plan– Enroll in math and English as stated in their SEP– Attend all tutoring meetings and check-in appointments– Maintain a 2.0 GPA– Enroll in 12 or more units per semester (depending on SEP)


• How are students selected for the LLP?– Need to place into one level below transfer English

or one level below transfer level math– Need to apply

The Idea for Two Comparison GroupsCollaborative Model of Institutional Research

Faculty/Staff/Student

Researcher

Joint Activity

Primary Responsibility

Key Features:• Dialogue-rich• Jointly-driven processes

Source: Dr. Rob Johnstone and Priya Chaplot, RP Group Conference Presentation

Raju HegdeDean of Math, English, Reading, and Instructional Support

Kirsten ColveyDean of Student Services, Counseling, and Matriculation

The Two Comparison Groups

• Left Lane Students– 258 out of 276 (93%) earned a GOR in Fall 2012– 82% graduated high school in 2012– 63% Female– 49% Hispanic

• First Comparison Group (Last Year High School was 2012)

• Second Comparison Group (Last Year High School was 2012 and did not participate in LLS)


Characteristic Left Lane Students

1st Comparison Group2012 High School

Graduates

2nd Comparison Group2012 HS Graduate and Did not

Participate in LL Strategy

N 258 583 357

Graduated HS 82% 100% 100%

Female 63% 48% 49%

Hispanic 49% 44% 48%

SOA3R Yes Varied No

LC Yes Varied No

SI Varied Varied No


Characteristic Left Lane Students

1st Comparison Group2012 High School

Graduates

2nd Comparison Group2012 HS Graduate and Did not

Participate in LL StrategyN 258 583 357

Placed Basic Skills Math 22% 26% 30%

Placed Developmental Math 77% 64% 62%

Placed Transfer Level Math 1% 10% 8%

Placed Basic Skills English 36% 37% 40%

Placed Developmental English 40% 34% 32%

Placed Transfer Level English 24% 29% 28%

Limitations

• The success rate comparisons do not control for instructor or by when the section was offered

• The comparison groups do not control for skill level (i.e. math and English placements)

Percent of Students Earning a Grade on Record in English and Math

GOR in English GOR in Math0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

77%

89%

45% 48%

26%34%

Left LaneFirst GroupSecond Group

Left Lane students were substantially (Effect Sizes were > .67) and statistically significantly (p values were < .001) more likely to earn GOR in English and Math.

Percent of Students Successfully Completing English and Math

Success in English Success in Math0%

10%20%30%40%50%60%70%80%90%

100%

82%

64%

78%

59%69%

51%

Left Lane First Group Second Group

• Left Lane students were slightly more likely to successfully complete English (ES = .10, p = .308) and math (ES = .11, p = .196) than students in the first comparison group

• Left Lane students were substantially and statistically significantly more likely to successfully complete English (ES = .32, p = .020) and math (ES = .27, p = .017) than students in the second comparison group

Percent of Students Successfully Completing English, Math, and/or Reading

Female African American Hispanic0%

10%20%30%40%50%60%70%80%90%

100%

81% 81% 79%73%

61%70%69%

45%

66%


• Left Lane female (ES = .20, p < .001, 81% to 73%), African American (ES = .43, p = .007, 81% to 61%), and Hispanic (ES = .19, p < .001, 79% to 70%) students were substantially and statistically significantly more likely to successfully complete courses than students in the first comparison group

• Left Lane female (ES = .31, p < .001, 81% to 68%), African American (ES = .77, p < .001, 81% to 45%), and Hispanic (ES = .27, p < .001, 79% to 66%) students were substantially and statistically significantly more likely to successfully complete courses than students in the second comparison group

English, Math, and Overall Retention (i.e. Persistence – earning GOR) from Fall to Spring

English Math Overall Retention0%

10%20%30%40%50%60%70%80%90%

100%

65%56%

87%

52%46%

80%

46% 43%

75%


• Left Lane English (ES = .25, p = .007), Math(ES = .19, p = .034), and overall (ES = .19, p < .012) retention was substantially and statistically significantly higher for Left Lane students than students in the first comparison group

• Left Lane English (ES = .39, p = .002), Math(ES = .27, p = .018), and overall (ES = .31, p < .001) retention was substantially and statistically significantly higher for Left Lane students than students in the second comparison group

English and Math Improvement Rates from Fall to Spring

Success in Subsequent English

Success in Subsequent Math

Overall Success in Subsequent Course

0%10%20%30%40%50%60%70%80%90%

100%

74%

61%67%

76%66%

71%79%

64%71%


Left Lane students did not perform as well in their subsequent English and/or math course after successfully completing an English and/or math course in their first semester

1. Gather Data / Assess

Process for Continuous Quality Improvement

ContinuousQuality

Improvement


2. Review, Discuss, & Gather

Feedback



Feedback

3. Identify Strategies for Improvement



Feedback


4. Implement Improvements

Communicate Results

Communicate Feedback

Communicate Strategies

Communicate Improvements

Left Lane Program

Summer Bridge Evaluation Surveys

Initial Research after the first Fall semester

More in-depth research after the first year


2. Review, Discuss, &

Gather Feedback



http://www.craftonhills.edu/~/media/Files/SBCCD/CHC/About%20CHC/Research%20and%20Planning/Research%20Briefs/Academic%20Success%20Studies/1213_2012FA_LeftLane_Success2.pdf

http://www.craftonhills.edu/~/media/Files/SBCCD/CHC/About%20CHC/Research%20and%20Planning/Research%20Briefs/Academic%20Success%20Studies/2012FA_LeftLane_Success3.pdf

Left Lane Program




Evaluation surveys and interviews indicated that students wanted both English and math during Summer Bridge

39% of non-Left Lane students participated in SOA3R, a learning community, or supplemental instruction. As a result, first comparison group had success rates 70% or higher.

LLP students are more likely to enroll in math and English in the first term and they were less likely to enroll in and successfully complete math and English in subsequent spring semester.



Gather Feedback





A Note on Facilitating Continuous Quality Improvement

• Traditional approach to facilitating evidence-based decision making– Provide the research results to constituents only in reports, briefs, and

dashboards– Discuss the possible implications and interpretations verbally

• Example– Left Lane students were substantially (ES = 1.10) and statistically

significantly (p < .001) more likely to earn a GOR in a math course their first semester (89%) at Crafton than non-Left Lane students (34%)

– Left Lane students were substantially (ES = 1.03) and statistically significantly (p < .001) more likely to earn a GOR in an English course their first semester (77%) at Crafton than non-Left Lane students (26%)

A Note on Facilitating Continuous Quality Improvement

New approach to facilitating evidence-based decision making• Provide the research results to constituents in reports, briefs, and dashboards, and…• Provide the research results with an interpretation that focuses on the possible implications of the research

Left Lane Comparison Groups and Possible Improvement AreasWhen preparing the comparison groups to examine the relationship between Left Lane Students and student outcomes one of the most interesting findings was the number of Fall 2012 first-time Crafton students who graduated from high school in 2012 who did not participate in Left Lane, but had participated in one of the Left Lane Strategies (LLS). Thirty-nine percent (n = 226) of these students participated in SOA3R, a learning community, or in supplemental instruction. As a result, the first comparison group of incoming students had success rates that were 70% or higher. Moreover, the Left Lane students often did not have a statistically significantly higher likelihood of achieving the outcomes examined than the first comparison group. Suggesting the possibility that one method for improving the LLP is to focus on ensuring that as many students as possible participate in the LLS rather than solely on LLP.

One of the strategies for decreasing the average amount of time it takes students to complete an AA/AS Degree involves increasing how quickly students are able to successfully complete transfer level English and math. The initial results of the research illustrated here indicates that Left Lane students are more likely to enroll in math and English courses in the fall, but in some instances they were not as likely to enroll in math and English in the subsequent semester and/or successfully complete the subsequent English and/or math course in the summer semester. LLP may need to require students to participate in the LLS in the spring semester and devote additional resources to services in the spring semester.

Left Lane Program







Combine Summer Bridge so that students are prepared for both English and math

One method for improving the LLP is to focus on ensuring that as many students as possible participate in the LL strategies.

Require students to participate in LL strategies in the spring along with enrolling in math and English in the spring.



Gather Feedback





Left Lane Program







Combine Summer Bridge so that students are prepared for both English and math

One method for improving the LLP is to focus on ensuring that as many students as possible participate in the LL strategies.

Require students to participate in LL strategies in the spring along with enrolling in math and English in the spring.

Combined the Summer Bridge Program to include both English and Math

Required Students to complete a 2 semester SEP to stay enrolled in math and English

New LLP proposal to SBCCD Chancellor emphasized increasing the capacity of the LLP and its strategies: counseling, SEPs , SOA3R and SI. In addition, also emphasized creation of DLAs in math and English, additional sections in math and English in both fall and spring.

LL strategies are also being included in new Title COOP grant proposal.



Gather Feedback





Additional Questions

the creation of two comparison groups to facilitate evidence-based decision making

Documents