Session Three:Identify Measures of Success

and Plan Cycles of InquiryLauren Bryant and Allison Black-Maier

In this session...1. Revisit logic model from Session 2. Identify key outcomes and measures of

success.2. Check the practicality of your measures of success.3. Establish a plan for conducting collaborative inquiry.

Revisiting our Logic ModelsWithin your project team, consider the following:

What are the key outcomes in your logic model? In other words, what outcomes will be the most important for you to track and measure? Why?

Practical Measures

Today we’re going to focus on two kinds of practical measures: outcome measures and primary driver measures (Bryk et al., 2015).

Practical Measures

Outcome measures operationalize your outcomes, precisely defining the common aim which allows project team members to have explicit conversations about the work of the project. This is the “end-of-the-line” indicator of the success of your project.

Examples of Outcome MeasuresFor an RPP focusing on improving a developmental math program at the community college level:

● The percentage of students who enrolled in the program and passed both semesters with a C or better.

Examples of Outcome MeasuresFor an RPP focusing on increasing eighth grade students’ participation in engineering:

● The percentage of students who took a year-long Introduction to Engineering course in eighth grade who go on to take engineering-related courses in high school.

More long-term possibility:

● The percentage of students who took a year-long Introduction to Engineering course in eighth grade who go on to declare an engineering major in college.

Practical MeasuresWithin your project team, consider the following:

What are possible outcome measures of your key outcome(s)? What would operationalize your key outcome and serve as an end-of-the-line indicator of the success of your project?

Practical Measures

Primary driver measures gather information on the major mechanisms that are going to result in your key outcomes. These measures should:

1. Predict your outcome measure.2. Be sensitive enough to change to give you timely

feedback.3. Provide guidance on what to modify to improve.4. Be folded into cycles of collaborative inquiry.

Primary Driver Measure ExamplesFor an RPP focusing on improving a developmental math program at the community college level:

● A three-minute, 25 item questionnaire to measure students’ productive persistence.

Examples of Primary Driver MeasuresFor an RPP focusing on increasing eighth grade students’ participation in engineering:

● A five-minute, 10-item questionnaire to measure students’ sense of belonging in engineering and interest in engineering.

Practical MeasuresWhat are possible primary driver measures for your project? What measures would meet the following guidelines?

1. Predict your outcome measure.2. Be sensitive enough to change to give you timely feedback.3. Provide guidance on what to modify to improve.4. Be folded into cycles of collaborative inquiry.

Collaborative Cycles of Inquiry

An important aspect of primary driver measures is that they won’t do you any good if they are not integrated into collaborative cycles of inquiry. However, collaborative cycles of inquiry do not evolve naturally in an RPP environment...

Collaborative Cycles of Inquiry

Large project teams are not immune to the Second Law of Thermodynamics - that closed systems tend to move toward chaos.

Reversing this tendency toward chaos requires the input of energy!

Collaborative Cycles of InquiryCollaborative cycles of inquiry are basically smaller research cycles that you build into your project so that you can improve as you go.

This approach is useful (arguably even necessary) for projects that are trying to tackle complex problems for which few to no workable solutions exist (Bryk et al., 2015; Patton, 2010).

Building Collaborative Cycles of InquiryTeam members should ask themselves the following questions:

● How often will you need to collect data using your outcome measure(s)?● How often will you need to collect data using your primary driver

measure(s)?● Who should collect the data? Who will the data be collected from?● Who should the results be shared with, and when? Includes people within

your RPP and people external to your project.● How and when will the results be used? What aspects of your project may

change based on the results?

Examples of Collaborative Cycles of InquiryFor an RPP focusing on improving a developmental math program at the community college level:

Year One

Students take 3-minute survey after every class, to be administered via email push by Mark from the research team. Reminders for survey given by Professors Smith and Brown.

Student data are analyzed every 3 weeks by Jesse from the research team. Results are shared via email to all members of the RPP.

Results are used immediately by Professors Smith and Brown to plan subsequent lessons, and by campus administrators in standing bi-weekly meeting to modify and strengthen external supports to students.

Rapid Cycle

Student grade data is collected one week after the close of the semester by Mark from the research team and analyzed by Jesse from the research team. Results are shared via email to the entire RPP team.

RPP members will meet 3 weeks after the close of the semester to review the student grade data and the final round of survey data. Team will discuss together what changes to make for the following year to better reach their key outcomes.

Changes and modifications will be in place one week before the start of Year Two. Professors Smith and Brown will be responsible for changes to the curriculum, campus administrators will be responsible for changes to external supports.

Annual Cycle

Build Your Collaborative Cycles of Inquiry● How often will you need to collect data using your outcome measure(s)?● How often will you need to collect data using your primary driver

measure(s)?● Who should collect the data? Who will the data be collected from?● Who should the results be shared with, and when? Includes people within

your RPP and people external to your project.● How and when will the results be used? What aspects of your project may

change based on the results?

Gallery Walk

ReferencesBryk, A. S., Gomez, L. M., Grunow, A., & LeMahieu, P. G. (2015). Learning to improve: How america’s schools can get better at getting better. Harvard Education Press: Cambridge, MA.

Fishman, B. J., Penuel, W. R., Allen, A. R., & Cheng, B. H. (Eds.). (2013). Design-based implementation research: Theories, methods, and exemplars. Teachers College, Columbia University.

Patton, M. Q. (2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. Guilford Press.

Additional Resources on Cycles of Inquiry● http://researchandpractice.org/toolkit-gather-and-use-evidence/● https://www.carnegiefoundation.org/blog/revisiting-purposes-practical-measur

ement-improvement-learning-bten-measurement-system/● https://www.carnegiefoundation.org/blog/improvement-discipline-in-practice/● http://www.thecorecollaborative.com/impact-teams● http://hepg.org/hel-home/issues/25_3/helarticle/improving-teaching-and-learni

ng-through-instructio