koontz 713 program evaluation full report
TRANSCRIPT
-
7/28/2019 Koontz 713 Program Evaluation Full Report
1/85
Running Head: CHECK-IN CHECK-OUT
Check-in Check-out Program Evaluation
Full Program Evaluation Report
Joseph Koontz
Submitted in partial fulfillment of the requirements for:
EDLP 713 Evidence-informed Perspectives on Practice II
April 17, 2013
-
7/28/2019 Koontz 713 Program Evaluation Full Report
2/85
CHECK-IN CHECK-OUT 2
In order to protect the anonymity of the participants in this evaluation, the school being
evaluated will be referred to as Suburban Elementary. Suburban Elementary is a pre-
kindergarten through grade five school with approximately 500 students located in Central
Virginia. The majority of the students at Suburban Elementary receive free or reduced lunch,
and Suburban Elementary receives Title I funding. While it has not performed well in the past, it
has recently made great gains in student performance. A portion of that recent success may be
attributed to the school-wide positive behavior support program. This program evaluation will
focus on a component of that school-wide system called Check-in/ Check-out (CICO).
History and Background
From 2008 - 2010, Suburban Elementary failed to earn Virginia Department of Education
(VDOE) academic accreditation. Schools that failed to earn state accreditation were required to
implement a specific school improvement plan model. There were many academic improvement
plans, as well as plans to reduce discipline. Disruptive behaviors lower overall student
achievement because they impede the learning of all students and monopolize a significant
portion of teachers and administrators time. (Anderson, 2009)
In 2009, Suburban Elementary implemented a school-wide discipline program. This
program is based on a system developed by VDOEs Training and Technical Assistance Center
(TTAC). This school-wide program is designed to create a universal set of expectations and
consequences for students behavior. In September 2010, Suburban Elementary implemented
phase II of TTACs program. This second phase consisted of a positive behavior support system
for students who needed individualized help beyond the school-wide discipline plan.
Positive behavior support is a proactive way of dealing with student behavior.
Traditional methods of discipline, such as suspension, time outs, and loss of privileges are
-
7/28/2019 Koontz 713 Program Evaluation Full Report
3/85
CHECK-IN CHECK-OUT 3
reactive. Positive behavior support is intended to reduce the exclusionary practices of traditional
discipline in favor of proactive behavior management. (Luiselli, Putnam, Handler, & Feinberg,
2005)
Suburban Elementary realized that student behavior was taking up a large portion of
teachers and administrators time. By reducing discipline, Suburban Elementary hopes to
increase the time students are on task and learning. Their school-wide discipline program and
the CICO portion of that program are part of the larger school improvement plan aimed at raising
overall student achievement. Suburban Elementary is starting the third year of the CICO
program and is ready to evaluate the program to determine if the time and resources invested are
helping toward their goal of increased student achievement.
Program Goals
The ultimate goal of CICO is to prevent disruptive behavior in students who need support
beyond the effective school-wide discipline program. The program is designed for students to
check-in with an adult in the morning, carry an individualized behavior plan throughout the day,
and check-out with the same adult at the end of the day. The check-in is intended to be a
positive start to the day. The adult conferences with the student and together they talk about the
expectations for the day and set behavior goals. The student then takes a behavior chart with him
or her that teachers use to record their behavior at different intervals throughout the day. The
intervals are designed to be short time periods with specific behavior expectations. These
behavior expectations are written in simple terms with a rating scale that has only three
indicators for the teacher to circle or mark. The individualized behavior charts are designed to
be easy for the student to understand and the teacher to document. At the end of the day, the
students again conference with the same adult and they chart their individual progress toward
-
7/28/2019 Koontz 713 Program Evaluation Full Report
4/85
CHECK-IN CHECK-OUT 4
goals they set with their adult mentor. Luiselli et al. (2005) summarizes the goals of positive
behavior support systems:
1. Setting behavior expectations.
2. Teaching interpersonal skills.
3. Provide reinforcement for reaching.
4. Monitors behavior through data analysis.
5. Involves all stakeholders including the students, parents, teachers, and administrators.
The effective school-wide discipline program is beneficial for the majority of students
who attend Suburban Elementary. For those students who need more personalized attention, the
CICO system helps minimize their negative behavior.
Logic Model
Figure 1 shows the logic model for the CICO program at Suburban Elementary. This
logic model was developed with input from the client and it is a helpful guide when analyzing
the program.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
5/85
CHECK-IN CHECK-OUT 5
Resources/
InputsActivities Outputs Outcomes Impact
Check-in/
Check-out
positivebehavior support
systemframework
documentation
School
Administration
Personnel
Student
incentives
Individualizedstudent behavior
plans
Funds for
student
incentives.
Identification of
students who are
unsuccessfulusing the Tier 1
School-wideDiscipline Plan
Pairing upstudents in
CICO with staff
mentor/advisor
Progress
monitoring astudent in the
CICO program
Daily check-in
and check-outbetween
students and
administrator
Counseling/ goal
setting withstudents in theCICO program
Rewardingstudents for
reaching set
behavior goals
Teachers fill out
students daily
behavior chart
Student
behavior plan
completed withteacher
documentation
Student referrals
Qualitative
information
from daily
studentconferences
Discipline Data
Students build
relationships
with an adult inthe school
building
Students take
ownership oftheir own
behavior
Decrease instudent
disciplinereferrals
Less teacher
and
administrativetime is spent on
student
discipline
Decrease in
negative
studentbehavior
Increased
student
achievement
Improved
school climate
Figure 1. Check-in / Check-out Logic Model for Suburban Elementary
-
7/28/2019 Koontz 713 Program Evaluation Full Report
6/85
CHECK-IN CHECK-OUT 6
Context
Political context. Suburban Elementary was on a school improvement plan that was
supervised by the state of Virginia. The county of Henrico and the state of Virginia invested a
myriad of resources into Suburban Elementary. Academic and behavioral data was scrutinized.
This type of scrutiny brings political pressure because public funding is involved. Part of the
reason Suburban Elementary implemented this program was because it was approved by the
VDOE through TTAC.
In recent months, the political pressure on school discipline has increases because of a
series of articles run in the Richmond Times-Dispatch. Virginia State Sen. A. Donald McEachin,
D-Henrico, expressed concerns in an October 2, 2013 letter co-written by Henrico County
District Supervisor Tyrone E. Nelson about the suspension rate of African American males.
(Martz, 2012) The principal made an appearance before the school board to discuss her
discipline practices. This CICO program is a critical part of the process for reducing their
suspensions.
Cultural context. Student discipline has the potential to have cultural implications.
Cultural and ethnic groups view discipline consequences differently. Furthermore, some parents
may see this type of individualized plan as punitive, rather than preventative. Parents may feel
their child is being singled out or they are somehow being stereotyped by this plan. The other
students in the school could also identify the students in the CICO program. There is the
potential for teasing and bullying based on participation in this program because the students
have a behavior chart they must carry to all classes. This differentiates these students from the
other students who do not have a plan. This program evaluation will help determine if the
positive outcomes outweigh the potential negative consequences of CICO.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
7/85
CHECK-IN CHECK-OUT 7
Evaluation Purpose
The administrative team at Suburban Elementary has already confirmed that this program
is successful at reducing the negative behavior of students. The data is clear and for that reason
this will not be a quantitative study looking at suspension rates or discipline referrals only. This
will be a responsive evaluation using mostly qualitative data to look at the efficiency of the
program. Suburban Elementary invests substantial resources in the CICO program, which is part
of the School-wide Positive Behavior Support (SPBS) program. Two of the three members of the
administration meet with students every morning and afternoon. The schools principal would
like to know if this investment is as effective and efficient as possible. The purpose of this
evaluation was to find areas in need of improvement in the CICO process. The evaluation was
guided by the following questions.
1. Are there aspects of the Check-in Check-out program that can be improved or run moreefficiently?
2. What are the perceptions of the Check-in Check-out program from the differentstakeholders?
This was a responsive evaluation where the first goal was to understand the CICO
program from the perspective of different stakeholders to be able to make recommendations for
improvement.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
8/85
CHECK-IN CHECK-OUT 8
Literature Review
Thousands of schools across the country have implemented school-wide positive
behavior support (SPBS) programs as a way to decrease problem behavior and improve school
climate. (Cohen, Kincaid, & Childs, 2007) These programs are designed to take a strategic,
proactive approach to student discipline and school climate. Traditionally, school discipline
policies tendto be mostly reactive and consequence driven. SPBS is a proactive way of dealing
with student behavior. It focuses on positive interactions, clear expectations, and consistent
consequences. SPBS programs also recognize that student behavior and school climate are
directly related to student achievement. SPBS programs are often part of schools
comprehensive improvement plans, which tie directly to state and local accreditation
requirements. This program evaluation will focus on a component of the SPBS called check-in/
check out.
School-wide positive behavior support programs are part of a larger initiative called
Response to Intervention (RTI). RTI can have academic or behavioral based programs that have
multiple implementation tiers. Tier-one implementation refers to school-wide expectations and
outcomes. Tier-one is the foundation for a successful SPBS program, but there will be a portion
of the school population that will need additional support beyond the tier-one program. Those
students would be served through tier-two intervention, also know as targeted intervention. This
literature review will focus on the research-supporting tier-one SPBS as well as tier-two, targeted
intervention.
Education Research Complete Database and Google Scholar were used to find peer-
reviewed articles dealing with school-wide positive behavior support programs. The following
key word search phrases were used to locate articles:
-
7/28/2019 Koontz 713 Program Evaluation Full Report
9/85
CHECK-IN CHECK-OUT 9
School-wide Behavior Program Positive Behavior Support Programs School-wide Positive Behavior Support Targeted Behavior Intervention Tier Two Behavior Intervention School-wide Behavior Support Evaluation
Articles were selected for review based on their relevance to the topic and based on the
type of study that was conducted. Articles that simply described SPBS and did not conduct or
review any study were rejected.
Tier-one: School-Wide Positive Behavior Support
A 2011 research article by Caldarella, Shatzer, Gray, Young, and Young used a quasi-
experimental design to prove the effectiveness of a SPBS on middle school climate and student
outcomes. The researchers used a non-equivalent two-group model with a pretest and posttest to
measure outcomes. This study included more than 300 teacher responses and 10,000 student
responses. Two schools were studied with one school serving as the control over a four-year
period.
In the school that implemented a SPBS program, there were statically significant
improvements in how teachers rated the school as well as a decrease in office referrals. The
study also found there was a statically significant correlation between the SPBS program and
student tardies and unexcused absences. If students are present on a regular basis, there is less of
a chance they fall behind due to missed work. In addition to the positive behavior outcomes, it
seems student participation increases. This is not a surprising correlation, but it does highlight
how behavior and school climate affect morethan just the number of office referrals.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
10/85
CHECK-IN CHECK-OUT 10
The SPBS program in this study had similar key features to many programs like it. The
program promotes school-wide teaching of social skills, praise notes from teachers to students,
posting of school rules, proactive screening for students at risk for emotional and behavioral
disorders, and referrals of at-risk students for targeted intervention. (Paul, et al., 2011) One of
the ways SPBS is different than traditional discipline programs is the emphasis on positive adult
interactions. Furthermore, there are clear expectations for the students and the teachers. This
study also talks about targeted intervention, which will be covered in subsequent articles. SPBS
programs recognize that not all students are receptive to the school-wide model. This study
indicates 15% of students may require a more intensive and individualized plan. These
students would require tier-two intervention.
A 2005 study by Luiselli, Putnam, Handler, and Feinberg highlights the link between
behavior and academic performance. This study sites research that suggests that behavior
problems in young children predict maladjustment later in life. Luiselli et al. (2005) contend that
behavior has a direct link to academic performance and that early intervention is critical to
success. The authors of this study use a before and after design model using a suburban
elementary school of approximately 600 students. The school volunteered for the study because
the administration had a desire for improved academic achievement and reduced discipline. The
school would implement a positive school-wide behavior support program and the discipline and
academic data would be tracked for a three-year period to show if the SPBS program was
effective. This study also used a questionnaire to gauge the social validity of the SPBS program.
Teachers were given a questionnaire in years two and three of the study to assess perceptions of
the program effectiveness and the impact on student learning.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
11/85
CHECK-IN CHECK-OUT 11
The results of this study affirmed the effectiveness of a school-wide behavior program.
Luiselli et al. stated, by virtue of reducing discipline problems, teachers can devote more time to
instruction and other learning opportunities that maximize educational progress. (2005, p. 195)
Over a three-year period, the study noted a decrease in suspensions and office referrals. The
study also noted an increase in student performance measured by the Metropolitan Achievement
TestSeventh Edition. The teachers also saw a positive effect on the school climate. The
number of teachers that indicated, the school discipline is effective (Luiselli et al., 2005,
p.194) went from less than 50% in year 1 to 100% in year 2. This study is limited because it
only looks at one school, but it does illustrate a successful school-wide implementation.
According to these studies and many others, SPBS programs have a clear benefit for
students, teachers, and administrators and have a positive impact on a schools climate and
achievement.
The previous studies have proved the value of positive school-wide behavior support
programs. Once a SPBS program is adopted, implementation quality and fidelity are critical
factors to a successful SPBS program. An article by Cohen, Kincaid, and Childs (2007) uses a
qualitative pilot study to look at the most effective ways to measure school-wide positive
behavior support implementation. According to their findings, more than 4,000 schools have
implemented some type of SPBS program. Cohen et al. credit these programs for decreasing
student discipline referrals and increasing positive climate. The authors contend the degree to
which a school implements a program with fidelity has an effect on the quality of the program.
The School-wide Benchmark of Quality (SBQ) is a tool to evaluate a SPBS program.
The SBQ is a 53 question rating scale that measures the degree of fidelity with which a
school is implementing a SPBS program. (Cohen et al., 2007, p. 204) 47 schools in two states
-
7/28/2019 Koontz 713 Program Evaluation Full Report
12/85
CHECK-IN CHECK-OUT 12
piloted the use of the SBQ. Their study finds this tool to be reliable and valid tool for evaluating
SBPS implementation. Their data suggests there is a direct correlation between a high score
(>70) and a decrease in office discipline referrals. For school administrators, having some way
to quantify the fidelity of implementation of programs is important.
Miramontes, Marchant, Heath, and Fishcer (2011) contend that social validity is an
important factor to the success of a school-wide discipline program. According to Miramontes et
al., social validity describes the value that society places on something. Miramontes et al. (2011)
contend that programs should be evaluated on more than goals, procedures, and outcomes and
should go beyond clinical judgment. (p. 446) The higher the degree of social validity, the more
individuals will buy into and be satisfied with a program. Their study attempts to measure the
social validity of positive behavior intervention and support programs using a questionnaire to
measure the social validity of stakeholders who were involved with a SPBS program.
One of the most significant findings from the study is the relationship between fidelity of
implementation and social validity. Miramontes et al. (2011) state, small yet statistically
significant correlations were found between faithful program implementation and increased
social validity, indication that the more accurately a program was implemented, the more
valuable the consumers perceived the procedures. (p.463) Implementation fidelity is a
consistent theme with several studies. This article highlights the need for stakeholders to find the
program relevant and practical in their everyday practice.
Tier-two: Targeted Intervention
Once a school has established an effective SPBS program, the next step is implementing
a tier two intervention program, also referred to as targeted intervention. Students who are not
responsive to SPBS will require targeted intervention. This targeted intervention for a small
-
7/28/2019 Koontz 713 Program Evaluation Full Report
13/85
CHECK-IN CHECK-OUT 13
percentage of students will look different for each student. Most targeted intervention involves
some type of check-in / check-out system. Students check-in with an adult in the morning, they
follow an individualized behavior plan during the day, and check-out with that same adult in the
afternoon.
According to a 2010 study by Cheney, Lynass, Flower, Waugh, Iwaszuk, Mielenz, and
Hawken, children with emotional and behavioral disabilities are at riskand often interfere
with teachers efforts to educate other children in the classroom. (p.152) Targeting these
students will not only benefit the students with significant behavior challenges, it will also help
the other students in the class by limiting disruptions. Cheney et al. (2010) also note that these
students are more likely to fail courses, drop out of high school, and are more likely to have
difficulties with relationships and employment when they are older.
This meta-analysis by Cheney et al. uses data from 18 urban elementary schools in the
Seattle metropolitan area. This article notes that their research shows 70% of students in targeted
intervention respond favorably, but 1015% of the students may need even more services such
as coaching on problem-solving skills and conflict resolution. There will be a small percentage
of students who do not respond to the increased support and they may have significant
disabilities that require special education services.
A study by Hawken and Horner (2003) prove the effectiveness of targeted behavior
support as a component of a SPBS program. This study shows a relationship between
intervention implementation and a reduction in problem behavior. This study confirms the
proportion of students who typically do not respond to the tier-one program.
This study looked at a group of students within a middle school in the pacific northwest
who had five or more office referrals and who were currently not receiving targeted intervention.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
14/85
CHECK-IN CHECK-OUT 14
The students were added to the CICO program where they check-in with an adult for a quick
conference upon arrival, they carry with them a data collection tool for their teachers to fill out
about their behavior, and they check-out at the end of the day with the same adult. In this study,
there was also a daily home-school report for reinforcement by the parents.
This study found while overall levels of problem behavior were reduced following the
intervention, the larger effect was that students became more consistent in participating in class
without problem behavior. (Hawken & Horner, 2010, p. 237) The study also had some
important suggestions for schools implementing a targeted intervention program. They found
that the match up between the students and the adult had to be a good fit. Hawkin and Horner
(2010) found that if students do not find adult attention reinforcing the BEP [targeted
intervention] will be less effective. (p.237) This study acknowledges the importance of the
quality of the support provided by the adult in the CICO system. Further study would be needed
to identify the degree to which to adult verses the behavior chart had more effect on the students
behavior.
A study by Medley, Little, and Akin-Little (2008) illustrates the importance of having a
strong tier-one system before implementing a tier-two system. Their study finds that schools
with a SPBS as a foundation write more effective targeted assistance plans. This study used an
instrument called the Behavior Support Plan-Quality Evaluation (BSP-QE) to rate targeted
assistance plans. They found using the BSP-QE that schools the foundation that is created by the
school-wide program creates an environment that is more conducive to targeted assistance.
Unfortunately, just because a school has a SPBS program in place, does not ensure a quality
targeted intervention program. The study found that all schools could improve their intervention
-
7/28/2019 Koontz 713 Program Evaluation Full Report
15/85
CHECK-IN CHECK-OUT 15
plans. The implications for administrators would be that they should use a tool like the BSP-QE
to evaluate their own intervention plans.
A multi-tiered behavior intervention system can have potential negative outcomes if not
properly implemented. A 2009 study by McIntosch, Campbell, Carter, and Dickey studied 34
students in a school district that were identified for tier-two behavior support. The findings
clearly showed a positive correlation between a tier-two intervention program and a reduction in
negative behavior, the study also highlighted some areas of caution for school personnel. The
authors of this study highlight the potential for over-identifying students for special education
services. Students who are not successful with a tier-two intervention such as CICO are often
referred for special education.
The authors contend there is not enough research regarding the move between tier-two
and tier-three or special education services. McIntosch et al. (2009) caution schools to avoid the
one-trick pony phenomenon where schools only have one form of tier-two intervention. (p.89)
Just as the school-wide plan does not fit all students, a check-in/check-out program might not fit
all students participating in a tier-two program. Furthermore, just because a student is
unsuccessful in a specific tier-two program like CICO, does not necessarily mean they should be
referred for special education. McIntosch et al. (2009) states a lack of student response to a tier
two intervention may be misconstrued as evidence of a disability when an equally likely
explanation may be that there was a poor match between the intervention and the function of the
students behavior. (p.89)
Conclusion: Impact for School Leaders
Current research is in agreement that School-wide Positive Behavior Support Programs
are an effective way to decrease negative student behavior and increase positive school climate.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
16/85
CHECK-IN CHECK-OUT 16
The research also indicates for some students, this school-wide program is not enough. For those
students, tier-two or targeted intervention is needed. The check-in/check out model is proven to
be effective for many of those students who need extra support.
For school administrators, there are several key themes that emerged from the current
literature. There is a need to evaluate school programs. There are tools available that can
quantify the fidelity of implementation and effectiveness of the programs. Administrators need
to go beyond the number of discipline referrals and results from climate surveys. There is also a
clear theme of implementation fidelity. Having all staff implement a SPBS program correctly
and consistently is critical to the success of the program. Finally, school leaders cannot forget
about the need to create buy-in from their stakeholders. The research shows the importance of
social validly to the success of a program.
Method
When this study was first conceptualized, it was originally a quantitative study looking at
the effectiveness of the CICO program in terms of the discipline data such as office referrals and
suspensions. After an initial conversation with the client, it was clear that this evaluation would
be much more qualitative in nature. The client has a clear understanding of the quantitativedata.
She knows that this program is effective in reducing the number of office referrals and
suspensions. She was more interested in the perceptions of the program and the efficiency of the
daily process. For this reason, the evaluation became a responsive evaluation using a mixed
method approach.
Participant Characteristics
-
7/28/2019 Koontz 713 Program Evaluation Full Report
17/85
CHECK-IN CHECK-OUT 17
There are many stakeholders involved in the CICO program. The administrators at
Suburban Elementary are responsible for checking-in and checking-out with the students in this
program each day. The students in the program are the most significant stakeholders. They are
the reason for the program and they are the students with the most significant behavior
challenges. They also have a substantial impact on the overall instruction and climate of the
school. Their behavior effects instruction and the teachers ability to effectively run his or her
classroom. During this program evaluation, all of these stakeholders may need to be involved
because each play a different role in the program.
The principal is ultimately responsible for all programs at Suburban Elementary. That
being said, the principal is intentionally removed from the CICO process for several reasons.
The adults involved in the CICO system need to maintain a positive relationship with the
students participating. The principal needs to maintain a certain detachment in order to be the
disciplinary figure if students in the CICO program need immediate consequences. Furthermore,
the responsibilities of a principal make it difficult to be available every morning and every
afternoon. The adults involved with the CICO system need to have consistent availability.
The school counselor and the resource teacher are the two primary individuals
responsible for this program. They each have a caseload of students who check-in and check-out
with them each day. They are also responsible for determining which students participate in the
program and for how long. They are also responsible for progress monitoring these students to
ensure the interventions are working. Any stakeholder has the opportunity to refer a student to
the CICO program, but the associate principal and resource teacher make the final determination.
The teachers are responsible for completing the individualized behavior charts throughout
the day for the children in this program. It is important for the teachers to implement the
-
7/28/2019 Koontz 713 Program Evaluation Full Report
18/85
CHECK-IN CHECK-OUT 18
requirements of the program with fidelity. They need to follow the daily behavior chart and
make accurate reports in order for the student to be successful. Cohen, Kincaid, and Childs
(2007) found the higher the degree of implementation fidelity, the more successful a school-wide
discipline is.
Parents are also significant stakeholders, although in conducting the study, it is clear that
parents are not a significant part of the CICO program in its current form.
The evaluation used enterprise documents, interviews, focus groups, and surveys to
collect information to help evaluate the CICO program at Suburban Elementary.
Evaluation Design
An initial interview with the schools principal and a review of the schools internal
documents helped shape the initial understanding of the program. With the help of the client, an
initial logic model was created and a tentative plan for conducting an evaluation was outlined.
The literature review was clear that any behavior intervention program is only as good as
the people implementing it. Therefore, a responsive evaluation method was chosen. Robert
Stake states, it [responsive evaluation] is not particularly responsive to program theory or stated
goals; it is responsive to stakeholder concerns. (Stake, 204, p.89) In the initial evaluation
design, stakeholder identification and interaction was emphasized. A meta-evaluation was also
important. A consultation with the principal before each data collection activity was built into
the design. The principal approved all questions and processes for collecting data from
stakeholders.
Interviews. Interviewing key stakeholders was part of the initial understanding process.
After an initial interview with the principal, an interview was conducted with the resource
teacher and the school counselor. Both of these individuals were chosen for interviews because
-
7/28/2019 Koontz 713 Program Evaluation Full Report
19/85
CHECK-IN CHECK-OUT 19
they are the stakeholders who run the CICO program on a daily basis. The students in the
program check-in and check-out with these two individuals. Interviewing these two people
would help get a better understanding of the process. Furthermore, the opinions of these two key
stakeholders would be valuable. For that reason, they were interviewed individually, not in a
focus group. The transcription of these interviews can be found in Appendices A and B.
Focus Groups. Focus groups are an efficient method for gathering data from a group of
stakeholders. Focus groups also allow for interplay between the stakeholders that can also
provide valuable information and context. Two focus groups were conducted with teachers from
Suburban Elementary. A complete transcription of the two focus groups can be found in
Appendices C and D.
Surveys. In designing this evaluation, it was important to have a way of gathering some
anonymous information from the stakeholders. In focus groups and interviews, stakeholders
might hold back or not be truthful out of fear of repercussions or because they are simply not
comfortable. Surveys allow individuals a way to be as truthful as they want. A survey was
designed for the teachers and the students with an area on the teacher survey for open ended
responses. A copy of the teacher and student surveys can be found in Appendices E and F
respectively.
Initially, a parent survey was developed, but after consulting with the client, it was
abandoned. For various reasons, the client did not want to involve the parents in this program
evaluation.
Steps taken to enhance credibility. There were several aspects of the evaluation design
that enhanced the credibility of the qualitative data collection.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
20/85
CHECK-IN CHECK-OUT 20
Multiple data sources. Having multiple data sources from multiple stakeholders was a
critical aspect of the program design that enhanced the credibility. Having interviews, focus
groups, observations, and surveys helped to enhance the credibility of each individually.
Member checks. Before and after each data collection process, a member check was
conducted with the client. This helped shape the next data collection tool and it allowed for
clarification of perceptions and interpretations. These member checks are referenced in the data
collection flow chart in Figure 2.
Progressive subjectivity. The research journal and reflections of the evaluator serve as a
way to increase credibility. Novice evaluators will learn as they go and the reflective thought
throughout the data collection process helps to drive the credibility of the evaluation.
Steps taken to enhance dependability. There were several features built into the design
of the study that enhanced the dependability of the data collected. First, the data was collected
sequentially. This allowed for the data collection questions and methods to evolve with new
understanding. If the surveys, interviews, and focus groups had been conducted concurrently,
there would not have been the opportunity to incorporate new understanding. When there was
more than one interview or focus group that were meant to be identical, planning was done to
make sure they were conducted as close together as possible. Both administrator interviews
were conducted on the same day and the teacher focus groups were conducted only one day a
apart.
Also, the evaluator made sure to transcribe the data in a timely manner, but waited to
code the transcriptions until the end of the evaluation. By waiting to code at the end, it ensures
that the same codes and thought process is used throughout.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
21/85
CHECK-IN CHECK-OUT 21
Data Collection
A flow chart of the data collection process can be seen in Figure 2 below. After each
data collection process, there was a member check with the schools principal. This allowed for
the data collection process to evolve with increased understanding.
Data Collection Process Flow Chart
Figure 2. Flow chart depicting the process for data collection, check for understanding, and plan
modification throughout the evaluation.
The initial interviews were extremely helpful in shaping the focus group and survey
process. One of the first discussions centered on how to survey the students. The principal did
not want to have to get permission from parents for the evaluator to conduct a focus group. The
evaluator and the client decided that someone from within the school would administer a brief
-
7/28/2019 Koontz 713 Program Evaluation Full Report
22/85
CHECK-IN CHECK-OUT 22
survey to the students. This would be someone the students were confortable with. A simple,
three-question survey was developed and administered to the students in the program. While this
was not ideal, it was a compromise that the client was comfortable with.
The teacher survey was sent out using Google Docs. The survey provided data that
helped shape the focus group questions. After discussing the results with the client, it was
decided to add additional questions about the Tier One, school-wide behavior support program.
The research indicated that having a strong Tier One program was important for the success of a
Tier Two program. The evaluator wanted to make sure there was some data collected about the
Tier One program.
There was also a compromise made on the teacher focus groups. Initially, the plan called
for separate focus groups for teachers who had students in the CICO program and teachers who
did not have students in the program. This was initially set up, but weather cancelations forced
the groups to be rescheduled. With the short notice, the focus groups had to be a mix of teachers.
While this was not the original plan, it ended up enriching the discussion. The evaluator relied
on the client to set up the focus groups. If these were conducted again, it would have been
helpful for the evaluator to have a list of the participants and to be able to touch base with them
ahead of time. It should also be noted that during one of the focus groups, the power went out.
There was a window in the room, so the blinds were opened and the focus group continued. The
participants indicated that they did not mind and this event did not seem to interfere with the data
collection.
Given more time, a follow up survey and focus group would have been helpful. Time
and logistics did not allow for this, but further questions about the students/teacher interaction
during the day would have been helpful.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
23/85
CHECK-IN CHECK-OUT 23
Data Analysis and Interpretation
Several overriding themes emerged from the data analysis. The overriding themes of
Time and Communication were evident throughout the multiple data sources. Within the theme
of Time, subthemes of equity, fidelity, and consistency all surfaced. Within the theme of
Communication, oral, written, and parent communication were prominent subthemes. Each of
these themes will be addressed within the different data collection methods below where they
apply.
Observations
Time. Time was the major theme that surfaced through the observations. The students
and CICO administrators seemed rushed through the process. Both CICO administrators later
confirmed this observation. These observations of the process were conducted early on in the
evaluation. If time had permitted, observations would have been beneficial after the interviews
and focus groups because the evaluator had a better understanding of the process. Also,
observing the teacher in the classroom filling out the behavior chart would have been helpful.
Time and scheduling did not permit such observations.
Interviews
The interviews were an invaluable part of the evaluation. Through the interviews, the
evaluator was able to gain an in-depth understanding of the CICO process. The themes of time,
communication, and parent involvement were all evident.
Time. The initial interviews with the program administrators confirmed some of my
observations. The interviewers agreed that time was a major factor. Both indicated that more
time with the students would improve the CICO program, especially at the end of the day. They
-
7/28/2019 Koontz 713 Program Evaluation Full Report
24/85
CHECK-IN CHECK-OUT 24
also indicated that they wanted more time to counsel the students. One example came up where
the interviewee talked about a report that he got from the teacher. He indicated that if he had the
time, he would have made sure the student understood the ramifications of her actions. When
each CICO administrator was asked what was the most effective aspect of the CICO program,
they both indicated the relationships they formed with the students. When asked what could
strengthen the program, they both indicated a desire for more individual time with the students.
Relations and time go hand in hand and there seem to be an agreement by both administrators
that time is an issue.
Communication. There are several subthemes that surfaced in the interviews
surrounding communication. The administrators wished they could communicate more with the
students but they both felt like that had the relationships formed with the students for them to
communicate effectively. Communication between the teacher and the CICO administrator is
done primarily through the students daily behavior chart. Significant emphasis is placed on the
score the students receive on this behavior chart. See Appendix I for a sample student behavior
chart. Students are striving for a 54 each day, but anything over a 40 is considered a good day.
Teachers can also leave anecdotal notes about the students day. These notes can help provide
some context for the score, but not all teachers leave a note.
Communication consistency also emerged as a theme. One interviewee mentioned that
he had modified his students form so that it worked for him. This could create some
inconsistencies for the teacher who is filling out the forms. The interviewees did not indicate if
there was a consistent communication between the teacher and the CICO administrator.
The topic of parent involvement and communication came up with both interviewees.
There were inconsistencies between the two interviews. One interviewee sent home the
-
7/28/2019 Koontz 713 Program Evaluation Full Report
25/85
CHECK-IN CHECK-OUT 25
students behavior chart each week, and one did not. They both talked about calling parents
when they needed to for follow up at home.
The same inconsistency occurred with the communication about expectations for the
kids. Each CICO administrator had different number expectations for the behavior chart. One
administrator wanted the students to strive for a perfect score; while the other set the goals lower
and had the students work up to a larger goal.
When each CICO administrator was asked about training and what training they received
to administer this program, they had different answers. One administrator said she went to a
conference five or six years ago. The other administrator said he had not received any formal
training. Both indicated that they did not communicate regularly with each other about the CICO
program.
Focus Groups
Two focus groups were conducted. The first focus group had five participants and the
second had eight. There was a mix of teachers from all grade levels and subjects. The focus
shifted slightly from the interview and survey questions. The focus group discussion started by
asking the participants about the Tier One behavior program at suburban elementary. This was
done strategically based on the literature review. During the literature review phase, it became
clear that a school needed to have a strong Tier One program in order for their Tier Two program
to be successful. When the evaluator was not charged with reviewing the Tier One program,
gathering some information about that program could be helpful in evaluating the Tier Two
program. The focus group data indicated that the Tier One program at Suburban Elementary was
strong and they all felt like it was effective at dealing with the behaviors of the majority of the
students. A common theme that emerged as a strength of the Tier One program was consistent
-
7/28/2019 Koontz 713 Program Evaluation Full Report
26/85
CHECK-IN CHECK-OUT 26
communication and expectations. Several times the participants mentioned the common
language and consistent message that the Tier One program provided. The client may be able
to take some of the strengths of the Tier One program and incorporate them into the Tier Two
program.
Time. Similar to the interview responses, time is a factor for the teachers in several
ways. One of the questions asked the teachers if they thought the student chart was manageable
from the teacher perspective. The teachers who had one student in CICO said yes, it was
manageable and there was enough time. Three teachers had more than one student in the
program and they all indicated that it was much more difficult to manage when they had multiple
students in the program. This will be something the school administration will want to consider
when placing students in the program.
The teachers also talked about the lost instructional time at the end of the day. The
students leave their classrooms early at the end of the day in order to check-out with their
assigned administrator. The teachers questioned wheather the students were being as efficient as
they could with this time. Several teachers also had concerns that some of their biggest behavior
problems had unsupervised time as they transitioned from their classroom to the check-out
location.
A theme that emerged across both focus groups was the concept of equity versus
equality. The teachers seemed to understand that these students needed a higher level of support,
but they also had some cognitive dissonance about rewarding students for mediocre behavior.
They also vocalized a desire to recognize and reward the students who consistently demonstrate
good behavior. The teachers worried that they were not setting the students up for the real
world where individuals are not rewarded for every correct behavior.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
27/85
CHECK-IN CHECK-OUT 27
Time for more classroom guidance emerged as a theme during the focus groups. The
teachers talked about a need for specific guidance with all students on how to interact with one
and other. This would include small group and classroom guidance.
Communication. There seemed to be a misunderstanding about the student chart and
what goes home to parents. Some teachers were under the impression that the behavior chart
went home to parents each day. Some teachers had been writing notes each day thinking they
were making it home to parents. There did not seem to be a clear consensus among the teachers
about what should be communicated on the chart each day. The teachers also indicated that they
had little interaction with the CICO administrator. School-wide, teachers use the students
agenda books to communicate with parents about student behavior. Some teachers still use this
means of communication with their CICO students and some did not. This will be something the
school administration should clarify with the teachers.
All the teachers who have students in the CICO use the student behavior sheet. This is
the most consistent way they communicate their students behavior on a dailybasis. This
quantifiable data point is subjective. This seemed to be another area where the teachers did not
agree. The behavior scale used goes from one to three, where a one is poor behavior and a three
represents good behavior. How one teacher grades a three could be very different than another
teacher. The teachers expressed a desire to have more clarification about how to use the chart so
they are consistent between classes. The teachers talked about the proscribed nature of the Tier 1
program. They expressed a desire to have the same consistency using the CICO program.
The teachers were also unsure about how to formally refer a student to the CICO
program. Some teachers voiced a concern that a teacher who writes up a student many times
gets preference. The teachers do not want their students to be overlooked because they have not
-
7/28/2019 Koontz 713 Program Evaluation Full Report
28/85
CHECK-IN CHECK-OUT 28
written them up enough. They expressed a desire for more transparency in the CICO selection
process.
The teachers had many suggestions for how the CICO could be improved. These ideas
will be fully discussed in the Discussion section of this report.
Quantitative Measures
Two surveys were conducted. One with the students in the CICO program and one was
given to the whole staff. An employee at Suburban Elementary gave the survey to 13 students.
The staff survey was conducted using a Google Doc form. It was sent to 38 staff members with
29 responses. The data from the teacher and student responses can be found in Appendices G
and H respectively.
Student survey. The student survey only had three questions. The first question asked,
What do you like about the Check-in/Check-out program? Six of the 13 students indicated
they like the candy or prizes. Three students indicated they like that the program helped them
control their behavior and the final four students responded with various other answers. 12 of
the 13 students were able to articulate that they behaved better because they were in the CICO
program.
Teacher survey. The teacher survey categorized the teachers into three groups, teachers
who currently have students in the CICO program, teachers who have previously had students in
the program, and teachers who have never had students in the program. The intent was to see if
there was a difference in the perception of the program among these groups. On nine out of the
10 questions on the survey, there was a difference between the teachers who have or have had
students in the CICO program and the teachers who have never had students in the program. The
largest difference among the groups was on question three which asked, The students behavior
-
7/28/2019 Koontz 713 Program Evaluation Full Report
29/85
CHECK-IN CHECK-OUT 29
chart is easy for the classroom teacher to complete. The teachers responded using a five-point
scale where a 1 represented disagree and 5 represented complete agreement. The teachers
who have students currently in the program rated this question almost a full point higher than
teachers who have never had a student in the program. That indicates there is a misperception
among the teachers who have never been a part of the program.
One question that all three groups rated lower than the other question had to do with
parent involvement. The question asked the teachers, I feel the level of parent communication
in the Check-in/Check-out program is sufficient. The teachers with students in the CICO
program gave this question an average rating of 2.93, the lowest of any category. One of the
survey respondents left the following message that may be of interest to the school
administration. He or she wrote, I do not know about the parent communication so am
interested in learning more. Also maybe clearer guidelines about when a student leaves the
program. Another respondent wrote, In the past when I have had students participate it
seemed that there was not clear communication between the teacher and the mentor about what
the expectations were. Both of these statements speak to a lack of communication and
understanding that could be addressed by the school administration.
When asked about the behavior chart itself, the teachers who had students in the CICO
program responded favorably to the question that asked if the behavior chart was easy for the
students to understand. The teachers agreed with this statement more than all other statements
with an average response of 4.14 on the five-point scale.
Figure 1 below displays the average response to each of the ten questions in the CICO
survey. The teacher responses are grouped by the type of teacher. The open-ended responses
can be found in Appendix G.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
30/85
CHECK-IN CHECK-OUT 30
Table 1
Average Response by Teacher Groups from CICO Survey
Teacher
Group
Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10
Currently
Using CICO
4.00 3.43 4.07 4.29 4.14 2.93 3.43 4.07 3.36 3.93
Used CICO
in the Past
4.25 3.38 3.63 3.75 3.75 3.13 3.38 3.50 3.25 3.50
No CICOExperience
3.57 3.71 3.14 3.57 3.43 3.14 3.14 3.86 3.14 3.29
Discussion
Time and communication were overarching themes that surfaced among all stakeholders
in this evaluation. The CICO program at Suburban Elementary is constrained by the hours in the
school day and the available resources. The program is also influenced by external factors such
as home environment and parental involvement that are beyond the control of the school. There
are areas where time and resources could be used more efficiently and there are channels of
communication that could be more productive.
Stakeholder Perceptions of the Check-in / Check-out Program
There are mixed perceptions of the CICO program at Suburban Elementary. The staff
perceptions of the program definitely differ with their level of involvement. When asked if the
CICO program is a worthwhile investment of time by the school administration, the teachers
who currently have students in the program more strongly agreed with that statement. The
-
7/28/2019 Koontz 713 Program Evaluation Full Report
31/85
CHECK-IN CHECK-OUT 31
qualitative feedback was positive overall. There are a few teachers who do not see the value in
the program. This seems to be because of a bad experience with the program or because they
have multiple students in the program and the added work load makes it less effective.
The students perception of the program was almost all positive. All but one student who
was surveyed indicated that they behaved better because they were in the CICO program.
During observations of the check-out process, it was clear the relationship the CICO
administrator and the students had. It is hard to quantify the value of a relationship, but it was
evident. Furthermore, when the CICO administrators were asked what the most valuable part of
the CICO program was, the first response from both administrators was they relationship they
have with the students.
This program evaluation did not specifically focus on the Tier One school wide behavior
program but, the perception of that program from the focus groups was positive. Both groups
talked about the common language that it creates from grade level to grade level. Phrases such
as, we dont do that here were mentioned several times. Many teachers commented how the
Bones allowed them to reward students throughout the school. The current research indicates
that it is essential to have a solid Tier One program in order for the Tier Two program to be
effective.
Improving perception. Suburban elementary may want to consider additional training
and information sessions about the CICO program. As new teachers come on, they will need to
be trained, even if they do not have students in the program. You may also want to consider
having teachers sit in on the Effective School-wide Discipline (ESD) meetings on a rotating
basis. Through the interviews and focus groups, there seems to be a perception that the ESD
group had substantial input regarding the CICO program. Having teachers participate in those
-
7/28/2019 Koontz 713 Program Evaluation Full Report
32/85
CHECK-IN CHECK-OUT 32
meetings would let them feel like they are a part of the process and it may also help them
understand the limited resources the school administration is dealing with.
Parent involvement and communication was also a recurring theme. Having the parents
more involved in the CICO program may also increase the positive perception by the teachers.
The teachers understand that the students have difficult home lives and they want the parents to
be aware of their challenging behavior and be part of the solution, if possible.
Overall communication transparency may also help with the perception of the program.
Letting teachers know more about the selection process and why certain students get in the
program and why certain students leave the program are important. The teachers had varied
understandings of the program and misinformation can lead to misunderstanding. That was clear
from the survey results and the discrepancy between the teachers who had students in the
program and those that did not.
Program Improvement and Efficiency
The Check-in Check-out program at Suburban Elementary seems to be running smoothly
and does not need major changes. There were areas that could be modified, based on the
feedback from the stakeholders in this evaluation. Any suggestions for improvement will want
to be carefully considered and studied prior to implementation.
Time considerations. The teachers and CICO administrators all indicated that there was
not enough time to counsel the students as much as they would like. While it is not realistic to
have the students leave class earlier than they already do, perhaps there could be an extended
check-out period once per week. The additional time could be for the whole group or on a
rotating basis with individual students. That extended block does not necessarily have to be at
the end of the day. It could be during lunch or some other consistent time. The purpose of this
-
7/28/2019 Koontz 713 Program Evaluation Full Report
33/85
CHECK-IN CHECK-OUT 33
extended blockwould not be to go over that days behavior chart, but instead go over a weeks
worth of data or spend extra time talking about the students behavior goals for the future. It will
take some planning and organization by the school administration, but an extended time period
would be a helpful modification to the CICO program.
The school administration will want to look closely at the time this program takes for
teachers who have more that one student in the program. In both focus groups, the teachers who
had more than one student in CICO indicated that keeping up with multiple student charts was
difficult. One student and one chart seemed to be manageable. The school administration will
want to further investigate alternative reporting methods for teachers with multiple students in
the program.
During the check-out process, there is a limited amount of time. The students have to
catch the bus or day care van and the CICO administrator may have an afternoon duty. The
more students the CICO administrator has, the less individual attention the administrator can
provide. The school administration may want to consider identifying more staff to serve as
CICO administrators. This could be any staff member that is not a classroom teacher. By
reducing the number of students in each group, it creates more time to work with each student at
the end of the day. This would take more coordination and oversight, but it could have a positive
effect and help alleviate some of the time constraints at the end of the day.
Communication. There are several areas where communication methods and processes
can be improved throughout the CICO program. The two major areas for potential improvement
are parent communication and staff communication.
There seems to be a misunderstanding and some inconsistencies regarding parent
communication. The CICO behavior sheet has a place for a parent signature, shown in Appendix
-
7/28/2019 Koontz 713 Program Evaluation Full Report
34/85
CHECK-IN CHECK-OUT 34
I. One CICO administrator does not send home the behavior sheets and the other CICO
administrator sends them home once per week. The teachers in the focus group definitely had a
misunderstanding about how to communicate with parents through this program. One teacher
stated in a focus group, see Ive been writing notes all year thinking parents were seeing the
notes at least on Fridays but I guess theyre not. The other participants also seemed unsure
about wheather or not their parents saw the CICO behavior sheets. This could be easily fixed
with a clear policy about the level of parent communication.
The teachers also indicated a desire for the parents to more involved in the process.
There were several suggestions such as having the parents come in for an orientation to the
program. When a child enters this program, they have been a consistent behavior problem and
most likely have been suspended. The parents should be motivated to come in and learn about
the program if it will keep their child from being suspended. This could also build some good
will between the school and the parent because they will see the time and personal attention that
the school is committing to their son or daughter.
There also could be more consistent communication between the teacher and the CICO
administrator. The CICO administrator plays a significant role in the students education. The
teachers indicated a desire for more consistent communication to provide some connection to the
behavior sheets. The numbers on the behavior sheets often do not tell the whole story about the
students behavior. Also, the teacher or the CICO administrator may find things out about the
students personal life that they would not put on the chart, but would want the other person to
know. Along those same lines, there needs to be consistency in the teachers reporting on the
sheet. The numbers on the sheet are the main way the teacher communicates the behavior.
Having the teachers using the CICO program participate in a norming exercise may be helpful.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
35/85
CHECK-IN CHECK-OUT 35
The teachers could be presented with a scenario and they could discuss how they would score
that on the behavior sheets. This would help them be more consistent. It may also be helpful to
create a forum for the teacher in the CICO program. These teachers all have students who have
been identified as needing Tier Two behavior intervention. Creating a forum for these teachers
to share ideas and have a support network may also be helpful. This could be in the form of a
quarterly meeting or some other digital communication forum.
Program and school-wide considerations. The school administration may want to
consider several modifications to the CICO program. One of the ideas that surfaced through the
focus groups was having a different model for the younger students and the older students. The
teachers seemed to think that the program and the behavior chart was better suited for the older
students. Further study will need to be conducted to see if there is an alternative behavior sheet
or check-out model for the students in the younger grades.
The behavior chart itself can be a trigger for students. The whole point of the CICO
program is to reduce negative behavior, but the chart can be a trigger at the end of the day for
students who did not get the score they wanted. For certain students, the CICO administrators
may want to consider an alternative communication method to prevent the chart from becoming
a trigger for negative behavior.
An area where the teachers had a great deal of cognitive dissonance revolved around
focusing so much energy on the most disruptive students and rewarding them for normal
behavior. One of the focus group participants stated:
I think were setting some of them up for a really harsh reality check for when they get
out to real world kind of things and theres not a cop with a fanny pack full of lollipops,
-
7/28/2019 Koontz 713 Program Evaluation Full Report
36/85
CHECK-IN CHECK-OUT 36
because theyre expecting to get some type of reward for doing every little thing.
(Appendix D, 2013)
The teachers also wanted to recognize the students who consistently exibit positive behavior.
The teachers stated a desire to recognize these students more than every nine weeks. The school
administration may want to involve the teachers in coming up with ways to recognize the most
well behaved students as well.
A final recommendation that bridges the gap between Tier One and Tier Two behavior
support is the need for more classroom guidance. One teacher stated, We need more [guidance
lessons]. With as many issues and when I look at my class, I have twenty-six students, more than
half of my students have anger management issues. The teachers agreed that some type of
weekly guidance lesson that was consistent would be helpful. This would take a significant time
and financial commitment by the school and may be something that is provided by a grant or
some other outside funding source.
Conclusion
The school administration at suburban elementary will have many decisions to make on
how to implement the recommendations from this evaluation or how to further evaluate different
aspects of the Check-in Check-out program. The limits of this evaluation should be considered
when considering the findings.
Limitations of this evaluation. Time and resources were a factor in this evaluation.
There was only one evaluator conducting this research and therefore the suggestions and findings
are biased by a single point of view. Furthermore, because of the confidential nature of this
evaluation, access to certain stakeholders was limited. The evaluator worked directly with the
school principal and did not go through the school system central administration. This was
-
7/28/2019 Koontz 713 Program Evaluation Full Report
37/85
CHECK-IN CHECK-OUT 37
mutual decision between the client and the evaluator, but it meant that there were limitations to
how the students could be surveyed and interviewed. Also, not including the parents in this
evaluation was a strategic decision made by the school administration. This evaluation mostly
focused on the perceptions of teachers and administration at suburban elementary and therefore
should not be considered a complete representation of all stakeholders.
The evaluator only conducted one round of interviews, focus groups, and surveys. This
was done out of respect forthe school staffs time at a busy time of the year for school personnel.
Ideally, there would be follow up with multiple groups over time.
Lastly, to truly evaluate the CICO program, it would need to be evaluated within an
evaluation of the entire school-wide behavior support program. The evaluator only included a
small portion of the questioning on the Tier One program.
Recommendations for future evaluations. The school administration at Suburban
Elementary may wish to further the investigation of the behavior programs. Based on the results
of the current program evaluation, the following students could be considerations for follow up
studies.
Implementation fidelity. The research indicates that a Check-in Check-out program is
only as good as the degree to which the school staff implements the program with fidelity.
Future studies should focus on observing the daily behavior sheet implementation. This current
study did not look at the teacher-student interaction, only the CICO administrator-student
interaction. A future study designed to track the fidelity of the implementation by the teachers
would be a logical follow-up.
Multiple students per class. One of the concerns that surfaced during the focus groups
was the effectiveness of the program when there were multiple students in the program within a
-
7/28/2019 Koontz 713 Program Evaluation Full Report
38/85
CHECK-IN CHECK-OUT 38
single class. A follow up study should be conducted to attempt to identify if there is a drop off in
effectiveness when more than one student in a class enters the CICO program.
Parent and student perceptions. A follow up study should be conducted with the proper
approval from the central office administration that involves more students and the parents of the
students in the CICO program. The data from the student survey was limited and no parents
were involved in the current evaluation. These are two important stakeholder groups that were
marginalized and should be included in future evaluations.
Longitudinal study. The administration at Suburban Elementary should consider
tracking the long-term outcomes of the CICO program. This could include students staying in
the program for multiple years as well as students who graduate from the program. This type of
study would investigate the outcomes and impact referenced in the log model in Figure 1.
Conclusion
The findings from this study will be helpful to the school administrators at Suburban
Elementary or to a school administration looking to implement a Tier Two behavior support
program such as Check-in Check-out. This type of program relies on relationships that are
formed between adults and students. Relationships take time and are difficult, if not impossible
to quantify. The current research and the findings from this study both confirm the need for
teachers and staff that are dedicated to students and willing to follow a program. Even if the
program is not implemented with complete consistency and fidelity, the relationship are what
make the difference in childrens lives.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
39/85
CHECK-IN CHECK-OUT 39
References
Anderson, E. (2009). The effectiveness of a proactive school-wide discipline plan on office
discipline referrals at the elementary school level. (Unpublished doctoral
dissertation). Liberty University, Virginia.
Caldarella, P., Shatzer, R. H., Gray, K. M., Young, K., & Young, E. L. (2011). The Effects
of School-wide Positive Behavior Support on Middle School Climate and Student
Outcomes.Research In Middle Level Education Online, 35(4), 1-14.
Cheney, D., Lynass, L., Flower, A., Waugh, M., Iwaszuk, W., Mielenz, C., & Hawken, L.
(2010). The Check, Connect, and Expect Program: A Targeted, Tier 2 Intervention
in the Schoolwide Positive Behavior Support Model.Preventing School Failure,
54(3), 152-158.
Cohen, R., Kincaid, D., & Childs, K. (2007). Measuring School-wide Positive Behavior
Support Implementation: Development and Validation of the Benchmarks of
Quality.Journal Of Positive Behavior Interventions, 9(4), 203-213.
Hawken, L. S., & Horner, R. H. (2003). Evaluation of a Targeted Intervention Within a
Schoolwide System of Behavior Support.Journal Of Behavioral Education, 12(3),
225-240.
Horner, R., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A., & Esperanza, J. (2009).
A randomized, wait-listed controlled effectiveness trial assessing school-wide
positive behavior support in elementary schools.Journal of positive behavior
interventions, 11(3), 133-144.
Hoyle, C. G., Marshall, K. J., & Yell, M. L. (2011). Positive Behavior Supports: Tier 2
Interventions in Middle Schools.Preventing School Failure, 55(3), 164-170.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
40/85
CHECK-IN CHECK-OUT 40
Luiselli, J., Putnam, R., Handler, M., & Feinberg, A. (2005). Whole-school positive
behavior supports: Effects on student discipline problems and academic
performance.Educational Psychology, 25(April-June), 183-198.
Martz, M. (2012, November 9). Henrico points to habitual offenders for high student
suspension rate: School suspensions examined.Richmond Times-Disbatch. Retrieved
from http://www.timesdispatch.com/news/henrico-points-to-habitual-offenders-for-high-
student-suspension-rate/article_2657cb19-24c4-535a-9b29-ee9bc83c0ec1.html
Medley, N. S., Little, S. G., & Akin-Little, A. (2008). Comparing Individual Behavior
Plans from Schools With and Without Schoolwide Positive Behavior Support: A
Preliminary Study.Journal Of Behavioral Education, 17(1), 93-110.
Miramontes, N. Y. & Marchant, M. & Heath, M. A. & Fischer, L.(2011). Social Validity
of a Positive Behavior Interventions and Support Model.Education and Treatment
of Children 34(4), 445-468. West Virginia University Press.
McIntosh, K., Campbell, A., Carter, D., & Dickey, C. (2009). Differential effects of a tier two
behavior intervention based on function of problem of behavior. Journal of positive
behavior interventions, 11(2), 82-93.
Stake, R. E. (2004). Standards-based and responsive evaluation. Thousand Oaks, CA: Sage.
Todd, A., Campbell, A., Meyer, G., & Horner, R. (2008). The effects of targeted intervention
to reduce problem behaviors: Elementary school implementation of check in - check
out. Journal of positive behavior interventions, 10(1), 46-55.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
41/85
CHECK-IN CHECK-OUT 41
Appendix A
Interviewee: Resource TeacherInterviewer: Joe Koontz, Program Evaluator
Location: Highland Springs Elementary
Date: February 4, 2013
Koontz: Tell me about the check in check out process from beginning to end.
DV: It really is for our heavy hitters. It tends to be our kids from St. Josephs Villa, but itcan be othersour heavy hitters. They come it in the morning when they get off the bus.
They get their breakfast and then they come to check-in. The check in process is fairly
simple. They come and they fill out their charts, we talk about the goals for the day and
then they leave. Im pretty structured so its almost independent. Because Im out on thebus loop, the kids can come in the fill out their thing, the date is already up there they fill
in their name the date and generally they tend to pick the same thing. These are the target
behaviors right here (refers to the CICO sheet)
Koontz: What do you mean they pick?
DV: They pick their goals for the day. A lot of them tend to pick self control becausethats their biggest issue. Ive started to
Koontz: Self control would be keep your hands to yourself?
DV: Yea but it could also be as simple as self control and they write it here. With me
what they do name date and we do their goal. Now if you were to fill this out and you
wrote that your goal was self control, I would ask you before you go out the doorwhatdoes that mean? What does that mean to you? How does that look? Thats where they
would say to me, it looks like I would keep my hand to myself, Im not talking back, if
somebody hits me Im not going to fight back. Then I say, alight, youre ready. Have agood day. Then they give this thing (CICO Sheet) to their teacher. Their teacher then, all
different parts of the day can rate them along these three categories, but also keeping in
mind how their goal goes into that. Three is the highest score.and then.someteachers dont total it up. Im ok with doing it. The highest you can get is a 54. When
they come back to me in the afternoon, we look at it. We kind of talk about what where
their challenges and difficulties and things like that. The teacher sometimes writes
comments.
DV: Let me show you a completed one. The teacher wrote.
Koontz: Is there a target that they are trying to reach?
DV: 54, 54 is the highest you can get. We always say we want a 54 day. Is what I say to
them and when they come in in the afternoon they go to that thing behind (pointing to thecharts) you and they record their scores.
-
7/28/2019 Koontz 713 Program Evaluation Full Report
42/85
CHECK-IN CHECK-OUT 42
Koontz: For the month?
DV: We just go week to week. Today is Monday, so well start a brand new line. That girl
(name deleted) you can tell that her scores have fall all over the place.
Koontz: So is a 53 still a pretty good day?
DV: A 53 is still a pretty good day. I dont necessarily group them like thatyou might
have gotten a 45, but your teacher said, you worked really hard. You had a roughmorning, but you got it together. I think thats a pretty good day. For my reward system,
the person with the highest score gets to go to my treat box or gumball machine. Its a
little friendly competition just between them. If multiple people get a 54, whoever that is
gets to go to the treat basket. So, if you get a 54 every day, you get to go to my treat boxevery single day, if not, we just go by who has the highest score. Then they lean them
(the sheets) in their folders. They dont take them home because if they took them home
Ill never see them again. Then their right here for the next day.
Koontz: When a kid come in with a 40, how does that go with keeping track of the points?
Does it makes it better or worse? The competition of it?
DV: Before they walk in and before I see their folder, I can tell by the way they walk in
the door what kind of day they had. If theyre really excited then theyll come in and tell
you, I got a 54! Or, they come inthis one particular kids just kind of come in and pushis folder down and watches me and then I kind of knownot too goo ha? Sometimes
Ill look at it and say, thats not too bad. (Student name) for instance I had to call his
mom because I kept telling him, if you dont get out of the 30s, Im going to have to call
your mom. We were working on.his last 39, I said I need today to be a 40 day. I justwant you to aim for a 40. A number with a 4 in front. When he came in he was like, I
got it I got a 4 he knew hes in second grade he knew he had because he could tell and
he was excited. Of course you can see the next day he went down to a 29.
Koontz: How are students selected for the program?
DV: Based on the most number of behavior referrals. I hate to say reputation, but
reputation. And what I mean by that is (name) for instance. He is currently suspended.
He hasnt been suspended in over a year, but he got into a fight on Friday. (Name) does
not have any suspensions and he didnt have any referrals, so from his past history fromlast year and the year before that, thats why we keep him on a tight leash (in the
program), plus him and I have a great relationship, we fear that if we take him off he
might start to regress. A lot of it is prevention, but some of it is you need to be in here.
Koontz: Is there a referral process for teachers?
DV: These are kids who need a little extra. How does a teacher refer. There is a teacher
-
7/28/2019 Koontz 713 Program Evaluation Full Report
43/85
CHECK-IN CHECK-OUT 43
in first grade she wanted me to take on a new kid because a kid had graduated from the
program who kept getting 54s. The teacher was like, Ill take him back if you take a
different one. We told her she couldnt just do that. We told her she neededdocumentation that says he needs to be in there because this is tier 3. He who youre
referring is not really ready for this yet, but I can check in with him when he needs me or
call for me and Ill come check in with him, he needs to get in a little more trouble beforeentering this program.Koontz: Child study team ever refer a student as a strategy out of child study?
DV: It could come up. I dont know how many of them this year. We just look at thenumber of time in the office.
Koontz: Is there a formal process for that like when they hit a certain number of referrals?
DV: It big picture, its anyone who goes to New Bridge (suspension intervention
program) because we want to keep tabs on them. There is a kid right now (name) who is
a 4
th
grader he is not in CICO, but when he comes back he will be because he has hadmultiple issues this school year. Hes been there about a month now.
Koontz: What do you think is the most effective part of the CICO program?
DV: The relationship building.
Koontz: Tell me a little more about that.
DV: What I mean is Im kind of one of those teachers that is kind of cool. I dont think that
any of my child that I have in CICO think that it is a bad thing or because their bad. They
are excited to come to me, they want to show me their folders. They take responsibilityfor it. I instill an expectation that you have to take care of this folder. Ive bade it pretty
much easy for them. So, relationship building and it gives them a sense of responsibility.
It helps them to monitor their own behaviors because they know they have to report abouttheir day.they may think about it a little more. I see kids throughout the day that
sayI got all 3s so far! They want to tell me.
Koontz: It sounds like it gives you a vocabulary to quantify a qualitative action?
DV: It gives them something to strive for something to keep in the front of their mind. I need
to make sure my day is going good because they know they need to answer me at the endof the day. It also helps when A kid hasnt been doing so well, when the teacher picks up
the folder to mark it sometimes their behavior can go down.
Koontz: What are the biggest challenges with the program?
DV: I would say consistency. Because if a teacher hasnt bought into it , if the teacher is not
consistent enough to keep this folder at a certain part and fill it out at the end of the day.I dont want to wait till the end of the day and their just circling things because you just
-
7/28/2019 Koontz 713 Program Evaluation Full Report
44/85
CHECK-IN CHECK-OUT 44
want to circle something. I would like it to be meaningful where they can actually talk
about or either give a kid a chance so their having a little off task and you are repeating
and talking to them to many times. You say now if I speak to you one more time Imgoing to mark you a 1 on your chart. At least thats what I would do so were having that
dialog where Ive spoken to you and you have an understanding of my understanding. So
consistency and teacher buy-in and teacher application for them to use it effectively in theclassroom.
Koontz: So the teacher has more to do with this than you do. Do you have any expectations
for the teachers in terms of a teacher that is getting a kid for the first time? Anytraining?
DV: Yes, (name)s teacher is new. He is coming off a weeks suspension. I went to the
teacher and said, he is having a meltdown in the morning when it comes to filling thisout. I told her, dont send him to me, Im going to bring it to you and you and him can
work on it whenever you get a chance. If you dont fill it out all the way, thats ok
because hes in kindergarten so it frustrating for him to write and he, when his medicinewas off he was flipping tables over. So, I go on a case by case basis. There is no initial
training because theyve all seen it.
Koontz: Is it more effective with the older students?
DV: Yes, because they definitely take more ownership. The younger student couldnt get that
is was a daily thing.
Koontz: Can kids ever exit the program?
DV: Yes. We meet once a quarter to look at their average scores. For a kid who averages a36, hes not really ready. A student who averages a 52 like (name) who was only a
behavior problem when he was hungry. We dont want to hold that against him. When
he was on, he was on. Talking to his teacher he could um, not be in CICO. We hersupport, we could take him off and so he graduated out. All the other kids would say
where is (name)? I would say, he graduated from the program. They kids would then
ask, when do I graduate?
Koontz: When kids graduate, do the teachers keep them on some type of behavior plan?
DV: I dont think so. I dont think they do.(Interruption here)
DV: No I dont think Ms. Hunt has him on any specific behavior thing. She knows that
(name) and I have a good relationship. Sh