Equity Audit and Root Cause Analysis

Does student performance on STAR and DIBELS testing correlate with passing the Reading Ohio Achievement Test?

Caitlin Reash

University of

Mount Union

MAEL

Why was this topic chosen to analyze?

o The school has been designated as Continuous Improvement for a number of years.

o Math achievement has been on the rise, but reading has not.

o Progress monitoring is completed through usage of the STAR and DIBELS testing.

o Questions I considered:o What is done to “catch” students who are on or below

expected reading levels?o If students score well on STAR and/or DIBELS, will they

pass the Reading OAA?o What can the school do to offer support early on in

order to increase achievement?

Assessment BackgroundSTAR, DIBELS, OAA

The STAR test is administered by grades K-4. At the end of Kindergarten, students take the STAR Early Literacy to determine their reading level entering First Grade.

Results are made available to teacher immediately due to the electronic format of the tests.

The scores are then analyzed by teachers, including Title One math, in order to monitor progress and determine intervention groupings.This also helps to determine the skills to be addressed during both math

and reading interventions. Based on math scores, the Title One teachers conduct mini lessons

(pull-out) in addition to the push-in math intervention time. Student scores and analysis are shared with parents after each time

the test is administered. Teachers interviewed are pleased with the types of questions used, the

speed/ease of getting results, and the data/interventions suggested based on the student scores.

Teachers also find that this is an effective way to monitor progress.Second grade teachers are considering using this test monthly because of

the data provided and how its use in effective interventions.

STAREarly Literacy, Reading, and Math

End of K-4th Sept, Jan, & MayProgress Monitoring/Intervention

Groupings

**Slide taken from Data Analysis Presentation

DIBELS is administered by Title One reading teachers three times a year.

Results are made available to teachers the same day as the test is given to his/her class.

The information collected is analyzed by the Title One teachers at the completion of testing for grades K-4.

Based on the students’ scores, reading intervention groups are determined to provide supports to at-risk students.

Currently, a two day pull-out, two day push-in model is being used to provide necessary interventions.

In Kindergarten only, parents receive a notice as to whether their child has met benchmark or not.

Title One teachers, along with classroom teachers, are currently questioning the effectiveness of the current push-in model because of the success/student improvement that was achieved through the five day pull-out model. The main reason for this is that in the current model, the student to teacher ratio is

13:1, whereas in the pull out model, the ratio is at most 4:1.

DIBELSDynamic Indicators of Basic Early Literacy Skills

Kindergarten-4th Sept, Jan, & AprLiteracy Skills/Title One Reading

Intervention

**Slide taken from Data Analysis Presentation

Starting in third grade, the OAA is administered to track student achievement.

In October, students are given the reading portion as a practice test. The actual test is not given until May, and this is when all portions

are administered. Results from the practice test are used to determine areas of need,

as well as to provide a pre test to use for student growth in the Spring.

After testing is complete, results are compiled by an outside testing service and returned to the school within a month typically.

Teachers are working to improve student achievement through intervention strategies and the implementation of a new reading series.

This is only type of Value-Added data currently provided.The data is provided on the State Report Card

OAAOhio Achievement Assessment

Grades 3-4 Practice: OCTActual: MAY

Student Achievement / Value Added

**Slide taken from Data Analysis Presentation

Data CollectionSTAR, DIBELS, OAA

OAA Data CollectionEach color represents a designated level.

STAR Data CollectionThe bands of color represent the students’ OAA designated

level.

DIBELS Data CollectionThe bands of color represent the students’ OAA designated

level.

FindingsAfter analyzing OAA, STAR, and DIBELS data:

STARo STAR results were mixed. Scoring in the Spring of 3rd grade did

not provide a consistent

representation of passing the OAA.

o Some with below level skills passed, while

others who were at or above grade level did

not.

DIBELSo DIBELS results were also mixed,

and a clear indication as to passing the OAA was not evident.

o Basic and Proficient designations were

scattered.

Overall Conclusionso For students who scored Accelerated or Advanced,

generally they also scored the highest on STAR and DIBELS assessments.

o Students who were at grade level (4) on the STAR in the Spring, generally passed with Accelerated or Advanced, however, for those who scored Proficient, this was not a determining factor.

o On the DIBELS, those who were designated as Limited also had some of the lowest scores on DIBELS; however, proficient and basic were scattered.

o Gender did not play a role.o For students with disabilities, DIBELS data aligned more

with OAA designations than STAR.

Projections For current 2nd and 3rd grade students

2nd Grade Data CollectionStudents are categorized as Above, Target, or Below based on

how their overall average scores correlate with the overall average scores of the students who took the OAA last year.

3rd Grade Data CollectionStudents are categorized as Above, Target, or Below based on

how their overall average scores correlate with the overall average scores of the students who took the OAA last year.

In order to gain a better understanding of peoples’ feelings and ideas about assessments, interventions, and reading in general, I conducted a

survey.

Survey

Survey Results

Distributed

Returned %Teachers: 18 8 44%Administrators: 1 0

0%Parents: 30 13

43%Overall: 49 21 42.8%Slip to receive results of data: 49 5 (parents) 10.2%

Survey Analysis

o Most teachers discuss STAR and DIBELS data during conferences (5 out of 8).o Reading at home is seen as very important.o Most teachers are unsure if the new reading curriculum will increase OAA scores (5 out

of 8).o “Over time, students will show improvement.”o “More technology is needed in order for it to succeed to its full capabilities.”o Others say it is too difficult for students will low achievement in reading.

o Results from DIBELS and STAR are used to determine students who are in need of intervention either push-in or pull-out (or both). o DIBELS provides “a year end goal to work towards” and deals with fluency.o STAR is prescriptive and can be used for intervention within the room.

o Interventions that have been found to be effective include:o Paired reading, echo reading, vocabulary skill lessons, small group reading, one-on-one

instruction, graphic organizers, and centers. o Pull-out Title One services as opposed to the current push-in model are preferred by teachers for

the following reasons:o Less distraction for students, smaller groups, and better achievement at this point in previous years.

Teacher Results(Section for Teachers only)

Survey Analysis: Continued

o Most parents have received a copy of their child’s STAR scores, but of those that did, some did not receive the prescriptive section.

o Of those that received the prescriptive section, almost all found it to be helpful or very helpful and used it to: o Provide intervention at homeo Understand where the child can improveo Increase communication between parents and teachers as to the skills to focus on

o Reading at home is done often, as most participants answered daily to 4-5 times/week that they read with their child.

o Activities that have proven to be helpful in improving/developing reading are:o Going to the library for programs and activities (Summer Reading Program)o Reading environmental print, newspapers, shopping lists, mail, cards, magazines, signs, etc.o Right to Read Weeko Reading togethero Finding challenging books to increase interest

Parents Results(Section for Parents only)

o Reading is found to be an important aspect of education and so is a home school connection.

o Feelings are varied when it comes to the purpose of STAR and DIBELS, but over half of participants feel they understand it well to very well.oThe purpose of the OAA is less clear than other assessments.

o Gender is not seen as a contributing factor as much as disability is.

o A correlation between assessments and the OAA is varied with most participants in the middle range of there being a link.

Overall Results

(From all participants)

Survey Analysis: Continued

Two veteran teachers who have a wealth of knowledge about assessments were interviewed. Both have over 25 years of experience.

Interviews

Interview Protocol

Mrs. W Mrs. T

o Progress monitoring once a month for intervention purposes for all learners

o Progress monitoring is done in math and then additional practice for students apart from the classroom interventions is provided, so a similar practice in reading could prove beneficial

o Kids at all levels are capable of success, so it is our job to help them build on wherever they are at.

o Additional intervention that is ability and skill based would be a good addition for reading.

How could we support students who are right on level, as opposed to only those that are below?

Interview Protocol

Mrs. W Mrs. T

o They provide us with knowing what critical skills they’re lacking through an item analysis.

o STAR also does the grouping for you electronically, which is helpful in developing intervention groups/activities

o It gives teachers a place to start and helps teachers to more effectively cover what students need.

o Enrichment for above level students is also needed and STAR is a “great tool” for guiding it.

How has the STAR prescription aided in improving students at various reading levels?

Interview Protocol

Mrs. W Mrs. T

o They (both) typically provide an accurate representation, but can be unreliable.

o STAR’s projections are not always accurate, but the way testing is headed, computer-based assessment practice will be beneficial.

o DIBELS is a good tool for fluency, and STAR is skill based, so together, they can help to provide more comprehensive look when used together.

o DIBELS is “down and dirty” fluency and has always been accurate for children with reading disabilities since it is 1 on 1 testing, and they can’t guess (like on the STAR).

o DIBELS is for fluency and STAR covers more skills.

Do you feel that both DIBELS and STAR provide an accurate representation of students’ abilities? Does one more than the other? What are the benefits of

using both assessments?

Interview Protocol

Mrs. W Mrs. T

o Social and emotional issues play a much greater role than gender.

o Behavior can be a factor and early on, girls mature faster than boys.

o As far as innate ability, there is no difference.

o Behaviorally it can, but not only based on gender.

o Boys are more hands-on in the early stages, so certain tests can be more difficult for them since girls mature quicker.

o Through experience though, gender alone is not a determining factor.

Do you feel that gender plays a role in student performance?

*After asking this question, I shared the results of my data when sorted by gender, and that not even one student moved when sorted by gender as opposed to score.

This supported their responses.

Interview Protocol

Mrs. W Mrs. T

o They maybe didn’t perform as well on other tests.

o If they were special education students, accommodations may not have been in place for the other assessments.

o They may prefer testing alone as opposed to on a computer or with an adult 1 on 1.

o Judgments can’t solely be made on assessments and some other factors could have had an effect on performance on the other assessments.

o The OAA is much quieter and done in a more controlled setting.

Some students passed the OAA, but were not on grade level for STAR or DIBELS. What could account for this?

Interview Protocol

Mrs. W Mrs. T

o Huge! It’s the foundation, and it starts on home.

o They need to be read to, talked to, and socialized from the early ages.

o Extremely.o Those who have

learned to love books will learn to love reading.

How important is parental involvement when it comes to reading ability?

Interview Protocol

Mrs. W Mrs. T

o Consistent, daily intervention is needed, which is not what the school is currently doing (because of what administration has said needs to be done).

o Evidence based interventions are needed.

o The new series has more rigor and once kids adjust to it, it will make a difference.

o Strategies like Readwell, the “Rewards” program for older students, and Sing, Spell, Read, & Write are all intervention programs that seem to be successful.

o As long as the books are not too advanced in order to reach even the students with the lowest ability level, then gains should show.

What intervention/strategies have proven to help most to improve student reading achievement, and do you feel the new reading series will help to

boost scores?

Interview Protocol

Mrs. W Mrs. T

o It should be used on a more flexible basis with grouping.

o During small group time

o During instruction via a push-in model in addition to other intervention times

o Administration need to look at the scores and use them to make decisions that will benefit the school.

o Data needs to be used to identify problems and then used to solve those same problems.o At both the

administrative level and for classroom instruction

How would you like to see DIBELS and STAR data being used to guide instructional practices?

After gathering and analyzing data, making projections, and conducting surveys and interviews, I have developed the steps I would take to

increase reading achievement.

Plan of Action

Plan of Action

o Teachers need to know:o The purpose of each

assessmento How to effectively use the

results of the assessmentso Ways to drive instruction

through data gained from the results

o Have a representative from the reading series present

o From the Data Collections project and through surveys, teachers do not know if the reading series will boost scores.

o Many teachers are not comfortable using the series and have questions.

o Small group aspect of the curriculum is an important piece to increasing reading achievement

Step 1: Professional Development

for Assessments

Step 2: Professional Development

on Reading Curriculum

Plan of Action

o Teachers would analyze scores and group students into Above, Targeted, or Below

o These would be used for intervention purposes

o Follow a consistent five day pull out program

o With pull out, DIBELS scores were higher last year than compared to the push in model this year

Step 3: Analyze Scores Step 5: DIBELS Intervention

o Ensure that curricular decisions match the students’ needs

o Is the targeted instruction varied from the above instructional so that all students are challenged?

Step 4: Examine Decision

o Use prescriptive measures to drive intervention within the classroom

o Focus on students’ individual needs during small group reading time

Step 6: STAR Intervention

Plan of Action

o Parents in the classroom during small group time to focus on individual needs

o Prescriptions from STAR sent home to increase communication of necessary skills to work on

o Family Literacy Night:o At least once at the beginning of the yearo Teachers modeling how to read a story and ask comprehension

questionso Teachers sharing activities, games, and resources that focus on

sight words, blending, and vocabulary to increase fluencyo Sharing what the public library has to offer and possibly having a

representative from the library with card applications

Step 7: Community Involvement

Reading at home is important to success, and the more educated parents are on how to help, the more they can educate their

children.

Synthesis and ReflectionBased on my findings throughout this process, the school is administering many

assessments, but may not be utilizing them or the results to their full potential. Understanding seems to be lacking and that has to do with the purpose of assessments not being made clear to teachers. “Despite the importance of assessments in education today, few teachers receive much formal training in assessment design or analysis” (Guskey, 2003). When teachers understand why they are administering the assessments and how to analyze results, more effective instructional decisions will likely follow. In an article from ASCD, Jennifer Morrison explains the importance of data in decision making, “Remaining unaware of the range of assessments and how to use them and accepting (frequently inadequate) standardized tests as the single measure of success is irresponsible” (2009). Even in Kindergarten when we are analyzing a graph, I tell them that without data, our findings don’t “mata.” That is something that teachers need to keep in mind when making instructional decisions.

Through the survey, I also found the home-school connection could be strengthened through communication of assessment results and educating parents on how to help their children. “Teachers can engage parents in early learning when they share children’s progress and growth using best assessment practices. Regular updates on children’s growth can encourage families to engage in more learning-related activities at home” (Ross, NAEYC). Communication of skills and expectations is an important component to success.

Interventions are currently being used at every grade level, but I feel that the format of them needs to change. For Title One reading, a pull-out program needs to replace the push-in model that has not proven as successful with the students’ achievement this year. While many schools are proponents of the push-in model and have had success with it, our school has not due to the distractions in a larger classroom (Rich, 2010). Either professional development on the correct ways to push-in need to be provided, or a pull out model needs to be put back in place. For STAR prescriptive recommendations, students can be serviced in the classroom during intervention time/small group instruction time. While the students would still be getting pulled out for DIBELS intervention, they would also be serviced in the room during this intervention time.

Synthesis and ReflectionIf I were to do this project again, I would likely follow a similar

approach although I would like to get the teachers more involved with the analysis. Currently, our morale in the school is low and communication between teachers at various grade levels is lacking. If the teachers were more open to change, I likely would have sat down with each grade level to analyze the data, help to determine where the students are in regards to being on track to pass the OAA, and what in-class interventions could take place to help boost achievement. I think it would be interesting for them to see the results and make inferences based on them, but of the survey results that were returned, no teachers requested to see the results, which demonstrates to me that they may not care to know what needs improved.

The roadblocks that I encountered have to do with the low morale of the school. I personally feel that they have not experienced much success with assessments over the years and are now more defensive than anything. From an administrative point of view, I feel that it is important for our principal to step up and create a more supportive and sharing environment similar to a Positive Learning Community or PLC. Among other things, a PLC can “support professional learning that informs good teaching and classroom practice…increase morale and job satisfactions; and create stronger commitment to making significant and lasting changes and higher likelihood of undertaking fundamental systemic change” (NASP, 2009).

I would like to know how to better improve communication between parents and teachers for the whole school because I feel that would make a tremendous difference.The most important factor in improving reading achievement is being able to reach the students at their levels and bring them up from there. This requires a commitment from teachers and parents and a commitment from teachers to parents and vice versa.

ReferencesGuskey, T. R. (2003). How classroom assessments improve learning. Educational

Leadership, 60(5). Retrieved from http://www.ascd.org/publications/educational-leadership/feb03/vol60/num05/How-Classroom-Assessments-Improve-Learning.aspx.

Morrison, J. M. (2009). Why teachers must be data experts. Educational Leadership, 66(4). Retrieved from http://www.ascd.org/publications/educational-

leadership/dec08/vol66/num04/Why-Teachers-Must-Be-Data-Experts.aspx.

Rich, S. R. (2010). The effects of pull-out and push-in reading intervention models on first and second grade students. (Unpublished doctoral dissertation). Baker University, Kansas. Retrieved from http://www.bakeru.edu/images/pdf/SOE/ EdD_Theses/Rich_Starr.pdf.

Snow, K. (2012). Research news you can use: Family engagements and early childhood education. Retrieved from http://www.naeyc.org/content/research-news-family-engagement.

Supporting teachers’ ability to teach in stressful times: Tips for administrators and teachers. (2009). National Association of School Psychologists. Retrieved

from http://www.nasponline.org/educators/Supporting_Teachers_Under_Stress_FINAL.pdf.