1 | P a g e

Submitted to: Created by:

Michelle Springer Samara OrkinMiddle School PrincipalBellevue Middle School

Cathy HancockEducational SpecialistBellevue Middle School

Bellevue Middle iChange Collaborative Unit Evaluation December 7, 2012

II. Evaluation Introduction

A. Organizational Context of the Evaluation

Two years ago, Bellevue Middle School (BMS) gave their students a survey called CSCI report. When BMS got the scores back, they noticed that 79% of students did not identify positively for social and emotional security and 55% of students did not identify positively regarding feeling physical safe in the school building. In addition, only 57% of the student body admitted to having respect for diversity. The questions and implementation of the survey are unknown, but the results that most students did not feel physically, socially, and emotionally safe caused the administration to want to act. This evaluation tool used the Sense of Social and Emotional Security tool to guide much of the evaluation (Appendix A).

BMS is a Metro Nashville Public School with 700 students and 37 teachers serving 5th grade through 8th grade. BMS is in good standing according to NCLB requirements. Therefore, it is a school of choice for schools in the district labeled as “failing” or “high-priority” because they have enough children passing the state test demonstrating proficiency in grade level content. As a result, children are bussed to Bellevue from North Nashville (an extremely impoverished and under-educated area: 60% have a high school diploma or less, and the average income is $12,000).

The ultimate challenge is to find a program to help improve the number of students who feel safe in the school building through bringing attention to student differences and educating children on those differences especially with the number of students being bussed in increasing. Dr. Michelle Springer is the principal of the school. This is her third year with BMS (1st year as principal), and this is an issue that is very close to her. She feels very strongly about running a school where students feel safe and comfortable being themselves. Dr. Springer also states, “cultural awareness was an area that students and staff addressed in a survey last year, and we know that cultural awareness for all stakeholders will be an intricate part of losing the achievement gap.”

BMS identified lack of knowledge on cultural differences to be a main source as to why students do not feel socially or emotionally safe. As a result, BMS connected with Martha Caldwell for support. The iChange Collaborative Unit is the unit on the cultural awareness curriculum that BMS will be using in an adapted form specifically for them. The program was originally created and adapted by Martha Caldwell and Oman Frame for their school in Atlanta.

The nature of work for this evaluation is to see if a curriculum called “Inquiry into Identity: A Middle School Curriculum for Race, Class and Gender Issues” from the iChange Collaborative Unit is able to improve student’s sense of emotional and social security in the school environment. This evaluation will determine if the educational program can create awareness in cultural differences that will eventually and hopefully help students feel safer at school.

2 | P a g e

B. Objectives of the Inquiry into Identity: A Middle School Curriculum for Race, Class and Gender Issues iChange Collaborative Unit

The objectives of this program are to educate children on their own cultures as well as those of other students with culture involving race, class, and gender. Through open discussion, reflection, and activities, students are exposed to conversation and awareness in an explicit and safe manner. This program has 10 lessons that the students will attend in their Owl Time, which is every held every two to three weeks on Friday afternoons. The lessons cover a wide range of topics:

Table 1:Lesson TopicsLesson Topics

Lesson One: Introduction to Owl GroupsLesson Two: Who Are You?Lesson Three: Discovering the Power of EmotionsLesson Four: Guidelines for Listening and SharingLesson Five: What is Learning?Lessons Six and Seven: Family History Interview ProjectsLesson Eight: What’s Great and What’s Not about Being a GirlLesson Nine: What’s Great and What’s Not about Being a BoyLesson Ten: Talking about Oppression

The key component in achieving these objectives is through the teachers who must follow the lessons and curriculum. Additionally, the school administration must support the teachers in this endeavor. Finally, many students leave early on Fridays and that is when the lessons are taught. Therefore, buy-in from all stakeholders will be important in providing a worthwhile context for the evaluation.

The ultimate objectives:

1. Increase student sense of social and emotional security by 10%. 2. Ensure transfer of knowledge of cultural awareness across grades and settings by an increase of

10%.

III. The Evaluation Process

A. Evaluation Methodology, Stakeholders, Key Questions and Logic Model:

i. Purpose of the Evaluation

The purpose of this evaluation is to see if the iChange Collaborative Unit actually creates a high sense of security, a more open-minded student population, and transfer of knowledge into social and academic settings. Being that the curriculum is only given to the 8th graders this year, the 8th graders are essentially piloting it. The evaluation will provide information to show if this program can yield success and if it is worth the investment, time and energy that will go into the evaluation and the unit. Looking closer at the key questions listed below, the evaluation will answer those questions and the

3 | P a g e

answers will indicate the next steps for BMS to take to create a more socially and emotionally safe place for students.

ii. Key Questions Guiding the Evaluation

From conversations with Dr. Springer and Cathy, the main concerns are bolded below. Using the main concerns, questions were created that the evaluation plan will answer.

1. Sense of Security: Did the IChange Collaborative Unit increase students emotional and social security at school?

2. Transfer of Knowledge: Does the iChange Collaborative Unit transfer to more than just the students being taught the unit & does it transfer to different settings?

iii. Evaluation Method

After analyzing the needs and goals of the iChange Collaborative Unit and learning the purpose of the implementation, Kirkpatrick seems to be a great fit for the model of evaluation. With Kirkpatrick, the true reaction, learning, behavior, and results can be evaluated. When working with children, it is important to see reactions to what they learned as well was the actual content learned. With a curriculum about content, the reaction of the students will be a key component to measure because reaction can help to understand the learning. Also, behavior is valuable with middle school children to see if they can actually change behavior based on learning in the program. Finally, the results will be key to see how this unit impacts the school as a whole to help look to see if it should be repeated and what could be improved. Kirkpatrick will allow a true picture of the success of the unit because it is thorough in looking at immediate reaction, what is learned, and what changes, to then look at bigger results proving that this model of evaluation is a strong choice. Below in Table 1 is the outline of the approach that will be used with Kirkpatrick.

Table 2: Evaluation MethodologyiChange Collaborative Unit Kirkpatrick Evaluation Methodology

Level 1-Reaction

How do the kids react? How do the teachers react? How was this unit perceived by participating parties? Was there relevancy in the context?

Level 2-Learning

What is learned about culture? Has knowledge and awareness increased?

Level 3-Behavior

Have social and emotional behaviors changed? What kind of behaviors?

Level 4-Results

Do students feel a stronger sense of social and emotional safety in the building?

Quasi-Experimental: Because there was no Treatment, but measurements were taken. Two years ago the survey called CSCI was given to the students showing that students did not feel socially and emotionally safe, which is where the need for a treatment came from. Now there is a treatment and then more in depth measurements to determine if students can feel safer.

4 | P a g e

iv. Stakeholders

The table below outlines the different stakeholders for the iChange Collaborative Unit. From talking with Bellevue Middle School administration, the stakeholders are only those that have significant influence in the success of the program. Whether it is be directly giving or receiving the unit or benefiting from the results and helping to build buy-in and support the program, the stakeholders that will be most affected are listed below.

Table 3: StakeholdersStakeholders

Stakeholder RoleMichelle Springer BMS Principal8th Grade Teachers/Unit Teachers Ones who are teaching the unit8th Grade Students Recipients of unitParents Recipients of improved awareness and safety from

childrenBMS other teachers/staff Observe the difference in the 8th graders to other grades

(related arts/special teacher—music/art/Spanish/technology/etc.

5 | P a g e

v. BMS Logic ModelTable 4: Logic Model

6 | P a g e

B. Evaluation Design:

Using Kirkpatrick’s design, the evaluation’s design will attempt to provide BMS with the appropriate information to determine if the effects of the unit are changing students identifying themselves as feeling safe at their school. Addressing the administration’s key questions in how the students and teachers perceive the unit (reaction), what is actually learned in the lessons (learning), how behaviors change (behavior), and actual impact and results the unit has on the students and school as a whole (results). See section C. Data Collection Plan and Instruments for specific questions associated with each level of the Kirkpatrick model, collection instruments used to assess for them, and the timing for administration.

PLEASE NOTE: All evaluation instruments will be surveys due to time constraints in the school setting.

The following are key components & factors impacting the evaluation’s design - Figure 3 provides a summary view:

Structure / Timing1. The evaluation design looks at the iChange Collaborative Unit as whole unit containing

10 detailed lessons with activities within the lessons. The teaching of the lessons will spread throughout a semester.

2. The evaluation design uses a mixture of tools to evaluate the effectiveness of the unit. This design will provide Bellevue Middle School with quantitative and qualitative data. a. Reaction Survey: mid-way through (5 lessons in) and at the conclusionb. Learning Survey: After every two lessonsc. Behavior Survey: After completion of unit (pre-post assessment)d. Result Survey: After completion of unit (post assessment)

Sampling1. 100% of 8th grade students (learning, reaction, result)2. 100% of those who teach the unit (learning, reaction, result)3. 100% of Related Arts/Specials teachers and 8th grade homeroom teachers (behavior)

Use of Results 1. All answers to the surveys and assessments will track trends from the responses to

determine the effects of the unit. Section D. Coding Plan and Data Analysis provides details regarding the analysis of instrument results.

2. The results each evaluation tool will be compiled into a single report for the administration of Bellevue Middle School for reviewing

3. The results will be used to align with the key questions and make programming decisions about repetition, population to give the unit to, and safety in the building.

7 | P a g e

Summary of Evaluative Design’s Annual Treatment Cycle

M1 Behavior Assessment (Pre-test) TL1TL2 M2 Learning Survey1

TL3TL4 M3 Learning Survey2 TL5 M4 Reaction Survey TL6 TL7 M5 Learning Survey3

TL9TL8 M6 Learning Survey4 TL10 M7 Learning Survey5 & M8 Reaction Survey

M9 Behavior Assessment (Post-test) + M10 Results Survey

T = Component Treatment; L = Lesson; M = Measurement

C. Data Collection Plan and Instruments:i. Data Collection Plan

Table 5: Data Collection PlanEvaluation

LevelWho (Data

Source)Objective Questions Evaluation

InstrumentTiming Responsible

PartyReaction-

The reaction of the student and

teacher participants?

8th Grade Student participants

How did participants perceive this unit?

Do participants foresee using the content?

What is missing from the lessons?

Reaction Survey After 5 lessons & after all 10 lessons

Principal, Unit Director, Unit Teachers

Teacher’s Teaching Unit

How do the teachers react? How is it perceived?

What is missing from the lessons?

Was there relevancy in the context?

Reaction Survey After 5 lessons & after all 10 lessons

Principal, Unit Director, Unit Teachers

Learning-The knowledge

learned and retained after

the lesson

8th grade Student participants

Are students learning the information?

Learning Survey Immediately after every one or two lessons.L1L2 togetherL3L4 togetherL5L6L7 together, L8L9 together, L10 alone

Principal, Unit Director, 8th grade homeroom teachers

Behavior-The behaviors in comparison to other grades not receiving treatment and the behaviors

that are transferring

outside of the classroom the environment

where the treatment is

given.

Related Arts/Specials Teachers

How do students behave with social issues involving culturally different students?

How do the behaviors compare to the other grades taught?

Behavior Assessment

Before the unit starts & at the conclusion of the unit 1 month after

Principal, Unit Director

8th Grade Teachers (teachers of participants)

How do students behave with social issues involving culturally different students?

Are behaviors consistent in different settings demonstrating transfer of knowledge?

Behavior Assessment

Before the unit starts & at the conclusion of the unit 1 month after

Principal, Unit Director

Results-The results in comparison to

the CSCI results from two years

ago to see if students are

feeling socially and emotionally

safer in the building

8th Grade Student participants

Do students feel a stronger sense of social and emotional safety in the building?

Results Survey (based on the survey used to establish the need for this kind of a unit)

One month after the conclusion of the unit

Principal, Unit Director

8 | P a g e

ii. Instrument 1 - Reaction Surve y

Objective: To gauge the perception/reaction all of the students and teachers who either taught or were a participant for the iChange Collaborative Unit

Responsible Party: Principal, Unit Administrator, Unit Teachers,

Given to: All those directly in contact with the treatment, 8th grade students at BMS & Teacher’s teaching the iChange Collaborative Unit, 80-100% response goal

Timing of Administration: The same survey immediately following:1. Lesson 5 2. Lesson 10

Collection Method: Hardcopy pen/paper survey

Survey DesignThis survey will be administered using a 5-point Likert scale, which should be attached to each of the survey statements below. There is one survey designated for teacher’s and one survey designated for students.

ScaleStrongly Agree ☐ Agree ☐ Neutral ☐ Disagree ☐ Strongly Disagree ☐ N/A ☐

Survey Statements

Which Lesson did you just complete? (Check One)

____ Lesson 5 (half-way point for the unit)____ Lesson 10 (completed the unit)

Student Statements:

1. I know more about cultural difference than I did before.2. The lessons helped answer questions that I had about cultural differences. 3. Information was not left out. 4. I already plan to use these lessons in my own life. 5. These lessons help me to better understand my peers. 6. These lessons help me feel safer at BMS. 7. Understanding more about culture, also taught me about my own culture. 8. These lessons will help other students feel safer at BMS. 9. These lessons will get rid of some of the bullying that goes on at BMS. 10. I think that these lessons will help create a safer school.

Teacher Statements:

1. The lessons helped answer questions that students had about cultural differences.

9 | P a g e

2. There was not any missing content. 3. I can foresee that students will use these lessons in their own lives. 4. These lessons helped students to better understand their peers. 5. These lessons will help students feel safer at BMS. 6. These lessons will help to get rid of some of the bullying that goes on at BMS. 7. Teaching these lessons is worth my time. 8. The lessons are appropriate for the students. 9. These lessons are good for BMS students. 10. I think that these lessons will help create a safer school.

10 | P a g e

(Fill in the blank)

iii. Instrument 2 – Learning Assessment

Objective: Gauge the extent participants retained content from the treatments by measuring how much they actually learned.

Responsible Party: Principal, Unit Director, 8th grade homeroom teachers

Given to: All program participants, 8th grade students at BMS, 80-100% response goal

Timing of Administration: Six different tools will be given after various grouping of lessons (groupings based on content pairs in unit):

Learning Assessment 1 after Lessons 1 & 2 Learning Assessment 2 after Lessons 3 & 4 Learning Assessment 3 after Lessons 5, 6, and 7 Learning Assessment 4 after Lessons 8 & 9 Learning Assessment 5 after Lesson 10

[Please Note: There is no additional cumulative assessment necessary for the learning assessment section of instruments. Cumulative measures will take place in the results and behavior surveys.]

Collection Method: Hardcopy pen/paper survey

Survey Design: Each Learning Assessment has 5 Questions with guidelines above in the Timing of Administration section explaining when to give the assessment. Questions can be presented on a screen to the class, and students can write answers on paper numbered 1-5 with lesson numbers at the top and turn it in.

Learning Assessment 1 (Lessons 1 & 2):

1. The purpose of the Owl Time is to explore various aspects of identity. (True or False)2. Which one is NOT a way that you can contribute a high quality of work?

a. Commitment to excellenceb. Communication with teachersc. Communication with classmatesd. Commitment to technology

3. In Owl Time we will learn about how race, class, gender, family background, religion and learning styles affect only one level of society. (True or False)

4. Honesty, openness, courage, caring, commitment, respect, and trust will contribute to a safe place during Owl Time. (True or False)

5. We can only talk about the lesson plans in class. (True or False)

Learning Assessment 2 (Lessons 3 & 4):

1. The word emotion comes from the same root as the word __________________. Our emotions are like fuel; they energize and motivate us.

2. It is good to ignore emotion. (True or False)

11 | P a g e

3. For good listening, it is important to listen even if you don’t agree. (True or False)4. The brain releases chemicals that process a healthy identity when some truly feels

accepted. (True or False)5. Crying is a way of expressing emotion. (True or False)

Learning Assessment 3 (Lessons 5, 6, & 7):

1. Learning only goes on in school. (True or False)2. Phone interviews are the best way to collect data for family report project. (True or

False)3. The Internet will be your best source for info. (True or False)4. Learning is a natural process. (True or False)5. Unique stories are not ideal for the family report project. (True or False)

Learning Assessment 4 (Lessons 8 & 9):

1. Gender roles are always the same. (True or False)2. Oppression against women & girls is when girls are treated positively. (True or False)3. Boys and girls deal with oppression. (True or False)4. Boys and girls have all the same issues. (True or False)5. Gender roles are shaped by society. (True or False)

Learning Assessment 5 (Lesson 10):

1. The way that society assigns defines people based on culture does not limit the access political and economic power and social privileges for certain groups of people. (True or False)

2. People make accommodations to avoid punishment. (True or False)3. Resisting oppression is more successful when individuals act alone than when groups of

people organize together. (True or False)4. Which one is NOT a kind of oppression that we learned about:

a. Internalized Groupb. Personally Mitigatedc. Self-Institutionalizedd. Institutionalizede. Self-Internalized

5. By educating yourself on cultural issues and studying history, you can help to resist oppression. (True or False)

12 | P a g e

iv. Instrument 3 – Behavior Survey

Objective: To gauge the behaviors of the participants’ (8th graders) compared to other grades’ behaviors as well to evaluate if behaviors are transferring outside of the Owl Time Classroom where treatment is taking place.

Responsible Party: Principal, Unit Director

Given to: All those who teach 8th grade students, the related arts teachers and 8th grade teachers, 80-100% response goal

Timing of Administration: The same survey given twice:1. Before the Unit begins 2. One month after the Unit is complete

Collection Method: Hardcopy pen/paper survey

Survey DesignThis survey will be administered using a 5-point Likert scale, which should be attached to each of the survey statements below. There is one designated for Unit Teachers and one designated Related Arts Teachers.

[Please Note: A Likert Scale is being used rather than asking how many times a student exhibits behaviors because it is make analyzing trends more comprehensible.]

ScaleStrongly Agree ☐ Agree ☐ Neutral ☐ Disagree ☐ Strongly Disagree ☐ N/A ☐

Survey Statements: Which behavior survey is this? (Check One)

____ Before the unit is taught ____ One month after the unit is taught

Related Arts Teachers (Teachers who teach more than just one grade): 1. 8th grade students behave exceptionally better with social issues involving culture

compared other grades. 2. I have observed 8th grade students continually demonstrating cultural awareness in

related arts classes. 3. Cultural differences do not create any divides among all my 8th grade students. 4. I see a great deal more social divisions with my 5th, 6th, and 7th grade students than

my 8th grade students. 5. 8th grade student behaviors are transferring to many other students I teach.

Only given with the 2nd administration of this survey: 6. Optional: Can you give any specific examples of positive changes in behaviors

from your 8th grade students?

13 | P a g e

7. Optional: Can you give any specific examples of transfer of behaviors from 8th grade students to other grades?

8th Grade/Unit Teachers1. I have observed many students demonstrating strong understanding of cultural

awareness and differences in Owl Time. 2. Cultural differences do not create any divides among the 8th grade students. 3. Students feel socially and emotionally safe throughout the building. 4. Students behave appropriately with all forms of social issues in various settings

during a school day. 5. Students openly accept those who are culturally different than themselves.

Only given with the 2nd administration of this survey: 6. Optional: Can you give any specific examples of changes in behaviors from your

8th grade students, please make sure to consider if the changes are occurring in various settings like lockers, hallway, lunch, academic class, etc.?

14 | P a g e

v. Instrument 4 – Results Survey

Objective: To gauge if results of the iChange Collaborative Unit actually increase students feeling socially and emotionally safer in the school building based.

Responsible Party: Principal, Unit Director

Given to: All program participants, 8th grade students at BMS, 80-100% response goal.

Timing of Administration: One month after the Unit is complete

Collection Method: Hardcopy pen/paper survey

Survey DesignThis survey will be administered using a 5-point Likert scale, which should be attached to each of the survey statements below. This survey is designed to resemble the CSCI survey (Appendix A) that served as the reason this treatment was given using the Sense of Social-Emotional Security Survey (NISCC, 2006). (Appendix B)

PLEASE NOTE: In order truly analyze the trend in student sense of social and emotional security, this survey is replicated exactly from the lowest scoring section of the NSCC that the students took two years ago, to serve as the Post-Test.

ScaleStrongly Agree ☐ Agree ☐ Neutral ☐ Disagree ☐ Strongly Disagree ☐ N/A ☐

Survey Statements1. Many students at my school go out of their way to treat other students badly. 2. I have been insulted, teased, harassed, or otherwise verbally abused more than

once in this school. 3. There are many students in my school who seem to be made fun of a lot by other

students4. Most students I my school act in a way that is sensitive to the feelings of other

students. 5. Students at my school will try to stop students from insulting or making fun of

other students. 6. Very few students insult or make fun or other students. 7. There are groups of students in the school who exclude others and make them feel

bad for not being part of the group. 8. I have seen other students insulted, teased, harassed or otherwise verbally abused

more than once by other students in this school. 9. Most students in my school try to treat other students the way they’d want to be

treated.

15 | P a g e

D. Coding Plan and Data Analysis:

The coding plans below are detailed explanations of how to use the results of the instruments and how they align with key objectives for the treatment of the iChange Collaborative Unit. There will be excel documents emailed to the contacts where the data can be directly inserted to create graphs displaying clear data.

Sample Graph Key:

o In the 100% stacked bar graphs the green parts of the bars are the ones aligned with the desired outcome. These are the ones to look for the 10% increase to align with original goals.

o All other colors show the distribution other responses.

16 | P a g e

i. Instrument 1: Reaction Survey

Purpose:

The results of this survey gauge initial reactions of those directly involved in the iChange Collaborative Unit, unit teachers and student participants. These results will serve as leading indicators for the 10% increase that the Logic Model assumes for sense of security and transfer of knowledge.

This survey looks at the buy-in to the content, which will directly influence the potential for success. This buy-in level will be measured based on the quantity of students and teachers identifying reactions at a Strongly Agree or Agree level.

Coding & Data Analysis: The following describes how to code and analyze the data captured by the Reaction Survey.

Once the results have been calculated, they will be placed into an excel table like below that will chart them using “100% stacked charts” for easy analysis (See below). These results can then be used to serve as the leading indicators for the 10% increases noted in the Logic Model.

Table 6: Sample Reaction Survey Data

17 | P a g e

Table 7: Sample Graph of Reaction Survey Data

Coding Outline Overview of Steps: 1. Separate student responses from teacher responses. 2. The responses will have to be recorded for how many of

answer choice exists.3. The responses will be entered into excel to create a

graph. (use excel file: reaction results)

Coding Outline Overview of Steps: 1. Grade the Learning Assessments 2. Record the number of student scoring an 80% of above

and those scoring below an 80%3. Enter the scoring results into excel to create a graph.

(Use excel file: Learning Assessment Results)

ii. Instrument 2 : Learning Assessment

Purpose:

The results of this survey evaluate the participants’ learning of the unit content. These results will also serve as leading indicators for the 10% increase that the Logic Model assumes for sense of security and transfer of knowledge.

Evaluating the learning of content is a key step for increasing the percentage of students who feel safe in the building and will change and transfer behaviors because they need the knowledge base first. This survey will be evaluated like a test, looking at scores that students get on the iChange Collaborate Content. Therefore, we are looking for at least 80% of students to score at least an 80% (4/5 questions) on each Learning Assessment.

Coding & Data Analysis: The following describes how to code and analyze the data from the learning assessments.

Answers Learning Assessment 1 (Lessons 1 & 2): 1. True2. D. Commitment to technology3. False4. True5. False

Answers Learning Assessment 2 (Lessons 3 & 4): 1. Motion2. False3. True4. True5. True

Answers Learning Assessment 3 (Lessons 5, 6, & 7): 1. False2. False3. False4. True5. True

Answers Learning Assessment 4 (Lessons 8 & 9):1. False

18 | P a g e

2. False3. True4. False5. True

Answers Learning Assessment 5 (Lesson 10):1. False2. True3. False4. C. Self-Institutionalized5. True

Once the results have been calculated, they will be placed into an excel table like below that will chart them using “100% stacked charts” for easy analysis (See below). These results can then be used to serve as the leading indicators for the 10% increases noted in the Logic Model.

Table 8: Learning Assessment Sample Data

19 | P a g e

iii. Instrument 3: Behavior Survey

Purpose:

The results of this survey gauge changes in behaviors of the student participants of the iChange Collaborative Unit from their teachers, both related arts/specials and unit teachers. These results will serve as key indicators to 10% increase that the Logic Model assumes for transfer of knowledge both as compared to other grades (related arts teachers) and transfer to other settings (unit teachers).

This survey will be measured based on the quantity of responses at a Strongly Agree or Agree level. Using a Pre and Post timeline, the Behavior Survey will evaluates the trends to show the 10% assumed increase in the transfer categories explained above.

[Please Note: Related Arts Teacher Behavior Survey accounts for the transfer of knowledge to different grades & the Unit Teacher Behavior Survey accounts for transfer of knowledge in various settings compiling both components of the transfer of knowledge objective.]

Coding & Data Analysis: The following describes how to code and analyze the data captured by the Behavior Survey.

Once the results have been calculated, they will be placed into an excel table like below that will chart them using “100% stacked charts” for easy analysis (See below). These results can then be used to serve as the key indicators for the 10% increases about transfer of knowledge in the Logic Model. The results will be evaluated looking for the trend from the pre & post of each behavior survey for the 10% increase.

IMPORTANT NOTE: In order to prevent overly skewing comparative results, the evaluation team should strive to achieve a response rate for each administration of the survey within +/- 5% of one another. For example, if 20 teachers respond to the Pre-Survey but only 11 respond to the Post-Survey, the ability to meaningfully compare the “trend” is diminished. Also, the two different surveys are separate the Related Arts needs +/-5% to be valid while independently the Unit Teacher need +/-5% to be valid because they are both analyzing different aspects of transfer of knowledge.

20 | P a g e

LA 1 LA 2 LA 3 LA 4 LA 5Students

0%

20%

40%

60%

80%

100%

<80%≥ 80%

*LA=Learning Assessment

Learning Assessment

Coding Outline Overview of Steps: 4. Separate two different survey responses 5. The responses will have to be recorded for how many of

each answer choice exists.6. The responses will be entered into excel to create a

graph. (Use excel file: behavior results)7. Use optional questions as anecdotal data that can used

however deemed appropriate by BMS.

Table 9: Learning Assessment Sample Graph

Table 10: Behavior Survey Sample Data

iv. Instrument 4 – Results Survey

21 | P a g e

Table 11: Behavior Survey Sample Graph

Purpose:

The results of this survey gauge the changes in student participants sense of social and emotional security after the completion of the iChange Collaborative Unit looking at the results. These results will serve as key indicators to 10% increase that the Logic Model assumes for sense of security based on certain responses only to replicate the original population used from the CSCI data as closely as possible.

This survey will be measured based on the quantity of responses at a Strongly Agree or Agree level from some and Strongly Disagree or Disagree for other (the division will be shown below). Using the CSCI survey from January 2011 as the pre data, this survey is a replica of that one to serve as a post-test that can demonstrate a trend to show the 10% assumed for student sense of security.

[Please Note: the two buckets for questions responses using both positive and negative answers for positive results are due to the wording from the CSCI original questions.]

Coding & Data Analysis: The following describes how to code and analyze the data captured by the Behavior Survey.

IMPORTANT NOTE: The +/-5% is not necessary to consider for this trend due to the two-year gap and unknown aspect from the pre-test. This evaluation is looking at the big picture for overall sense of security, so as many responses as possible are ideal.

Strongly Agree & Agree Designate Positive Outcomes Strongly Disagree & Disagree Designate Positive Outcomes4. Most students I my school act in a way that is sensitive to the feelings of other students.5. Students at my school will try to stop students from insulting or making fun of other students.6. Very few students insult or make fun or other students.9. Most students in my school try to treat other students the way they’d want to be treated.

1. Many students at my school go out of their way to treat other students badly.2. I have been insulted, teased, harassed, or otherwise verbally abused more than once in this school.3. There are many students in my school who seem to be made fun of a lot by other students7. There are groups of students in the school who exclude others and make them feel bad for not being part of the group.8. I have seen other students insulted, teased, harassed or otherwise verbally abused more than once by other students in this school.

22 | P a g e

Coding Outline Overview of Steps: 1. First, look at the questions desiring answers of strongly

agree and agree versus those desiring answers of strong disagree or agree responses.

2. The responses will have to be recorded for how many of each answer choice exists.

3. The responses will be entered into excel to create a graph. (Use excel file: results results)

4. Use optional questions as anecdotal data that can used

Once the results have been calculated, they will be placed into two different excel tables and graphs (based on the two groups above). Notice the charts and tables below using “100% stacked charts” for easy analysis (See below). These results can then be used to serve as the key indicators for the 10% increases about transfer of knowledge in the Logic Model. The results will be evaluated looking for the trend from the pre & post for the assumed 10% increase (specifically looking for 10% increase in the green sections).

23 | P a g e

Table 12: Results Sample Data (Agree)

Table 13: Results Data Sample Graph (Agree)

Table 14: Results Sample Data (Disagree)

Table 15: Results Sample Graph (Disagree)

E. Presentation Plan:

As a final tool, I have created two different outlines that the client can use to help them present this information to two different parties. The first outline will be for Dr. Michelle Springer, the BMS Principal, to see how the unit met her expectations, and the second will be the 8th grade teachers/unit teachers to help to look for improvements and reflect on the gains. The outline for Dr. Springer will focus on the data and percentages aligning with the key questions that are based on her original goals for the program. The outline for the teachers will focus on the reactions and content to help improve and better create the program as well as the results and their thoughts on them. The teachers should be done first as a check before going to the Principal. Also, these should be done as close to after the data is taken as possible.

[Please Note: This is only a recommendation]

Table 16: Evaluation Results Communication PlanCommunications by Stakeholder Audiences

Stakeholder Role/Stake Communication Plan

24 | P a g e

8th Grade Teachers/Unit Teachers

As the teachers responsible for teaching and implementing the unit, it is important to discuss with them the outcomes. They worked closest with the unit and have the most knowledge of the unit and students. It is important to keep them in the look and get their thoughts, especially if information seems accurate to them.

Summary of briefing package:o Summary of key evaluative findings

displayed by the graphs dividing by instruments

o Ask for feedback and opinions on resultso Ask for thoughts for improvement.

Principal: Dr. Michelle Springer

Dr. Springer is the one who wanted the program from the beginning. As quoted in the beginning, she feels that cultural awareness will solve the problems of students not feeling safe in the building. Her stake is huge as the principal and her own personal values.

o A recommended briefing package:o Key questions addressed by the

evaluationo Provide the timeline of when the instruments will

be given with:o Rational for each instrumento How instruments align with the key

questions.o Copies of instrumentso Summary of key evaluative findings displayed by

the graphs dividing by instrumentso Main focus on Results and Behavior

sectionso Explain the green is where to focus to

look for 10% gains

25 | P a g e

Appendix A:

Appendix B:

26 | P a g e

[Please Note: The client has the original in color where the data is clearly displayed, this is just used to be a point of reference.]

References

27 | P a g e

NISCC, 2006. Bellevue Middle School CSCI Report. Schoolclimate.org (January 2011).

Caldwell, M. & Frame, O. (2012). iChange Collaborative Unit. Inquiry into Identity: A Middle School Curriculum for Race, Class and Gender Issues.

Hancock, C. & Springer, M. (2012). Personal Communication.

28 | P a g e