academic program assessment report - kean universityncate/caep accreditation 2017/slo... ·...
TRANSCRIPT
Page 1 of 20
Academic Program Assessment Report INSTRUCTIONS: Page 1 of this document serves as the program’s annual assessment plan. Please complete page 1 by October 31, 2015. Pages 2-3 serve as the program’s annual assessment report. Please complete pages 2-3 by June 30, 2016.
COLLEGE: College of Education
ENTER PROGRAM NAME: Elementary Education Initial Certification (K-6, K-6/5-8)
ACADEMIC YEAR: 2015-2016
REPORT AUTHOR: Gail Verdi
PROGRAM STUDENT LEARNING OUTCOMES (CHECK OFF THE SLOs BEING ASSESSED):
☐ SLO1: Knowledge (KU 2, 4; GE S4) -- Candidate has understanding and knowledge of subject matter, and of national and New Jersey state standards.
☐ SLO2: Knowledge (KU 2, 4; GE S4, V4) -- Candidate demonstrates knowledge of how elementary students learn and develop.
☐ SLO3: Skills (KU 2, 4; GE S4) -- Candidate plans instruction based on knowledge of subject matter, and of national (CCSS), professional and New Jersey Core Curriculum Content Standards.
☐ SLO4: Skills (KU1, 4; GE S4, V4) -- Candidate uses formal and informal assessment strategies to measure student progress and adjust instruction as needed.
☐ SLO5: Dispositions (KU1, 2, 4; GE K2, S4, V4) – Candidate appreciates individual, cultural, and linguistic differences.
☐ SLO6: Dispositions (KU 2, 4; GE S@, S4, V2, V4) – Candidate takes responsibility for establishing a positive climate in the classroom. DIRECT MEASURE:
☐ SLO1: Knowledge (KU 2, 4; GE S4) -- Candidate has understanding and knowledge of subject matter, and of national and New Jersey state standards.
☐ SLO2: Knowledge (KU 2, 4; GE S4, V4) -- Candidate demonstrates knowledge of how elementary students learn and develop. Praxis II In order for K-6 Elementary Education teacher candidates to qualify for their professional internship semester (student teaching), they must take and pass the Praxis II: Multiple Subjects Test (5001) designed by ETS (http://www.ets.org/s/praxis/pdf/5001.pdf)/. This standardized test measures teacher candidates’ content knowledge of Language Arts, Mathematics, Social Studies, and Sciences, as well as their understanding of how students at different grade levels acquire and develop specific skills. For example, when reviewing descriptors for foundational skills in reading, it is noted on page 6 of the Praxis Study Companion (see link above) that teacher candidates “must understand the role of phonological awareness in literacy development” as well as “identify and provide examples of phonemes, syllables, onsets, and rimes.” Therefore, this assessment not only assesses teacher candidates understanding and knowledge of subject matter linked to national and NJ State standards, but that they also have to demonstrate how
Page 2 of 20
students learn and develop these forms and knowledge and skills over time. The Praxis II: Multiple Subject test is a requirement of the New Jersey Department of Education for all Elementary Education K-6 initial teacher candidates. TARGET:
Minimal scores required for initial teacher certification candidates in the state of New Jersey are as follows:
Content Test Test Code Minimal Passing Score
1. Reading and Language Arts Subtest (5002) 157
2. Mathematics (5003) 157
3. Social Studies (5004) 155
4. Science (5005) 159
Please note that these are minimal scores required as of November 9, 2015. The cut off scores may be revised during a given academic year. Therefore, scores may change in subsequent reports
DATA COLLECTION AND RESULTS:
Over the 2012-2013, 2013-2014, 2014-2015 academic years, ETS revised the Elementary Education Praxis II exam three
times. Below you will find Table 1, listing a breakdown of the years each exam was counted towards NJ Elementary
Education K-6 certification, the passing scores for the three different versions of the test, and the years that data for the
each exam were collected:
Elementary Education Content Knowledge Test (0014/5014)
Elementary Education Multiple Subject Tests (5031-combined) a. Language Arts (5032) b. Math (5033) c. Social Studies (5034) d. Science (5035)
Elementary Education Multiple Subject Test Revised LA & Math (5001 combined) e. Language Arts (5002) f. Math (5003) g. Social Studies (5004) h. Science (5005)
Page 3 of 20
Table 1.
Academic Years Test Administered 2012-2013 2013-2014 2014-2015
Elementary Education Content Knowledge (0014/5014) Passing score = 141
I - Reading/Language Arts X X X
II - Mathematics X X X
III - Social Studies X X X
IV - Science X X X
Elementary Education Multiple Subject Language Arts (5032) Passing Score = 165
I - Reading X X X
II - Language Writing & Communication X X X
Elementary Education Multiple Subject Mathematics (5033) Passing Score = 164
I - Number Operations and Algebraic Thinking X X X
II - Geometry, Measure Data and Interpretation X X X
Elementary Education Multiple Subject Social Studies (5034) Passing Score = 155
I - US History, Government and Citizenship X X X
II - Geography, Anthropology and Sociology X X X
III - World History and Economics X X X
Elementary Education Multiple Subject Science (5035) Passing Score = 159
I - Earth Science X X X
II - Life Science X X X
III - Physical Science X X X
Elementary Education Multiple Subject Language Arts Subset (5002) Passing Score = 167
I - Reading X
Page 4 of 20
II - Language Writing & Communication X
Elementary Education Multiple Subject Mathematics Subset (5003) Passing Score = 167
I - Numbers and Operations X
II - Algebraic Thinking X
III - Geometry, Measure Data and Interpretation X
Elementary Education Multiple Subject Social Studies Subset (5004) Passing Score = 155
I - US History , Government and Citizenship X
II - Geography, Anthropology and Sociology X
III - World History and Economics X
Elementary Education Multiple Subject Science Subset (5005) Passing Score = 159
I - Earth Science X
II - Life Science X
III - Physical Science X
Narrative Discussion of Changes Across Versions of Elementary Education Praxis II Exams
1.1 Elementary Education Content Knowledge Praxis II (0014/5014): The passing score for both Kean University and
The State of New Jersey for this version of the test is 141. This test, like the later versions of Multiple Subject Exam, was
designed for prospective teachers of children in primary through upper elementary school grades. The 120 multiple-
choice questions focused on four major subject areas: language arts/reading, mathematics, social studies, and science.
There are 30 questions for each subject area. Test questions were arranged in the test book by subject area. Each of the
four content areas constitutes 25 percent of the test. This test is no longer being administered because it was found that
the composite score of 141 did not require teacher candidates to exhibit proficiency in all four content areas. In other
words, you could score high on the language arts, social, studies, and science portions of the exam, but miss a majority
of math questions and still pass the test by receiving a combined score of 141. This would mean that elementary
teachers would be certified even though they lacked the appropriate background knowledge in one or more of the four
key content areas assessed through the Elementary Education Content Knowledge Praxis II (0014/5014):
Praxis II Content Categories No. of Questions % of Exam
Reading/Language Arts 30 25%
Mathematics 30 25%
Page 5 of 20
Social Studies 30 25%
Science 30 25%
1.2 Elementary Education Multiple Subject Praxis II Subtests (5031 Series/5032, 5033, 5034, 5035) were instituted in
the 2012-2013 academic year: The passing scores for both Kean University and the State of New Jersey for this version
of the test were now disaggregated by content area (subtests) to resolve the issues found in the previous Content
Knowledge Praxis II:
Content Area Passing Score Code #Qs % of Exam
Language Arts 165 (5032) 65 100%
Math 164 (5033) 40 100%
Social Studies 155 (5034) 55 100%
Science 169 (5035) 50 100%
The major shift in the Praxis II 5031 Series test construct was that it required teacher candidates to pass each section
separately to qualify for state certification. However, ETS now made it easier for universities and employers to review
data that indicated the specific strengths and weaknesses under subcategories of a content area by cohort. Table 1
above shows how each content area section of the Praxis II tests specific skills. For example, under Language Arts (5032)
teacher candidates are tested on their knowledge of reading as well as language, writing, and communication. When
looking at the actual data, we can see the scoring trends of a university cohort under subcategories of the Language Arts
(5032) subtest.
1.3 Elementary Education Multiple Subject Praxis II Exams (5001 Series/5002, 5003, 5004, 5005) were instituted in the
2014-2015 academic year: For the third and most recent version of the Elementary Education Praxis II, passing scores
for both Kean University and the State of New Jersey for this version of the test continued to be disaggregated by
content area (subtests) to resolve the issues found in the Content Knowledge Praxis II (0014/5014) and continued to
include the same subcategories of knowledge under each subtest, According to a 2014 Multistate Standard-Setting
Technical Report issued by Educational Testing Services regarding the Language Arts and Math Subtests, the Language
Arts and Math subtests were revised to reflect the new National Common Core Standards; therefore, the number of
items of each subtest for LA and Math were increased and the scores on both these tests were lowered from 165 to 157
for Language Arts & from 164 to 157 for Math to reflect the revisions made for this purpose. The Social Studies and the
Science subtests scores were not changed:
Page 6 of 20
Content Area Passing Score Code #Qs 5
Language Arts 157 (5002) 80
Math 157 (5003) 50
Social Studies 155 (5004) 55
Science 169 (5005) 50
1. Alignment with national ACEI Standards
Table 2
Academic Years 2012-2013 2013-2014 2014-2015 ACEI Standards Qs Percent Qs Percent Qs Percent
Elementary Education Content Knowledge (0015/5014) Passing score = 141
I Language Arts 30 25% 2.1
II - Mathematics 30 25% 2.3
III - Social Studies 30 25% 2.4
IV - Science 30 25% 2.2
Elementary Education Multiple Subject Language Arts (5032) Passing Score = 165
I - Reading 32 49% 2.1
II – Lang. Writing & Communication
33 51% 2.1
Elementary Education Multiple Subject Mathematics (5033) Passing Score = 164
I – Number Operations and Algebraic Thinking
26 65% 2.3
II - Geometry, Measure Data and Interpretation
14 35% 2.3
Elementary Education Multiple Subject Social Studies (5034) Passing Score = 155
I - US History, Govern. & Citizenship
25 45% 2.4
II - Geography, Anthrop. and Sociology
16 30% 2.4
III - World History and Economics
14 25% 2.4
Elementary Education Multiple Subject Science (5035) Passing Score = 159
I - Earth Science 16 32% 2.2
II - Life Science 17 34% 2.2
III - Physical Science 17 34% 2.2
Elementary Education Multiple Subject Language Arts (5002) Passing Score = 157
I - Reading 38 47% 2.1
II - Language Writing & Communication
42 53% 2.1
Elementary Education Multiple Subject Mathematics (5003) Passing Score = 157
I - Numbers and 20 40% 2.3
Page 7 of 20
Operations
II - Algebraic Thinking 15 30% 2.3
III - Geometry, Measure Data and Interpretation
15 30% 2.3
Elementary Education Multiple Subject Social Studies (5004) Passing Score = 155
I - US History , Govern. Citizenship
25 45% 2.4
II - Geography, Anthrop. and Sociology
16 30% 2.4
III - World History and Economics
14 25% 2.4
Elementary Education Multiple Subject Science Subset (5005) Passing Score = 159
I - Earth Science 16 32% 2.2
II - Life Science 17 34% 2.2
III—Physical Science 17 34% 2.2
As indicated in Table 2 above, all three of the Praxis II tests: Praxis II Elementary Content Knowledge (0014/5014), Praxis
II Elementary Education Multiple Subjects (Series 5031), and Revised Praxis II Elementary Education Multiple Subjects
(Series 5001) are aligned with the Association of Childhood Education International (ACEI) Specialized Professional
Association (SPA) standards.
Content Categories ACEI Standards
Reading/Language Arts 2.1
Mathematics 2.3
Social Science 2.4
Science 2.2
Similarly, the subcategories of the 4 content areas have remained consistent across the three tests:
Language Arts covers Foundations of Reading, Language in Writing, Communication Skills (Speaking,
Listening, and Viewing).
Mathematics covers Mathematical Processes, Number Sense and Numeration, Algebraic Concepts,
Informal Geometry and Measurement, and Data Organization and Interpretation.
Social Studies covers Geography, Anthropology, Sociology, World History, United States History,
Government, Citizenship, and Economics
Science covers Earth Science, Life Science, and Physical Science
However, what did change drastically was the shift in test construction from one test (0014/5014) to 4
separate tests (5031) of equal weight, to a realignment of the Language Arts and Math tests (5002) and (5003)
to the National Common Core Standards.
Page 8 of 20
DISCUSSION OF FINDINGS
2. Brief analysis of data findings A review of the Praxis II data from 2012-2015 indicates that our candidates achieved a 100% pass rate for all three
versions of the Praxis II Elementary Education Exams. Therefore, it can be argued that our candidates possess the
overall content knowledge to score well on the Praxis II and that currently we are well placed in terms of our passing
rate when compared to state and national averages. The results reported in Table 3 below focus on the number of
correct answers on each of the subtests of each version of the Praxis II
Table 3: Test Results 2012-2015 - Percent of Correct Answers
Test Results (Percent Correct Answers) National New Jersey Kean University Elementary Education Content Knowledge (0014/5014) Passing score = 141 (2012-2015) N=143 I - Reading/Language Arts 77 68 75 II - Mathematics 75 66 74 III - Social Studies 64 57 68 IV - Science 70 61 72 Elementary Education Multiple Subject Reading Arts (5032) Passing Score = 165 (2012-2015) N=95 I - Reading 76 71 80 II - Language Writing & Communication 78 74 82 Elementary Education Multiple Subject Mathematics (5033) Passing Score = 164 (2012-2015) N=95 I - Number Operations and Algebraic Thinking 73 70 79 II - Geometry, Measure Data and Interpretation 67 63 75 Elementary Education Multiple Subject Social Studies (5034) Passing Score = 155 (2012-2015) N=95 I - US History, Government and Citizenship 66 61 69 II - Geography, Anthropology and Sociology 71 67 75 III - World History and Economics 68 61 67 Elementary Education Multiple Subject Science (5035) Passing Score = 169 (2012-2015) N=95 I - Earth Science 66 62 62 II - Life Science 76 73 77 III - Physical Science 58 54 54 Elementary Education Multiple Subject Language Arts (5002) Passing Score = 167 (2014-2015) N=3 I - Reading 67 64 68 II - Language Writing & Communication 65 61 64 Elementary Education Multiple Subject Mathematics (5003) Passing Score = 167 (2014-2015) N=3 I - Numbers and Operations 74 68 75 II - Algebraic Thinking 63 60 63 III - Geometry, Measure Data and Interpretation 68 65 69 Elementary Education Multiple Subject Social Studies (5004) Passing Score = 155 (2014-2015) N=3 I - US History , Government and Citizenship 60 55 75 II - Geography, Anthropology and Sociology 70 66 57 III - World History and Economics 57 54 90 Elementary Education Multiple Subject Science (5005) Passing Score = 159 (2014-2015) N=3 I - Earth Science 67 65 68 II - Life Science 76 75 69 III - Physical Science 61 59 72
Page 9 of 20
Comparison of Number of Correct Answers from National, State, and Kean University Data 2012-2015
As noted in Table 3 above, the aggregated data outlines the number of teacher candidates from Kean Union, Kean
Ocean County, and Kean’s Post Bac Program that took one of the following three tests: For 014 the total number of test
takers is 143, for the 5031 series the total is 95, and for 5001 series the total is 3. Given the extremely low number of
test results from the 5001 series, the data from the newest version of the Praxis II Multiple Subjects test is the least
reliable, but when we compare data from the 5031 series and the 5001 we can see that Kean’s teacher candidates
remain on par with national averages and score higher on the majority of the subtests than their New Jersey state peers.
Language Arts Data (Number of Correct Answers) Across Exams (014, 5032, and 5002)
In 2009, when comparing the number of correct answers on the 014 Language Arts subtest to National data, it
was reported that 5% of Kean teacher candidates scored below the national average. In the years 2012-2015 only 2% of
teacher candidates taking the 014 Language Arts subtest scored below the National average and 10% of Kean teacher
candidates scored above the New Jersey State average. Therefore, Kean teacher candidates test score results on this
portion of the 014 Praxis II exam rose 3% when compared to the nation and state data showed that our students held a
significant lead in results when compared to their counterparts at other New Jersey Teacher Education Programs.
When reviewing the data from 5032 Language Arts subtest, the data shows that Kean graduates scored 5%
higher than the national average and scored 12% higher compared peers at other higher education institutions in New
Jersey. Similarly, the data from the 5002 Language Arts subtest demonstrate that our Kean University students
outperform their state cohort (6% higher) and are on par with the national cohort.
Math Data (Number of Correct Answers) Across Exams (014, 5032, and 5002)
In 2009, when comparing scores on the 014 Math subtest to National data, it was reported that 5% of Kean
teacher candidates scored below the national average. In the years 2012-2015 only 1% of teacher candidates taking the
014 Math subtest scored below the National average and 12% of Kean teacher candidates scored above the New Jersey
State average. Therefore, Kean teacher candidates test score results on this portion of the 014 Praxis II exam rose 4%
when compared to the nation and state data showed that our students held a significant lead in results when compared
to their counterparts at other New Jersey Teacher Education Programs.
When reviewing the data from 5032 Math subtest, the data shows that Kean graduates scored 10% higher than
the national average and scored 1% higher compared peers at other higher education institutions in New Jersey.
Similarly, the data from the 5002 Math subtest demonstrate that our Kean University students outperform their state
cohort (7% higher) and are on par with the national cohort.
Social Studies Data (Number of Correct Answers) Across Exams (014, 5032, and 5002)
Page 10 of 20
In 2009, when comparing scores on the 014 Social Studies subtest to National data, it was reported that 6% of
Kean teacher candidates scored below the national average. In the years 2012-2015 only 6% of teacher candidates
taking the 014 Social Studies subtest scored above the National average and 19% of Kean teacher candidates scored
above the New Jersey State average. Therefore, Kean teacher candidates test score results on this portion of the 014
Praxis II exam rose 12% when compared to the nation and state data showed that our students held a significant lead in
results when compared to their counterparts at other New Jersey Teacher Education Programs.
When reviewing the data from 5032 Social Studies subtest, the data shows that Kean graduates scored 2%
higher than the national average and scored 11% higher compared peers at other higher education institutions in New
Jersey. Similarly, the data from the 5002 Social Studies subtest demonstrate that our Kean University students
outperform their state cohort and are on par with the national cohort - except in one instance. The three students that
took the 5002 Social Studies subtest scored 22% lower than the national average and 15% lower than the state cohort
on the subcategory of Geography, Anthropology and Sociology.
Science Data (Number of Correct Answers) Across Exams (014, 5032, and 5002)
In 2009, when comparing scores on the 014 Science subtest to National data, it was reported that 6% of Kean
teacher candidates scored below the national average. In the years 2012-2015 2% of Kean teacher candidates taking the
014 Math subtest scored above the National average and 18% of Kean teacher candidates scored above the New Jersey
State average. Therefore, Kean teacher candidates test score results on this portion of the 014 Praxis II exam rose 8%
when compared to the nation and state data showed that our students held a significant lead in results when compared
to their counterparts at other New Jersey Teacher Education Programs.
When reviewing the data from 5032 Science subtest, the data shows that Kean graduates scored 4% lower than
the national average and scored 1% higher compared peers at other higher education institutions in New Jersey.
Similarly, the data from the 5002 Science subtest demonstrate that our Kean University students outperform their state
cohort (4% higher) and are on par with the national cohort.
CLOSING THE LOOP
3. Interpretation of how data provides evidence for meeting ACEI standards: The consistent pass rate of 100% indicates the strengths of the Elementary Education Program and our candidates to
provide evidence of their ability to meet ACEI Standards. These results are indicative of our candidates’ ability to
perform well on a nationally standardized exam. While the data shows our students possess the content knowledge to
score well on the Praxis II, this is not a performance-based assessment of the candidates’ abilities to demonstrate
mastery of the integration of this knowledge with teaching or candidates’ professional attitudes. It also shows that we
have steadily improved as result of steps taken, e.g., more opportunities on-campus for Praxis preparation, an increased
focus on content standards have impacted the scores of Kean’s teacher candidates. What this doesn’t show us,
however, are the scores of students that have not passed the Praxis II exam. This is important data for the NJDOE to
supply to colleges of education across the state so that we can realistically accommodate the needs of students that do
not meet the academic requirements of their teacher education programs. The pass rate is 100% because students do
Page 11 of 20
not become program completers if they do not do so. This ultimately skews the data and denies us from getting a truly
clear picture of how our students are doing on these exams. Questions arise here such as: What do we do to assist our
teacher education candidates that do not pass one of more sections of the Praxis II? Faculty need to be involved in
analyzing the types of thinking and the forms of knowledge teacher candidates need to have acquired to pass the Praxis
II Multiple Subjects Exam. For example, what can be done to increase students knowledge of the Geographical,
Anthropological, and Sociological Concepts need to increase scores on that section of the Social Studies exam? One
place Kean’s College of Education has begun to explore answers to these questions is by having faculty take the exams
their students must take in order to become teachers. This has helped us begin to take part in valuable dialogues
around how teacher testing and evaluation impede or inform new teacher preparation programs.
SUPPORTING EVIDENCE: SEE ATTACHMENT A
Page 12 of 20
DIRECT MEASURE:
☐ SLO3: Skills (KU 2, 4; GE S4) -- Candidate plans instruction based on knowledge of subject matter, and of national (Common Core for Math and Language Arts), professional and New Jersey Core Curriculum Content Standards.
TWS
In Professional Internship (Student Teaching), a candidate’s instructional planning based on subject matter and national (Common Core) and NJDOE Standards is assessed using the COE Teacher Work Sample (TWS) Assessment Rubric. Course instructors and clinical supervisors use the TWS Rubric to measure each teacher candidate’s ability to develop appropriate learning goals for a given grade level (including differentiation), construct an assessment plan to determine if students have met goals, design an instructional unit based on those learning goals. Therefore, I am going to focus on three subsections of the Teacher Work Sample Assessment Rubric to analyze data from Direct Measure SLO3 listed above. The two subsections I will focus on for the 2015-2016 academic year are:
1. Learning Goals: The teacher candidate sets significant, challenging, varied, and appropriate learning goals
2. Assessment Plan: The teacher candidate uses multiple assessment modes and approaches aligned with learning goals to assess student learning before, during, and after instruction.
3. Design for Instruction: The teacher candidate designs instruction for specific learning goals, student characteristics and needs, and learning contexts.
The rubric consists of 10 components/criteria (Intro to Portfolio, Philosophy Statement, Contextual Factors, Learning Goals, Assessment Plan, Design for Instruction, Instructional Decision Making, Analysis of Student Learning, Reflection and Self-Evaluation and Writing Mechanics and Appearance). Each component includes 3-4 descriptors to assist course instructors and clinical supervisors in providing detailed feedback to our Professional Interns as they work towards becoming highly qualified teachers.
TARGET:
***For the 2014-2015 Elementary Education Annual Academic Program Assessment Report. I used the following criteria to determine whether teacher candidates met minimum competency on a dispositional measure based on the total score:
A candidate must meet a minimum score between 40-44 (Capable) to be recommended for K-6 Initial certification in the state of New Jersey. Since this is a culminating experience of all K-6 teacher candidates, it is expected that 100% of Professional Intern Candidates meet the minimal competencies outlined in the TWS Assessment Rubric. If a Professional Intern fails to meet the minimal criteria (Capable), a remediation plan is outlined by the cooperating teacher, the clinical supervisor, the Senior Seminar course instructor, and the Director of the Teaching Performance Center. Generally, a Professional Intern in need of a remediation plan will be expected to hold back graduation, and to repeat the Professional Internship experience with further support and guidance.
Page 13 of 20
However, this year I plan to review individual components of the TWS since I believe that select component focus on a various SLOS. Therefore, for 2015-2016 Elementary Education Annual Academic Program Assessment Report I plan to look at data for:
1. Learning Goals: The teacher candidate sets significant, challenging, varied, and appropriate learning goals
2. Assessment Plan: The teacher candidate uses multiple assessment modes and approaches aligned with learning goals to assess student learning before, during, and after instruction.
3. Design for Instruction: The teacher candidate designs instruction for specific learning goals, student characteristics and needs, and learning contexts.
This data will specifically align with SLO 3 listed above. This data will be assessed on a 5 point scale for each item listed above. The criteria are as follows:
A description of how this assessment specifically aligns with the standards
The Level III Teacher Work Sample Portfolio is aligned with both the College of Education Learning Goals based on
the SPECTRUM Model (Knowledge, Skills and Dispositions) and the ACEI standards. The alignment of the
components of the TWS and the College of Education Learning Goals is reflected in the document on page 28 of A
Guide for Compiling a Teacher Work Sample Portfolio titled “Integration of Teacher Work Sample Process and COE
Learning Goals: Teaching Processes Assessed by the Kean COE Teacher Work Sample. The ACEI standards are
indicated in the rubrics used to evaluate each component of the TWS.
For example, the Philosophy Statement aligns with ACEI Standard 1 that requires candidates to know and
understand major concepts, principles and theories related to the development of children and adolescents. The
Contextual Factors assignment spans two ACEI standards 3 and 5: Standards 3.2 and 3.3 requiring clinical interns to
make adaptations for diverse learners and to understand the need to use a variety of teaching strategies to support
learning and to make accommodations for students with specific learning needs. Standard 5.2 require clinical
interns to demonstrate an understanding of the needs of the community they work in and to develop opportunities
1 = Unacceptable
(Not Competent)
Teacher candidate
demonstrates little
or no competence.
2 = Beginning
(Beginning Competence)
Teacher candidate
demonstrates competence
with significant assistance
and prompting.
3 = Developing
(Developing Competence)
Teacher candidate
demonstrates developing
competence with some
assistance and
prompting.
4 = Capable
(Competent)
Teacher candidate
consistently
demonstrates
competence without any
assistance or prompting.
5 = Accomplished
(Highly Competent)
Teacher candidate
consistently demonstrates a
high degree of competence
functioning independently.
Page 14 of 20
to engage parents, colleagues and community members in conversations about teaching and learning. The
Contextual Factors piece helps candidates to compile information they can use to create communication plans and
to help parents and community members become partners in students’ education.
The Learning Goals and Assessment Plan supports ACEI standards 2.1 to 2.7 by aligning learning goals and
assessment plans for tasks to specific content area standards or by integrating cross-curricula units of study and ACEI
standard 4.0 by designing both formal and informal assessments to collect and analyze data from pre and post
assessments and formative assessments.
Similarly, the Design for Instruction covers ACEI standards 2.1-2.7 and 3.1, 3.2, and 3.4. In developing units of
instruction, clinical interns must decide what content area or cross-curricular thematic unit they will teach. In doing
this, the clinical interns will cover one or more of the ACEI standards that focus on curricular design. In addition, this
task invites clinical interns to explore a variety of instructional techniques, therefore, utilizing ACEI standard 3.1
(Integrating and applying knowledge of instruction) and ACEI standard 3.3 (Developing critical thinking and problem
solving). When clinical interns complete the section of the TWS on Instructional Decision-Making, they must
consider the types of modifications they might need to make for their original Design for Instruction based on the
background knowledge students bring to the lessons (ACEI standard 5.1), the need for individualizing instruction
based on findings (ACEI standard 3.2), and the units alignment with learning goals and standards (ACEI standards
1.0, 3.2, & 5.1).
Through the Analysis of Student Learning clinical interns continue to use assessment to plan and evaluate
instructions (ACEI 4.0) and through the Reflection and Self-Evaluation Task they cover ACEI standard 5.1
(Candidates are aware of and reflect on their practice) and continue to explore standards 2.1-2.7 (Curriculum design)
by rethinking the methods and materials used and consider alternative actions that might lead to more probable
success for student learning.
Page 15 of 20
DATA COLLECTION AND RESULTS
3. Brief Analysis of Data Findings
Section I
The data set focusing on clinical interns’ performance during Level III Teacher Work Sample was collected using The
Teacher Work Sample (TWS) Portfolio Assessment Form (attached). The following data table (see Teacher Work
Sample Portfolio Assessment Data attached) were designed by the Teacher Performance Center to summarize
findings collected over four semesters (spring 2014 to fall 2015):
Scores comparing competency levels from both the University Supervisor (EMSE 4800 Professional Internship) and the Course Instructor for Senior Seminar (EMSE 4900)
Scores comparing undergraduate (UG) and post-baccalaureate (PB) clinical interns competency levels Scores comparing competency levels from the Union Campus (Union) and Ocean County Campus (OCC)
Each table includes the ten Process Indicators described in sections 1 and 2 of this report as well as the number of
candidates and competency levels from each subset listed above.
Section II
In addition, during the most recent program review cycle, the elementary education program and the Teaching
Performance Center collected a second source of data for assessment 5 to determine the impact instruction by
clinical interns had on students’ learning. Therefore, for each semester (spring 2015 to fall 2015) data was collected
on the following:
Instructional Unit Number of lessons
taught in the Unit
PreTest Average
Class Score
(Percent Correct)
PostTest Average
Class Score
(Percent Correct)
Percent Increase
(Decrease) in P-12
Student Learning
However, since both data sets (TWS and Student Learning Pretest/Postest Comparisons) are collected via cohorts
and there is no access to individual clinical intern data, we can not determine if there is a correlation between
specific scores on the TWS and the success or failure to produce higher posttest scores by any individual clinical
intern.
Page 16 of 20
4. DISCUSSION OF FINDINGS: Interpretation of the data and how it provides evidence of meeting standards
Section I
As noted in sections 1 & 2 of this report, the ten process indicators listed in The TWS Portfolio Assessment are closely
aligned with the ACEI standards. Therefore, the links between the indicators, descriptors and the ACEI standards
show that the data collected represent high levels of content validity. The cumulative scores submitted by
instructors and supervisors for all subsets described above (UG, PB, Union, OCC) from the spring of 2014 to the fall
of 2015 indicate that majority of competency levels ranged from 4 (Capable) to 5 (Accomplished) for individual
Process Indicators 1-10 with a small percentage of undergraduate clinical interns from both the Union and Ocean
Campuses being scored 3 (Developing). No clinical interns in the Post Baccalaureate Program received a score lower
than a 4 (Capable).
However, upon review of the data across semesters (spring 2014 to fall 2015) there are no trends in specific areas
that need improvement for the majority of undergraduate clinical interns at either the Union or OCC Campuses. For
example, the data doesn’t show that the number of students scored a 3 (Developing) on the Introduction to
Portfolio in the Spring 2014 remained constant over time (to fall 2015). If it had, this would have informed the
faculty that more emphasis would have to be placed on how to write an Introduction in the EMSE 4900 course
during class workshops. Although individual students, may need more support when constructing the TWS, the
cohort as a whole performed at a level consistent with a highly qualified clinical intern.
Sample of Undergraduate Program Data from the Union Campus
Below you will find a sampling of data for two components – Assessment Plan and Design for Instruction –
submitted by the EMSE 4900 Course Instructors and the University Supervisors—for the Undergraduate Program on
the Union Campus from spring 2014 to fall
2015:
Process/Component S14
Instr
S14
Super
F14
Instr
F14
Super
S15
Instr
S15
Super
F15
Instr
F15
Super
Assessment Plan 2
5%
5
11%
0 2
5%
0 1
3%
0 0
Design for
Instruction
1
3%
1
2%
0 2
5%
0 0 0 0
Page 17 of 20
As indicated earlier in this report, university supervisors have a tendency to score the TWS a bit lower than the EMSE
4900 Course Instructor, but this is not significant in the overall assessment of individual cohorts over time. When we
look at the data across semesters, there actually appears to be an improvement in scoring over time. However, this
may be misleading and depend more on variables that may not be determined through this data such as classroom
observations and input from the cooperating teacher that the Course Instructor may not have access to. In addition,
the Course Instructor spends most of his/her time assisting the clinical interns in drafting and organizing the TWS
Portfolio where the University Supervisor’s evaluation of the TWS may be colored by classroom observations of the
clinical intern. Course Instructors tend to view themselves more as advocates for the clinical interns while the
University Supervisor may perceive their role as more of a judge. Since it is difficult to truly determine what is going
on in the minds of the Course Instructor and the University Supervisor, it might be more helpful if we can follow a
few individual students over time to get a sense of how they are progressing and deepening their skills as they move
through the program. It is our hope that the new assessment system, EdTPA, that the NJDOE will be requiring for
clinical interns to complete in order to get certified will help close the gap in our methods of gathering data so that
we can get a better sense of how to measure an individual clinical interns impact on student learning.
Sample of Undergraduate Program Data at the Union and OCC Campuses
Below you will find a sampling of data for two components – Assessment Plan and Design for Instruction –
submitted by University Supervisors—for the Undergraduate Programs on both the Union and OCC Campuses from
spring 2014 to fall
2015:
Process/Component S14
Union
S14
OCC
F14
Union
F14
OCC
S15
Union
S15
OCC
F15
Union
F15
OCC
Assessment Plan 4
9%
1
10%
1
3%
1
10%
1
3%
0 0 0
Design for
Instruction
1
2%
0 1
3%
1
10%
0 0 0 0
As you can see, there are semesters when clinical interns at both campuses receive 3 (Developing) scores on the
Assessment Plan and Design for Instruction components. Given that the data shifts to 0% at both campuses in the
Fall of 2015 might indicate that there is a trend towards students being more prepared for clinical internship prior to
entering their student teaching semester.
Page 18 of 20
These findings make sense in light of the increased GPA required for students to declare Elementary Education as a
major. It also might hint at how the integration of components of the Teacher Work Sample into other courses
starting in the sophomore field experiences have improved clinical interns’ ability to score higher on these
assessments. The faculty learned, from the previous assessment cycle, that it is important to introduce clinical
interns to the components of the final TWS Portfolio that they are expected to complete during their Professional
Internship prior to their final semester:
1. Assessment #3-The Interdisciplinary Unit/Level II Teacher Work Sample requires teacher candidates in Pre-Professional Internship to Develop a Unit of Study and to teach a part or all of the unit to students.
2. Assessment #7-The Language Arts and Literacy Development Case Study is an assignment that assists teacher candidates in acquiring the skills of assessing student language proficiency and developing a plan to help students improve reading and writing.
Although instructors and supervisors may not always agree on whether Kean clinical interns are producing
accomplished work, the general trend in the Final Recommendation scores indicate that Kean is producing clinical
interns that can develop curriculum, assess student learning, and reflect on their teaching in order to improve
instruction. Across the four semesters of data presented above, the majority of clinical interns, UG and PB, as well
as UGs at Union and OCC have received Final Recommendation scores at either the Accomplished (5) or Capable (4)
levels. When revisiting scores for the spring 2014 through fall 2015 semesters percentages of students reaching
either the Accomplished or Capable competency levels ranged from between 95% and 100% across subsets (UG &
PB) and (Union & OCC). Therefore, 100% of clinical interns reported on in this report, both Undergraduates and
Post-Baccalaureates at the Union Campus as well as 100% of clinical interns at OCC were recommended for
certification.
Section II
At the end of the 2013 academic year, faculty determined that although the TWS was an excellent way of aligning
the ACEI Standards with the work clinical interns were completing in the field, they also came to the conclusion that
the assessment of the components of TWS was not enough to measure Kean clinical interns’ impact on student
learning. Given the extensive reform efforts taking place at both the national and local levels requiring teachers to
measure growth via SGOs (Student Growth Outcomes), the COE Teaching Performance Center began collection data
on Pretest/Posttest Comparison Scores to determine whether the instruction taking place during the Professional
Field Experience was showing that students’ were acquiring the knowledge and skills necessary to prepare them for
the next grade and to be College and Career Ready. Below you will see a table reviewing the Average growth % for
the total cohort for semesters Spring 2014 to Fall 2015 (Full Section II Reports are found of the end of this report):
Page 19 of 20
Semester Elementary Programs
UG/PB or Union & OCC
# of Clinical Interns Average % of Score
Increase
Spring 2014 UG/Post Bac/OCC 51 45%
Fall 2014 UG/Post Bac/OCC 44 39%
Spring 2015 UG/Post Bac/OCC 42 48%
Fall 2015 UG/Post Bac/OCC 22 45%
The data collected from Spring 2014 to Fall 2015 is not disaggregated by the subsets of Undergraduate/Post Bac and
Union/OCC; therefore, the data doesn’t present differences in the percentage of average score across the subsets of
UG, PB, and OCC. When averaging the percentage of score increase on posttest scores across the 4 semesters listed
above, the average increase equals 44%. When looking at the data by individual unit topic and Pretest/Posttest
scores, we can reason that, for the most part, the higher the pretest scores the lower the percentage of score
increase. This makes sense when you consider that the more knowledge a student already possess on a given topic,
the less he/she needs to learn in order to pass an assessment. Here are some examples of units that produced low
percentage increases across semesters – leading to a decrease in the overall percentage of average increase:
Semester Topic PreTest Score PostTest Score Increase %
Spring 2014 Geomentry 89.77 90.16 0%
Spring 2014 Reading Non-
Fiction
77 85 9%
Fall 2014 Weather 84 94 11%
Fall 2014 Health-Identifying
Personal Traits
83 92 10%
Fall 2014 Social Studies:
Map Skills
83 93 10%
Spring 2015 Butterflies 74 88 16%
Spring 2015 Making
Connections
78 95 18%
Fall 2015 Objects in the Sky 82 85 4%
Page 20 of 20
CLOSING THE LOOP
Although the data indicates that students working with our clinical instructors increase the understanding of content
by an average of 44%, that doesn’t really tell us what individual students are experiencing in these classrooms. One
option might be for clinical interns to move away from the Pretest/Posttest model to one of working with Student
Growth Objectives geared towards developing realistic instructional goals for students at various levels of
proficiency at a given grade level.
Below is the link to the Achieve NJ Student Growth Objectives Guidebook that outlines procedures for creating
outcomes that run across a continuum of learning. This enables a student that enters a teachers class with little to
no knowledge or skill in a particular area to still show marked improvement over a marking period:
http://www.state.nj.us/education/AchieveNJ/teacher/SGOGuidebook.pdf
Here are examples of different types of Student Growth Objectives (p.4 – SGO Guidebook):
This doesn’t mean that our clinical interns aren’t doing this when they are teaching their units. They are being
required to measure student learning based on these design criteria, but we can’t access that data from the
information we have collected. Therefore, as indicated earlier in this report, it would be helpful if we can find a way
to follow several students longitudinally and find a way to correlate the true impact individual candidates have on
student learning through the TWS. It is important to note that the TWS will be replaced by edTPA in the fall of 2017
as a certification requirement. We will revise our imbedded assessments to assistant clinical interns in designing
and implementing the edTPA, but the edTPA portfolio will be assessed by trained outside evaluators.
SUPPORTING EVIDENCE: SEE ATTACHMENT B