Assessment Analysis and Critique

Carly GuinnCollege of William and MaryCRIN 550, Dr. HindmanAssessment Analysis and Critique

1Purpose of the TestAdministered: March 7, 2013Matoaka Elementary School3rd grade21 studentsA review of second grade concepts: Ancient Egypt and ChinaTest designed to meet the new rigor of SOL testsMultiple correct responsesSorting through multiple layersAdministered during the 40-minute block they are used to using for Social Studies

2Table of Specifications(Select-Response Questions)

Question # (% correct)

3Table of Specifications(Supply-Response Questions)Student writes a paragraph using complete sentences, accurate punctuation, indentation, and appropriate spelling (2 points) (90%)Student identifies one plausible way in which the culture of Ancient China was affected by its environment/land (4 points) (43%)Student identifies one plausible way in which the culture of Ancient Egypt was affected by its environment/land (4 points) (62%)Student accurately identifies an invention of Ancient Egypt or Ancient China that was built due to the environment of the respective region (3 points) (86%)*(If more than one invention is provided, points will only be given for one correct answer)Student provides the correct usage of the invention identified (2 points) (67%)Rubric Item (point value) (% correct)

4A Final Note on ScoringQuestions 1-10, 15-27Select-responseOne point each13 points totalQuestions 11-14Multiple-response questionsOne point per correctly-selected item; points deducted from collected points for incorrect response8 points totalQuestion 28Supply-response: short answer question5 rubric items; all or nothing for each item15 points total

46 points total

5Students in the AggregateClass ScoresHigh score 45 = 95%Low score24 = 52%Mean36.95/46 = 80%Mode32, 41, 45 = 70%, 89% 98%(3 responses each)Median39 = 85%Results by GenderFemale (11)Mean: 36.6 = 80%Mode: 32, 41, 42 = 70%, 89%, 91%(2 responses each)Median: 39 = 85%Male (10)Mean: 37.3 = 81%Mode: 35, 40, 45 = 76%, 87%, 98%(2 responses each)Median: 37.5 = 82%Results by EthnicityWhite(18)Mean: 37.6 = 82%Mode: 41, 45 = 89%, 98% (2 responses each)Median: 40 = 87% Hispanic(3)Mean: 33 = 72%Mode: n/a(one response for each score)Median: 33 = 72%

Discrepancy b/w mean and median in female shows that females had more significant outliers than males didReason for choosing race and only having HispanicDid score lowerBased on personal experience with students, their scores seem in line with their typical scores on testsTest was examined for cultural biasOnly 3 responses could have impacted analysis6Students in the AggregateClass ScoresHigh score 45 = 95%Low score24 = 52%Mean36.95/46 = 80%Mode32, 41, 45 = 70%, 89% 98%(3 responses each)Median39 = 85%Results by Reading LevelHigh readers (above grade level) (9)Mean: 40.3 = 88%Mode: 45 = 98% Median: 41 = 89% Average readers (on grade level) (9)Mean: 36.8 = 80% Mode: 32, 35, 40 = 70%, 76%, 87%(2 responses each) Median: 35 = 76% Low readers (below grade level) (3)Mean: 27.3 = 59% Mode: n/a(one response for each score)Median: 27 = 59%

Inferences!

Obviously this test will be affected by students reading levels that was a concern of its validity, that while it is written on the reading level of a middle-of-the-year third grader, not every student is at this level; it would assess students reading and writing levels more than their content knowledge. The writing element especially hurt students who may not be high readers. Some high readers, though, were affected by just reading the directions. Many students did not pay attention to directions and missed points because of it (especially for sections like questions 11-14).7Student Profiles: Student AStudent DescriptionCaucasian femaleLow reader (below grade level), works with reading specialistLow-SES (receives free/reduced lunch)Very little family support: rarely returns notes signed, does not receive help with homework

Student As ScoresQuestions 1-108/10Questions 15-279/13Questions 11-147/8Short Answer7/15TOTAL31 = 67%Student missed questions: 5, 9, 13, 22, 24Student missed short answer items 2 and 3All questions missed were at the comprehension/application levelStudent was significantly impacted by having to write a short answer response

Why I chose student A: the lowest reader8Student Profiles: Student BStudent DescriptionCaucasian maleAverage reader (reads just below mid-year third grade level)Does not receive free/reduced lunch; active in sports and extracurricular activitiesHigh level of family support; parents often participate in school-related functionsFence sitter

Student Bs ScoresQuestions 1-109/10Questions 15-2712/13Questions 11-146/8Short Answer13/15TOTAL40 = 87%Student missed questions: 6, 13, 14, 22Student missed short answer item 5Student missed most questions from SOL 2.4b; no questions were missed from the knowledge levelStudent was affected by questions that were designed to reflect the rigor of the new SOLs

9Principles of assessmentQuestion formatsScoring decisions

Benefits for students: short- and long-term

Creating a valid and reliable assessmentInstructional Decisions

Principles of AssessmentJudge student learningProvided with detailed information regarding students:Knowledge of contentAbility to respond to various question formatsAbility to apply their knowledge of content to written responsesImprove student learningTeacher understands what material must be reviewedFinds gaps in learning and understandingOpportunity to discuss new question formats with studentsPrepares students for SOLsEmphasizing reading directions!Review strategies and techniques for addressing writing promptsTeacher-lead think aloud

Question FormatsMultiple choiceMultiple responseStudents required to select more than one answer on three of four questionsStudents needed to read directions carefullyMatchingMulti-level matchingStudent needed to first understand what a contribution wasDetermine if it was from Egypt or ChinaShort-AnswerTake content knowledge and apply it to a writing promptAnalysis-level thinkingAlso graded on third-grade writing

12Scoring DecisionsMultiple response itemsNeeded to give points to accurately address question formatStudents accumulate points throughout the test inability to lose points for each questionShort answer itemsRubric was clearly defined for students in the question itself (expectations were clear)Points per rubric item were proportionally related to the intended difficulty level and thought-time requirementAssessment of student writing expected to help students

13Short-TermLong-TermReviewing second-grade knowledge relevant for SOLAssessment tells students what they know and what they should review prior to the SOL

Provides practice with new question formats and expectations Think aloud following test review provides information about ways to address writing promptsBenefits for Students

ValidityReliabilityTest items were aligned with:Blooms TaxonomyVA standards (SOL Blueprints)Instructional timeLooked like a test (especially like practice SOL tests)Test was administered directly following instruction of material

Question design reflected rules specified by Gareis and GrantQuestions accurately evaluated students understanding of content unless student guessed correctlyTest was reviewed by another W&M CRIN 550 studentPossible threatsRandom errorStudents discomfort with question formatTaking Precautions

Edits and Modifications for the FutureEnsure ample instruction is given on most difficult topicsShort answer questionsGive students instruction regarding question formats prior to test

Carly GuinnAssessment Analysis and Critique

17