august 2007science bias & sensitivity review1 new england common assessment program necap...
Post on 01-Jan-2016
217 Views
Preview:
TRANSCRIPT
August 2007 Science Bias & Sensitivity Review 1
New England Common Assessment Program
NECAP Science
Bias & Sensitivity ReviewWith Reading Passage Review
Supplement
August 14-16, 2007
Killington, VT
August 2007 Science Bias & Sensitivity Review 2
State Departments of Education Staff:• Michael Hock – Vermont• Deb Fleurant – New Hampshire• Heather Heineke Agnew – Rhode Island
Measured Progress Staff:• Harold Stephens – NECAP Program Director• Elliot Scharff – NECAP Science Program
Manager• Amanda Smith – NECAP Program Manager• Josh Evans – NECAP Program Manager
Welcome and Introductions
August 2007 Science Bias & Sensitivity Review 3
• Committee Member Expense Reimbursement Form
• NECAP Nondisclosure Form• Bias & Sensitivity Guidelines for NECAP item
development
Logistics
August 2007 Science Bias & Sensitivity Review 4
What Is Item Bias?
Bias is the presence of some characteristic of an assessment item that results in the differential performance of two individuals of the same ability but from different student subgroups.
Bias is not the same thing as stereotyping (but we don’t want stereotypes in NECAP either).
We need to ensure that ALL students have an equal opportunity to demonstrate their knowledge and skills.
August 2007 Science Bias & Sensitivity Review 5
The ability to answer an item correctly should reflect a student’s learning opportunities, not cultural opportunities or life experiences.
No student should be disadvantaged by item content that is insensitive or disrespectful to the student's ethnic, religious, or cultural background.
Think of "culture" in broad enough terms to include disability, language proficiency, socio-economic status, and regional differences.
Item insensitivity is not the same thing as bias, but it is a potential source of bias that should be identified and eliminated.
What is the purpose of sensitivity review?
August 2007 Science Bias & Sensitivity Review 6
The “Big Idea” of Bias-Sensitivity Review
NECAP should measure the results of learning opportunities NOT life experiences
August 2007 Science Bias & Sensitivity Review 7
Item DevelopmentBias-Sensitivity Review Item ReviewPilot-Testing FeedbackPilot-Testing Data Analysis (DIF)
How Do We Prevent Item Bias?
August 2007 Science Bias & Sensitivity Review 8
Review assessment items Complete independent ratings of each
item and record comments Be prepared to discuss “Mixed
Reviews” and “Problem Items” with your review team; Finalize independent ratings
Please Note: The Bias-Sensitivity Review Committee plays an advisory role. Final decisions on which passages and items to
include in NECAP will be made by the NH, RI and VT DOEs.
Role of the Bias-Sensitivity Review Committee
August 2007 Science Bias & Sensitivity Review 9
Sensitivity to different cultures, religions, ethnic and socio-economic groups, and disabilities
Balanced gender roles that are free of gender stereotypes Use of positive language, situations, and images Use of context, situations, images, and examples that are
universally understood across groups and cultures Overall, that the test is free of items and text that may elicit
strong emotions in specific groups of students, and as a result prevent those groups of students from accurately demonstrating their skills and knowledge
Overall, that the test is free of items that measure life experiences instead of learning experiences
Role of the Bias-Sensitivity Review Committee
The Bias-Sensitivity Review Committee DOES need to make recommendations that will help ensure…
August 2007 Science Bias & Sensitivity Review 10
Reading Level Grade Level Appropriateness GSE Alignment Language Structure and Complexity Instructional Relevance Accessibility Overall Item Design
Role of the Bias-Sensitivity Review Committee
The Bias-Sensitivity Review Committee DOES NOT need to make recommendations concerning…
The Item Review Committees Are
Addressing These Issues and Factors.
August 2007 Science Bias & Sensitivity Review 11
Process: Day 1 – (1) Training, (2) Practice, (3) Clarification of
Guidelines and Procedures, (4) The entire committee will review grade 8 science, (5) Break into two groups for independent review of item sets—Group 1 will review science for grade 4 and Group 2 will review science for grade 11
Day 2 – (1) Break into small groups for re-evaluation and discussion of “Problem Items” (2) Finalize Independent Ratings, (3) Evaluate the Review Process and Provide Feedback
Day 3 – (1) Training on reading passage review (2) The entire committee will review grade 11 reading passages (3) Break into two groups for independent review of reading passages—Group 1 will review grades 4, 6, and 8 and Group 2 will review grades 3, 5, and 7
The Bias-Sensitivity Review Process
August 2007 Science Bias & Sensitivity Review 12
Avoid issues and topics considered sensitive by the general public.
Keep in mind that some topics that are appropriate for classroom discussion may not be appropriate for assessment.
Consider context, setting, and instructional relevance; Bias-Sensitivity Review is not a “literal” process.
Think of “bias” as an assessment issue as well as a cultural issue, that is, a specific topic or feature of a passage or item that may disadvantage one of two otherwise equally matched groups of students.
General Guidelines
August 2007 Science Bias & Sensitivity Review 13
Abortion Birth control Child abuse or
neglect Creationism Divorce Incest Occult/Witchcraft
RapeReligious doctrineSexual orientationSuicideQuestioning/
ignoring parent/teacher authority
Topics to Avoid in Science Test Items
August 2007 Science Bias & Sensitivity Review 14
Death Impact of humans
on the environment
Natural disasters Conservation,
logging, fisheries Extinction Climate change,
global warming
Sexuality, pregnancy, reproduction, sexually transmitted disease
ViolenceHuman or animal
research subjects, animal rights
PollutionObesity
Science Topics that Need Careful Consideration
August 2007 Science Bias & Sensitivity Review 15
Awareness Issues
Avoid economic, regional, cultural, or gender bias in questions, prompts, identified situations, graphics, or reading selections.
Avoid profanity unless it is used in a relevant direct quote.
Ensure that items and passages emphasize academic programs and learning rather than life experiences.
Remember that we are proud of our states and region and expect to see that pride reflected in assessment passages and items.
Remember that items should reflect scientific thinking, processes, and content and not political or personal opinions. Nor should items require students to present political or personal opinions on scientific issues.
August 2007 Science Bias & Sensitivity Review 16
New England Common Assessment ProgramBias & Sensitivity Review Committee
Item Review Evaluation Form
Read the statement below and select the rating that best describes your evaluation of the item for potential bias and sensitivity concerns.
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Description
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
Comments
Please provide specific comments to explain your evaluation. Your comments should address your particular bias/sensitivity concerns, the subgroup(s) that you feel may be
impacted, and the extent of the potential impact within the subgroup(s).
Item Review Rating Form
August 2007 Science Bias & Sensitivity Review 17
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 18
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 19
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 20
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 21
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 22
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 23
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 24
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 25
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 26
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
2 In a model of our solar system, a tennis ball is used to represent Earth. A marble would be a good representation of Mercury in this model because compared to Earth, Mercury is —A brighterB lighterC smallerD more solid
August 2007 Science Bias & Sensitivity Review 27
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 28
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 29
Bias-Sensitivity Examples and Non-Examples
“This item does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
If a woodland is cut down to make room for a large shopping mall, which of the following would be the most likely outcome for birds living in the area?
a. They will build nests on rooftops
b. They will move to a new woodland habitat
c. They will add garbage to their diet
d. They will stop mating
August 2007 Science Bias & Sensitivity Review 30
Tim Kurtz Director of AssessmentNH Department of Education(603) 271-3846
Mary Ann Snider Director of Assessment and Accountability
Rhode Island Department of Elementary and
Secondary Education(401) 222-8492
Michael Hock Director of Educational AssessmentVermont Department of
Education(802) 828-3115
Information, Questions and Comments
August 2007 Science Bias & Sensitivity Review 31
New England Common Assessment Program
Bias & Sensitivity Review Reading
Passage Supplement
August 2007 Science Bias & Sensitivity Review 32
The “Big Idea” of Bias-Sensitivity Review
NECAP should measure the results of learning opportunities NOT life experiences
August 2007 Science Bias & Sensitivity Review 33
Reading Passages are Selected by Item Developers (who are trained to recognize potential sources of bias)
Sensitivity Review of Reading Passages Reading and Math Items are Developed Bias-Sensitivity Review of Reading & Math
Items Item Review of Reading and Math Items Field-Testing Feedback Pilot-Testing Data Analysis (DIF)
How Do We Prevent Item Bias?
August 2007 Science Bias & Sensitivity Review 34
Abortion Birth Control Child Abuse/Neglect Creationism Divorce Incest Occult/Witchcraft Questioning
Parental Authority
Rape Religious Doctrine Sex/Sexuality Sexual Orientation Sexually Transmitted
Diseases Suicide Obesity
Topics to Avoid in Reading Passages & Test Items
August 2007 Science Bias & Sensitivity Review 35
Death Drugs, Alcohol &
Tobacco Family Issues Guns / Gun Control Handicapping
Conditions Holidays Homelessness
Misuse of Animals/ Animal Rights
Murder Pregnancy Racism, Sexism &
Ageism Religion (Except in
Historical Context) Violence
Topics Which May or May Not Be Acceptable
August 2007 Science Bias & Sensitivity Review 36
Passage Review Rating FormNew England Common Assessment Program
Bias/Sensitivity Committee Meeting Passage Review
Grad Level:
Passage # Title
Passage Evaluation
Read the statement below and select the rating that best describes your evaluation of the passage for potential bias and sensitivity concerns.
“This passage does not raise bias and/or sensitivity concerns that will interfere with the performance of a group of students.” Rating Description
4 Strongly Agree 3 Agree 2 Disagree 1 Strongly Disagree
Comments/Explanation
Please provide specific comments to explain your evaluation. Your comments should address your particular bias/sensitivity concerns, the subgroup(s) that you feel may be impacted, and the extent of the potential impact within the subgroup(s)
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students”
Grade Level:
August 2007 Science Bias & Sensitivity Review 37
My sister met Kyle at a high school dance and nine months later I was an uncle.
When dad brought the broken-down mare home from the livestock auction, we all thought he had thrown his money away. Now there’s a handsome colt in the next stall, so I guess the old horse was a good deal after all.
Bias-Sensitivity Examples and Non-Examples
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 38
What’s the Point?
“Topics to Avoid: Sex/Sexuality”
Not okay because thepassage alludes to
extra-marital sex betweenhigh school students
Okay because our generalpublic standard generally
allows for the notion ofanimal parents and babies
My sister met Kyle at a high school dance and nine months later I was an uncle.
When dad brought the broken-down mare home from the livestock auction, we all thought he had thrown his money away. Now there’s a handsome colt in the next stall, so I guess the old horse was a good deal after all.
Bias-Sensitivity Examples and Non-Examples
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 39
The Christmas Tree Fern is a common sight in the New England woodlands.
Christmas is the most wonderful time of the year.
Bias-Sensitivity Examples and Non-Examples
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 40
What’s the Point?
“Topics Which May or May Not Be Acceptable: Holidays, Religion”
Okay because “Christmas” refers to the name of the
plant; Not a specific reference to religion
Not okay because thereference is based on the
writer’s cultural and religiousvalues (not shared by all)
The Christmas Tree Fern is a common sight in the New England woodlands.
Christmas is the most wonderful time of the year.
Bias-Sensitivity Examples and Non-Examples
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 41
Once, I got my head stuck in our porch railing. If that man with a cell phone hadn’t stopped and called the fire department, I’d probably still be there. Cell phones can save lives.
Everybody at school has a cell phone. Once, during an English test, somebody actually got a call and in a regular speaking voice, said, “Sorry, I can’t talk now because I’m taking an English test.”
Bias-Sensitivity Examples and Non-Examples
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 42
What’s the Point?
“Avoid ECONOMIC, regional, CULTURAL or gender bias in questions, prompts, identified situations…”
Okay because the writer’sculture and economic status
are not factors in the passage
Not okay because the passageassumes a cultural and
economic status that won’tbe shared by all test-takers
Once, I got my head stuck in our porch railing. If that man with a cell phone hadn’t stopped and called the fire department, I’d probably still be there. Cell phones can save lives.
Everybody at school has a cell phone. Once, during an English test, somebody actually got a call and in a regular speaking voice, said, “Sorry, I can’t talk now because I’m taking an English test.”
Bias-Sensitivity Examples and Non-Examples
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 43
It was ten years after Uncle Ned’s death when we found his old sea trunk in the attic.
Mary cried and cried when she was told Uncle Ned had died.
Bias-Sensitivity Examples and Non-Examples
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 44
What’s the Point?
“Topics Which May or May Not Be Acceptable: Death”
Okay because the deathreference is used to set the
passage’s time and place but is not a key plot feature
Not okay because death isa key plot feature; deals
with emotions that may beupsetting to some test-takers
It was ten years after Uncle Ned’s death when we found his old sea trunk in the attic.
Mary cried and cried when she was told Uncle Ned had died.
Bias-Sensitivity Examples and Non-Examples
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 45
Pa and Uncle Jimmy drove the final nails into the new barn roof while Ma and I laid out supper. For settlers on the Great Plains in the 1840’s, a barn raising was hard work, but also a time for celebration
Never give a girl a hammer ‘cause she’s liable to clobber herself or anybody in reach.
Bias-Sensitivity Examples and Non-Examples
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 46
What’s the Point?
“Avoid economic, regional, cultural or GENDER BIAS in questions, prompts, identified situations…”
Okay because the genderspecific roles in the passageare historically accurate and
do not, per se, represent bias
Not okay because thewriter’s perspective
represents a clear and unacceptable gender bias
Pa and Uncle J immy drove the final nails into the new barn roof while Ma and I laid out supper. For settlers on the Great Plains in the 1840’s, a barn raising was hard work, but also a time for celebration
Never give a girl a hammer ‘cause she’s liable to clobber herself or anybody in reach.
Bias-Sensitivity Examples and Non-Examples
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 47
“Never give a girl a hammer ‘cause she’s liable to clobber herself or anybody in reach.” That’s what I used to think until the day I spied Mary Jean McCubbins up in a tree, building the coolest tree fort I ever saw.
Bias-Sensitivity Examples and Non-Examples
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students.”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
August 2007 Science Bias & Sensitivity Review 48
What’s the Point?
Okay because the writeridentifies the gender bias inprior beliefs and attempts to correct the stereotype
“Avoid economic, regional, cultural or GENDER BIAS in questions, prompts, identified situations…”
“Never give a girl a hammer ‘cause she’s liable to clobber herself or anybody in reach.”That’s what I used to think until the day I spied Mary J ean McCubbins up in a tree, building the coolest tree fort I ever saw.
Bias-Sensitivity Examples and Non-Examples
“This passage does not raise bias and/or sensitivity concerns that would interfere with the performance of a group of students”
Rating Descriptions
4 Strongly Agree
3 Agree
2 Disagree
1 Strongly Disagree
top related