laura martin edutl 7645: test construction project the...
TRANSCRIPT
1
Laura Martin
EDUTL 7645:
Test Construction Project
The Ohio State University
2
Table of Contents
Statement of the Problem.............................................................................................................. 1
Construct ...................................................................................................................................... 3
Content ......................................................................................................................................... 3
Content-Based Required Knowledge ................................................................................. 5
Language-Based Required Knowledge .............................................................................. 6
Operations ........................................................................................................................ 6
Types of Texts Produced ................................................................................................... 6
Addressees of Texts .......................................................................................................... 6
Length of Texts ................................................................................................................. 6
Topics ............................................................................................................................... 7
Readability........................................................................................................................ 7
Structural Range ............................................................................................................... 7
Vocabulary Range............................................................................................................. 7
Dialect/Accent/Style ......................................................................................................... 7
Speed of Processing .......................................................................................................... 7
Test Structure, Timing, Medium, and Techniques ........................................................................ 8
Test Structure.................................................................................................................... 8
Number of Items ............................................................................................................... 8
Number of Passages .......................................................................................................... 8
Timing .............................................................................................................................. 8
Medium/Channel .............................................................................................................. 9
Techniques........................................................................................................................ 0
Criterial Levels of Performance .................................................................................................... 9
Scoring Procedures ....................................................................................................................... 10
Opinion Paragraph Scoring Rubric .................................................................................... 11
Compare/Contrast Essay Scoring Rubric ........................................................................... 12
Validity of the Test Administered ................................................................................................. 13
Reliability of the Test Administered ............................................................................................. 13
Practicality of the Test Administered ............................................................................................ 14
Listening Script................................................................................................................. 15
Rationale ...................................................................................................................................... 17
References .................................................................................................................................... 21
1
Class: English as a Second Language (ESL) Social Studies Grade: 4th
Level: Ohio LEP Intermediate Level / TESOL Level 3 (Developing)
Topics: The U.S. Constitution; Comparing & Contrasting Vocabulary and Skills
STATEMENT OF THE PROBLEM
English Language Learners (ELLs) have the unique task of developing both language and
content-area skills simultaneously. It is essential that as ESL classes assist ELLs with language
development, they also promote grade-appropriate content understanding. In the state of Ohio,
learning standards are determined by the Ohio Department of Education, which has modified the
Ohio standards to fit the Common Core State Standards. The standards for Social Studies are set
forth in Ohio’s New Learning Standards: K-12 Social Studies (“Ohio’s New,” 2012). English
language standards for all students are described in the Common Core State Standards for
English Language Arts. Specific standards for ELLs are stated in both the Ohio English
Proficiency Standards for Limited English Proficient Students and the TESOL PreK-12 English
Language Proficiency Standards (“National Governors,” 2010b; “Ohio English,” 2012;
“TESOL”, 2006).
According to the Common Core State Standards, “teachers should recognize that it is
possible to achieve the standards… without manifesting native-like control of conventions and
vocabulary” (“National Governors,” 2010a, para. 2). Thus, because the fourth grade standards
for Social Studies in Ohio dictate that students must acquire understanding of the U.S.
Constitution, students in fourth-grade ESL Social Studies classrooms must also meet this
standard. By aligning learning objectives in this way, students are better prepared for the TLU
domain—a mainstream content-area classroom.
In this ESL Social Studies class, students will spend approximately three weeks on a unit
about the U.S. Constitution. They then must take a progress achievement test to: (a) demonstrate
what they have learned in this thematic unit and, (b) provide the teacher with feedback about
learner understanding and continued areas for necessary improvement.
Regarding content-area knowledge, this progress achievement test will measure student
understanding of key facts related to the U.S. Constitution. Regarding language development,
this test will measure learners’ abilities to analyze content knowledge, write passages that
compare and contrast branches of government, and write an opinion paragraph about First
Amendment rights. It will also measure the students’ abilities to listen and understand short
monologues in English related to the Social Studies curriculum.
Although this is not a high-stakes test, the results must be sufficiently detailed and
accurate so as to give both the teacher and learners an accurate picture of the students’ current
levels of understanding and how much progress was achieved as a result of this unit. These
results will provide the teacher with essential information that will likely result in positive
washback, as it can help her tailor the content of future lessons.
This test is limited to a 50-minute time constraint, as each ESL Social Studies class meets
for 55 minutes daily (and five minutes should be allotted for both routine classroom procedures
and the distribution/collection of exams). Although most students will complete the exam on a
2
pre-determined date, the test must be arranged in such a way that any students who happen to be
ill/absent that day can take the exam during a remedial period at a later date. Furthermore, as
much as possible, the test must be free of cultural bias, as the students in this class come from a
wide range of countries (e.g. Saudi Arabia, Somalia, El Salvador, Cuba, Vietnam, and China).
This unit and its corresponding progress achievement test must address the following
standards:
Ohio English Language Proficiency (ELP) Standards
Intermediate Level
Listening Standard 1.4 Comprehend the meaning of academic and/or specialized vocabulary
when spoken
Reading Standard 3.2 Identify the meaning of written vocabulary
Writing Standard 4.1 Write for varied purposes and audiences, with appropriate tone and voice,
using various media
Writing Standard 4.2 Write using a range of vocabulary, sentence structures, and verb tenses
TESOL PreK-12 English Language Proficiency Standards
Standard 2: English language learners communicate information, ideas, and concepts necessary
for academic success in the area of language arts.
Standard 5: English language learners communicate information, ideas, and concepts necessary
for academic success in the area of social studies.
Common Core State Standards for English Language Arts
Writing: Grade 4
W4.1: Write opinion pieces on topics or texts, supporting a point of view with reasons and
information.
W4.2: Write informative/explanatory texts to examine a topic and convey ideas and information
clearly.
Ohio’s New Learning Standards for K-12 Social Studies
Within the “Government Strand” of 4th-grade standards, students need to know:
Content Statement 19: The U.S. Constitution establishes a system of limited government and
protects citizens’ rights; five of these rights are addressed in the First Amendment.
Content Statement 20: A constitution is a written plan for government. Democratic
constitutions provide the framework for government in Ohio and the United States.
Content Statement 21: The Ohio Constitution and the U.S. Constitution separate the major
responsibilities of government among three branches.
Note: This unit is only designed to address the U.S. Constitution. Future lessons will draw on the
knowledge acquired in this unit to make connections between the U.S. and Ohio Constitutions.
3
CONSTRUCT
In this ESL Social Studies class, knowledge of the U.S. Constitution is defined as an
overall understanding of the basic history, purposes, and organization of the Constitution, and the
framework of government it provides for the United States.
English language ability in this context is defined as an ability to read, listen, write, and
speak about a variety of content-related topics. The manifestation of this is demonstrated through
the following abilities: (a) recognizing, understanding, and producing related key vocabulary; (b)
writing short sentences about content-specific information; (c) comparing and contrasting aspects
of this theme; and (d) sharing through writing an opinion about an issue related to the
Constitution.
CONTENT
Content-Based Required Knowledge
Historical Background of the Constitution
Americans formed a new national government under the Articles of Confederation in
1777. However, the Articles of Confederation were ineffective for running a new
nation.
Because of the ineffectiveness of the Articles of Confederation, early colonists held a
Constitutional Convention in 1787. Representatives from twelve states attended.
At the Constitutional Convention, colonists wrote the Constitution. It became an
official law in the United States on June 21, 1788 after 2/3 of the states voted for it.
Structure of the Constitution
The Constitution of the United States allowed for changes to be made to it when
necessary.
The first ten amendments are called the Bill of Rights.
The First Amendment to the Constitution provides the freedoms of religion, speech,
press, petition, and assembly.
o Freedom of Religion means that a person may choose his/her religion and how
or if he/she practices it.
o Freedom of Speech means that the government cannot prevent people from
saying what they want to or from expressing their beliefs, thoughts, and
opinions.
o Freedom of Press means that the government cannot control the news that is
reported to the public.
4
o Freedom of Assembly means that U.S. Americans can come together and join
groups without fear that the government will stop them.
o Freedom of Petition means that U.S. Americans can request changes in the
law or government.
The Constitution is organized into three major sections: (1) the Preamble; (2) the
Articles; and (3) the Amendments.
The Preamble is the introduction to the Constitution. It describes the purpose of the
Constitution.
There are seven articles in the Constitution. These articles establish how the
government will be organized and how the Constitution can be changed.
Amendments are changes made to the Constitution.
Collectively, the first 10 amendments are called the Bill of Rights.
Key Facts about the Constitution:
A constitution is a written document that describes the way a government is
organized and how its power is distributed.
The U.S. Constitution provides a framework for government, describing what it may
and may not do.
The Constitution protects the rights of citizens.
The Constitution limits the power of government.
The Constitution promotes the common good of all people.
Citizens of the United States elect government officials and representatives and make
decisions by voting. Thus, the U.S. people/citizens are the source of the government’s
authority and power.
Three Branches of Government:
The Constitution divides the power in the government between three different
branches so that one branch of government does not have too much power or control.
This system is called “separation of powers.”
The Legislative Branch makes laws. Officials in this branch of government can create
a proposal for a new bill, approve a bill, and pass it on to the executive branch.
o A law is a rule enforced by government that citizens must follow.
o The Senate and the House of Representatives are part of the Legislative Branch.
5
The Executive Branch carries out and enforces the laws. This branch has the power to
sign a bill and make it a law.
o The President of the United States is the head of the Executive Branch. He
gets help from the Vice President. The Executive Branch also includes the
Presidential Cabinet, government agencies, and the armed forces.
The Judicial Branch decides on the meaning of laws and applies the laws. This branch
has the power to determine whether or not a law follows the Constitution.
o The Supreme Court is part of the Judicial Branch.
All branches work together to uphold the Constitution. They all are based in
Washington, D.C., and they all carry out the responsibilities that are outlined for them
in the Constitution.
Language-Based Required Knowledge
Compare/Contrast Key Vocabulary
Contrast: to tell how two things are different
Compare: to tell how two things are alike or similar
Vocabulary to Describe Similarities
in the same way
both …and…
in addition
likewise
similarly
too
have in common
the same as
Vocabulary to Describe Differences
different from
compared with
unlike
but
conversely
in contrast
however
although
Key Question/Answer Formations
How are _________ and _________ alike?
o _________ and _________ are alike because….
How is _________ different from _________?
o _________ is different from _________ in its/their ….
How is _________ similar to _________?
o _________ is similar to _________ in its/their…..
What is a similarity between _________ and _________?
o One similarity between _________ and _________ is…
What is the difference between _________ and _________?
o One difference between_________ and _________ is….
6
Graphic Organizers
Venn diagrams are useful tools to show comparisons and contrasts graphically. To
complete a Venn diagram, draw two circles on the page. Write differences in the parts
of the circles that do not overlap and write similarities in the overlapping portion of
the circle.
Operations
Complete definitions (through gap-filling) of key vocabulary and concepts
Compare and contrast two of the three branches of government
State and support one’s opinion in writing about an aspect of the First Amendment
List the purposes of the U.S. Constitution
Identify and describe the major responsibilities of the branches of government
Organize information about the duties of the branches of government on a Venn diagram
Explain the significance of the first three words of the Constitution (“We the People…”)
Listen to a person describe his/her job and identify which branch of government he/she works for
Write short answers to content-specific questions about key ideas
Use a chart to sort useful “compare/contrast” vocabulary words and phrases into two categories
Types of Texts Produced
Gap-filling
Short answer responses
Opinion paragraph
Graphic organizer
2-paragraph compare/contrast essay
Addressees of Texts
Students are writing with awareness that the individual reading their text will be their
elementary ESL teacher.
The teacher, who is familiar with her students’ proficiency levels, will be the author of all
components of the test. In this way, she can ensure student comprehension is not affected
adversely by the test’s readability level or the types of tasks required of students.
Length of Texts
Gap-filling: 1-3 words per gap
Short answer: 1-2 sentences per question (sentence frames included to provide modification)
Opinion paragraph: 3-6 sentences
Compare/contrast essay: 2 paragraphs (6-12 sentences)
7
Topics
The U.S. Constitution (brief history of, structure, purpose)
Branches of government as outlined by the U.S. Constitution
Helpful vocabulary and techniques for compare/contrast writing
Readability
Texts throughout the exam are written at a level that is at or near a 4th-grade readability
level according to the Flesch-Kincaid Readability Score.
All parts of the test are written using Comic Sans (at least 12 pt. font), as this is the font
which most closely replicates print and is typically easiest for beginning ESL students to
understand (Epstein & Ormiston, 2007).
Structural Range
Structures included in the texts are high-frequency structures that will not impede
understanding for students of an intermediate level.
Vocabulary Range
Vocabulary utilized on the test consists of mostly high-frequency words. Some
exceptions are made for content-specific words that are related to this unit.
Dialect/Accent/Style
For the listening portion of the test, students listen to a pre-recorded track of three native
speakers of Standard American English.
On the writing portion of the exam, they will be expected to write short answers and
paragraphs in a simplified academic voice in a manner that is in keeping with the benchmarks
described in the Ohio English Language Proficiency Standards for intermediate-level learners.
Speed of Processing
The listening passages are reduced to a pace that is slower than a typical native speaker’s
speed of speech (around 70-80 words per minute).
8
TEST STRUCTURE, TIMING, MEDIUM, AND TECHNIQUES
Test Structure
Day 1
Part One: Listening Practice about Branches of Government
Part Two: Gap-Filling about Constitution/Key Terms
Part Three: Short Answer about the Constitution’s Purposes and Related Key Facts (sentence
frames provided for modification)
Part Four: Compare/Contrast Vocabulary Graphic Organizer Word Sort (word bank included)
Part Five: Opinion Paragraph about First Amendment Rights (visual support provided via
pictures of each right in order to assist student comprehension)
Day 2
Part One: Compare/Contrast Venn Diagram about Two Branches of Government (word bank
provided for extra support)
Part Two: Compare/Contrast Essay about Two Branches of Government
Number of Items
Day 1
Part One: Three questions; write which branch of government each speaker represents
Part Two: Seven gap-filling questions related to key vocabulary/concepts
Part Three: Three short answer questions about key concepts
Part Four: Ten words/phrases to “sort” into one of two columns
Part Five: One paragraph (3-6 sentences) sharing student’s opinion about First Amendment
Day 2
Part One: Test takers choose two of the three branches of government to compare/contrast in a
graphic organizer (Venn diagram)
Part Two: Two paragraphs about the similarities and differences between two branches of
government
Number of Passages
N/A
Timing
Day 1
Part One: 7 minutes
Parts Two, Three, Four, and Five: 43 minutes
Day 2
Parts One and Two: 50 minutes
9
Medium/Channel
Hand-written
Techniques
Technique Used Skills/Sub-skills Measured
Listening & Fill-in-the-Blank Response Listening; Recall duties of the Legislative,
Judicial, and Executive Branches
Short Answer (with sentence frames) Understanding of key concepts related to U.S.
Constitution
Gap-Filling Understanding key concepts related to U.S.
Constitution
Paragraph Writing Abilities to: (a) write for varied purposes; (b)
share opinion via writing; (c) compare/contrast
two main ideas
Graphic Organizer Word Sort Recognition of key compare/contrast
vocabulary words/phrases; Ability to
differentiate between similarities/differences
and various words used to talk about each
Venn Diagram Knowledge of how to sort items in a Venn
diagram; Ability to analyze two topics and
compare/contrast them; Understanding the
similarities/differences between the 3 branches
of government
CRITERIAL LEVELS OF PERFORMANCE
This test is criterion-referenced. The total number of points possible is 70. In order to
achieve an acceptable level, a student should obtain a score of at least 80% or a raw score of 56.
Students who score below this level will need to have a one-on-one meeting with the teacher to
address problem areas and set goals for future improvement.
10
SCORING PROCEDURES
On day one of the test, the total score possible is 50 points; on day two of the test, the
total score possible is 20 points. Scores from both days of the test will be added together to make
a total test value of 70 points.
The teacher will score all parts of the exam and will distribute results to students within
one week of the original test date. Students will receive their original exam papers and the
rubrics used to score both pieces of paragraph/essay writing.
Day 1
Each question in parts one (listening), two (gap-filling), and three (short answer) is worth
2 points. No partial credit will be given. Answers are based on content only and spelling does not
count against students unless meaning is impeded.
On part four of the exam (similarities/differences vocabulary chart), students get 1 point
for each word/phrase they correctly categorize in the graphic organizer.
Part five of the exam (opinion paragraph) is worth 10 points and is scored using the
analytic rubric on page 11.
Day 2
Part one (Venn diagram) is worth a total of 8 points. Students receive one point for listing
each of the names of the branches of government at the top of the diagram (for a total of 2
points). They receive one point for listing each of the two differences for both branches of
government (for a total of 4 points). Lastly, they receive one point for each of two similarities
between the branches of government (for a total of 2 points).
Part two (compare/contrast essay) is worth 12 points and is scored using the analytic
rubric on page 12.
11
Student: ____________________
Opinion Paragraph: First Amendment Rights
Scoring Rubric
0 1 2
Statement of
Opinion
Student does not give his/her opinion
about the most important 1st
Amendment right
Student gives his/her opinion about
the most important 1st Amendment
right
X
Understanding
of 1st
Amendment
Rights
It is clear that the student does not
understand the meaning of the 1st
Amendment right selected; or student
has not written about any of the 1st
Amendment rights
Student demonstrates that he/she has
at least a basic understanding of the
meaning of the 1st Amendment right
selected
X
Supporting
Reason #1
Student does not support his/her
opinion with a 1st reason
Student gives a 1st reason to support
his/her opinion that is unclear, illogical
and/or unrelated to the 1st Amendment
right selected
Student gives a 1st reason to support
his/her opinion that is mostly clear,
logical, and related to the 1st
Amendment right selected
Supporting
Reason #2
Student does not support his/her
opinion with a 2nd reason
Student gives a 2nd reason to support
his/her opinion that is unclear, illogical
and/or unrelated to the 1st Amendment
right selected
Student gives a 2nd reason to support
his/her opinion that is mostly clear,
logical, and related to the 1st
Amendment right selected
Supporting
Reason #3
Student does not support his/her
opinion with a 3rd reason
Student gives a 3rd reason to support
his/her opinion that is unclear, illogical
and/or unrelated to the 1st Amendment
right selected
Student gives a 3rd reason to support
his/her opinion that is mostly clear,
logical, and related to the 1st
Amendment right selected
Language Use
Student writing sample contains an
abundance of errors;
Meaning is incomprehensible
Student does not respond in paragraph
form/does not use complete sentences;
Several errors in spelling/grammar;
More than one error impedes meaning
Student responds in paragraph form
with complete sentences;
Few errors in spelling and grammar;
Errors do not impede meaning
Total Score: ________ / 10
12
Student: ______________________________
Compare/Contrast Essay: Two Branches of Government
Scoring Rubric
0 1 2 3
Content:
Differences
Does not include any
differences
Includes 1
difference with 0
errors in content;
OR includes 2
differences with
content errors in
both
Attempts to include
2 differences but
has a content error
in 1 of them
Includes 2
differences;
No errors in content
Content:
Similarities
Does not include any
similarities
Includes 1 similarity
with 0 errors in
content; OR includes 2
similarities with
content errors in
both
Attempts to include
2 similarities but
has a content error
in 1 of them
Includes 2
similarities;
No errors in content
Organization
Is not written in
paragraph form;
Is entirely
unorganized and
illogical;
No use of
transitions
Is not written in
paragraph form;
OR is in prose but
does not distinguish
between paragraphs;
Illogical
organization;
Poor transitions
Includes 2
paragraphs but does
not organize content
according to
similarities and
differences;
Transitions are
lacking or somewhat
confusing
Includes 2 distinct
paragraphs, using
1for similarities and
1 for differences;
Makes good
transitions between
sentences and
paragraphs
Language
Use
Language is
incomprehensible;
Full of spelling and
grammar errors;
Almost all errors
impede meaning;
Vocabulary is
extremely limited
Language is
somewhat
incomprehensible;
Several
spelling/grammar
errors that impede
meaning; Vocabulary
is very limited
Language is mostly
clear and
comprehensible;
Several
spelling/grammar
errors but they do
not impede meaning;
Vocabulary is limited
Language is clear
and comprehensible;
Few
spelling/grammar
errors are made;
Errors do not
impede meaning;
Utilizes a wide range
of vocabulary
Total Score: _____ /12
13
VALIDITY OF THE TEST ADMINISTERED
For this classroom-based progress achievement test, a high content validity is critical.
Because the test designer is also the classroom teacher, she will be able to guarantee high content
validity by ensuring that all topics addressed on the exam have been adequately covered in class.
Additionally, content validity is guaranteed because all test items correlate directly to
information that is included in the test specifications (Hughes, 2010).
It is also important that this test has a high face validity. As this is an ESL Social Studies
class, the test must address issues of both language and content. For this exam, the test designer
is able to maintain high face validity through the incorporation of both elements throughout the
exam. By dividing the test into sub-parts (five on the first day; 2 on the second day) with specific
focuses, the face validity is enhanced, as it becomes apparent that both language and content are
being tested.
Another validity-related concern is that content-specific segments of the test are, in fact,
assessing content (and not language) and vice versa. In order to address this concern, the test
designer has included several modifications. For example, in the short answer section of the
exam, the test designer has included sentence frames for students. In this way, the student is
supplied with all of the language he/she needs to demonstrate his/her knowledge. Additionally,
in the opinion paragraph segment of the test, pictures have been included to give students a
visual representation of each First Amendment right. On day two of the test, students are once
again provided with the language they need to demonstrate their knowledge through the
inclusion of a word/idea bank. In this way, students can focus their attention more on aspects of
compare/contrast essay writing and less on producing content-specific language.
RELIABILITY OF THE TEST ADMINISTERED
One way the test designer has addressed the reliability of this test is by ensuring that students
are “familiar with [the] format and testing techniques” (Hughes, 2010, p. 47). Because this three-
week unit will occur in the middle of the school year and after students have had several other
exams from the same teacher, it is safe to assume that students are already very familiar with the
testing procedures included in this test (listening questions, gap-filling, short answer, graphic
organizers, essay writing, etc.). Furthermore, prior to the exam date, students will have
participated in a review period, in which the teacher prepares them for the test by discussing the
basic format of this exam.
Another manner in which the reliability of this exam has been addressed is through the
provision of multiple samples of assessment (Hughes, 2010). Over the course of three weeks,
students will be assessed daily through in- and out-of-class assignments. Thus, upon completion
of this exam, the teacher will be able to compare students’ scores with their performances on
other assessments.
In the event that a student is absent on the first day of the exam and has to make up the
test on a later date, the test designer has taken measures to guarantee reliability in the listening
section of the exam by ensuring that all students have access to the same information and
14
materials. To do this, she has created a pre-recorded listening track that follows the script on
page 16.
The test designer has also attempted to ensure reliability by providing “clear directions”
and “unambiguous items” (Hughes, 2010, pp. 46-47). In an effort to ensure that questions are
clear, the test designer has solicited feedback from other colleagues.
Lastly, reliability of this exam is enhanced through the provision of an objective set of
scoring criteria. The majority of the exam has one clear, objective answer. For the more
subjective writing segments of the exam, analytic rubrics have been created to ensure intra-rater
reliability (Hughes, 2010).
PRACTICALITY OF THE TEST ADMINISTERED
This test is quite practical, as it does not require extensive time or resources. The entire
test is paper-based, and although color copies of some pages of the exam are required (to ensure
clarity), the small number (18) of students taking the exam makes this feasible.
Aside from copies of the test, the only other required materials are a pre-recorded audio
track for the listening section and lined pieces of paper on which the students will complete their
opinion paragraphs. The pre-recorded listening track will be recorded as an .mp3 file by the
teacher and three of her coworkers (to imitate three different voices) and can be done easily with
the technology included on a basic laptop.
The two-day timeframe for this test is reasonable, as it covers a three-week unit of class
and addresses both language and content standards. Although it is preferable for the tests to be
conducted on back-to-back days, it is possible for the second day of the exam to be completed
several days after the administration of the first day’s test if necessary.
15
ESL Social Studies Test
The U.S. Constitution
Part 1: Listening
Listening Script
(To be read at a pace of about 70-80 words per minute)
Narrator (Teacher’s voice):
This is the Listening part of your test. I will now read the directions. Follow along as I
read them.
Directions: Listening to people talk about their jobs. You will hear each person speak two
times. After each person speaks, I will read the question in bold to the class. To answer
the question, write the correct branch of government on the line.
Are there any questions? (Pause)
Let’s begin.
We will now begin question number one.
Speaker #1 (Female Voice):
My name is Sonia. I went to law school at Harvard Law School. Now, I am part of the
U.S. Supreme Court. It is my responsibility to decide if a law follows the Constitution or
not.
(Pause. Then repeat.)
Narrator (Teacher’s Voice):
Question #1: Which branch of government does Sonia work for? (Pause)
We will now move on to question number two.
Speaker #2 (Male Voice):
My name is Joe. I am the Vice President of the United States. I work in Washington D.C.
with the President. It is my responsibility to help carry out and enforce the laws in our
country.
(Pause. Then repeat.)
16
Narrator (Teacher’s Voice):
Question #2: Which branch of government does Joe work for? (Pause)
We will now move on to question number three.
Speaker #3 (Male Voice):
Hello! My name is Jim. I am from Ohio. Now, I work in Washington D.C. in the House
of Representatives. It is my duty to help make and pass laws for our country. It is an
important job. I enjoy being part of Congress.
(Pause. Then repeat.)
Narrator (Teacher’s Voice):
Question #3: Which branch of government does Jim work for? (Pause)
This is the end of the listening part of the test. You may now turn to page two and
complete the rest of the test.
17
RATIONALE
With my only formal teaching experience being in the field of EFL, ESL classrooms are
still a relatively “new world” to me. However, the time I have spent recently as a substitute
teacher in ESL classrooms has allowed me to obtain a better picture of what this environment is
like. I used these recent experiences as a starting point for my test construction project.
The intended audience I chose for this project is based on my current understanding of a
typical environment in an ESL classroom in Ohio. I chose elementary-aged learners and a
content-based assessment because: (a) I could model my “imaginary” classroom environment
after classes I have observed recently; and (b) I felt that it would be a significant learning
experience for me, having little formal teaching experience with such young learners and
content-based lessons.
Once I decided on my classroom environment and learners, I was then faced with the task
of determining the content on which I wanted to base my assessment. In order to do this, I
explored the Ohio Department of Education website and read the current English Language
Proficiency standards and the content standards for Social Studies and English Language Arts.
After finding some connections between the Social Studies and English Language Proficiency
standards, I decided to design an assessment based on a thematic unit about the U.S. Constitution
that utilized language as a tool to demonstrate understanding and analysis. Based on the
standards that I selected, I created a list of required content knowledge for students to learn
related to both language use and Social Studies.
With my list of required content knowledge complete, I then began to design a test to
assess my students’ progress and their understanding of this content. I found this to be far more
difficult than I originally anticipated. Designing a content-specific test that assessed students on
their knowledge of Social Studies and their ability to use language to listen, read, and write about
it was a difficult task! I repeatedly designed test items that I later deleted or modified because
they tested too many skills simultaneously. For example, some items that I intended to use to
assess student comprehension of the content area were ineffective because they required high
levels of reading or writing. Naturally, in an ESL Social Studies class, language will overlap with
content. However, my challenge was to assess content knowledge without allowing language to
be an unnecessary hurdle in the demonstration of this knowledge. Likewise, I sought to assess
language skills in a way that was not adversely affected by a limited content knowledge.
This complicated process of designing appropriate test items was quite recursive. I
frequently wrote and rewrote items to fit my objectives and modified my test construct to align
with my test items. Ultimately, I chose to include several different types of assessment (listening,
gap-filling, short answer, essay, graphic organizer, etc.) in an effort to obtain multiple samples
across a wide spectrum and give students many “fresh starts.” I also weighted each segment of
the test similarly, so that no student was “punished” for making multiple errors on one portion of
the test (whether due to lack of understanding of that content or merely to difficulty with a
specific format of assessment). However, the “content-specific” questions were weighted more
heavily than the “language-specific” portions of the test. This was intentional, as the class is a
content-based class where language is used as a tool to learn and demonstrate knowledge of
content-specific information.
My rationale for the readability, structural range, vocabulary range, speed of processing,
and dialect/accent was rooted in knowledge I have obtained from my graduate coursework and
interactions I have had with other ESL teachers. From these two sources, I have learned that
18
when testing ELLs in content areas, it is beneficial to limit the structural and vocabulary range to
high-frequency language features and words and keep the readability at or below the student’s
grade level so that language does not impede understanding. I chose to use native English
speakers with a Standard American accent for the listening passages because this dialect is most
representative of the students’ TLU domain (a mainstream classroom in an Ohio public school).
Although the speed of processing is slower than that which would be necessary in the TLU
domain, I made this modification in an effort to accommodate the specific needs of English
Language Learners.
The organization of my test items was another aspect of the test design process that was
recursive for me. I carefully debated in which order to put the various parts of the test and settled
on the current order for several reasons.
The test begins with the listening segment for practicality purposes. Since the whole class
must complete this part of the exam simultaneously, it is important to conduct this section first so
that students can use the remaining class time to complete the other parts of the test at their own
pace without being interrupted.
I debated between using either the gap-filling segment or the compare/contrast word sort
activity for the second part of the test. Initially, I considered including the compare/contrast word
sort in order to motivate the students, as this segment of the exam would most likely be easier for
most students than the gap-filling questions. However, in the end, I chose to place the gap-filling
questions in “part two” of the test because I felt that it was important to the test’s face validity
for the first non-listening part of the assessment to be directly related to Social Studies content.
Since the gap-filling exercise is the easiest set of content-based questions, I thought this was an
appropriate way to begin the written portion of the exam.
The opinion paragraph is the last question on the first day of the exam because I wanted
to ensure student awareness of the amount of time they have left to spend on paragraph writing. I
feared that if I included this item in the beginning of the test, students would spend too much
time writing and not have sufficient time to complete other portions of the test.
Lastly, for the second day of this test, I chose to require both a graphic organizer and an
essay because I thought that the Venn diagram would serve as a beneficial pre-writing tool for
students to refer to while writing their essays. Since they will often be required to do pre-writing
in the form of graphic organizers, webbing, or outlines in the TLU domain, it is useful for
students to also do pre-writing on this ESL Social Studies exam.
Another obstacle I encountered in my test design process was in determining the best way
to create a valid listening section. In one of the earliest versions of my test, I included visual
support for ELLs by providing pictures of each individual speaking. However, because I did not
want the questions to be inauthentic, I used only real people who are current government
officials. I later realized that the inclusion of the full names and pictures of these people posed a
threat to my test’s validity because it added an element of background knowledge that could not
be controlled. Although it was relatively unlikely, it was possible that some students could
recognize the names or photos of the government officials. If this occurred, students could
answer the questions without listening to the pre-recorded track and my test would no longer be
assessing listening skills. To address this, I removed the pictures and used only the first names of
three government officials (Sonia Maria Sotomayor, Joe Biden, and Jim Butler).
After creating several initial drafts of my test, I sent it to a currently-practicing teacher
who teaches 4th-grade ESL students in Iowa for feedback. She confirmed that the readability
level, types of test items, and length of test were appropriate for this level of learner. However,
19
she also suggested that I improve my test by adding visual support for students and making the
exam more aesthetically pleasing.
In response to her feedback, I continued to revise my test. I added more “white space,”
changed the font, and added pictures to several parts of the exam. For example, I added pictures
to represent each of the First Amendment rights in the opinion paragraph question in an effort to
provide students with a visual reminder of the meaning of each right and allow them to focus
their efforts more on writing and less on content. Likewise, on the second day of the test, I added
pictures to represent each of the branches of government. The pictures included are ones that
students would have seen repeatedly in class sessions and on lesson materials. I also added an
“idea bank” in which I listed the content that students would most likely want to include in their
Venn diagrams and compare/contrast essays. My motivation for doing this was to provide
students with the content-specific language they need to complete the task successfully. Since I
tested content in earlier parts of the exam, I did not feel that including this “idea bank” would
detract from the validity of the test or my ability to assess students’ content knowledge.
In addition to the 4th-grade ESL teacher in Iowa, I also sent my test to a 4
th-grade teacher
in North Carolina who works with low-income students. After receiving her feedback, I was
faced with a new challenge. She recommended that I add word banks for the gap-filling portion
of my exam to make it easier and also suggested that I make the test open-book. She claimed that
if the test were open-book, the students could work on their “finding proof” skills (a focus of the
Common Core State Standards for English Language Arts). After receiving her feedback, my
challenge was not to determine how to incorporate her suggestions, but whether to incorporate
her suggestions.
After much consideration, I chose not to use these two suggestions. I felt that if I were to
add a word bank for the gap-filling portion of the exam, I would no longer get an accurate idea of
my student’s content knowledge because they might be able to use “test wiseness” to determine
correct answers (based on the forms of words, singular/plural, beginning consonant v. vowel,
etc.). Additionally, I was more interested in observing students’ ability to produce content-
specific terms than their ability to simply recognize them from a list. In my opinion,
incorporating a word bank into this portion of the test would compromise my objectives.
Likewise, if I allowed the students to take this test as an open-book exam, it would change the
skills and abilities being tested. I would no longer be testing content knowledge, but rather the
ability to read from a text and apply the information to the exam.
Although this individual’s suggestions would be useful if applied to a different context, I
did not feel that they were applicable to my objectives and the situation as described in my
“statement of the problem.”
Throughout the development of this test construction project, I have addressed TESOL
Standard 4.a (“Issues of Assessment for English Language Learners”), which states that it is
necessary for candidates to, “[d]emonstrate understanding of various assessment issues as they
affect ELLs, such as accountability, bias, special education testing, language proficiency, and
accommodations in formal testing situations” in several ways (“TESOL,” 2010, p. 56).
First of all, I have addressed performance indicator 4.a.1, which states that candidates
should “[d]emonstrate an understanding of the purposes of assessment as they relate to ELLs and
use the results appropriately” (“TESOL,” 2010, p. 58). Although there are many different types
of assessments, I have chosen to create a progress achievement test for this test construction
project. Naturally, the purposes of this type of test and the ways in which I, the teacher, can use
the results vary considerably from other forms of assessment (i.e. diagnostic, proficiency, and
20
high-stakes tests). As I mentioned in my “statement of the problem,” I would hope to use this
assessment to provide me with information about my students’ understanding of the required
content knowledge. Additionally, I would use the results to inform future lessons in my ESL
Social Studies classroom. If few students achieved the criterial level of performance, it would
indicate to me that it was necessary to readdress specific aspects of the content so that my
students fully comprehend the information and can achieve the standards listed in my objectives.
In the development of my progress achievement test, I have also addressed performance
indicator 4.a.2, which states that candidates must be“[k]nowledgeable about and able to use a
variety of assessment procedures for ELLs” (“TESOL,” 2010, p. 58). I have designed a valid,
reliable, and practical exam that utilizes a variety of assessment techniques (listening, gap-filling,
short answer, graphic organizers, paragraphs, essays, etc.). Additionally, as is stated in my
description of the test’s reliability (pp. 13-14), I would also use formative assessments
accumulated throughout the course of the three-week unit to assess my students’ comprehension
and abilities related the aforementioned standards. By comparing students’ formative and
summative assessment results, I would obtain a clearer understanding of their progress.
Similarly, my discussion of this test’s validity, reliability, and practicality (pp. 13-14)
demonstrate that I am able to meet performance indicator 4.a.3, which states that candidates must
“[d]emonstrate an understanding of key indicators of good assessment instruments” (“TESOL,”
2010, p. 59). Through the recursive test design process, I have interacted with these three critical
characteristics of an assessment tool by evaluating my test in relation to each area. Likewise, I
have modified my assessment repeatedly in an effort to increase its validity, reliability, and
practicality.
Lastly, I have added multiple accommodations for ELLs to my exam in order to address
TESOL performance indicators 4.a.4 (candidates“[d]emonstrate an understanding of the
advantages and limitations of assessments including accommodations for ELLs”) and 4.a.5
(candidates “[d]istinguish among ELLs’ language differences, giftedness, and special education
needs”) (“TESOL,” 2010, p. 59). For example, in order to refrain from linguistic bias and to
address the linguistic limitations of the ELLs in my classroom, I have added “sentence frames”
to the short answer segment of this test. By doing this, I provided students with a large portion of
the language they need in order to answer the question. Likewise, recognizing that it would be
difficult for ELLs to produce extensive content-related vocabulary and then sort it in a graphic
organizer, I have included a word bank for students to use as a starting point for this part of the
test. Lastly, I have incorporated several pictures on the test in an effort to provide
accommodation via visual support for ELLs. Because I am aware that pictures are often a source
of cultural bias, the graphics I chose to use on my exam are culturally-sensitive and are ones that
my students would already be familiar with and would have seen in the context of in-class
worksheets and/or lesson materials.
The completion of this test construction project has been an eye-opening experience for
me. As a former EFL teacher, I created a multitude of performance achievement tests for my
English Language Learners. However, it was not until I began this project that I truly considered
elements of validity, reliability, practicality, and bias when creating an exam. Constructing a
good assessment tool is not something that can or should be done quickly. In addition, as I
worked on this project, I learned that assessments should not merely demonstrate student
learning, but should also contribute to student learning. Creating an assessment tool that meets
these criteria and those addressed in state standards is a difficult task. However, it is one that is
essential if I desire positive washback from my assessment.
21
References
Epstein, R. & Ormiston, M. (2007). Tools and Tips for Using ELT Materials. Ann Arbor, MI:
University of Michigan Press.
Hughes, A. (2010). Testing for language teachers (2nd
ed.). Cambridge: Cambridge University
Press.
National Governors Association Center for Best Practices & Council of Chief State School
Officers. (2010). Application of Common Core State Standards for English language
learners. Retrieved from http://www.corestandards.org/assets/application-for-english-
learners.pdf
National Governors Association Center for Best Practices & Council of Chief State School
Officers. (2010). English language arts standards: Writing: Grade 4. National Governors
Association Center for Best Practices & Council of Chief State School Officers:
Washington, DC.
Ohio English Language Proficiency (ELP) Standards. (2012). In Ohio Department of Education
Learning supports. Retrieved from http://education.ohio.gov/GD/Templates/Pages/ODE/
ODEDetail.aspx?page=3&TopicRelationID=500&ContentID=6621&Content=133989
Ohio's New Learning Standards: K-12 Social Studies. (2012). In Ohio Department of Education:
Teaching. Retrieved from http://www.ode.state.oh.us/GD/Templates/Pages/ODE/ODE
Detail.aspx?page=3&TopicRelationID=1706&ContentID=126270&Content=137752.
TESOL International Association. (2006). PreK-12 English language proficiency standards.
TESOL International Association: Alexandria, VA.
TESOL International Association. (2010). Standards for the recognition of initial TESOL
programs in P-12 ESL teacher education. TESOL International Association: Alexandria, VA.