assessing speaking skills using human raters in an ......toefl ibt® test speaking section • test...
TRANSCRIPT
![Page 1: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/1.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL
are registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Assessing speaking skills
using human raters
in an Internet-based environment
Joanna Wrzesińska ETS Global
![Page 2: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/2.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
What we will be doing today
1. Introduction: Review the iBT Speaking section content
2. Discuss rater training
3. Explain and use the scoring guides and
benchmarks for Speaking to rate real test-taker
responses
![Page 3: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/3.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL
are registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Speaking
![Page 4: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/4.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
TOEFL iBT® Test
Speaking Section • Test takers respond to 6 tasks
• 2 independent tasks about familiar topics
– Responses based on personal experience
• 4 integrated tasks
– 2 Reading/Listening/Speaking tasks (1 academic, 1 campus-based)
– 2 Listening/Speaking tasks (1 academic, 1 campus-based)
– Responses based on what was read and heard
• Time: 20 minutes
![Page 5: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/5.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
TOEFL® Rater Training & Scoring
Apply Intensive ETS
Training Daily Calibration
Test
Monitoring by Scoring
Leader
Analyze Results
1 3 2 5 4
![Page 6: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/6.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Speaking Scores
• Holistic scoring
• Test takers are rated on:
– Delivery
– Language use
– Topic development
• Ratings are 0 - 4
– Converted to a scaled score of 0 - 30
![Page 7: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/7.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Speaking Benchmarks
• What is a Benchmark?
• Listen to the 4 Benchmarks for Speaking Task
#1 - Independent Speaking
![Page 8: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/8.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
![Page 9: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/9.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Most test takers should be able to respond to this topic easily, although it is likely that
very few will limit their discussion to just “one way” their life will be different. Even test
takers with responses at the highest level may introduce more than one aspect of their
lives that they think will be different. These responses are still eligible for a 4 as long as
some detail and development is included in their discussion.
Lower level respondents may merely list a number of ways that their lives may be
changed without much development of any of them (“I will be married, have children, a
different job,” etc.); these types of responses will most likely not be eligible for a score of
4 for topic development.
Some candidates may misinterpret the prompt and talk about the world in general
(technology, etc.) without connecting it to their own experience, or misinterpret the prompt
in some other way (e.g., talking about ten years ago). Responses that misinterpret the
prompt generally will not be eligible for high marks. Consult your Scoring Leader
whenever you are unsure of the appropriate score when considering topic development.
Topic Notes
![Page 10: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/10.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
TOEFL iBT® Speaking Section
Benchmark speaking responses • Benchmarks – independent speaking
• Annotations provide narrative description of the level
• Each score represents a range of ability
– Level 4
– Level 3
– Level 2
– Level 1
![Page 11: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/11.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Now Try Rating the Independent Task
– Listen to each response
– Use the Independent Speaking Rubric
– Give a rating from 1 to 4; take notes on rationale (how it fits the descriptor)
– Discuss your rationale in groups or pairs
Test taker number 1
Test taker number 2
![Page 12: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/12.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Integrated Speaking Task • Task #4 - General/Specific
• A Reading/Listening/Speaking Academic Topic
• Consists of two sources: a short reading and a lecture, followed by a single question
– Test taker reads a paragraph (~100 words) about an academic topic
– Test taker hears an excerpt from a lecture illustrating the topic (~90 seconds)
– The test taker is asked to briefly combine and convey important information from the reading and lecture
• Note taking allowed
• Prep time: 30 seconds
• Response time: 60 seconds
![Page 13: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/13.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
![Page 14: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/14.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
![Page 15: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/15.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
![Page 16: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/16.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Key Points
• Relict behavior is a behavior that an animal displays that once served a
purpose (for survival) but no longer serves that purpose.
• In the past, the (North American) pronghorn’s speed was critical for its
survival because its predator, the lion, also ran very fast.
• Lions are now extinct (in North America), but the pronghorn still displays the
relict behavior of running extremely fast even though that is no longer
required for survival (as it is so much faster than its current predators,
bobcats and coyotes).
![Page 17: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/17.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL are
registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Use the Integrated Speaking Rubric to
match the characteristics of each response
to a score.
– Test Taker 1
– Test Taker 2
Now Try Rating the Integrated
Task
![Page 18: Assessing speaking skills using human raters in an ......TOEFL iBT® Test Speaking Section • Test takers respond to 6 tasks • 2 independent tasks about familiar topics –Responses](https://reader036.vdocuments.us/reader036/viewer/2022062414/5f1b241c533f824c22442a1a/html5/thumbnails/18.jpg)
Copyright © 2017 by Educational Testing Service. All rights reserved. ETS, the ETS logo, MEASURING the POWER of LEARNING., E-RATER, TOEFL, TOEFL IBT and PROPELL
are registered trademarks of Educational Testing Service (ETS) in the United States and other countries. 767906
Thank you!