cognitive approaches to the winograd schema challenge
TRANSCRIPT
Cognitive Approaches to the Winograd Schema Challenge:
Challenges and Complications
Jesse MuSupervised by Joshua Hartshorne and
Tobias Gerstenberg
A Mystery of Pronoun Resolution
Q: Who does “he” refer to?Pronoun resolution techniques fail!(Hobbs Algorithm, Centering Theory)
• A lost to B at tug-of-war although he was weak.• A didn’t beat B at tug-of-war because he was strong.etc…
Resolution encoded in meaning, not syntax
• A beat B at tug-of-war because he was strong.• A beat B at tug-of-war because he was weak.
Why Winograd Schema?
Empirical Evaluations
1. Troy (Novelty)2. College (Learning)
Empirical Evaluations
1. Troy (Novelty)2. College (Learning)
resolution = 0.913
strengthL = 0.347
strengthR = 0.802
resolution = 0.043
strengthL = 0.872
strengthR = 0.252
1. Christopher beat Nicholas at tug-of-war, because he is strong.2. Brian didn't beat Gabriel at tug-of-war, because he is strong.3. Andrew beat Joseph at tug-of-war, because he is weak.4. Adam didn't beat Jack at tug-of-war, because he is weak.5. Christian lost to Austin at tug-of-war, because he is strong.6. Daniel didn't lose to Seth at tug-of-war, because he is strong.7. Dylan lost to Ethan at tug-of-war, because he is weak.8. Richard didn't lose to Carlos at tug-of-war, because he is weak.
9. Daniel beat William at tug-of-war, although he is strong.10. Isaiah didn't beat Luis at tug-of-war, although he is strong.11. Brandon beat Ryan at tug-of-war, although he is weak.12. Marcus didn't beat Charles at tug-of-war, although he is weak.13. Benjamin lost to Noah at tug-of-war, although he is strong.14. Patrick didn't lose to Bryan at tug-of-war, although he is strong.15. Samuel lost to Robert at tug-of-war, although he is weak.16. Cole didn't lose to Jackson at tug-of-war, although he is weak.
1. A beat B at tug-of-war, because he is strong.2. A didn't beat B at tug-of-war, because he is strong.3. A beat B at tug-of-war, because he is weak.4. A didn't beat B at tug-of-war, because he is weak.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.
9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.
Nontroy Troy
5. Christian lost to Austin at tug-of-war,because he is strong.
1. A beat B at tug-of-war, because he is strong.2. A didn't beat B at tug-of-war, because he is strong.3. A beat B at tug-of-war, because he is weak.4. A didn't beat B at tug-of-war, because he is weak.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.
9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.
Nontroy Troy
4. Adam didn’t beat Jack at tug-of-war,because he is weak.
Pronoun Resolution (Because)Non-troy Troy
Strength Judgments (Because)Non-troy Troy
??Pronoun Resolution (Because)
Non-troy Troy
Strength Judgments (Because)Non-troy Troy
??
A Tale of Two Interpretations11. Brandon beat Ryan at tug-of-war,
although he is weak.
He = Brandon?He = Ryan?
Although [he=Brandon] is weak, Ryan is even weaker.
A Tale of Two Interpretations11. Brandon beat Ryan at tug-of-war,
although he is weak.
Although [he=Ryan] isweak, Brandon is even weaker.
Although [he=Brandon] is weak, the God of Weaklings helped him win.
Nontroy Troy
11. Brandon beat Ryan at tug-of-war,although he is weak.
r = 0.958
r = -0.985
1. A beat B at tug-of-war, because he is strong.2. A didn't beat B at tug-of-war, because he is strong.3. A beat B at tug-of-war, because he is weak.4. A didn't beat B at tug-of-war, because he is weak.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.
9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.
r = 0.958
r = -0.985
Empirical Evaluations
1. Troy (Novelty)2. College (Learning)
Willowbrook College
Pinehurst University
1. A beat B at tug-of-war, because he is strong.2. A didn't beat B at tug-of-war, because he is strong.3. A beat B at tug-of-war, because he is weak.4. A didn't beat B at tug-of-war, because he is weak.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.
Willowbrook College
Pinehurst University
17. A beat B at tug-of-war,because he is from Willowbrook College
??
1. A beat B at tug-of-war, because he is strong.2. A didn't beat B at tug-of-war, because he is strong.3. A beat B at tug-of-war, because he is weak.4. A didn't beat B at tug-of-war, because he is weak.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.
17. A beat B at tug-of-war, because he is from Willowbrook College.18. A didn't beat B at tug-of-war, because he is Willowbrook College.3. A beat B at tug-of-war, because he is weak.4. A didn't beat B at tug-of-war, because he is weak.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.
17. A beat B at tug-of-war, because he is from Willowbrook College.18. A didn't beat B at tug-of-war, because he is from Willowbrook College.19. A beat B at tug-of-war, because he is from Pinehurst University.20. A didn't beat B at tug-of-war, because he is from Pinehurst University.5. A lost to B at tug-of-war, because he is strong.6. A didn't lose to B at tug-of-war, because he is strong.7. A lost to B at tug-of-war, because he is weak.8. A didn't lose to B at tug-of-war, because he is weak.9. A beat B at tug-of-war, although he is strong.10. A didn't beat B at tug-of-war, although he is strong.11. A beat B at tug-of-war, although he is weak.12. A didn't beat B at tug-of-war, although he is weak.13. A lost to B at tug-of-war, although he is strong.14. A didn't lose to B at tug-of-war, although he is strong.15. A lost to B at tug-of-war, although he is weak.16. A didn't lose to B at tug-of-war, although he is weak.
“College” Questions17. A beat B at tug-of-war, because he is from Willowbrook College.18. A didn't beat B at tug-of-war, because he is from Willowbrook College.19. A beat B at tug-of-war, because he is from Pinehurst University.20. A didn't beat B at tug-of-war, because he is from Pinehurst University.21. A lost to B at tug-of-war, because he is from Willowbrook College.22. A didn't lose to B at tug-of-war, because he is from Willowbrook College.23. A lost to B at tug-of-war, because he is from Pinehurst University.24. A didn't lose to B at tug-of-war, because he is from Pinehurst University.25. A beat B at tug-of-war, although he is from Willowbrook College.26. A didn't beat B at tug-of-war, although he is from Willowbrook College.27. A beat B at tug-of-war, although he is from Pinehurst University.28. A didn't beat B at tug-of-war, although he is from Pinehurst University.29. A lost to B at tug-of-war, although he is from Willowbrook College.30. A didn't lose to B at tug-of-war, although he is from Willowbrook College.31. A lost to B at tug-of-war, although he is from Pinehurst University.32. A didn't lose to B at tug-of-war, although he is from Pinehurst University.
1-16: Control17-32: CollegeProcedure
½ Control + ¼ College
Diagnostic Message
½ Control + ¾ College
Order Effects for Control Questions
1. Christopher beat Nicholas at tug-of-war, because he is strong.
Collapse into Control/Pre-diagnostic/Post-diagnostic1. A beat B at tug-of-war, because he is strong/from Willowbrook College.2. A didn't beat B at tug-of-war, because he is strong/from Willowbrook College.3. A beat B at tug-of-war, because he is weak/from Pinehurst University.4. A didn't beat B at tug-of-war, because he is weak/from Pinehurst University.5. A lost to B at tug-of-war, because he is strong/from Willowbrook College.6. A didn't lose to B at tug-of-war, because he is strong/from Willowbrook College.7. A lost to B at tug-of-war, because he is weak/from Pinehurst University.8. A didn't lose to B at tug-of-war, because he is weak/from Pinehurst University.9. A beat B at tug-of-war, although he is strong/from Willowbrook College.10. A didn't beat B at tug-of-war, although he is strong/from Willowbrook College.11. A beat B at tug-of-war, although he is weak/from Pinehurst University.12. A didn't beat B at tug-of-war, although he is weak/from Pinehurst University.13. A lost to B at tug-of-war, although he is strong/from Willowbrook College.14. A didn't lose to B at tug-of-war, although he is strong/from Willowbrook College.15. A lost to B at tug-of-war, although he is weak/from Pinehurst University.16. A didn't lose to B at tug-of-war, although he is weak/from Pinehurst University.
1. A beat B at tug-of-war,because he is strong/from
Willowbrook College.
(Left) (Right)
(hypothetical)
Pronoun resolution
1. A beat B at tug-of-war,because he is strong/from
Willowbrook College.
(Left) (Right)
(hypothetical subject bias)
Pronoun resolution
1. A beat B at tug-of-war,because he is strong/from
Willowbrook College.
(Left) (Right)
(results)
Pronoun resolution
Collapse into Control/Pre-diagnostic/Post-diagnostic1. A beat B at tug-of-war, because he is strong/from Willowbrook College.2. A didn't beat B at tug-of-war, because he is strong/from Willowbrook College.3. A beat B at tug-of-war, because he is weak/from Pinehurst University.4. A didn't beat B at tug-of-war, because he is weak/from Pinehurst University.5. A lost to B at tug-of-war, because he is strong/from Willowbrook College.6. A didn't lose to B at tug-of-war, because he is strong/from Willowbrook College.7. A lost to B at tug-of-war, because he is weak/from Pinehurst University.8. A didn't lose to B at tug-of-war, because he is weak/from Pinehurst University.9. A beat B at tug-of-war, although he is strong/from Willowbrook College.10. A didn't beat B at tug-of-war, although he is strong/from Willowbrook College.11. A beat B at tug-of-war, although he is weak/from Pinehurst University.12. A didn't beat B at tug-of-war, although he is weak/from Pinehurst University.13. A lost to B at tug-of-war, although he is strong/from Willowbrook College.14. A didn't lose to B at tug-of-war, although he is strong/from Willowbrook College.15. A lost to B at tug-of-war, although he is weak/from Pinehurst University.16. A didn't lose to B at tug-of-war, although he is weak/from Pinehurst University.
5. Christian lost to Austin at tug-of-war, because he is strong/from
Willowbrook College.
(Left) (Right)
(hypothetical)
Pronoun resolution
5. Christian lost to Austin at tug-of-war, because he is strong/from
Willowbrook College.
(Left) (Right)
(hypothetical subject bias)
Pronoun resolution
5. Christian lost to Austin at tug-of-war, because he is strong/from
Willowbrook College.
(Left) (Right)
(results)
Pronoun resolution
1. Pre-diagnostic subject bias
2. Post-diagnostic moves in direction of control resolution
1. A beat B at tug-of-war,because he is strong/from
Willowbrook College.
(Weak) (Strong)
(hypothetical)
StrengthL StrengthR
(Weak) (Strong)
1. A beat B at tug-of-war,
1. A beat B at tug-of-war,because he is strong/from
Willowbrook College.
(Weak) (Strong)
(hypothetical)
StrengthL StrengthR
(Weak) (Strong)
5. Christian lost to Austin at tug-of-war, because he is strong/from
Willowbrook College.
(Weak) (Strong)
(hypothetical)
StrengthL StrengthR
(Weak) (Strong)
5. Christian lost to Austin at tug-of-war, because he is strong/from
Willowbrook College.
(Weak) (Strong)
(hypothetical)
StrengthL StrengthR
(Weak) (Strong)
r = 0.323
Pre/Control Correlation
Post/Control Correlation
r = 0.990
Comparison
Empirical Evaluations
1. Troy (Novelty)2. College (Learning)
Empirical Evaluations
1. Troy (Novelty)2. College (Learning)
Cognitive Approaches to the Winograd Schema Challenge:
Challenges and Complications
Jesse MuSupervised by Joshua Hartshorne and
Tobias Gerstenberg