Upload File

commonsense benchmarks - university of washingtonmsap/acl2020...small commonsense benchmarks i hung...

  • Home
  • Documents
  • Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The
65
Commonsense benchmarks Or how to measure that your model is actually doing some commonsense reasoning

Upload: others

Post on 25-Mar-2021

7 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Commonsense benchmarks

Or how to measure that your model is actually doing some commonsense reasoning

Page 2: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

How do you know that a model is doing commonsense reasoning?

Page 3: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

How do you know that a model is doing commonsense reasoning?

Unsupervised:• Observe behavior,• Probe representations,• etc.

Page 4: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

How do you know that a model is doing commonsense reasoning?

Unsupervised:• Observe behavior,• Probe representations,• etc.

Benchmarks: knowledge-specific tests (w/ or w/o training data)

Page 5: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

How do you know that a model is doing commonsense reasoning?

Unsupervised:• Observe behavior,• Probe representations,• etc.

Benchmarks: knowledge-specific tests (w/ or w/o training data)

QA format: easy to evaluate (e.g., accuracy)

Page 6: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Step 1: Determine type of reasoning

https://leaderboard.allenai.org/

https://leaderboard.allenai.org/
Page 7: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Step 1: Determine type of reasoning

https://leaderboard.allenai.org/

Abductive reasoning

https://leaderboard.allenai.org/
Page 8: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Step 1: Determine type of reasoning

https://leaderboard.allenai.org/

Visual commonsense

reasoningAbductive reasoning

https://leaderboard.allenai.org/
Page 9: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Reasoning about Social Situations

Page 10: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Reasoning about Social Situations

Alex spilt food all over the floor and it made a huge mess.

What will Alex want to do next?

Page 11: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Reasoning about Social Situations

Alex spilt food all over the floor and it made a huge mess.

What will Alex want to do next?

run around in the mess mop up the mess

Page 12: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Reasoning about Social Situations

Alex spilt food all over the floor and it made a huge mess.

What will Alex want to do next?

run around in the mess mop up the mess

less likely more likely

Page 13: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

effects

stativecauses

no intent

drink too muchclumsy

embarrassed clean it up slip on the spill

fall over

careless

upsetget a broom

gets dirty

PersonX spills ___ all over the floor

X is seen as

has effect on X

X will want

X will feel

X wanted to

X needed to

Knowledge tested in SOCIAL IQA: ATOMIC

Page 14: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Step 2: Choosing a benchmark size

Small scale Large scale

Creation Expert-curated Crowdsourced/automatic

Coverage Limited coverage Large coverage

Training Dev/test only Training/dev/test

Budget Expert time costs Crowdsourcing costs

Page 15: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Step 2: Choosing a benchmark size

Small scale Large scale

Creation Expert-curated Crowdsourced/automatic

Coverage Limited coverage Large coverage

Training Dev/test only Training/dev/test

Budget Expert time costs Crowdsourcing costs

Winograd Schema Challenge (WSC), Choice of Plausible Alternatives (COPA)

Page 16: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Small commonsense benchmarks

The city councilmen refused the demonstrators a permit because they advocated violence. Who is “they”?

(a)The city councilmen(b)The demonstrators

The city councilmen refused the demonstrators a permit because they feared violence. Who is “they”?

(a)The city councilmen(b)The demonstrators

Winograd Schema Challenge (WSC)

273 examples

Choice of Plausible Alternatives (COPA)

500 dev, 500 test

Page 17: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Small commonsense benchmarks

I hung up the phone. What was the cause of this?

(a)The caller said goodbye to me.(b)The caller identified himself to me.

The toddler became cranky. What happened as a result?

(a)Her mother put her down for a nap.(b)Her mother fixed her hair into pigtails.

Winograd Schema Challenge (WSC)

273 examples

Choice of Plausible Alternatives (COPA)

500 dev, 500 test

Page 18: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Step 2: Choosing a QA benchmark size

Small scale Large scale

Creation Expert-curated Crowdsourced/automatic

Coverage Limited coverage Large coverage

Training Dev/test only Training/dev/test

Budget Expert time costs Crowdsourcing costs

Challenge: do to collect positive/negative answers?

Page 19: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Challenge of collecting unlikely answers

Page 20: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Challenge of collecting unlikely answers

Goal: negative answers have to be plausible but unlikely

Page 21: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Challenge of collecting unlikely answers

Goal: negative answers have to be plausible but unlikely

• Automatic matching?• Random negative sampling won’t work, too topically different

• “smart” negative sampling isn’t effective either

Page 22: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Challenge of collecting unlikely answers

Goal: negative answers have to be plausible but unlikely

• Automatic matching?• Random negative sampling won’t work, too topically different

• “smart” negative sampling isn’t effective either

• Need better solution… maybe we can ask crowd workers?

Page 23: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Collecting answers from crowdworkers

Alex spilt food all over the floor and it made a huge mess.

What will Alex want to do next?

WHAT HAPPENS NEXT

Context and Question

Page 24: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Collecting answers from crowdworkers

Alex spilt food all over the floor and it made a huge mess.

What will Alex want to do next?

WHAT HAPPENS NEXT

Context and Question

Page 25: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Collecting answers from crowdworkers

Alex spilt food all over the floor and it made a huge mess.

What will Alex want to do next?

WHAT HAPPENS NEXT✔mop up✔ give up and order take out

✘ leave the mess✘ run around in the mess

Handwritten ✔ and ✘ Answers

Context and Question

Free Text Response

Page 26: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Collecting answers from crowdworkers

Alex spilt food all over the floor and it made a huge mess.

What will Alex want to do next?

WHAT HAPPENS NEXT✔mop up✔ give up and order take out

✘ leave the mess✘ run around in the mess

Handwritten ✔ and ✘ Answers

Context and Question

Free Text Response

Problem: handwritten unlikely answers are too easy to detect

Page 27: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Problem: annotation artifacts

Page 28: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Problem: annotation artifacts

• Models can exploit artifacts in handwritten incorrect answers• Exaggerations, off-topic, overly emotional, etc.

• See Schwartz et al. 2017, Gururangan et al. 2018, Zellers et al. 2018, etc.

Page 29: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Problem: annotation artifacts

• Models can exploit artifacts in handwritten incorrect answers• Exaggerations, off-topic, overly emotional, etc.

• See Schwartz et al. 2017, Gururangan et al. 2018, Zellers et al. 2018, etc.

• Seemingly “super-human” performance by large pretrained LMs (BERT, GPT, etc.)

Page 30: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Problem: annotation artifacts

• Models can exploit artifacts in handwritten incorrect answers• Exaggerations, off-topic, overly emotional, etc.

• See Schwartz et al. 2017, Gururangan et al. 2018, Zellers et al. 2018, etc.

• Seemingly “super-human” performance by large pretrained LMs (BERT, GPT, etc.)

Page 31: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

How to make unlikely answers robust to annotation artifacts?

Page 32: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

How to make unlikely answers robust to annotation artifacts?

SOCIAL IQA, COMMONSENSEQA:Modified answer collection

Page 33: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

How to make unlikely answers robust to annotation artifacts?

SOCIAL IQA, COMMONSENSEQA:Modified answer collection

HellaSwag & AF-lite:Adversarial filtering of artifacts

Page 34: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Question-Switching Answers (SOCIAL IQA)

Alex spilt food all over the floor and it made a huge mess.

What will Alex want to do next?

WHAT HAPPENS NEXT

✔mop up✔ give up and order take out

Original Question

✘ have slippery hands✘ get ready to eat

Page 35: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Question-Switching Answers (SOCIAL IQA)

Alex spilt food all over the floor and it made a huge mess.

What will Alex want to do next?

WHAT HAPPENS NEXT

Question-Switching Answer

✔mop up✔ give up and order take out

What did Alex need to do before this?

WHAT HAPPENED BEFORE

Original Question

✘ have slippery hands✘ get ready to eat

Page 36: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Question-Switching Answers (SOCIAL IQA)

Alex spilt food all over the floor and it made a huge mess.

What will Alex want to do next?

WHAT HAPPENS NEXT

Question-Switching Answer

✔mop up✔ give up and order take out

What did Alex need to do before this?

WHAT HAPPENED BEFORE

Original Question

✘ have slippery hands✘ get ready to eat

have slippery handsget ready to eat

✔ have slippery hands✔ get ready to eat

Page 37: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Question-Switching Answers (SOCIAL IQA)

Alex spilt food all over the floor and it made a huge mess.

What will Alex want to do next?

WHAT HAPPENS NEXT

Question-Switching Answer

✔mop up✔ give up and order take out

What did Alex need to do before this?

WHAT HAPPENED BEFORE

✔ have slippery hands✔ get ready to eat

Original Question

✘ have slippery hands✘ get ready to eat

Page 38: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Comparing incorrect/correct answers’ styles

More stylistically different from

correct

More stylistically

similar

Page 39: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Comparing incorrect/correct answers’ styles

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

Arousal Dominance Valence

Effect Size when comparing to Correct Answers

Handwritten Incorrect Question Switching

More stylistically different from

correct

More stylistically

similar

Page 40: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Comparing incorrect/correct answers’ styles

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

Arousal Dominance Valence

Effect Size when comparing to Correct Answers

Handwritten Incorrect Question Switching

More stylistically different from

correct

More stylistically

similar

Question switching answers are more stylistically similar to correct answers

Page 41: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

COMMONSENSEQA: pivot on knowledge graphs

Talmor et al. 2019

Page 42: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

COMMONSENSEQA: pivot on knowledge graphs

Talmor et al. 2019

Page 43: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

COMMONSENSEQA: pivot on knowledge graphs

Talmor et al. 2019

Page 44: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

COMMONSENSEQA: pivot on knowledge graphs

Talmor et al. 2019

Page 45: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

COMMONSENSEQA: pivot on knowledge graphs

Talmor et al. 2019

Page 46: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

COMMONSENSEQA: pivot on knowledge graphs

Talmor et al. 2019

Page 47: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Adversarial Filtering (lite)

Goal: remove examples with exploitable artifacts or spurious correlations

• Use pre-trained representations

• Iteratively remove data that’s easiest to predict by a linear classifier (e.g., logistic)

• Robust examples remain

HellaSwag (Zellers et al., 2019)AF-lite (Le Bras et al., 2019)

Unfiltered examples

Page 48: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Adversarial Filtering (lite)

Goal: remove examples with exploitable artifacts or spurious correlations

• Use pre-trained representations

• Iteratively remove data that’s easiest to predict by a linear classifier (e.g., logistic)

• Robust examples remain

HellaSwag (Zellers et al., 2019)AF-lite (Le Bras et al., 2019)

Unfiltered examplesUnfiltered examples

Page 49: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Adversarial Filtering (lite)

Goal: remove examples with exploitable artifacts or spurious correlations

• Use pre-trained representations

• Iteratively remove data that’s easiest to predict by a linear classifier (e.g., logistic)

• Robust examples remain

HellaSwag (Zellers et al., 2019)AF-lite (Le Bras et al., 2019)

Unfiltered examples

+

Unfiltered examples

Page 50: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Adversarial Filtering (lite)

Goal: remove examples with exploitable artifacts or spurious correlations

• Use pre-trained representations

• Iteratively remove data that’s easiest to predict by a linear classifier (e.g., logistic)

• Robust examples remain

HellaSwag (Zellers et al., 2019)AF-lite (Le Bras et al., 2019)

“Easy”examples

Unfiltered examples

+

Unfiltered examples

Page 51: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Adversarial Filtering (lite)

Goal: remove examples with exploitable artifacts or spurious correlations

• Use pre-trained representations

• Iteratively remove data that’s easiest to predict by a linear classifier (e.g., logistic)

• Robust examples remain

HellaSwag (Zellers et al., 2019)AF-lite (Le Bras et al., 2019)

“Easy”examples

Robust examples

Unfiltered examples

+

Unfiltered examples

Page 52: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Performance of models on the WikiHow portion of HellaSwag (Zellers et al., 2019) with different AF settings and different training models

Page 53: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Performance of models on the WikiHow portion of HellaSwag (Zellers et al., 2019) with different AF settings and different training models

Adversarial filtering removes examples with spurious correlations=> Task becomes harder

Page 54: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Model performance on SOCIAL IQA

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0%

Accuracy

Humans

Bert-large

Bert-base

GPT

Random

Page 55: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Model performance on SOCIAL IQA

0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0%

Accuracy

Humans

Bert-large

Bert-base

GPT

Random

>20% gap to improve on

Page 56: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Challenging SOCIAL IQA examples for BERT-large

Although Aubrey was older and stronger, they lost to Alex in arm wrestling.

How would Alex feel as a result?

ashamed

boastful

they need to practice more

how Aubrey would feel, not Alex

Page 57: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Challenging SOCIAL IQA examples for BERT-large

Remy gave Skylar, the concierge, her account so that she could check into the hotel.

What will Remy want to do next?

Although Aubrey was older and stronger, they lost to Alex in arm wrestling.

How would Alex feel as a result?

lose her credit card

arrive at a hotel

get the key from Skylar

ashamed

boastful

they need to practice more

how Aubrey would feel, not Alex

what Remy did before

Page 58: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Challenging SOCIAL IQA examples for BERT-large

Remy gave Skylar, the concierge, her account so that she could check into the hotel.

What will Remy want to do next?

Although Aubrey was older and stronger, they lost to Alex in arm wrestling.

How would Alex feel as a result?

lose her credit card

arrive at a hotel

get the key from Skylar

ashamed

boastful

they need to practice more

how Aubrey would feel, not Alex

what Remy did before

Need more robust, person-centric reasoning

Page 59: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Challenging SOCIAL IQA examples for BERT-large

Remy gave Skylar, the concierge, her account so that she could check into the hotel.

What will Remy want to do next?

Although Aubrey was older and stronger, they lost to Alex in arm wrestling.

How would Alex feel as a result?

lose her credit card

arrive at a hotel

get the key from Skylar

ashamed

boastful

they need to practice more

how Aubrey would feel, not Alex

what Remy did before

Need more robust, person-centric reasoning

Need better notion of causes vs. effects

Page 60: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Commonsense benchmarks

Social IQa Physical IQa

Abductive NLI

HellaSwag

VCR WinoGrandeCommonsenseQA JHU Ordinal

Commonsense

MCTaco

SWAG

Naïve Psychology

COPAWSC

ROC story

MultiRCCosmosQA

ReCORD

Page 61: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Social commonsense

Commonsense benchmarks

Social IQa Physical IQa

Abductive NLI

HellaSwag

VCR WinoGrandeCommonsenseQA JHU Ordinal

Commonsense

MCTaco

SWAG

Naïve Psychology

COPAWSC

ROC story

MultiRCCosmosQA

ReCORD

Page 62: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Physical commonsense Social commonsense

Commonsense benchmarks

Social IQa Physical IQa

Abductive NLI

HellaSwag

VCR WinoGrandeCommonsenseQA JHU Ordinal

Commonsense

MCTaco

SWAG

Naïve Psychology

COPAWSC

ROC story

MultiRCCosmosQA

ReCORD

Page 63: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Temporal commonsense

Physical commonsense Social commonsense

Commonsense benchmarks

Social IQa Physical IQa

Abductive NLI

HellaSwag

VCR WinoGrandeCommonsenseQA JHU Ordinal

Commonsense

MCTaco

SWAG

Naïve Psychology

COPAWSC

ROC story

MultiRCCosmosQA

ReCORD

Page 64: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Temporal commonsense Commonsense reading comprehension

Physical commonsense Social commonsense

Commonsense benchmarks

Social IQa Physical IQa

Abductive NLI

HellaSwag

VCR WinoGrandeCommonsenseQA JHU Ordinal

Commonsense

MCTaco

SWAG

Naïve Psychology

COPAWSC

ROC story

MultiRCCosmosQA

ReCORD

Page 65: Commonsense benchmarks - University of Washingtonmsap/acl2020...Small commonsense benchmarks I hung up the phone. What was the cause of this? (a)The caller said goodbye to me. (b)The

Temporal commonsense Commonsense reading comprehension

Physical commonsense Social commonsense

Commonsense benchmarks

Social IQa Physical IQa

Abductive NLI

HellaSwag

VCR WinoGrandeCommonsenseQA JHU Ordinal

Commonsense

MCTaco

SWAG

Naïve Psychology

COPAWSC

ROC story

MultiRCCosmosQA

ReCORD

Thanks! Questions?


ACL2020 Best Paper & Honorable Mention Papers

Commonsense Reasoning about Containers using Radically

Commonsense Framework to Measure Social Media

ThomasPaine CommonSense

COMET: Commonsense Transformers for Automatic Knowledge

Commonsense - cntw.nhs.uk

Pm Commonsense Approach

SCIENCE AND COMMONSENSE

Commonsense Justification for Action Explanation · commonsense knowledge about that action. The lack of commonsense knowledge is a major bottleneck in artificial agents which jeopardizes

Commonsense Ai

Commonsense Knowledge Base Population

Paine CommonSense

Commonsense Reasoning about Chemistry Experiments: Ontology and Representation Ernest Davis Commonsense 2009

Semantic Understanding and Commonsense Reasoning in an ...alumni.media.mit.edu/~hugo/publications/talks/masterworks-eecs-mastersthesis.pdfSemantic Understanding and Commonsense Reasoning

CSKG: The CommonSense Knowledge Graph

CommonSense Art Williams

From Recognition to Cognition: Visual Commonsense Reasoningopenaccess.thecvf.com/content_CVPR_2019/papers/Zellers_From... · From Recognition to Cognition: Visual Commonsense Reasoning

Shortcutted Commonsense: Data Spuriousness in Deep

Commonsense Physical Reasoning: Boxes and Pitchers

Moore Commonsense

WebChild: Harvesting and Organizing Commonsense Knowledge

Incorporating commonsense knowledge to vqa

Heterogeneous Graph Learning for Visual Commonsense Reasoningpapers.neurips.cc/...graph-learning-for-visual-commonsense-reasonin… · Heterogeneous Graph Learning for Visual Commonsense

BPN Radical CommonSense Cookbook

Logic and Commonsense Reasoning - hal.archives-ouvertes.fr

ConceptNet — a practical commonsense reasoning tool-kitalumni.media.mit.edu/~hugo/publications/papers/BTTJ-ConceptNet.pdf · ConceptNet — a practical commonsense reasoning tool

Commonsense 01

Commonsense Judgments of Infanticide

Differentiable Open-Ended Commonsense Reasoning

01 commonsense political thinking booklet

Commonsense 02

Commonsense Social Media Measurement

CoSePost - CommonSense

Commonsense Cookery Notes

NLog-like Inference and Commonsense Reasoning