![Page 1: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/1.jpg)
PeTra: A Sparsely Supervised Memory Modelfor People Tracking
Shubham ToshniwalAllyson Ettinger Kevin Gimpel Karen Livescu
github.com/shtoshni92/petra
![Page 2: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/2.jpg)
People Tracking
Amelia Shepherd, M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone. In her debut appearance, Amelia visited her former sister-in-law.
![Page 3: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/3.jpg)
People Tracking
Amelia Shepherd1 , M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone2 . In her1 debut appearance, Amelia1 visited her1 former sister-
in-law.
![Page 4: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/4.jpg)
People Tracking
Amelia Shepherd1 , M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone2 . In her1 debut appearance, Amelia1 visited her1 former sister-
in-law.
Coreference Resolution for people references
![Page 5: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/5.jpg)
Coreference Resolution
![Page 6: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/6.jpg)
Mention Ranking Models
Amelia Shepherd Caterina Scorsone her Amelia
her
![Page 7: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/7.jpg)
Mention Ranking Models
Amelia Shepherd Caterina Scorsone her Amelia
her
![Page 8: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/8.jpg)
Mention Ranking Models
Amelia Shepherd Caterina Scorsone her Amelia
her
Impractical for long documents!
![Page 9: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/9.jpg)
Entity-Mention Models
Entities
...
Caterina Scorsone
Amelia Shepherd
her
![Page 10: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/10.jpg)
Entity-Mention Models
Entities
...
Caterina Scorsone
Amelia Shepherd
her
![Page 11: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/11.jpg)
Entity-Mention Models
Entities
...
Caterina Scorsone
Amelia Shepherd
her
Number of entities can be quite large!Fixed-size Memory
![Page 12: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/12.jpg)
PeTra
![Page 13: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/13.jpg)
Ideal Run
Amelia
Amelia
OW
Shepherd
CR
, portrayed by
. . .
IG
Caterina
OW
Scorsone
CR
, visits
. . .
IG
her
CR
friend
IG
IG = Ignore, OW = Overwrite, CR = Coreference
![Page 14: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/14.jpg)
Ideal Run
Amelia
Amelia
OW
Shepherd
CR
, portrayed by
. . .
IG
Caterina
OW
Scorsone
CR
, visits
. . .
IG
her
CR
friend
IG
IG = Ignore, OW = Overwrite, CR = Coreference
![Page 15: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/15.jpg)
Ideal Run
Amelia
Amelia
OW
Shepherd
CR
, portrayed by
. . .
IG
Caterina
OW
Scorsone
CR
, visits
. . .
IG
her
CR
friend
IG
IG = Ignore, OW = Overwrite, CR = Coreference
![Page 16: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/16.jpg)
Ideal Run
Amelia
Amelia
OW
Shepherd
CR
, portrayed by
. . .
IG
Caterina
OW
Scorsone
CR
, visits
. . .
IG
her
CR
friend
IG
IG = Ignore, OW = Overwrite, CR = Coreference
![Page 17: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/17.jpg)
Ideal Run
Amelia
Amelia
OW
Shepherd
CR
, portrayed by
. . .
IG
Caterina
OW
Scorsone
CR
, visits
. . .
IG
her
CR
friend
IG
IG = Ignore, OW = Overwrite, CR = Coreference
![Page 18: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/18.jpg)
Model Details
BERT Encoder
wt
ht
Mt−1 Mt. . . . . .
Controller
et, ot, ct
. . . . . .
. . . . . .
GRU HiddenStates
Input Tokens
Memory
ControllerOutputs
Key Outputs:et Entity Probabilityot Overwrite Prob. Vectorct Coreference Prob. Vector
![Page 19: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/19.jpg)
Sparse Supervision
Amelia Shepherd1 , M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone2 . In her1 debut appearance, Amelia visited her former sister-in-
law.
![Page 20: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/20.jpg)
Auxiliary Loss
Amelia Shepherd1 , M.D. is a fictional character on Grey’s Anatomy, portrayed byCaterina Scorsone2 . In her1 debut appearance, Amelia visited her former sister-in-
law.
Most tokens don’t refer to people!Penalize high entity probability outside labeled spans.
![Page 21: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/21.jpg)
Results
![Page 22: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/22.jpg)
GAP Results
2 4 6 8 10 12 14 16 18 20
# of memory cells
80
81
82
83
84
85
86F
-sco
re
Referential Reader
![Page 23: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/23.jpg)
GAP Results
2 4 6 8 10 12 14 16 18 20
# of memory cells
80
81
82
83
84
85
86F
-sco
re
Referential Reader
![Page 24: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/24.jpg)
GAP Results
2 4 6 8 10 12 14 16 18 20
# of memory cells
80
81
82
83
84
85
86F
-sco
re
Referential Reader
PeTra
![Page 25: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/25.jpg)
Memory Log Visualization
CROW
CROW
CROW
[CLS
]Am
elia
Shep
herd , M . D . is a
fictio
nalch
arac
ter .
.....
portra
yed by
Cat##er
ina Sc
##ors
##one . In her
debut
appea
rance in
seas
onth
ree ,
Ameli
avis
ited
herfo
rmer
......
[SEP
]
CROW
PeTra can resolve references beyond pronouns!
![Page 26: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/26.jpg)
Interpretability
![Page 27: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/27.jpg)
Tracking People
Are the models actually tracking people?
![Page 28: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/28.jpg)
Tracking People
Are the models actually tracking people?
What does it mean for memory model to “track people”?
Detect all references to people.One-to-one correspondence between people and memory cells.
![Page 29: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/29.jpg)
Human Evaluation
Neef took an individual silver medal at the 1994 European Cup behind Rus-sia’s Svetlana Goncharenko and returned the following year to win gold. Shewas a finalist individually at the 1994 European Championships and camesixth for Scotland at the 1994 Commonwealth Games.
CROW
CROW
CROW
CROW
CROW
CROW
CROW
[CLS
] N##
ee ##f
took an
indi
vidua
lsil
ver
med
al at the
1994
Euro
pean Cup
behi
ndRu
ssia ' s S
##ve
t##
lana Go
##nc
h##
are
##nk
oan
dre
turn
ed the
follo
win
gye
ar to win
gold .
She
was a
final
istin
divid
ually at the
1994
Euro
pean
Cham
pion
ship
san
dca
me
sixth for
Scot
land at th
e19
94
Com
mon
wea
lthGa
mes
.[S
EP]
CROW
Mem
ory
Cells
CROW
CROW
CROW
CROW
CROW
CROW
CROW
[CLS
] N##
ee ##f
took an
indi
vidua
lsil
ver
med
al at the
1994
Euro
pean Cup
behi
ndRu
ssia ' s S
##ve
t##
lana Go
##nc
h##
are
##nk
oan
dre
turn
ed the
follo
win
gye
ar to win
gold .
She
was a
final
istin
divid
ually at the
1994
Euro
pean
Cham
pion
ship
san
dca
me
sixth for
Scot
land at th
e19
94
Com
mon
wea
lthGa
mes
.[S
EP]
CROW
Mem
ory
Cells
![Page 30: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/30.jpg)
Human Evaluation
Neef took an individual silver medal at the 1994 European Cup behind Rus-sia’s Svetlana Goncharenko and returned the following year to win gold. Shewas a finalist individually at the 1994 European Championships and camesixth for Scotland at the 1994 Commonwealth Games.
CROW
CROW
CROW
CROW
CROW
CROW
CROW
[CLS
] N##
ee ##f
took an
indi
vidua
lsil
ver
med
al at the
1994
Euro
pean Cup
behi
ndRu
ssia ' s S
##ve
t##
lana Go
##nc
h##
are
##nk
oan
dre
turn
ed the
follo
win
gye
ar to win
gold .
She
was a
final
istin
divid
ually at the
1994
Euro
pean
Cham
pion
ship
san
dca
me
sixth for
Scot
land at th
e19
94
Com
mon
wea
lthGa
mes
.[S
EP]
CROW
Mem
ory
Cells
CROW
CROW
CROW
CROW
CROW
CROW
CROW
[CLS
] N##
ee ##f
took an
indi
vidua
lsil
ver
med
al at the
1994
Euro
pean Cup
behi
ndRu
ssia ' s S
##ve
t##
lana Go
##nc
h##
are
##nk
oan
dre
turn
ed the
follo
win
gye
ar to win
gold .
She
was a
final
istin
divid
ually at the
1994
Euro
pean
Cham
pion
ship
san
dca
me
sixth for
Scot
land at th
e19
94
Com
mon
wea
lthGa
mes
.[S
EP]
CROW
Mem
ory
Cells
PeTra Ref. Reader Neutral74% 8% 18%
![Page 31: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/31.jpg)
Conclusion
- Propose a memory model for tracking people with sparse pronoun resolutionannotations.
- Outperforms previous approach on both the end task and interpretability measures.
- Future work: Extend this work to longer documents.
![Page 32: PeTra: A Sparsely Supervised Memory Model for People Trackingshtoshni/papers/petra_slides.pdf · Auxiliary Loss Amelia Shepherd 1, M.D. is a ctional character on Grey’s Anatomy,](https://reader036.vdocuments.us/reader036/viewer/2022071117/60029070b6bbe3686467adb6/html5/thumbnails/32.jpg)
References
Fei Liu, Luke Zettlemoyer, and Jacob Eisenstein.The Referential Reader: A recurrent entity network for anaphora resolution.ACL, 2019.Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge.Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns.TACL, 2018.Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun.Tracking the world state with recurrent entity networks.ICLR, 2017.Alex Graves et al.Hybrid computing using a neural network with dynamic external memory.Nature, 2016.