self-assemblying hypernetworks for cognitive learning of linguistic memory int. conf. on cognitive...
Post on 20-Jan-2016
215 Views
Preview:
TRANSCRIPT
Self-Assemblying Hypernetworks for Self-Assemblying Hypernetworks for Cognitive Cognitive
Learning of Linguistic MemoryLearning of Linguistic Memory
Int. Conf. on Cognitive Science, CESSE-2008, Feb. 6-8, 2008, Int. Conf. on Cognitive Science, CESSE-2008, Feb. 6-8, 2008, Sheraton Hotel, Cairo, EgyptSheraton Hotel, Cairo, Egypt
Byoung-Tak Zhang and Chan-Hoon Park
Biointelligence LaboratorySchool of Computer Science and Engineering
Cognitive Science, Brain Science, and Bioinformatics ProgramsSeoul National University
Seoul 151-744, Korea
btzhang@bi.snu.ac.krhttp://bi.snu.ac.kr/
Talk Outline
A Language Game Learning the Linguistic Memory
The Hypernetwork Model of Language
Sentence Recall Experiments Extension to Multimodal Memory Game
(Language + Vision) Conclusion
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
3
The Language Game PlatformThe Language Game Platform
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
4
A Language GameA Language Game
? still ? believe ? did this. I still can't believe you did
this.
We ? ? a lot ? gifts. We don't have a lot of gifts.
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
5
Text Corpus: TV Drama SeriesText Corpus: TV Drama Series
Friends, 24, House, Grey Anatomy, Gilmore Girls, Sex and the City
289,468 Sentences
(Training Data)
700 Sentences with Blanks(Test Data)
I don't know what happened.Take a look at this.…
What ? ? ? here.? have ? visit the ? room.
…
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
6
Step 1: Learning a Linguistic Step 1: Learning a Linguistic MemoryMemory
Display
Color
Text
News
Monitor
Computer
Network
Price
Computer network is rapidly increased. The price of computer network installing is very cheap. The price of monitor display is on decreasing. Nowadays, color monitor display is so common, the price is not so high. This is a system adopting text mode color display. This is an animation news networks. ...
Price
Computer Network
Computer Price
Computer Display
Computer Monitor Display
News Network
Text News
Monitor Display
Monitor Price
Color Monitor
Color Monitor
Color Display
Text Display
Color
Display
Computer Network Price
Price
Color
Text
k = 2k = 2k = 3k = 4 …
HypernetworkMemory
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/7
Step 2: Recalling from the Step 2: Recalling from the MemoryMemory
He is my best friend
best friendmy bestis myHe is
He is a ? friend
He is a strong boy
Strong friend likes pretty girl
He is is a a strong strong boy
Strong friend friend likes likes strong pretty girl
Strong friend
a strong
best friend
He is a strong friend
X7
X6
X5
X8
X1
X2
X3
X4
Recall
Self-assembly
Storage
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
8x8 x9
x12
x1x2
x3
x4
x5
x6
x7x10
x11
x13
x14
x15
x1 =1
x2 =0
x3 =0
x4 =1
x5 =0
x6 =0
x7 =0
x8 =0
x9 =0
x10 =1
x11 =0
x12 =1
x13 =0
x14 =0
x15 =0
y
= 1
x1 =0
x2 =1
x3 =1
x4 =0
x5 =0
x6 =0
x7 =0
x8 =0
x9 =1
x10 =0
x11 =0
x12 =0
x13 =0
x14 =1
x15 =0
y
= 0
x1 =0
x2 =0
x3 =1
x4 =0
x5 =0
x6 =1
x7 =0
x8 =1
x9 =0
x10 =0
x11 =0
x12 =0
x13 =1
x14 =0
x15 =0
y
=14 sentences (with labels)
x4 x10 y=1x1
x4 x12 y=1x1
x10 x12 y=1x4
x3 x9 y=0x2
x3 x14 y=0x2
x9 x14 y=0x3
x6 x8 y=1x3
x6 x13 y=1x3
x8 x13 y=1x6
1
2
3
1
2
3
x1 =0
x2 =0
x3 =0
x4 =0
x5 =0
x6 =0
x7 =0
x8 =1
x9 =0
x10 =0
x11 =1
x12 =0
x13 =0
x14 =0
x15 =1
y
=14
x11 x15 y=0x84
Round 1Round 2Round 3
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
9
x1x2
x3
x4
x5
x6
x7
x8 x9
x10
x11
x12
x13
x14
x15
The Hypernetwork Memory The Hypernetwork Memory
)( 2121...21
21
21...21
321
321321
21
2121
321
321321
21
2121
2 ,...,,
)()()()(
2 ,...,,
)()()()(
,,
)()()()3(
,
)()()2(
)()(
,,
)()()()3(
,
)()()2()(
...)(
1exp)Z(
isfunction partition thewhere
,...)(
1exp
)Z(
1
...6
1
2
1exp
)Z(
1
)];(exp[)Z(
1 )|(
ondistributiy probabilit The
...6
1
2
1 );(
rkhypernetwo theofenergy The
m kkiiikiii
k
kiiikiii
iiiiiiiiii
iiiiiiiiii
K
k iii
mmmk
K
k iii
nnnk
iii
nnn
ii
nn
nn
iii
nnn
ii
nnn
xxxwkc
W
xxxwkcW
xxxwxxwW
WEW
WP
xxxwxxwWE
x
xx
x
Nn
K
iii
i
D
WWWW
SkXSSS
xxxX
WSXH
I
1)(
)()3()2(
,...,,
}{
:set Training
),...,,(
|| , ,
)(
),,(
as defined isrk hypernetwo The
21
x
[Zhang, DNA12-2006]
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
10
Molecular Self-Assembly of HypernetworksMolecular Self-Assembly of Hypernetworks
xi xj y
X7
X6
X5
X8
X1
X2
X3
X4
Hypernetwork Representation
x1 x3 Class
x1 x2 x4 Classx2 x3 Class
x1 x4 Class
x1 x3 Class
x1 x3 Class
x1 x2 x4 Class
x1 x2 x4 Class
x2 x3 x4 Class
x2 x3 x4 Class
x2 x3 x4 Class
x2 x3 Class
x2 x3 Class
x1 x4 Class
x1 x4 Class
x1 Class
x2 Class
x1 x2 Class
x1 x3 Class
x1 xn Class…
x1 Class
x1 Class
x2 Class
x1 x2 Class
x1 x2 Class
x1 x3 Class
x1 x3 Class
x1 x3 Class
x1 xn Class…
x2 Class
x2 Class
x1 x3 Class
x1 x3 Class
Molecular Encoding
DNA Computing
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
11
Experimental SetupExperimental Setup
The order (k) of an hyperedge Range: 2~4 Fixed order for each experiment
The method of creating hyperedges from training data Sliding window method Sequential sampling from the first word
The number of blanks (question marks) in test data Range: 1~4 Maximum: k - 1
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
12
Learning Behavior Analysis (1/3)Learning Behavior Analysis (1/3)
The performance monotonically increases as the learning corpus grows. The low-order memory performs best for the one-missing-word problem.
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
13
Learning Behavior Analysis (2/3)Learning Behavior Analysis (2/3)
The medium-order (k=3) memory performs best for the two-missing-words problem.
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
14
Learning Behavior Analysis (3/3)Learning Behavior Analysis (3/3)
The high-order (k=4) memory performs best for the three-missing-words problem.
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
15
The Language Game: ResultsThe Language Game: Results
Why ? you ? come ? down ? Why are you go come on down here
? think ? I ? met ? somewhere before I think but I am met him somewhere before
? appreciate it if ? call her by ? ? I appreciate it if you call her by the way
I'm standing ? the ? ? ? cafeteria I'm standing in the one of the cafeteria
Would you ? to meet ? ? Tuesday ? Would you nice to meet you in Tuesday and
? gonna ? upstairs ? ? a shower I'm gonna go upstairs and take a shower
? have ? visit the ? room I have to visit the ladies' room
We ? ? a lot ? gifts We don't have a lot of gifts
? ? don't need your ? If I don't need your help
? ? ? decision to make a decision
? still ? believe ? did this I still can't believe you did this
What ? ? ? here What are you doing here
? you ? first ? of medical school Are you go first day of medical school
? ? a dream about ? In ? I had a dream about you in Copenhagen
Extension to Multimodal Memory Game
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
ImageImage SoundSound TextText
But, I'm getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?But if you are, call me tonight.
But, I'm getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?But if you are, call me tonight.
Memorizing Images to Retrieve Texts
Memorizing Images to Retrieve Texts
Memorizing Textsto Retrieve Images
Memorizing Textsto Retrieve Images
Hint Hint TextText
But, I'm getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?But if you are, call me tonight.
But, I'm getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?But if you are, call me tonight.
But, I'm getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?But if you are, call me tonight.
But, I'm getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?But if you are, call me tonight.
But, I'm getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?But if you are, call me tonight.
But, I'm getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?But if you are, call me tonight.
But, I'm getting married tomorrowWell, maybe I am...I keep thinking about you.And I'm wondering if we made a mistake giving up so fast.Are you thinking about me?But if you are, call me tonight.
Image Image HintHint
Image Generation GameImage Generation GameImage Generation GameImage Generation GameText Generation GameText Generation GameText Generation GameText Generation Game
Machine LearnerMachine Learner
How can it be done?
How can it be done?
Scene1 He is best friend
Scene2 She is strong boy
Scene3 friend likes pretty girl
Scene1 a1 a2 a3 a4
Scene2 b1 b2 b3 b4
Scene3 c1 c2 c3 c4
TextText ImageImage
Text: N sequential samplesImage: N random samples
He is a1 a3 a4
best friend a1 a3 a2
Multimodal HNMultimodal HN
She is b1 b3 b5
is strong b2 b3 b1
He is best friendText QueryText Query
He is a1 a3 a4
best friend a1 a3 a4
is best b2 b3 b1
MatchingMatching
He is best friendText QueryText Query
He is a1 a3 a4
best friend a1 a3 a4
is best b2 b3 b1
MatchingMatching
Generating an Image
Mapa1=2b1=1
b2=1a3=2b3=1
a4=2 voting Map a1 b2 a3 a4
Map a1 b2 a3 a4
Scene1 a1 a2 a3 a4
Scene2 b1 b2 b3 b4
Scene3 c1 c2 c3 c4
ImageImage
Hamming Distance
Scene1 a1 a2 a3 a4Map a1 b2 a3 a4
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
18
Image-to-Text Crossmodal RecallImage-to-Text Crossmodal RecallText
Learningby Viewing
Image- Where am I giving birth- You guys really don't know anything- So when you guys get in there- I know it's been really hard for you- …
User TextCorpus
Question:
Answer:
Where am I giving birth
Where ? I giving ?
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
19
Text-to-Image Crossmodal RecallText-to-Image Crossmodal RecallText
Learningby Viewing
Image Corpus User
Question:
Answer:
Image
You've been there
- Where am I giving birth- You guys really don't know anything- So when you guys get in there- I know it's been really hard for you- …
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
20
The Multimedia (Movie) CorpusThe Multimedia (Movie) Corpus
Dataset: 2 dramas Images and the corresponding scripts Titles
Friends, Prison Break
Training data: 2,808 images and scripts Image size: 80 x 60 = 4800 pixels Vocabulary: 2,579 words
Where am I giving birth
I know it's been really hard for you
So when you guys get in there
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
21
Experimental SetupExperimental Setup
The order (k) of memory units Text: k = 2, 3, 4 Image: k = 10, …, 340
Constructing hyperedges from training data Text: Sequential sampling from a random position Image: Random sampling from 4,800 pixel positions
The number of repetitive samples from an image-text pair N = 150, …, 300
AnswerQuery
I don't know what happened
There's a kitty in my guitar case
Maybe there's something I can do to make sure I get pregnant
Maybe there's something there's something I … I get pregnant
There's a a kitty in … in my guitar case
I don't know don't know what know what happened
Matching &Completion
Image-to-Text Recall ExamplesImage-to-Text Recall Examples
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
Query Matching &Completion
I don't know what happened
Take a look at this
There's a kitty in my guitar case
Maybe there's something I can do to make sure I get pregnant
Answer
Text-to-Image Recall ExamplesText-to-Image Recall Examples
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
24
Image to Text (Recall Rate)Image to Text (Recall Rate)
Note: In the tolerant recall, the generated sentence is evaluated correct if the number of mismatches is within the specified tolerance level (here two words).
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
25
Text to Image (Recall Rate)Text to Image (Recall Rate)
Note: The retrieved image is evaluated correct if its hamming distance to the target image is the smallest.
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
26
ConclusionConclusion Hypernetworks are a random graph model employing higher-order edges
and allowing for a more natural representation for learning higher-order interactions.
We introduce a linguistic memory model based on a self-organizing hypernetwork inspired by mental chemistry.
The hypernetwork stores the sentences in random fragments and recalls a sentence by self-assemblying them given a partial, query sentence.
Applied to a sentence corpus of 290K sentences, we obtain a recall performance of 90-100%, depending on the difficulty of the task.
Cognitive plausibility: “Multiple representations of partially overlapping micromodules which are
partially active simultaneously” [Fuster, 2003] Neural microcircuits [Grillner et al, 2006] Cognitive schema or cognitive code [Tse et al., 2007]
Data Acquisition and Experimentation Text: Ha-Young Jang Image: Min-Oh Heo Sun Kim Joo-Kyoung Kim Ho-Sik Seok Kwonil Kim Sang-Yoon LeeSupported by - National Research Lab Program of Min. of Sci. & Tech. (2002-2007)
- Next Generation Tech. Program of Min. of Ind. & Comm. (2000-2010)
- BK21-IT Program of Min. of Education (2006-2009)
- SK Telecom (2007-2008)
More Information at - http://bi.snu.ac.kr/ Research MMG (to be open soon)
Acknowledgements
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
29
The Hypernetwork Model of The Hypernetwork Model of LearningLearning
Nn
K
iii
i
D
WWWW
SkXSSS
xxxX
WSXH
I
1)(
)()3()2(
,...,,
}{
:set Training
),...,,(
|| , ,
)(
),,(
as defined isrk hypernetwo The
21
x
)( 2121...21
21
21...21
321
321321
21
2121
321
321321
21
2121
2 ,...,,
)()()()(
2 ,...,,
)()()()(
,,
)()()()3(
,
)()()2(
)()(
,,
)()()()3(
,
)()()2()(
...)(
1exp)Z(
isfunction partition thewhere
,...)(
1exp
)Z(
1
...6
1
2
1exp
)Z(
1
)];(exp[)Z(
1 )|(
ondistributiy probabilit The
...6
1
2
1 );(
rkhypernetwo theofenergy The
m kkiiikiii
k
kiiikiii
iiiiiiiiii
iiiiiiiiii
K
k iii
mmmk
K
k iii
nnnk
iii
nnn
ii
nn
nn
iii
nnn
ii
nnn
xxxwkc
W
xxxwkcW
xxxwxxwW
WEW
WP
xxxwxxwWE
x
xx
x
[Zhang, 2006]
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
30
Deriving the Learning RuleDeriving the Learning Rule
N
n
K
k iii
nnnk
N
n
Kn
Nn
WZxxxwkc
WWWP
WP
k
kiiikiii1 2 ,...,,
)()()()(
1
)()3()2()(
1)(
)(ln...)(
1exp
),...,,|(ln
)|}({ln
21
21...21
x
x
)|}({ln 1)(
)(
...21
WPw
Nns
siii
x
N
n
Nn
WP
WP
1
(n)
1)(
)|(
)|}({
x
x
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
31
Derivation of the Learning RuleDerivation of the Learning Rule
xx
x
x
x
x
)|(......
...1
...
where
......
......
)(ln...)(
1exp
)(ln...)(
1exp
)|}({ln
2121
2121
2121
2121
...2121
21...21
...21
21
21...21
...21
...21
)|(
1
)()()(
)|(
1)|(
)()()(
1)(
2 ,...,,
)()()()()(
1 2 ,...,,
)()()()()(
1)(
)(
WPxxxxxx
xxxN
xxx
xxxxxxN
xxxxxx
WZw
xxxwkcw
WZxxxwkcw
WPw
siiis
siiis
ss
ssiii
siiik
kiiikiii
siii
k
kiiikiii
siii
siii
WPiii
N
n
nnn
Dataiii
WPiiiDataiii
N
nWPiii
nnn
N
ns
K
k iii
nnnks
N
n
K
k iii
nnnks
Nns
Molecular Self-AssemblyMolecular Self-Assembly
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
33
Encoding a Hypernetwork with Encoding a Hypernetwork with DNADNA
z1 :
z2 :
z3 :
z4 :
b)
x1
x2
x3
x4
x5
y
0
1
where
z1 : (x1=0, x2=1, x3=0, y=1)z2 : (x1=0, x2=0, x3=1, x4=0, x5=0, y=0)z3 : (x2=1, x4=1, y=1)z4 : (x2=1, x3=0, x4=1, y=0)
a)
AAAACCAATTGGAAGGCCATGCGG
AAAACCAATTCCAAGGGGCCTTCCCCAACCATGCCC
AATTGGCCTTGGATGCGG
AATTGGAAGGCCCCTTGGATGCCC
GG
AAAA
AATT
AAGG
CCTT
CCAA
ATGC
CC
Collection of (labeled) hyperedges
Library of DNA molecules corresponding to (a)
© 2008, SNU Biointelligence Lab, http://bi.snu.ac.kr/
34
DNA Molecular ComputingDNA Molecular Computing
Self-assembly
Heat
Cool
Polymer
Repeat
Self-replication
Molecular recognitionNanostructure
top related