Unsupervised Mining of Analogical Frames by Constraint Satisfaction

Lance De Vine Shlomo Geva Peter BruzaQueensland University of Technology

Brisbane, Australia{l.devine, s.geva, p.bruza}@qut.edu.au

Abstract

It has been demonstrated that vector-basedrepresentations of words trained on large textcorpora encode linguistic regularities that maybe exploited via the use of vector space arith-metic. This capability has been extensivelyexplored and is generally measured via taskswhich involve the automated completion oflinguistic proportional analogies. The ques-tion remains, however, as to what extent it ispossible to induce relations from word embed-dings in a principled and systematic way, with-out the provision of exemplars or seed terms.In this paper we propose an extensible and effi-cient framework for inducing relations via theuse of constraint satisfaction. The method isefficient, unsupervised and can be customizedin various ways. We provide both quantitativeand qualitative analysis of the results.

1 Introduction

The use and study of analogical inference andstructure has a long history in linguistics, logic,cognitive psychology, scientific reasoning and ed-ucation (Bartha, 2016), amongst others. The useof analogy has played an especially importantrole in the study of language, language change,and language acquisition and learning (Kiparsky,1992). It has been a part of the study of phonology,morphology, orthography and syntactic grammar(Skousen, 1989), as well as the development ofapplications such as machine translation and para-phasing (Lepage and Denoual, 2005).

Recent progress with the construction of vec-tor space representations of words based on theirdistributional profiles has revealed that analogi-cal structure can be discovered and operationalisedvia the use of vector space algebra (Mikolov et al.,2013b). There remain many questions regardingthe extent to which word vectors encode analog-ical structure and also the extent to which this

structure can be uncovered. For example, we arenot aware of any proposal or system that is fo-cussed on the unsupervised and systematic discov-ery of analogies from word vectors that does notmake use of exemplar relations, existing linguisticresources or seed terms. The automatic identifica-tion of linguistic analogies, however, offers manypotential benefits for a diverse range of researchand applications, including language learning andcomputational creativity.

Computational models of analogy have beenstudied since at least the 1960’s (Hall, 1989;French, 2002) and have addressed tasks relatingto both proportional and structural analogy. Manycomputational systems are built as part of inves-tigations into how humans might perform analog-ical inference (Gentner and Forbus, 2011). Mostmake use of Structure Mapping Theory (SMT)(Gentner, 1983) or a variation thereof which mapsone relational system to another, generally using asymbolic representation. Other systems use vectorspace representations constructed from corpora ofnatural language text (Turney, 2013). The analo-gies that are computed using word embeddingshave primarily been proportional analogies and areclosely associated with the prediction of relationsbetween words. For example, a valid semanticproportional analogy is “cat is to feline as dog isto canine” which can be written as “cat : feline ::dog : canine.”

In linguistics proportional analogies have beenextensively studied in the context of both in-flectional and derivational morphology (Blevins,2016). Proportional analogies are used as partof an inference process to fill the cells/slots in aword paradigm. A paradigm is an array of mor-phological variations of a lexeme. For example,{cat, cats} is a simple singular-noun, plural-nounparadigm in English. Word paradigms exhibitinter-dependencies that facilitate the inference of

Lance De Vine, Shlomo Geva and Peter Bruza. 2018. Unsupervised Mining of Analogical Frames by ConstraintSatisfaction. In Proceedings of Australasian Language Technology Association Workshop, pages 34−43.

new forms and for this reason have been studiedwithin the context of language change. The in-formativeness of a form correlates with the degreeto which knowledge of the form reduces uncer-tainty about other forms within the same paradigm(Blevins et al., 2017).

In this paper we propose a construction whichwe call an analogical frame. It is intended to elicitassociations with the terms semantic frame andproportional analogy. It is an extension of a lin-guistic analogy in which the elements satisfy cer-tain constraints that allow them to be induced in anunsupervised manner from natural language text.We expect that analogical frames will be usefulfor a variety of purposes relating to the automatedinduction of syntactic and semantic relations andcategories .

The primary contributions of this paper are two-fold:

1. We introduce a generalization of proportionalanalogies with word embeddings which wecall analogical frames.

2. We introduce an efficient constraint satisfac-tion based approach to inducing analogicalframes from natural language embeddings inan unsupervised fashion.

In section 2 we present background and relatedresearch. In section 3 we present and explain theproposal of Analogical Frames. In section 4 wepresent methods implemented for ensuring searchefficiency of Analogical Frames. In section 5 wepresent some analysis of empirical results. In sec-tion 6 we present discussion of the proposal and insection 7 we conclude.

2 Background and Related Work

2.1 Proportional Analogies

A proportional analogy is a 4-tuple which we writeas x1 : x2 :: x3 : x4 and read as “x1 is to x2 asx3 is to x4”, with the elements of the analogy be-longing to some domain X (we use this notationas it is helpful later). From here-on we will usethe term “analogy” to refer to proportional analo-gies unless indicated otherwise. Analogies can bedefined over different types of domains, for ex-ample, strings, geometric figures, numbers, vec-tor spaces, images etc. (Stroppa and Yvon, 2005)propose a definition of proportional analogy overany domain which is equipped with an internal

composition law⊕making it a semi-group (X ,⊕).This definition also applies to any richer algebraicstructure such as groups or vector spaces. In Rn,given x1, x2 and x3 there is always only one pointthat can be assigned to x4 such that proportionalityholds. (Miclet et al., 2008) define a relaxed formof analogy which reads as “x1 is to x2 almost asx3 is to x4”. To accompany this they introduce ameasure of analogical dissimilarity (AD) which isa positive real value and takes the value 0 whenthe analogy holds perfectly. A set of four pointsRn can therefore be scored for analogical dissimi-larity and ranked.

2.2 Word Vectors and ProportionalAnalogies

The background just mentioned provides a use-ful context within which to place the work on lin-guistic regularities in word vectors (Mikolov et al.,2013b; Levy et al., 2014). (Mikolov et al., 2013b)showed that analogies can be completed using vec-tor addition of word embeddings. This means thatgiven x1, x2 and x3 it is possible to infer the valueof x4. This is accomplished with the vector offsetformula, or 3COSADD (Levy et al., 2014).

argmaxx4

s(x4, x2 + x3 − x1) 3CosAdd

The s in 3COSADD is a similarity measure. Inpractice unit vectors are generally used with co-sine similarity. (Levy et al., 2014) introduced anexpression 3COSMUL which tends to give a smallimprovement when evaluated on analogy comple-tion tasks.

argmaxx4

s(x4, x3) · s(x4, x2)s(x4, x1) + ε

3COSMUL

3COSADD and 3COSMUL are effectively scor-ing functions that are used to judge the correctnessof a value for x4 given values for x1, x2 and x3.

2.3 Finding Analogies

Given that it is possible to complete analogies withword vectors it is natural to ask whether analogiescan be identified without being given x1, x2 andx3. (Stroppa and Yvon, 2005) considers an anal-ogy to be valid when analogical proportions holdbetween all terms in the analogy. They describe afinite-state solver which searches for formal analo-gies in the domain of strings and trees. As notedby several authors (Lavallee and Langlais, 2009;

35

Langlais, 2016; Beltran et al., 2015) a brute forcesearch to discovering proportional analogies iscomputationally difficult, with the complexity ofa naive approach being at least O(n3) and perhapsO(n4). A computational procedure must at leasttraverse the space of all 3-tuples if assuming thatthe 4th term of an analogy can be efficiently in-ferred.

For example, if we use a brute force approachto discovering linguistic analogies using a vocab-ulary of 100 000 words, we would need to ex-amine all combinations of 4-tuples, or 1000004,or 1020 combinations. Various strategies may be

x1 x2

x3 x4

Figure 1: A Proportional Analogy template, with 4variables to be assigned appropriate values

considered for making this problem tractable. Thefirst observation is that symmetries of an analogyshould not be recomputed (Lepage, 2014). Thiscan reduce the compute time by 8 for a single anal-ogy as there are 8 symmetric configurations.

Another proposed strategy is the construction ofa feature tree for the rapid computation and anal-ysis of tuple differences over vectors with binaryattributes (Lepage, 2014). This method was usedto discover analogies between images of Chinesecharacters. This has complexity O(n2). It com-putes the n(n+1)

2 vectors between pairs of tuplesand collects them together into clusters contain-ing the same difference vector. A variation of thismethod was reported in (Fam and Lepage, 2016)and (Fam and Lepage, 2017) for automatically dis-covering analogical grids of word paradigms us-ing edit distances between word strings. It is notimmediately obvious, however, how to extend thisto the case of word embeddings where differencesbetween word representations are real valued vec-tors.

A related method is used in (Beltran et al., 2015)for identifying analogies in relational databases.It is less constrained as it uses analogical dis-similarity as a metric when determining validanalogies.

(Langlais, 2016) extend methods from (Lepage,

2014) to scale to larger datasets for the purposeof machine translation, but also limit themselvesto the formal or graphemic level instead of moregeneral semantic relations between words.

2.4 Other Related Work

The present work is related to a number of re-search themes in language learning, relationallearning and natural language processing. We pro-vide a small sample of these.

(Holyoak and Thagard, 1989) introduce the useof constraint satisfaction as a key requirement formodels of analogical mapping. Their computerprogram ACME (Analogical Constraint MappingEngine) uses a connectionist network to balancestructural, semantic and pragmatic constraints formapping relations. (Hummel and Holyoak, 1997)propose a computational model of analogical in-ference and schema induction using distributedpatterns for representing objects and predicates.(Doumas et al., 2008) propose a computationalmodel which provides an account of how struc-tured relation representations can be learned fromunstructured data. More specifically to languageacquisition, (Bod, 2009) uses analogies over treesto derive and analyse new sentences by combin-ing fragments of previously seen sentences. Theproposed framework is able to replicate a range ofphenomena in language acquisition.

From a more cognitive perspective (Kurtz et al.,2001) investigates how mutual alignment of twosituations can create better understanding of both.Related to this (Gentner, 2010), and many others,argue that analogical ability is the key factor in hu-man cognitive development.

(Turney, 2006, 2013) makes extensive investi-gations of the use of corpus based methods fordetermining relational similarities and predictinganalogies.

(Miclet and Nicolas, 2015) propose the conceptof an analogical complex which is a blend of ana-logical proportions and formal concept analysis.

More specifically in relation to word embed-dings, (Zhang et al., 2016) presents an unsu-pervised approach for explaining the meaning ofterms via word vector comparison.

In the next section we describe an approachwhich addresses the task of inducing analogies inan unsupervised fashion from word vectors andbuilds on existing work relating to word embed-dings and linguistic regularities.

36

3 Analogical Frames

The primary task that we address in this paper isthe discovery of linguistic proportional analogiesin an unsupervised fashion given only the distri-butional profile of words. The approach that wetake is to consider the problem as a constraint sat-isfaction problem (CSP) (Rossi et al., 2006). Weincrease the strength of constraints until we canaccurately decide when a given set of words andtheir embeddings forms a valid proportional anal-ogy. At this point we introduce some terminology:

Constraint satisfaction problems are generallydefined as a triple P = 〈X,D,C〉 whereX = {x1, ..., xn} is a set of variables.D = {d1, ..., dn} is the set of domains associatedwith the variables.C = {c1, ..., cm} is the set of constraints to be sat-isfied.

A solution to a constraint satisfaction prob-lem must assign a single value to each variablex1, ..., xn in the problem. There may be multiplesolutions. In our problem formulation there is onlyone domain which is the set of word types whichcomprise the vocabulary. Each variable must beassigned a word identifier and each word is as-sociated with one or more vector space represen-tations. Constraints on the words and associatedvector space representation limit the values thatthe variables can take. From here-on we will weuse the bolded symbol xi to indicate the vectorspace value of the word assigned to the variablexi.

In our proposal we use the following fiveconstraints.

C1. AllDiff constraint. The Alldiff constraintconstrains all terms of the analogy to be distinct(The Alldiff constraint is a common constraint inCSPs), such that xi 6= xj for all 1 < i < j < n.

C2. Asymmetry constraint. The asymmetry con-straint (Meseguer and Torras, 2001) is used toeliminate unnecessary searches in the search tree.It is defined as a partial ordering on the values ofa subset of the variables. In the case of a 2x3 ana-logical frame (figure 2), for example, we define theordering as:

x1 ≺ x2 ≺ x3, and x1 ≺ x4.where the ordering is defined on the integer

identifiers of the words in the vocabulary.C3. Neighbourhood Constraint. The neigh-

bourhood constraint is used to constrain the value

of variables to the words which are within thenearest neighbourhood of the words to which theyare connected. We define this as:

xi ∈ Neight(xj) and xj ∈ Neight(xi)

whereNeight(xi) is the nearest neighbourhoodof t words of the the word assigned to xi, asmeasured in the vector space representation of thewords.

C4. Parallel Constraint. The Parallel Con-straint forces opposing difference vectors to havea minimal degree of parallelism.

x2 − x1 · x5 − x4 < pThreshold

where pThreshold is a parameter. For the paral-lel constraint we ensure that the difference vectorx2 − x1 has a minimal cosine similarity to thedifference vector x5 − x1. This constraint over-laps to some extent with the proportionality con-straint (below), however it serves a different pur-pose, which is to eliminate low probability candi-date analogies.

C5. Proportionality Constraint. The Propor-tionality Constraint constrains the vector spacerepresentation of words to form approximate ge-ometric proportional analogies. It is a quarternaryconstraint. For any given 4-tuple, we use the con-cept of “inter-predictability” (Blevins et al., 2017)to decide whether the 4-tuple is acceptable. Weenforce inter-predictability by requiring that eachterm in a 4-tuple is predicted by the other threeterms. This implies four analogy completion taskswhich must be satisfied for the variable assign-ment to be accepted.

x1 : x2 :: x3 : x⇒ x = x4

x2 : x1 :: x4 : x⇒ x = x3

x3 : x4 :: x1 : x⇒ x = x2

x4 : x3 :: x2 : x⇒ x = x1

(Stroppa and Yvon, 2005) use a similar ap-proach with exact formal analogies. With ourapproach, however, we complete analogies usingword vectors and analogy completion formulas(eg. using 3COSADD or 3COSMUL, or a deriva-tive).

The proportionality constraint is a relatively ex-pensive constraint to enforce as it requires manyvector-vector operations and comparison againstall word vectors in the vocabulary. This constraintmay be checked approximately which we discussin the next section.

37

3.1 The Insufficiency of 2x2 AnalogiesWhen we experiment with discovering 2x2 analo-gies using the constraints just described we findthat we can’t easily set the parameters of the con-straints such that only valid analogies are pro-duced, without severely limiting the types ofanalogies which we accept. For example, the fol-lowing analogy is produced “oldest : old :: earliest: earlier”. We find that these types of mistakes arecommon. We therefore take the step of expandingthe number of variables so that the state space isnow larger (figure 2).

x11

4

x2

4

2x3

4

3

x41

x52

x63

Figure 2: A 2x3 Analogical Frame

The idea is to increase the inductive support foreach discovered analogy by requiring that analo-gies be part of a larger system of analogies. Werefer to the larger system of analogies as an Ana-logical Frame. It is important to note that in figure2, x1 and x3 are connected. The numbers asso-ciated with the edges indicate aligned vector dif-ferences. It is intended that the analogical pro-portions hold according to the connectivity shown.For example, proportionality should hold such thatx1 : x2 :: x4 : x5 and x1 : x3 :: x4 : x6 andx2 : x3 :: x5 : x6. It is also important to notethat while we have added only two new variables,we have increased the number of constraints by al-most 3 times. It is not exactly 3 because there issome redundancy in the proportions.

3.2 New FormulasWe now define a modified formula for complet-ing analogies that are part of a larger systems ofanalogies such as the analogical frame in figure 2.It can be observed that 3COSADD and 3COSMUL

effectively assigns a score to vocabulary items andthen selects the item with the largest score. Wedo the same but with a modified scoring functionwhich is a hybrid of 3COSADD and 3COSMUL.3COSADD or 3COSMUL could both be used aspart of our approach, but the formula which wepropose, better captures the intuition of larger sys-tems of analogies where the importance is placedon 1) symmetry, and 2) average offset vectors.

When we are not given any prior knowledge, orexemplar, there is no privileged direction withinan analogical frame. For example, in figure 2, thepair (x2, x5) has the same importance as (x4, x5).

We first construct difference vectors and thenaverage those that are aligned.

dif2,1 = x2 − x1 dif4,1 = x4 − x1

dif5,4 = x5 − x4 dif5,2 = x5 − x2

dif6,3 = x6 − x3

difsum1 = dif2,1 + dif5,4

difsum2 = dif4,1 + dif5,2 + dif6,3

dif1 =difsum1

|difsum1|

dif2 =difsum2

|difsum2|The vector dif1 is the normalized average off-

set vector indicated with a 1 in figure 2. The vectordif2 is the normalized average offset vector indi-cated with a 4 in figure 2.

Using these normalized average difference vec-tors we define the scoring function for selecting x5given x1, x2 and x4 as:

argmaxx5

s(x5,x4) · s(x5,dif1) (1)

+ s(x5,x2) · s(x5,dif2)

The formulation makes use of all informationavailable in the analogical frame. Previous workhas provided much evidence for the linear com-positionality of word vectors as embodied by3COSADD (Vylomova et al., 2015; Hakami et al.,2017). It has also been known since (Mikolovet al., 2013a) that averaging the difference vectorsof pairs exhibiting the same relation results in adifference vector with better predictive power forthat relation (Drozd et al., 2016).

Extrapolating from figure 2 larger analogicalframes can be constructed, such as 2 x 4, 3 x 3,or 2 x 2 x 3. Each appropriately connected 4-tuplecontained within the frame should satisfy the pro-portionality constraint.

4 Frame Discovery and Search Efficiency

The primary challenge in this proposal is to effi-ciently search the space of variable assignments.We use a depth first approach to cover the searchspace as well several other strategies to make thissearch efficient.

38

4.1 Word Embedding Neighbourhoods

The most important strategy is the concentrationof compute resources on the nearest neighbour-hoods of word embeddings. Most analogies in-volve terms that are within the nearest neighbor-hood of each other when ranked according to sim-ilarity (Linzen, 2016). We therefore compute thenearest neighbour graph of every term in the vo-cabulary as a pre-processing step and store the re-sult as an adjacency list for each vocabulary item.We do this efficiently by using binary represen-tations of the word embeddings (Jurgovsky et al.,2016) and re-ranking using the full precision wordvectors. The nearest neighbour list of each vocab-ulary entry is used in two different ways, 1) for ex-ploring the search space of variable assignments,and 2) efficiently eliminating variable assignments(next section) that do not satisfy the proportionalanalogy constraints.

When traversing the search tree we use the ad-jacency list of an already assigned variable to as-sign a value to a nearby variable in the frame. Thebreadth of the search tree at each node is thereforeparameterized by a global parameter t assigned bythe user. The parameter t is one of the primary pa-rameters of the algorithm and will determine thelength of the search and the size of the set of dis-covered frames.

4.2 Elimination of Candidates by Sampling

A general principle used in most CSP solvingis the quick elimination of improbable solutions.When we assign values to variables in a frame weneed to make sure that the proportional analogyconstraint is satisfied. For four given variable as-signmentsw1, w2, w3 andw4 this involves makingsure that w4 is the best candidate for completingthe proportional analogy involving w1, w2 and w3.This is equivalent to knowing if there are any bet-ter vocabulary entries than w4 for completing theanalogy. Instead of checking all vocabulary en-tries we can limit the list of entries to the m near-est neighbours of w2, w3 and w4

1 to check if anyscore higher than w4. If any of them score higherthe proportional analogy constraint is violated andthe variable assignment is discarded. The advan-tage of this is that we only need to check 3 x mentries to approximately test the correctness ofw4.

1assuming w1 is farthest away from w4 and that the near-est neighbours of w1 are not likely to help check the correct-ness of w4

We find that if we set m to 10 then most incorrectanalogies are eliminated. An exhaustive check forcorrectness can be made as a post processing step.

4.3 Other Methods

Another important method for increasing effi-ciency is testing the degree to which opposing dif-ference vectors in a candidate proportional anal-ogy are parallel to each other. This is encapsulatedin constraint C4. While using parallelism as a pa-rameter for solving word analogy problems hasnot been successful (Levy et al., 2014), we havefound that the degree of parallelism is a good in-dicator of the confidence that we can have in theanalogy once other constraints have been satisfied.Other methods employed to improve efficiency in-clude the use of Bloom filters to test neighbour-hood membership and the indexing of discoveredproportions so as not to repeat searching.

4.4 Extending Frames

Frames can be extended by extending the initialbase frame in one or more directions and search-ing for additional variable assignments that satisfyall constraints. For example, a 2x3 frame can beextended to a 2x4 frame by assigning values to twonew variables x7 and x8 (figure 3).

x1 x2 x3 x7//

x4 x5 x6 x8//

Figure 3: Extending a 2x3 frame

As frames are extended the average offset vec-tors (equation 1) are recomputed so that the off-set vectors become better predictors for computingproportional analogies.

5 Experimental Setup and Results

The two criteria we use to measure our proposalinclude a) accuracy of analogy discovery, b) com-pute scalability. Other criteria are also possiblesuch as relation diversity and interestingness of re-lations.

The primary algorithm parameters are 1) Thenumber of terms in the vocabulary to search, 2)the size of the nearest neighbourhood of each termto search, 3) the degree of parallelism required foropposing vector differences and 4) whether to ex-tend base frames.

39

5.1 Retrieving Analogical Completions fromFrames

When frames are discovered they are stored in aFrame Store, a data structure used to efficientlystore and retrieve frames. Frames are indexed us-ing posting lists similar to a document index. Tocomplete an incomplete analogy, all frames whichcontain all terms in the incomplete analogy are re-trieved. For each retrieved frame the candidateterm for completing the analogy is determined bycross-referencing the indices of the terms withinthe frame (figure 4). If diverse candidate termsare selected from multiple frames, voting is usedto select the final analogy completion, or randomselection in the case of tied counts.

a1,1 : a1,3 :: a3,1 :?

a1,1 a1,2 a1,3a2,1 a2,2 a2,3a3,1 a3,2 a3,3

Figure 4: Determining an analogy completion froma larger frame

We conducted experiments with embeddingsconstructed by ourselves as well as with publiclyaccessible embeddings from the fastText web site2

trained on 600B tokens of the Common Crawl(Mikolov et al., 2018) .

We evaluated the accuracy of the frames by at-tempting to complete the analogies from the wellknown Google analogy test set.3 The greatestchallenge in this type of evaluation is adequatelycovering the evaluation items. At least three of theterms in an analogy completion item need to be si-multaneously present in a single frame for the itemto be attempted. We report the results of a typicalexecution of the system using a nearest neighbour-hood size of 25, a maximum vocabulary size of 50000, and a minimal cosine similarity between op-posing vector differences of 0.3. For this set ofparameters, 8589 evaluation items were answeredby retrieval from the frame store, covering approx-imately 44% of the evaluation items (table 1). Ap-proximately 30% of these were from the semanticcategory, and 70% from the syntactic. We com-pared the accuracy of completing analogies usingthe frame store, to the accuracy of both 3CosAdd

2https://fasttext.cc/docs/en/english-vectors.html3http://download.tensorflow.org/data/questions-words.txt

Table 1: Analogy Completion on Google Subset

3CosAdd 3CosMul FramesSem. (2681) 2666 2660 2673Syn. (5908) 5565 5602 5655Tot. (8589) 8231 8262 8328

and 3CosMul using the same embeddings as usedto build the frames.

Results show that the frames are slightly moreaccurate than 3CosAdd and 3CosMul, achieving96.9% on the 8589 evaluation items. It needs tobe stressed, however, that the objective is not tooutperform vector arithmetic based methods, butrather to verify that the frames have a high degreeof accuracy.

To better determine the accuracy of the discov-ered frames we also randomly sampled 1000 of the21571 frames generated for the results shown intable 2, and manually checked them. The raw out-puts are included in the online repository4. Theseframes cover many relations not included in theGoogle analogy test set. We found 9 frames witherrors giving an accuracy of 99.1%.

It should be noted that the accuracy of frames isinfluenced by the quality of the embeddings. How-ever, even with embeddings trained on small cor-pora it is possible to discover analogies providedthat sufficient word embedding training epochshave been completed.

The online repository contains further empiricalevaluations and explanations regarding parameterchoices, including raw outputs and errors made bythe system.

5.2 Scaling: Effect of Neighbourhood Sizeand pThreshold

Tables 2 and 3 show the number of frames discov-ered on typical executions of the software. The re-ported numbers are intended to give an indicationof the relationship between neighbourhood size,number of frames produced and execution time.The reported times are for 8 software threads.

5.3 Qualitative Analysis

From inspection of the frames we see that a largepart of the relations discovered are grammatical ormorpho-syntactic, or are related to high frequency

4https://github.com/ldevine/AFM

40

Table 2: Par Thres = 0.3 (Less Constrained)

Near. Sz. 15 20 25 30Frames 13282 16785 19603 21571Time (sec) 20.1 29.3 38.3 45.9

Table 3: Par Thres = 0.5 (More Constrained)

Near. Sz. 15 20 25 30Frames 6344 7995 9286 10188Time (sec) 10.4 15.7 21.1 26.0

entities. However we also observe a large num-ber of other types of relations such as synonyms,antonyms, domain alignments and various syntac-tic mappings. We provide but a small sample be-low.

eyes eye blindnessears ear deafness

Father Son HimselfMother Daughter Herself

geschichte gesellschaft musikhistoire societe musique

aircraft flew skies airships sailed seas naval

always constantly constantoften frequently frequent

sometimes occasionally occasional

faint weak

fainter weaker

bright strong

brighter stronger

(2 x 2 x 2 Frame)

We have also observed reliable mappings withembeddings trained on non-English corpora.

The geometry of analogical frames can also beexplored via visualizations of projections onto 2Dsub-spaces derived from the offset vectors. 5

6 Discussion

We believe that the constraint satisfaction ap-proach introduced in this paper is advantageousbecause it is a systematic but flexible and can makeuse of methods from the constraint satisfactiondomain. We have only mentioned a few of theprimary CSP concepts in this paper. Other con-straints can be included in the formulation such

5Examples provided in the online repository

as set membership constraints where sets may beclusters, or documents.

One improvement that could be made to the pro-posed system is to facilitate the discovery of re-lations that are not one-to-one. While we foundmany isolated examples of one-to-many relationsexpressed in the frames, a strictly symmetricalproportional analogy does not seem ideal for cap-turing one-to-many relations.

As outlined by (Turney, 2006) there are manyapplications of automating the construction and/ordiscovery of analogical relations. Some of theseinclude relation classification, metaphor detection,word sense disambiguation, information extrac-tion, question answering and thesaurus generation.

Analogical frames should also provide insightinto the geometry of word embeddings and mayprovide an interesting way to measure their qual-ity.

The most interesting application of the system isin the area of computational creativity with a hu-man in the loop. For example, analogical framescould be chosen for their interestingness and thenexpanded.

6.1 Software and Online RepositoryThe software implementing the proposed systemas a set of command line applications can be foundin the online repository6. The software is imple-mented in portable C++11 and compiles on bothWindows and Unix based systems without com-piled dependencies. Example outputs of the sys-tem as well as parameter settings are provided inthe online repository including the outputs createdfrom embeddings trained on a range of corpora.

7 Future Work and Conclusions

Further empirical evaluation is required. Theestablishment of more suitable empirical bench-marks for assessing the effectiveness of openanalogy discovery is important. The most inter-esting potential application of this work is in thecombination of automated discovery of analogiesand human judgment. There is also the possibilityof establishing a more open-ended computearchitecture that could search continuously foranalogical frames in an online fashion.

6 https://github.com/ldevine/AFM

41

ReferencesPaul Bartha. 2016. Analogy and analogical reasoning.

The Stanford Encyclopedia of Philosophy.

William Correa Beltran, Helene Jaudoin, and OlivierPivert. 2015. A clustering-based approach to themining of analogical proportions. In Tools with Ar-tificial Intelligence (ICTAI), 2015 IEEE 27th Inter-national Conference on, pages 125–131. IEEE.

James P Blevins. 2016. Word and paradigm morphol-ogy. Oxford University Press.

James P Blevins, Farrell Ackerman, Robert Malouf,J Audring, and F Masini. 2017. Word and paradigmmorphology.

Rens Bod. 2009. From exemplar to grammar: A prob-abilistic analogy-based model of language learning.Cognitive Science, 33(5):752–793.

Leonidas AA Doumas, John E Hummel, and Cather-ine M Sandhofer. 2008. A theory of the discoveryand predication of relational concepts. Psychologi-cal review, 115(1):1.

Aleksandr Drozd, Anna Gladkova, and Satoshi Mat-suoka. 2016. Word embeddings, analogies, and ma-chine learning: Beyond king-man+ woman= queen.In Proceedings of COLING 2016, the 26th Inter-national Conference on Computational Linguistics:Technical Papers, pages 3519–3530.

Rashel Fam and Yves Lepage. 2016. Morphologicalpredictability of unseen words using computationalanalogy.

Rashel Fam and Yves Lepage. 2017. A study of thesaturation of analogical grids agnostically extractedfrom texts.

Robert M French. 2002. The computational modelingof analogy-making. Trends in cognitive Sciences,6(5):200–205.

Dedre Gentner. 1983. Structure-mapping: A theo-retical framework for analogy. Cognitive science,7(2):155–170.

Dedre Gentner. 2010. Bootstrapping the mind: Ana-logical processes and symbol systems. CognitiveScience, 34(5):752–775.

Dedre Gentner and Kenneth D Forbus. 2011. Compu-tational models of analogy. Wiley interdisciplinaryreviews: cognitive science, 2(3):266–276.

Huda Hakami, Kohei Hayashi, and Danushka Bolle-gala. 2017. An optimality proof for the pairdiff oper-ator for representing relations between words. arXivpreprint arXiv:1709.06673.

Rogers P Hall. 1989. Computational approaches toanalogical reasoning: A comparative analysis. Ar-tificial intelligence, 39(1):39–120.

Keith J Holyoak and Paul Thagard. 1989. Analogicalmapping by constraint satisfaction. Cognitive sci-ence, 13(3):295–355.

John E Hummel and Keith J Holyoak. 1997. Dis-tributed representations of structure: A theory ofanalogical access and mapping. Psychological re-view, 104(3):427.

Johannes Jurgovsky, Michael Granitzer, and ChristinSeifert. 2016. Evaluating memory efficiency and ro-bustness of word embeddings. In European Con-ference on Information Retrieval, pages 200–211.Springer.

Paul Kiparsky. 1992. Analogy. International Encyclo-pedia of Linguistics.

Kenneth J Kurtz, Chun-Hui Miao, and Dedre Gentner.2001. Learning by analogical bootstrapping. TheJournal of the Learning Sciences, 10(4):417–446.

Philippe Langlais. 2016. Efficient identification of for-mal analogies.

Jean-Francois Lavallee and Philippe Langlais. 2009.Unsupervised morphological analysis by formalanalogy. In Workshop of the Cross-Language Eval-uation Forum for European Languages, pages 617–624. Springer.

Yves Lepage. 2014. Analogies between binary im-ages: Application to chinese characters. In Compu-tational Approaches to Analogical Reasoning: Cur-rent Trends, pages 25–57. Springer.

Yves Lepage and Etienne Denoual. 2005. Purest everexample-based machine translation: Detailed pre-sentation and assessment. Machine Translation,19(3-4):251–282.

Omer Levy, Yoav Goldberg, and Israel Ramat-Gan.2014. Linguistic regularities in sparse and explicitword representations. In CoNLL, pages 171–180.

Tal Linzen. 2016. Issues in evaluating seman-tic spaces using word analogies. arXiv preprintarXiv:1606.07736.

Pedro Meseguer and Carme Torras. 2001. Exploit-ing symmetries within constraint satisfaction search.Artificial intelligence, 129(1-2):133–163.

Laurent Miclet, Sabri Bayoudh, and Arnaud Delhay.2008. Analogical dissimilarity: definition, algo-rithms and two experiments in machine learning.Journal of Artificial Intelligence Research, 32:793–824.

Laurent Miclet and Jacques Nicolas. 2015. From for-mal concepts to analogical complexes. In CLA 2015,page 12.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jef-frey Dean. 2013a. Efficient estimation of wordrepresentations in vector space. arXiv preprintarXiv:1301.3781.

42

Tomas Mikolov, Edouard Grave, Piotr Bojanowski,Christian Puhrsch, and Armand Joulin. 2018. Ad-vances in pre-training distributed word representa-tions. In Proceedings of the International Confer-ence on Language Resources and Evaluation (LREC2018).

Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig.2013b. Linguistic regularities in continuous spaceword representations. In HLT-NAACL, volume 13,pages 746–751.

Francesca Rossi, Peter Van Beek, and Toby Walsh.2006. Handbook of constraint programming. El-sevier.

Royal Skousen. 1989. Analogical modeling of lan-guage. Springer Science & Business Media.

Nicolas Stroppa and Franccois Yvon. 2005. An ana-logical learner for morphological analysis. In Pro-ceedings of the Ninth Conference on ComputationalNatural Language Learning, pages 120–127. Asso-ciation for Computational Linguistics.

Peter D Turney. 2006. Similarity of semantic relations.Computational Linguistics, 32(3):379–416.

Peter D Turney. 2013. Distributional semantics beyondwords: Supervised learning of analogy and para-phrase. arXiv preprint arXiv:1310.5042.

Ekaterina Vylomova, Laura Rimell, Trevor Cohn, andTimothy Baldwin. 2015. Take and took, gaggle andgoose, book and read: Evaluating the utility of vec-tor differences for lexical relation learning. arXivpreprint arXiv:1509.01692.

Yating Zhang, Adam Jatowt, and Katsumi Tanaka.2016. Towards understanding word embeddings:Automatically explaining similarity of terms. In BigData (Big Data), 2016 IEEE International Confer-ence on, pages 823–832. IEEE.

43