Towards Understanding the Geometry of Knowledge Graph Embeddings

ChandrahasIndian Institute of Sciencechandrahas@iisc.ac.in

Aditya SharmaIndian Institute of Scienceadityasharma@iisc.ac.in

Partha TalukdarIndian Institute of Science

ppt@iisc.ac.in

Abstract

Knowledge Graph (KG) embedding hasemerged as a very active area of researchover the last few years, resulting in thedevelopment of several embedding meth-ods. These KG embedding methods rep-resent KG entities and relations as vectorsin a high-dimensional space. Despite thispopularity and effectiveness of KG em-beddings in various tasks (e.g., link pre-diction), geometric understanding of suchembeddings (i.e., arrangement of entityand relation vectors in vector space) is un-explored – we fill this gap in the paper.We initiate a study to analyze the geome-try of KG embeddings and correlate it withtask performance and other hyperparame-ters. To the best of our knowledge, this isthe first study of its kind. Through exten-sive experiments on real-world datasets,we discover several insights. For example,we find that there are sharp differences be-tween the geometry of embeddings learntby different classes of KG embeddingsmethods. We hope that this initial studywill inspire other follow-up research onthis important but unexplored problem.

1 Introduction

Knowledge Graphs (KGs) are multi-relationalgraphs where nodes represent entities and typed-edges represent relationships among entities. Re-cent research in this area has resulted in the de-velopment of several large KGs, such as NELL(Mitchell et al., 2015), YAGO (Suchanek et al.,2007), and Freebase (Bollacker et al., 2008),among others. These KGs contain thousands ofpredicates (e.g., person, city, mayorOf(person,city), etc.), and millions of triples involving such

predicates, e.g., (Bill de Blasio, mayorOf, NewYork City).

The problem of learning embeddings forKnowledge Graphs has received significant atten-tion in recent years, with several methods beingproposed (Bordes et al., 2013; Lin et al., 2015;Nguyen et al., 2016; Nickel et al., 2016; Trouil-lon et al., 2016). These methods represent enti-ties and relations in a KG as vectors in high di-mensional space. These vectors can then be usedfor various tasks, such as, link prediction, entityclassification etc. Starting with TransE (Bordeset al., 2013), there have been many KG embed-ding methods such as TransH (Wang et al., 2014),TransR (Lin et al., 2015) and STransE (Nguyenet al., 2016) which represent relations as trans-lation vectors from head entities to tail entities.These are additive models, as the vectors interactvia addition and subtraction. Other KG embed-ding models, such as, DistMult (Yang et al., 2014),HolE (Nickel et al., 2016), and ComplEx (Trouil-lon et al., 2016) are multiplicative where entity-relation-entity triple likelihood is quantified by amultiplicative score function. All these methodsemploy a score function for distinguishing correcttriples from incorrect ones.

In spite of the existence of many KG embed-ding methods, our understanding of the geometryand structure of such embeddings is very shallow.A recent work (Mimno and Thompson, 2017) an-alyzed the geometry of word embeddings. How-ever, the problem of analyzing geometry of KGembeddings is still unexplored – we fill this impor-tant gap. In this paper, we analyze the geometry ofsuch vectors in terms of their lengths and conicity,which, as defined in Section 4, describes their po-sitions and orientations in the vector space. Welater study the effects of model type and traininghyperparameters on the geometry of KG embed-dings and correlate geometry with performance.

We make the following contributions:

• We initiate a study to analyze the geometry ofvarious Knowledge Graph (KG) embeddings.To the best of our knowledge, this is the firststudy of its kind. We also formalize variousmetrics which can be used to study geometryof a set of vectors.

• Through extensive analysis, we discover sev-eral interesting insights about the geometryof KG embeddings. For example, we findsystematic differences between the geome-tries of embeddings learned by additive andmultiplicative KG embedding methods.

• We also study the relationship between geo-metric attributes and predictive performanceof the embeddings, resulting in several newinsights. For example, in case of multiplica-tive models, we observe that for entity vec-tors generated with a fixed number of neg-ative samples, lower conicity (as defined inSection 4) or higher average vector lengthlead to higher performance.

Source code of all the analysis tools de-veloped as part of this paper is availableat https://github.com/malllabiisc/kg-geometry. We are hoping that these re-sources will enable one to quickly analyze thegeometry of any KG embedding, and potentiallyother embeddings as well.

2 Related Work

In spite of the extensive and growing literature onboth KG and non-KG embedding methods, verylittle attention has been paid towards understand-ing the geometry of the learned embeddings. A re-cent work (Mimno and Thompson, 2017) is an ex-ception to this which addresses this problem in thecontext of word vectors. This work revealed a sur-prising correlation between word vector geometryand the number of negative samples used duringtraining. Instead of word vectors, in this paper wefocus on understanding the geometry of KG em-beddings. In spite of this difference, the insightswe discover in this paper generalizes some of theobservations in the work of (Mimno and Thomp-son, 2017). Please see Section 6.2 for more details.

Since KGs contain only positive triples, nega-tive sampling has been used for training KG em-beddings. Effect of the number of negative sam-ples in KG embedding performance was studied

by (Toutanova et al., 2015). In this paper, we studythe effect of the number of negative samples onKG embedding geometry as well as performance.

In addition to the additive and multiplicativeKG embedding methods already mentioned inSection 1, there is another set of methods wherethe entity and relation vectors interact via a neu-ral network. Examples of methods in this cate-gory include NTN (Socher et al., 2013), CONV(Toutanova et al., 2015), ConvE (Dettmers et al.,2017), R-GCN (Schlichtkrull et al., 2017), ER-MLP (Dong et al., 2014) and ER-MLP-2n (Rav-ishankar et al., 2017). Due to space limitations,in this paper we restrict our scope to the analysisof the geometry of additive and multiplicative KGembedding models only, and leave the analysis ofthe geometry of neural network-based methods aspart of future work.

3 Overview of KG Embedding Methods

For our analysis, we consider six representativeKG embedding methods: TransE (Bordes et al.,2013), TransR (Lin et al., 2015), STransE (Nguyenet al., 2016), DistMult (Yang et al., 2014), HolE(Nickel et al., 2016) and ComplEx (Trouillonet al., 2016). We refer to TransE, TransR andSTransE as additive methods because they learnembeddings by modeling relations as translationvectors from one entity to another, which results invectors interacting via the addition operation dur-ing training. On the other hand, we refer to Dist-Mult, HolE and ComplEx as multiplicative meth-ods as they quantify the likelihood of a triple be-longing to the KG through a multiplicative scorefunction. The score functions optimized by thesemethods are summarized in Table 1.Notation: Let G = (E ,R, T ) be a KnowledgeGraph (KG) where E is the set of entities, R isthe set of relations and T ⊂ E × R × E is the setof triples stored in the graph. Most of the KG em-bedding methods learn vectors e ∈ Rde for e ∈ E ,and r ∈ Rdr for r ∈ R. Some methods alsolearn projection matrices Mr ∈ Rdr×de for rela-tions. The correctness of a triple is evaluated usinga model specific score function σ : E × R× E →R. For learning the embeddings, a loss functionL(T , T ′; θ), defined over a set of positive triplesT , set of (sampled) negative triples T ′, and theparameters θ is optimized.

We use small italics characters (e.g., h, r) torepresent entities and relations, and correspond-

Type Model Score Function σ(h, r, t)

AdditiveTransE (Bordes et al., 2013) −‖h + r− t‖1

TransR (Lin et al., 2015) −‖Mrh + r−Mrt‖1STransE (Nguyen et al., 2016) −

∥∥M1rh + r−M2

r t∥∥1

MultiplicativeDistMult (Yang et al., 2014) r>(h� t)

HolE (Nickel et al., 2016) r>(h ? t)

ComplEx (Trouillon et al., 2016) Re(r>(h� t))

Table 1: Summary of various Knowledge Graph (KG) embedding methods used in the paper. Please seeSection 3 for more details.

ing bold characters to represent their vector em-beddings (e.g., h, r). We use bold capitalization(e.g., V) to represent a set of vectors. Matrices arerepresented by capital italics characters (e.g., M ).

3.1 Additive KG Embedding MethodsThis is the set of methods where entity and rela-tion vectors interact via additive operations. Thescore function for these models can be expressedas below

σ(h, r, t) = −∥∥M1

r h + r−M2r t∥∥1

(1)

where h, t ∈ Rde and r ∈ Rdr are vectors forhead entity, tail entity and relation respectively.M1

r ,M2r ∈ Rdr×de are projection matrices from

entity space Rde to relation space Rdr .TransE (Bordes et al., 2013) is the simplest addi-tive model where the entity and relation vectors liein same d−dimensional space, i.e., de = dr = d.The projection matricesM1

r = M2r = Id are iden-

tity matrices. The relation vectors are modeled astranslation vectors from head entity vectors to tailentity vectors. Pairwise ranking loss is then usedto learn these vectors. Since the model is simple,it has limited capability in capturing many-to-one,one-to-many and many-to-many relations.TransR (Lin et al., 2015) is another translation-based model which uses separate spaces for en-tity and relation vectors allowing it to address theshortcomings of TransE. Entity vectors are pro-jected into a relation specific space using the cor-responding projection matrix M1

r = M2r = Mr.

The training is similar to TransE.STransE (Nguyen et al., 2016) is a generalizationof TransR and uses different projection matricesfor head and tail entity vectors. The training issimilar to TransE. STransE achieves better perfor-mance than the previous methods but at the cost ofmore number of parameters.

Equation 1 is the score function used inSTransE. TransE and TransR are special cases of

STransE with M1r = M2

r = Id and M1r = M2

r =Mr, respectively.

3.2 Multiplicative KG Embedding Methods

This is the set of methods where the vectors inter-act via multiplicative operations (usually dot prod-uct). The score function for these models can beexpressed as

σ(h, r, t) = r>f(h, t) (2)

where h, t, r ∈ Fd are vectors for head entity, tailentity and relation respectively. f(h, t) ∈ Fd mea-sures compatibility of head and tail entities andis specific to the model. F is either real space Ror complex space C. Detailed descriptions of themodels we consider are as follows.DistMult (Yang et al., 2014) models entities andrelations as vectors in Rd. It uses an entry-wiseproduct (�) to measure compatibility betweenhead and tail entities, while using logistic loss fortraining the model.

σDistMult(h, r, t) = r>(h� t) (3)

Since the entry-wise product in (3) is symmetric,DistMult is not suitable for asymmetric and anti-symmetric relations.HolE (Nickel et al., 2016) also models entities andrelations as vectors in Rd. It uses circular correla-tion operator (?) as compatibility function definedas

[h ? t]k =

d−1∑i=0

hit(k+i) mod d

The score function is given as

σHolE(h, r, t) = r>(h ? t) (4)

The circular correlation operator being asymmet-ric, can capture asymmetric and anti-symmetricrelations, but at the cost of higher time complexity

Figure 1: Comparison of high vs low Conicity. Randomly generated vectors are shown in blue withtheir sample mean vector M in black. Figure on the left shows the case when vectors lie in narrow coneresulting in high Conicity value. Figure on the right shows the case when vectors are spread out havingrelatively lower Conicity value. We skipped very low values of Conicity as it was difficult to visualize.The points are sampled from 3d Spherical Gaussian with mean (1,1,1) and standard deviation 0.1 (left)and 1.3 (right). Please refer to Section 4 for more details.

(O(d log d)). For training, we use pairwise rank-ing loss.ComplEx (Trouillon et al., 2016) represents enti-ties and relations as vectors in Cd. The compati-bility of entity pairs is measured using entry-wiseproduct between head and complex conjugate oftail entity vectors.

σComplEx(h, r, t) = Re(r>(h� t)) (5)

In contrast to (3), using complex vectors in (5) al-lows ComplEx to handle symmetric, asymmetricand anti-symmetric relations using the same scorefunction. Similar to DistMult, logistic loss is usedfor training the model.

4 Metrics

For our geometrical analysis, we first define a term‘alignment to mean’ (ATM) of a vector v belong-ing to a set of vectors V, as the cosine similarity1

between v and the mean of all vectors in V.

ATM(v,V) = cosine

(v,

1

|V|∑x∈V

x

)We also define ‘conicity’ of a set V as the meanATM of all vectors in V.

Conicity(V) =1

|V|∑v∈V

ATM(v,V)

1cosine(u, v) = u>v‖u‖‖v‖

Dataset FB15k WN18#Relations 1,345 18#Entities 14,541 40,943

#TriplesTrain 483,142 141,440

Validation 50,000 5,000Test 59,071 5,000

Table 2: Summary of datasets used in the paper.

By this definition, a high value of Conicity(V)would imply that the vectors in V lie in a nar-row cone centered at origin. In other words, thevectors in the set V are highly aligned with eachother. In addition to that, we define the varianceof ATM across all vectors in V, as the ‘vectorspread’(VS) of set V,

VS(V) =1

|V|∑v∈V

(ATM(v,V)−Conicity(V)

)2

Figure 1 visually demonstrates these metrics forrandomly generated 3-dimensional points. Theleft figure shows high Conicity and low vectorspread while the right figure shows low Conicityand high vector spread.

We define the length of a vector v as L2-normof the vector ‖v‖2 and ‘average vector length’(AVL) for the set of vectors V as

AVL(V) =1

|V|∑v∈V‖v‖2

(a) Additive Models

(b) Multiplicative Models

Figure 2: Alignment to Mean (ATM) vs Density plots for entity embeddings learned by various additive(top row) and multiplicative (bottom row) KG embedding methods. For each method, a plot averagedacross entity frequency bins is shown. From these plots, we conclude that entity embeddings fromadditive models tend to have low (positive as well as negative) ATM and thereby low Conicity and highvector spread. Interestingly, this is reversed in case of multiplicative methods. Please see Section 6.1 formore details.

5 Experimental Setup

Datasets: We run our experiments on subsets oftwo widely used datasets, viz., Freebase (Bol-lacker et al., 2008) and WordNet (Miller, 1995),called FB15k and WN18 (Bordes et al., 2013), re-spectively. We detail the characteristics of thesedatasets in Table 2.

Please note that while the results presented inSection 6 are on the FB15K dataset, we reach thesame conclusions on WN18. The plots for our ex-periments on WN18 can be found in the Supple-mentary Section.Hyperparameters: We experiment with multiplevalues of hyperparameters to understand their ef-fect on the geometry of KG embeddings. Specif-ically, we vary the dimension of the generatedvectors between {50, 100, 200} and the numberof negative samples used during training between{1, 50, 100}. For more details on algorithm spe-cific hyperparameters, we refer the reader to theSupplementary Section.2

2For training, we used codes from https://github.

Frequency Bins: We follow (Mimno and Thomp-son, 2017) for entity and relation samples used inthe analysis. Multiple bins of entities and relationsare created based on their frequencies and 100 ran-domly sampled vectors are taken from each bin.These set of sampled vectors are then used for ouranalysis. For more information about samplingvectors, please refer to (Mimno and Thompson,2017).

6 Results and Analysis

In this section, we evaluate the following ques-tions.

• Does model type (e.g., additive vs multiplica-tive) have any effect on the geometry of em-beddings? (Section 6.1)

com/Mrlyk423/Relation_Extraction (TransE,TransR), https://github.com/datquocnguyen/STransE (STransE), https://github.com/mnick/holographic-embeddings (HolE) andhttps://github.com/ttrouill/complex (Com-plEx and DistMult).

(a) Additive Models

(b) Multiplicative Models

Figure 3: Alignment to Mean (ATM) vs Density plots for relation embeddings learned by various additive(top row) and multiplicative (bottom row) KG embedding methods. For each method, a plot averagedacross entity frequency bins is shown. Trends in these plots are similar to those in Figure 2. Mainfindings from these plots are summarized in Section 6.1.

• Does negative sampling have any effect onthe embedding geometry? (Section 6.2)

• Does the dimension of embedding have anyeffect on its geometry? (Section 6.3)

• How is task performance related to embed-ding geometry? (Section 6.4)

In each subsection, we summarize the mainfindings at the beginning, followed by evidencesupporting those findings.

6.1 Effect of Model Type on Geometry

Summary of Findings:Additive: Low conicity and high vector spread.Multiplicative: High conicity and low vectorspread.

In this section, we explore whether the type ofthe score function optimized during the traininghas any effect on the geometry of the resulting em-bedding. For this experiment, we set the numberof negative samples to 1 and the vector dimensionto 100 (we got similar results for 50-dimensionalvectors). Figure 2 and Figure 3 show the distribu-tion of ATMs of these sampled entity and relation

vectors, respectively.3

Entity Embeddings: As seen in Figure 2, thereis a stark difference between the geometries of en-tity vectors produced by additive and multiplica-tive models. The ATMs of all entity vectors pro-duced by multiplicative models are positive withvery low vector spread. Their high conicity sug-gests that they are not uniformly dispersed in thevector space, but lie in a narrow cone along themean vector. This is in contrast to the entity vec-tors obtained from additive models which are bothpositive and negative with higher vector spread.From the lower values of conicity, we concludethat entity vectors from additive models are evenlydispersed in the vector space. This observationis also reinforced by looking at the high vectorspread of additive models in comparison to that ofmultiplicative models. We also observed that addi-tive models are sensitive to the frequency of enti-ties, with high frequency bins having higher conic-ity than low frequency bins. However, no such pat-tern was observed for multiplicative models and

3We also tried using the global mean instead of mean ofthe sampled set for calculating cosine similarity in ATM, andgot very similar results.

Figure 4: Conicity (left) and Average Vector Length (right) vs Number of negative samples for entityvectors learned using various KG embedding methods. In each bar group, first three models are additive,while the last three are multiplicative. Main findings from these plots are summarized in Section 6.2

conicity was consistently similar across frequencybins. For clarity, we have not shown different plotsfor individual frequency bins.Relation Embeddings: As in entity embeddings,we observe a similar trend when we look at thedistribution of ATMs for relation vectors in Fig-ure 3. The conicity of relation vectors generatedusing additive models is almost zero across fre-quency bands. This coupled with the high vec-tor spread observed, suggests that these vectorsare scattered throughout the vector space. Re-lation vectors from multiplicative models exhibithigh conicity and low vector spread, suggestingthat they lie in a narrow cone centered at origin,like their entity counterparts.

6.2 Effect of Number of Negative Samples onGeometry

Summary of Findings:Additive: Conicity and average length are in-variant to changes in #NegativeSamples forboth entities and relations.Multiplicative: Conicity increases while av-erage vector length decrease with increasing#NegativeSamples for entities. Conicity de-creases, while average vector length remainsconstant (except HolE) for relations.

For experiments in this section, we keep thevector dimension constant at 100.Entity Embeddings: As seen in Figure 4 (left),the conicity of entity vectors increases as the num-ber of negative samples is increased for multi-plicative models. In contrast, conicity of the en-tity vectors generated by additive models is unaf-fected by change in number of negative samplesand they continue to be dispersed throughout the

vector space. From Figure 4 (right), we observethat the average length of entity vectors producedby additive models is also invariant of any changesin number of negative samples. On the other hand,increase in negative sampling decreases the aver-age entity vector length for all multiplicative mod-els except HolE. The average entity vector lengthfor HolE is nearly 1 for any number of negativesamples, which is understandable considering itconstrains the entity vectors to lie inside a unitball (Nickel et al., 2016). This constraint is alsoenforced by the additive models: TransE, TransR,and STransE.Relation Embeddings: Similar to entity embed-dings, in case of relation vectors trained using ad-ditive models, the average length and conicity donot change while varying the number of negativesamples. However, the conicity of relation vec-tors from multiplicative models decreases with in-crease in negative sampling. The average rela-tion vector length is invariant for all multiplica-tive methods, except for HolE. We see a surpris-ingly big jump in average relation vector lengthfor HolE going from 1 to 50 negative samples, butit does not change after that. Due to space con-straints in the paper, we refer the reader to the Sup-plementary Section for plots discussing the effectof number of negative samples on geometry of re-lation vectors.

We note that the multiplicative score betweentwo vectors may be increased by either increas-ing the alignment between the two vectors (i.e., in-creasing Conicity and reducing vector spread be-tween them), or by increasing their lengths. It isinteresting to note that we see exactly these ef-fects in the geometry of multiplicative methods

Figure 5: Conicity (left) and Average Vector Length (right) vs Number of Dimensions for entity vectorslearned using various KG embedding methods. In each bar group, first three models are additive, whilethe last three are multiplicative. Main findings from these plots are summarized in Section 6.3.

analyzed above.

6.2.1 Correlation with Geometry of WordEmbeddings

Our conclusions from the geometrical analysis ofentity vectors produced by multiplicative mod-els are similar to the results in (Mimno andThompson, 2017), where increase in negativesampling leads to increased conicity of word vec-tors trained using the skip-gram with negativesampling (SGNS) method. On the other hand, ad-ditive models remain unaffected by these changes.

SGNS tries to maximize a score function of theform wT ·c for positive word context pairs, wherew is the word vector and c is the context vector(Mikolov et al., 2013). This is very similar to thescore function of multiplicative models as seen inTable 1. Hence, SGNS can be considered as a mul-tiplicative model in the word domain.

Hence, we argue that our result on the increasein negative samples increasing the conicity of vec-tors trained using a multiplicative score functioncan be considered as a generalization of the one in(Mimno and Thompson, 2017).

6.3 Effect of Vector Dimension on Geometry

Summary of Findings:Additive: Conicity and average length are in-variant to changes in dimension for both entitiesand relations.Multiplicative: Conicity decreases for both en-tities and relations with increasing dimension.Average vector length increases for both entitiesand relations, except for HolE entities.

Entity Embeddings: To study the effect of vec-

tor dimension on conicity and length, we set thenumber of negative samples to 1, while varyingthe vector dimension. From Figure 5 (left), weobserve that the conicity for entity vectors gen-erated by any additive model is almost invariantof increase in dimension, though STransE exhibitsa slight decrease. In contrast, entity vector frommultiplicative models show a clear decreasing pat-tern with increasing dimension.

As seen in Figure 5 (right), the average lengthsof entity vectors from multiplicative models in-crease sharply with increasing vector dimension,except for HolE. In case of HolE, the average vec-tor length remains constant at one. Deviation in-volving HolE is expected as it enforces entity vec-tors to fall within a unit ball. Similar constraintsare enforced on entity vectors for additive modelsas well. Thus, the average entity vector lengths arenot affected by increasing vector dimension for alladditive models.

Relation Embeddings: We reach similar conclu-sion when analyzing against increasing dimensionthe change in geometry of relation vectors pro-duced using these KG embedding methods. Inthis setting, the average length of relation vectorslearned by HolE also increases as dimension is in-creased. This is consistent with the other meth-ods in the multiplicative family. This is because,unlike entity vectors, the lengths of relation vec-tors of HolE are not constrained to be less thanunit length. Due to lack of space, we are unable toshow plots for relation vectors here, but the samecan be found in the Supplementary Section.

Figure 6: Relationship between Performance (HITS@10) on a link prediction task vs Conicity (left) andAvg. Vector Length (right). For each point, N represents the number of negative samples used. Mainfindings are summarized in Section 6.4.

6.4 Relating Geometry to Performance

Summary of Findings:Additive: Neither entites nor relations exhibitcorrelation between geometry and performance.Multiplicative: Keeping negative samples fixed,lower conicity or higher average vector lengthfor entities leads to improved performance. Norelationship for relations.

In this section, we analyze the relationship be-tween geometry and performance on the Link pre-diction task, using the same setting as in (Bordeset al., 2013). Figure 6 (left) presents the effects ofconicity of entity vectors on performance, whileFigure 6 (right) shows the effects of average entityvector length.4

As we see from Figure 6 (left), for fixed num-ber of negative samples, the multiplicative modelwith lower conicity of entity vectors achieves bet-ter performance. This performance gain is largerfor higher numbers of negative samples (N). Addi-tive models don’t exhibit any relationship betweenperformance and conicity, as they are all clusteredaround zero conicity, which is in-line with our ob-servations in previous sections. In Figure 6 (right),for all multiplicative models except HolE, a higheraverage entity vector length translates to betterperformance, while the number of negative sam-ples is kept fixed. Additive models and HolE don’texhibit any such patterns, as they are all clusteredjust below unit average entity vector length.

The above two observations for multiplicativemodels make intuitive sense, as lower conicity andhigher average vector length would both translate

4A more focused analysis for multiplicative models is pre-sented in Section 3 of Supplementary material.

to vectors being more dispersed in the space.We see another interesting observation regard-

ing the high sensitivity of HolE to the number ofnegative samples used during training. Using alarge number of negative examples (e.g., N = 50or 100) leads to very high conicity in case of HolE.Figure 6 (right) shows that average entity vectorlength of HolE is always one. These two obser-vations point towards HolE’s entity vectors lyingin a tiny part of the space. This translates to HolEperforming poorer than all other models in case ofhigh numbers of negative sampling.

We also did a similar study for relation vectors,but did not see any discernible patterns.

7 Conclusion

In this paper, we have initiated a systematic studyinto the important but unexplored problem of an-alyzing geometry of various Knowledge Graph(KG) embedding methods. To the best of ourknowledge, this is the first study of its kind.Through extensive experiments on multiple real-world datasets, we are able to identify several in-sights into the geometry of KG embeddings. Wehave also explored the relationship between KGembedding geometry and its task performance.We have shared all our source code to foster fur-ther research in this area.

Acknowledgements

We thank the anonymous reviewers for their con-structive comments. This work is supported inpart by the Ministry of Human Resources Devel-opment (Government of India), Intel, Intuit, andby gifts from Google and Accenture.

ReferencesKurt Bollacker, Colin Evans, Praveen Paritosh, Tim

Sturge, and Jamie Taylor. 2008. Freebase: a collab-oratively created graph database for structuring hu-man knowledge. In Proceedings of the 2008 ACMSIGMOD international conference on Managementof data. AcM, pages 1247–1250.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko.2013. Translating embeddings for modeling multi-relational data. In Advances in neural informationprocessing systems. pages 2787–2795.

T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel.2017. Convolutional 2D Knowledge Graph Embed-dings. ArXiv e-prints .

Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, WilkoHorn, Ni Lao, Kevin Murphy, Thomas Strohmann,Shaohua Sun, and Wei Zhang. 2014. Knowledgevault: A web-scale approach to probabilistic knowl-edge fusion. In Proceedings of the 20th ACMSIGKDD international conference on Knowledgediscovery and data mining. ACM, pages 601–610.

Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, andXuan Zhu. 2015. Learning entity and relation em-beddings for knowledge graph completion. In AAAI.pages 2181–2187.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor-rado, and Jeff Dean. 2013. Distributed representa-tions of words and phrases and their compositional-ity. In Advances in neural information processingsystems. pages 3111–3119.

George A Miller. 1995. Wordnet: a lexical database forenglish. Communications of the ACM 38(11):39–41.

David Mimno and Laure Thompson. 2017. The strangegeometry of skip-gram with negative sampling. InProceedings of the 2017 Conference on EmpiricalMethods in Natural Language Processing. pages2863–2868.

T. Mitchell, W. Cohen, E. Hruschka, P. Talukdar, J. Bet-teridge, A. Carlson, B. Dalvi, M. Gardner, B. Kisiel,J. Krishnamurthy, N. Lao, K. Mazaitis, T. Mohamed,N. Nakashole, E. Platanios, A. Ritter, M. Samadi,B. Settles, R. Wang, D. Wijaya, A. Gupta, X. Chen,A. Saparov, M. Greaves, and J. Welling. 2015.Never-ending learning. In Proceedings of AAAI.

Dat Quoc Nguyen, Kairit Sirts, Lizhen Qu, and MarkJohnson. 2016. Stranse: a novel embedding modelof entities and relationships in knowledge bases. InProceedings of NAACL-HLT . pages 460–466.

Maximilian Nickel, Lorenzo Rosasco, and Tomaso A.Poggio. 2016. Holographic embeddings of knowl-edge graphs. In AAAI.

Srinivas Ravishankar, Chandrahas, and Partha PratimTalukdar. 2017. Revisiting simple neural networksfor learning representations of knowledge graphs.6th Workshop on Automated Knowledge Base Con-struction (AKBC) at NIPS 2017 .

M. Schlichtkrull, T. N. Kipf, P. Bloem, R. van den Berg,I. Titov, and M. Welling. 2017. Modeling RelationalData with Graph Convolutional Networks. ArXiv e-prints .

Richard Socher, Danqi Chen, Christopher D Manning,and Andrew Ng. 2013. Reasoning with neural ten-sor networks for knowledge base completion. In Ad-vances in Neural Information Processing Systems.pages 926–934.

Fabian M Suchanek, Gjergji Kasneci, and GerhardWeikum. 2007. Yago: a core of semantic knowl-edge. In WWW.

Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoi-fung Poon, Pallavi Choudhury, and Michael Gamon.2015. Representing Text for Joint Embedding ofText and Knowledge Bases. In Empirical Methodsin Natural Language Processing (EMNLP). ACLAssociation for Computational Linguistics.

Theo Trouillon, Johannes Welbl, Sebastian Riedel, EricGaussier, and Guillaume Bouchard. 2016. Complexembeddings for simple link prediction. In ICML.

Zhen Wang, Jianwen Zhang, Jianlin Feng, and ZhengChen. 2014. Knowledge graph embedding by trans-lating on hyperplanes. In AAAI. Citeseer, pages1112–1119.

Bishan Yang, Wen-tau Yih, Xiaodong He, JianfengGao, and Li Deng. 2014. Embedding entities andrelations for learning and inference in knowledgebases. arXiv preprint arXiv:1412.6575 .